User avatar
Inquisitor720
Posts: 5
Joined: Sun Dec 30, 2012 11:20 pm

IPC (Inter-Process Communications)

Mon Dec 04, 2023 4:38 pm

I'm building a server based program in C++ just because it's the language I've used for 40 years (on Windows). I will be adding a plug-in type feature for extensibility. Although, I'd probably be the only one writing the the plug-ins, I don't want to ham-string it based on my programming background. I'd like to learn Python some day when I have free-time. :lol:

I'd want absolute server stability no matter how badly a plug-in was written. This would eliminate things like whatever the equivalent is to a Window's Dynamic Link Libraries or COM. It would also make things less desirable like TCP, Named Pipes or Shared memory. At the moment, I believe the plug-ins would not need to register or send data to the server. They'd only be receivers of data. The data also is not critical in the sense of needing delivery assurance like TCP/IP. I also want it to be simple so that even a novice could start receiving and using data.

My initial thought is to use a simple UPD/IP broadcast. This way plug-ins (actually independent programs) could be on the same machine or even other machines on the LAN. I'd be interested, mainly, in the opinions of Python developers on the Raspberry Pi on what IPC solutions might be the easiest (or most commonly used)?

Thanks.

Heater
Posts: 19722
Joined: Tue Jul 17, 2012 3:02 pm

Re: IPC (Inter-Process Communications)

Mon Dec 04, 2023 6:06 pm

Strangely enough I am currently involved in a project that at a high level sounds like what you are describing. We have major parts of the code written in Rust rather than C++ but the principle is the same, other parts written in Python or potentially any other language.

Now, there is a large part of this code that is distributing data to other parts. Some of those other parts are essential to the system, others are optional, new ones can be added easily in the future.

Rather than thinking of "client" and "server" we have reduced everything to communication between peer processes. New peer processes can join this system at any time.

How do we do it? Easy, we use the NATS communications system https://nats.io

With NATS a data producer can "publish" data at any time and it will be received by "subscribers" to that same data item. Rather like MQTT. In addition processes can make "requests" for data, and other processes will see those and produce "responses". There are a few other tricks one can do with Rust. All this communication is mediate by a NATS server. In this set up everything can run on the same machine, or different parts can run on different machines. The NATS server itself is capable of running as a cluster distributed over many machines for performance and/or fault tolerance.

There is NATS client software available for pretty much all common languages so one can mix and match as much as one likes. All without ever having worry about maintaining network connections, fiddling with IP addresses/host names etc, etc. Application processes only have to think about the data they produce or consume.
Slava Ukrayini.

jalih
Posts: 271
Joined: Mon Apr 15, 2019 3:54 pm

Re: IPC (Inter-Process Communications)

Mon Dec 04, 2023 6:35 pm

For my own home automation software, I simply use Publisher/Subscriber pattern on top of the WebSockets. Authenticated clients belong into "groups" and can publish topics and subscribe into topics inside the group they belong into. Active topics are maintained inside the servers in-memory database.

Heater
Posts: 19722
Joined: Tue Jul 17, 2012 3:02 pm

Re: IPC (Inter-Process Communications)

Mon Dec 04, 2023 7:15 pm

jalih wrote:
Mon Dec 04, 2023 6:35 pm
For my own home automation software, I simply use Publisher/Subscriber pattern on top of the WebSockets. Authenticated clients belong into "groups" and can publish topics and subscribe into topics inside the group they belong into. Active topics are maintained inside the servers in-memory database.
When you say "simply" do you mean that you wrote your own pub/sub server?

Simpler might be to use use an existing one, like NATS. Then you application code gets as simple as:

Code: Select all

// Subscribe to some data
let subscription = nat_connection
                .subscribe("some.subject")?
                .with_handler(move |message| {
                    // Do whatever with received message.
                }); 


// Publish some data
nats_connection.publish("some.subject", some_message)?;
NATS also supports using MQTT style pub/sub, works over web sockets if need be, supports users, groups and so on. Secure with TLS if need be.

Previously we used a pub/sub system I created in Javascript and node.js using web sockets. It worked, but NATS is more performant and feature full and secure.
Slava Ukrayini.

jalih
Posts: 271
Joined: Mon Apr 15, 2019 3:54 pm

Re: IPC (Inter-Process Communications)

Mon Dec 04, 2023 8:17 pm

Heater wrote:
Mon Dec 04, 2023 7:15 pm
When you say "simply" do you mean that you wrote your own pub/sub server?

Simpler might be to use use an existing one, like NATS. Then you application code gets as simple as:

Code: Select all

// Subscribe to some data
let subscription = nat_connection
                .subscribe("some.subject")?
                .with_handler(move |message| {
                    // Do whatever with received message.
                }); 


// Publish some data
nats_connection.publish("some.subject", some_message)?;
NATS also supports using MQTT style pub/sub, works over web sockets if need be, supports users, groups and so on. Secure with TLS if need be.

Previously we used a pub/sub system I created in Javascript and node.js using web sockets. It worked, but NATS is more performant and feature full and secure.

Yes, it's a simple pub/sub server modified from 8th sample and with my websocket library fixes. I use a JSON based api and messages like:

Code: Select all

{ api: "subscribe", topic: ["/1_EVT", "/1_SYS2_1", "/1_OWD1", "/LVV"], remove: false }
And

Code: Select all

{ api: "publish", msg: "20.0 C", topic: "/1_OWD" } 
Nice thing about is that client can be a web page, mobile or desktop application. Also, it's simple to reconnect after connection drops and as my automation controllers send both periodic and realtime data, there is almost no need to save any state information locally.

Heater
Posts: 19722
Joined: Tue Jul 17, 2012 3:02 pm

Re: IPC (Inter-Process Communications)

Mon Dec 04, 2023 9:12 pm

Yep, looks about like what we have done. I love the way such a pub/sub system decouples components. From embedded systems to back end services to web pages. Everything basically connects up in the same simple way.
Slava Ukrayini.

User avatar
Inquisitor720
Posts: 5
Joined: Sun Dec 30, 2012 11:20 pm

Re: IPC (Inter-Process Communications)

Fri Dec 08, 2023 7:01 pm

Thank you all for your input.
Are any of you using Python on one end or the other or do you support Python developers wanting to access your server exposed interfaces?

Heater
Posts: 19722
Joined: Tue Jul 17, 2012 3:02 pm

Re: IPC (Inter-Process Communications)

Fri Dec 08, 2023 9:06 pm

Yep. In our current developments we have systems composed of processes written in Python and Rust. There are NATS clients provided for both and a lot of other languages besides. See here: https://docs.nats.io/using-nats/developer. They are all very simple to use.

Like I said above, we should be careful about terminally here. With NATS communications all our processes, in whatever language, are peers communicating through NATS. Here "server" is the NATS server, which does nothing but shunt messages around. All the communicating processes are clients of that server.
Slava Ukrayini.

Return to “General programming discussion”