Sharing data between python scripts through a LAN

[MrYsLab]

I rebuilt for pypi installation and now installs on both Jessie and Stretch without having to jump through hoops. Never had to use wheels before, but things have changed in that direction.

On Jessie, for python 2, first do a

Code: Select all

sudo apt-get install python-dev

Followed by:

Code: Select all

sudo pip2 install python-banyan

All others combinations - python 2 and python 3:

Code: Select all

sudo pip2 install python-banyan

Note on Jessie, it will install a newer version of numpy and this takes a while to build.

[asandford]

In the meantime, I tried asandford's recommendation. I did this:

Code: Select all

sudo apt-get install mosquitto
sudo pip install paho-mqtt

And these two things installed no probs.

I did some reading here:
https://pypi.python.org/pypi/paho-mqtt/1.3.0

but quickly got lost. Have not yet found a tutorial for dummies on how to make this work on my raspberry pi's running simple python scripts. I am just looking to pass strings and simple numbers between them.

If anybody knows of guides, I would appreciate direction.

I think I just need to see a few examples. To late to keep googling...

PPi.

Starting on a fresh stretch lite image:

Code: Select all

 sudo apt update; sudo apt upgrade -y
 sudo apt install -y mosquitto mosquitto-clients
 

You've now installed the mosquitto MQTT broker (think of it as a server) and the clients (publisher and subsriber)

Now type the following:

Code: Select all

pi@raspberrypi:~ $ mosquitto_sub -d -t hello/world
Client mosqsub/1389-raspberryp sending CONNECT
Client mosqsub/1389-raspberryp received CONNACK
Client mosqsub/1389-raspberryp sending SUBSCRIBE (Mid: 1, Topic: hello/world, QoS: 0)
Client mosqsub/1389-raspberryp received SUBACK
Subscribed (mid: 1): 0

Open another terminal or putty session, and type this:

Code: Select all

pi@raspberrypi:~ $ mosquitto_pub -d -t hello/world -m "Hello from Terminal window 2!"
Client mosqpub/1416-raspberryp sending CONNECT
Client mosqpub/1416-raspberryp received CONNACK
Client mosqpub/1416-raspberryp sending PUBLISH (d0, q0, r0, m1, 'hello/world', ... (29 bytes))
Client mosqpub/1416-raspberryp sending DISCONNECT
pi@raspberrypi:~ $

In your first window you should see:

Code: Select all

Client mosqsub/1389-raspberryp sending PINGREQ
Client mosqsub/1389-raspberryp received PINGRESP
Client mosqsub/1389-raspberryp sending PINGREQ
Client mosqsub/1389-raspberryp received PINGRESP
Client mosqsub/1389-raspberryp received PUBLISH (d0, q0, r0, m0, 'hello/world', ... (29 bytes))
Hello from Terminal window 2!
Client mosqsub/1389-raspberryp sending PINGREQ
Client mosqsub/1389-raspberryp received PINGRESP

So this shows that the broker and clients are working.

Crtl-C out of the 'subcriber' screen and type:

Code: Select all

pi@raspberrypi:~ $ mosquitto_sub -d -t ledStatus

Install pip and paho-mqtt from you 'publisher' console:

Code: Select all

 sudo apt install -y python-pip
  pip install paho-mqtt
Collecting paho-mqtt
  Downloading paho-mqtt-1.3.0.tar.gz (79kB)
    100% |████████████████████████████████| 81kB 1.0MB/s
Building wheels for collected packages: paho-mqtt
  Running setup.py bdist_wheel for paho-mqtt ... done
  Stored in directory: /home/pi/.cache/pip/wheels/a8/fb/b8/25cf89a4ebec3846c24b8feec8469f8402e7706c2c962c84b4
Successfully built paho-mqtt
Installing collected packages: paho-mqtt
Successfully installed paho-mqtt-1.3.0

Now create a test python program and run it:

Code: Select all

pi@raspberrypi:~ $ vi testmq.py
pi@raspberrypi:~ $ cat testmq.py
import paho.mqtt.publish as publish
import time
print("Sending 0...")
publish.single("ledStatus", "0", hostname="localhost")
time.sleep(1)
print("Sending 1...")
publish.single("ledStatus", "1", hostname="localhost")

pi@raspberrypi:~ $ python testmq.py
Sending 0...
Sending 1...
pi@raspberrypi:~ $

And on your 'subscription' console you should see:

Code: Select all

pi@raspberrypi:~ $ mosquitto_sub -d -t ledStatus
Client mosqsub/6104-raspberryp sending CONNECT
Client mosqsub/6104-raspberryp received CONNACK
Client mosqsub/6104-raspberryp sending SUBSCRIBE (Mid: 1, Topic: ledStatus, QoS: 0)
Client mosqsub/6104-raspberryp received SUBACK
Subscribed (mid: 1): 0
Client mosqsub/6104-raspberryp received PUBLISH (d0, q0, r0, m0, 'ledStatus', ... (1 bytes))
0
Client mosqsub/6104-raspberryp received PUBLISH (d0, q0, r0, m0, 'ledStatus', ... (1 bytes))
1

Loads more info here for all sorts of IoT devices.

Node-Red has inbuilt javascript clients and the msg.payload var contains just the text that is sent / received, and as it's non-blocking and async, you don't have to write horrible callbacks - it just works.

[pumpkinpi]

Oh, this is cool stuff. Thanks for all the help folks. It is going to take me a few weeks to start playing around with this in my spare time. This is definitely what I need for my little home control IOT project.

I'll report back on progress.

PPi.

[pumpkinpi]

Ran through your mini tutorial tonight @asandford and it WORKS. So cool. Replaced "localhost" with an ip address on my lan and can send messages between Pi's too. So cool.

Thanks!

[asandford]

Ran through your mini tutorial tonight @asandford and it WORKS. So cool. Replaced "localhost" with an ip address on my lan and can send messages between Pi's too. So cool.

Thanks!

Glad you found it useful.

I used just one Pi for convenience, but usually the clients and broker(s) are on different servers (although inter-process communication can be done fairly easily with MQs on a single host).

[pumpkinpi]

Ok, trying this inside some existing python scripts and I must be missing something.

So, this is a section from my computer (picontrol) that is subscribing to a topic and taking action if it gets a message:

Code: Select all

import datetime
import time
import paho.mqtt.client as paho

def on_message(pihmi1, userdata, msg):
        audiblealert()
        print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))

pihmi1 = paho.Client()
pihmi1.on_message = on_message
pihmi1.connect("192.168.0.4", 1883)
pihmi1.subscribe("kitchenoccupancy")
pihmi1.loop_start()

Basically, this is supposed to make an audible alert and print on the screen when somebody walks into the kitchen or it is deemed unoccupied by a motion sensor.

This is the publisher code (pihmi1):

Code: Select all

import time
import datetime
import RPi.GPIO as io
import paho.mqtt.client as paho
io.setmode(io.BCM)
io.setup(18, io.IN)

occupiedcount = 0
picontrol = paho.Client()
picontrol.connect("192.168.0.5", 1883)
picontrol.loop_start()
occupied = 'NO'
picontrol.publish("kitchenoccupancy", occupied)

while True:
    oldmotion = io.input(18)
    time.sleep(2)
    occupiedcount += 1
    if io.input(18):
        if occupied == 'NO':
            occupiedcount = 0
            occupied = 'YES'
            picontrol.publish("kitchenoccupancy", occupied)
    if occupiedcount > 1350:
        if occupied == 'YES':
            occupied = 'NO'
        picontrol.publish("kitchenoccupancy", occupied)

I tried to leave "optional" stuff out to see if it would work this way. Basically, the code does not crash, but I don't see any effect either.

I am running the first one with python3 and the second with python if that makes any difference. I would not expect so.

PPi

[mattmiller]

those are two different ip addresses- but lets ignore that as people never paste the code they are ACTUALLY using

Use another MQTT tool to make sure the broker is actually working and receiving and sending messages

I use MQTTBox Chrome app on my Win10 machine to do that

[pumpkinpi]

Oh, that is interesting. I thought the addresses had to be different. Should they be the address of the publisher computer or the subscriber?

They are not the actual addresses, but are different in my code.

I did do the stuff above (command line stuff) so I know it is working fine - just not in my code.

Thanks,
PPi

[mattmiller]

Oh, that is interesting. I thought the addresses had to be different. Should they be the address of the publisher computer or the subscriber?

The address on all machines needs to be the address of the broker you are using

MQTT is based on there being a broker running on a computer somewhere (can be on your own LAN or on the internet like test.mosquitto.org )

All computers that want to share data on a particular topic must publish and/or subscribe to the broker that is carrying that topic

Are you running a broker on your LAN?

[asandford]

Oh, that is interesting. I thought the addresses had to be different. Should they be the address of the publisher computer or the subscriber?

They are not the actual addresses, but are different in my code.

I did do the stuff above (command line stuff) so I know it is working fine - just not in my code.

Thanks,
PPi

In the mini tutorial, the clients and server are all installed on one host. All the references to localhost should refer to the hostname of the device running the broker. The client IP addressess do not need to be used, or even known, for a successfull connection. Think of the process like this forum. You need to know the address to post a message, and I need to know it to read it. You don't need to know who will read your post, and I don't need to know the poster - hence the forum is a broker of sorts.

There are advanced cases where clients can connect to different bokers, and the brokers are bridged, but I don't think you're at that stage yet

The use of a broker provides better QoS (the broker knows which clients have received a message and which ones have missed it) and can redeliver if need be. MQTT is designed to be lightweight on resources and there are more feature rich protocols available. I use the RabbitMQ package which can broker MQTT messages, as well as a host of other protocols.

[pumpkinpi]

Ok, so I got it to work by putting 'localhost' in the one machine (broker). In the other one, I left the IP address of the broker and it worked. Putting localhost in both, does not work.

This is cool stuff. I will just set one Pi on my network as the broker and have all other machines reference to that one. If I do that, can one "client" machine send messages to another "client" if they are both subscribed to the same topic?

I am having an issue with reading the message that comes back. It prints a b'YES'. I believe that means byte. I just want strings.

Code: Select all

kitchenoccupancy 0 b'YES'

Can't figure out how to just get strings to come through.

Thanks,
PPi

[pumpkinpi]

Figured out the conversion. It was just:

Code: Select all

if msg.payload.decode("utf-8") == "YES":

PPi

[asandford]

This is cool stuff. I will just set one Pi on my network as the broker and have all other machines reference to that one. If I do that, can one "client" machine send messages to another "client" if they are both subscribed to the same topic?

Yes. If one client publishes to a topic, all subscribers to that topic will receive it. It's worth pointing out that a client that is publishing may also be subscribing, so the flow of data can be complex.

[pumpkinpi]

IT WORKS! IT WORKS!

This is the coolest thing I have done on my Pi's. I wish I had figured this out years ago. I have about 12 Pi's in my house doing lots of different jobs and the one thing I have struggled with the most is getting them to coordinate with each other. I have a lot of code to sort through now and rewrite.

This is so much more powerful than other solutions I have used.

Thanks a million for the assistance folks.

I just can't believe how easy this is in the end.

PPi

[mattmiller]

I just can't believe how easy this is in the end.

I'm with you
The fact that its available to use and run from Desktop PCs to little ESP8266s is brilliant

[asandford]

IT WORKS! IT WORKS!

This is the coolest thing I have done on my Pi's. I wish I had figured this out years ago. I have about 12 Pi's in my house doing lots of different jobs and the one thing I have struggled with the most is getting them to coordinate with each other. I have a lot of code to sort through now and rewrite.

This is so much more powerful than other solutions I have used.

Thanks a million for the assistance folks.

I just can't believe how easy this is in the end.

PPi

Now try Node Red and MQTT and you can have asynchronous, non-blocking, event driven code. I have a green house monitoring system reading sensors and controlling switches with it. It also has easy database integration and message transformation, native gpio control on a Pi and Arduino support with firmata. It's already built in to Raspbian, and is developed by IBM.