Reading a serial signal from a USB port

[B-VE]

Hi,

I'm a beginner in Linux/ Python, so sorry for maybe asking something quite simple for some of you
I'd like to read from a component which is sending a serial signal and use the RPi2 as a data logger.
In the end the RPi should be a stand alone logger which at startup starts logging.

The signal is transferred with following parameters:
Baud rate: 19200, Data bits: 8, Parity: none, Stop bits: 1, Flow control: none

This is how the data is send: Message format
The device transmits blocks of data at 1 second intervals. Each field is sent using the following format:
<Newline><Field-Label><Tab><Field-Value>
The identifiers are defined as follows:
<Newline>A carriage return followed by a line feed (0x0D, 0x0A).
<Field-Label> An arbitrary length label that identifies the field. Where applicable, this will be the same as the label that is used on the LCD.
<Tab> A horizontal tab (0x09).
<Field-Value> The ASCII formatted value of this field. The number of characters transmitted depends on the magnitude and sign of the value.

I have a serial -> USB converter attached to the RPi2. I've got PyUSB installed. With lsusb I already found the connected converter and used to parameters in PyUSB to try to retrieve data. This works, because: device = usb.core.find(idVendor=0x0403, idProduct=0x6015) is TRUE and the device is recognized (a Future Technology Devices International, Ltd Bridge(I2C/SPI/UART/FIFO converter).
With Minicom I also got a dump of the data which is being send from the device...

Now I'd like to read from the device myself (writing in Python) and store this data in a file. Printing would already be sufficient I guess as I believe writing to a file will then not be so difficult.
On my search for PyUSB resources I did not find how to read from such a source. Can somebody help me (maybe a link or code) how to start reading from this USB converter? I do not need to write (at least I think ).
Thanks!

[gordon77]

Code: Select all

#!/usr/bin/python
import os, sys
import serial
import time

ser = serial.Serial('/dev/ttyUSB0',19200, timeout = 5)

# listen for the input, exit if nothing received in timeout period
while True:
   line = ser.readline()
   if len(line) == 0:
      print("Time out! Exit.\n")
      sys.exit()
   print line,

[DougieLawson]

You don't need to run it as a raw USB device. Plug it in and Linux should recognise it and make it available as /dev/ttyUSB0.

Then a simply python serial program like

Code: Select all

#!/usr/bin/python

import serial, string

output = " "
ser = serial.Serial('/dev/ttyUSB0', 4800, 8, 'N', 1, timeout=1)
while True:
  print "----"
  while output != "":
    output = ser.readline()
    print output
  output = " "

will read the data and print it.

[B-VE]

Hi Gordon and Doug,

thanks for the fast reply! That worked out quit nice, I changed the baud rate in doug's example... and both worked directly!

Helped my out a lot!

Regards,

[DougieLawson]

Sorry, my program was designed for grabbing some NMEA data from a GPS receiver hence the 4800 baud setting.

You may want to do some data parsing in the program reading the raw data (by scanning the input for 0x09, 0x0A and 0x0D).

[B-VE]

Hi Doug,

actually that's where I'm at right now... saving in a file is done. I need to be able to save it as a .csv file to be able to import it easily in Excel...
Amazing how strong the Raspberry platform is... Only got it running for 2 days now

[B-VE]

Hi Doug, and others,

I'd now like to parse the text in the .csv file:

Code: Select all

# Read & print/write data
ser = serial.Serial('/dev/ttyUSB0',19200, 8, 'N', 1, timeout = 5)
# Open ve_direct.csv
file_data = open('ve_data.csv', 'w')
print "Reading data and writing to ve_data.csv"
# Listen for the input, exit if nothing received in timeout period
output = " "
while True:
  while output != "":
    output = ser.readline()
    file_data.write(output)
    print output,
  output = " "

# Close file
print "Stopped writing to ve_data.csv"
file_data.close()

Can I somehow parse the data in the 'file_data.write(output)'.
I know that the dataline or in this code readline() is always:
<Newline><Field-Label><Tab><Field-Value>
where
<0x0D, 0x0A><XXX><0x09><XXX>

I now get the dataline in one row in the .csv file.
I can't seem to find the way to recognize the TAB and make it visual in the .csv file which I write.
Can this be done easily?

Here 2 screenshots; 1 from the Raspberry, the second from Excel afterwards...

screenshot.png (18.16 KiB) Viewed 76295 times

screenshot1.png (19.23 KiB) Viewed 76295 times

[DougieLawson]

Try including this somewhere in your program.

Code: Select all

import re
new_output = re.sub(r'\s+', ',', output)

new_output will be the output with the tabs, returns and line feeds removed.

[gordon77]

Isnt the problem the op cant Detect tabs not wants to lose them?

[B-VE]

Hi Gordon,
it is exactly how Doug wrote it..

seems to work... thanks Doug!
the TABs are back in by replacing all by a ','
Though I didn't want to loose the return/ line feed in the .csv?!
The data is now nicely devided in single cells, but the new dataline is not started in a new row..

Can I also search for the code '0x09' to replace this by a ','? I tried a couple of things, but it wasn't recognized

Kind regards

[DougieLawson]

Isnt the problem the op cant Detect tabs not wants to lose them?

He can use an alternative regex to detect them.

[rpdom]

Can I also search for the code '0x09' to replace this by a ','?

The sequence \t should match the tab character exactly.

[B-VE]

Thanx! That's seems to do the trick!

[achapman]

I would like to go further than this. Is it possible to build an in-line serial logger? The idea occurred to me when I saw this - https://store.hackaday.com/products/usb-tester-2-0. This only logs the voltage going through the usb cable, whereas what I would like to see is the serial data going through.
This would be a non-intrusive way of monitoring usb/serial comms without the need to install software at either end to sniff the data. It would be particularly useful with WINCE devices where loading software is difficult and when the device generating the serial stream needs to be sent specific initialisation data (unlike the GPS device) in order to generate the required data.
If such a device could be built it could not only record data streams but play back particular recorded data, which would be useful for testing purposes as well.

[rasbee]

New to this platform and hope that this is the right forum to ask my question.

How can I read raw data (bit pattern) from a usb port. (It is not serial protocol.)

TIA.

[scotty101]

New to this platform and hope that this is the right forum to ask my question.

How can I read raw data (bit pattern) from a usb port. (It is not serial protocol.)

TIA.

Impossible to tell you whether it is possible or how to do it without further information. Is the device actually USB data? What type of device is it?
If not it is more likely to be able to read the data through a GPIO pin than a USB port.

[rasbee]

Thanks for your reply.

It is just an ordinary stream of bits coming from the field, nothing special (not following any comm protocol).

I will write a Python App to make sense of the bit pattern. My problem is how I can fetch it from the USB port programmatically.

(I know that I can use GPIO, but the USB will provide a cleaner interface to the outside world.)

[scotty101]

Thanks for your reply.

It is just an ordinary stream of bits coming from the field, nothing special (not following any comm protocol).

I will write a Python App to make sense of the bit pattern. My problem is how I can fetch it from the USB port programmatically.

(I know that I can use GPIO, but the USB will provide a cleaner interface to the outside world.)

An 'ordinary stream of bits coming from the field' tells us nothing about the data.

Short answer for you is that you can't just connect some random string of data to the USB port and expect to be able to read it.
USB ports expect a defined communication protocol to be followed, certainly not a random stream of bits.

GPIO pins will likely be the only way to read a stream of bits in a custom (and as yet undefined) format. If you want a nice mechanical connection to the external device then there are any number of nice mechanical connectors that you could use.

[rasbee]

Short answer for you is that you can't just connect some random string of data to the USB port and expect to be able to read it.
USB ports expect a defined communication protocol to be followed, certainly not a random stream of bits.

Thanks, I am aware of the fact that the USB port expects a certain comms protocol. However, my intention is to bypass/disable that protocol and access the raw bit stream at the pins.

The Python sample codes (like below) suggest a well defined device at the port. However, I don't have a USB device connected to the port.

dev = usb.port.find(device_identifier)

I was hoping for suggestions like going to the kernel level (possibly using C, C++, etc, even if not Python) and accessing the raw data at the port pins. After all, the kernel does know how to access that data and I should be able to bypass/disable the kernel driver for the USB and access that raw data at the port.

[scotty101]

However, my intention is to bypass/disable that protocol and access the raw bit stream at the pins.

You can't!

The USB ports on a Pi 3 are connected via a combined 4 port USB hub and Ethernet device. That IC is then connected to a dedicated USB OTG port. You can't just 'disable' the protocol. Try another way.

[rasbee]

The USB ports on a Pi 3 are connected via a combined 4 port USB hub and Ethernet device. That IC is then connected to a dedicated USB OTG port. You can't just 'disable' the protocol. Try another way.

Thanks for your reply. Is there really any other way to read the raw data (bit pattern) at the USB port between the data pins programmatically (without having to write a driver for the external device to make it USB/Serial protocol compliant)?

[scotty101]

The USB ports on a Pi 3 are connected via a combined 4 port USB hub and Ethernet device. That IC is then connected to a dedicated USB OTG port. You can't just 'disable' the protocol. Try another way.

Thanks for your reply. Is there really any other way to read the raw data (bit pattern) at the USB port between the data pins programmatically (without having to write a driver for the external device to make it USB/Serial protocol compliant)?

I'll say this once more. There is no way to read a non-USB bitstream on the Pi's USB ports. You'd need to convert the bit stream to a USB protocol using an external IC.
Stop asking and use the GPIO pins instead.

[ingvildf]

Hi!
I think I am doing the exact same as you, were you trying to connect to a Victron product using the V.E. Direct Protocol?

Anyhow, I am really struggling with identifying the USB port that I want to communicate with.
I am experimenting connecting only a USB mouse , but using the dmesg command only informs me "USB disconnect, device number 7".

And when i list all tty's using ls /dev/tty* and then again after connecting mouse again, no new name appears, as this (https://oscarliang.com/connect-raspberr ... usb-cable/) post says it is supposed to be!


What am I doing wrong, is it because it is not possible to detect the right port using a simple mouse??
I am using a Raspberry Pi Model B....

[rasbee]

I think I am doing the exact same as you, were you trying to connect to a Victron product using the V.E. Direct Protocol?

I am not sure if we are trying to do the same thing. I understand you are trying to talk to a certain product (mouse) using usb. I was trying to bypass the rpi kernel level protocol to be able to interpret the bits coming the usb connection (as I desired). My investigation and replies in this forum indicated it is very difficult (even possible).

Now I am using an Arduino (Nano) to package up my bits and send them through the USB connection to RPi. (Pls don't ask me why I don't use the GPIO on RPi. It is a requirement to connect via USB to RPi!...)

Hence, I had to identify the Arduino connection at the RPi side which can be done very easily by using "serial.tools.list_ports" module.

You can use the following source to iterate thorugh the USB connections:

import serial.tools.list_ports
allPorts = list(serial.tools.list_ports.comports())
for aPort in allPorts:
aPort.vid == ARDUINO_VENDOR_ID

where ARDUINO_VENDOR_ID is 0x2341 for UNO and 0x1A86 for NANO.

They are unique vendor IDs. There must be one for Victron as well (I guess).

You can also find other properties of the connection such as aPort.device, aPort.location, etc.

Good luck!...

[ingvildf]

Hmm, okey, thank you!

But when i list the aPorts, I only get /dev/ttyAMAO, which I guess is to the GPIO pins??

I doesn't matter if I have the USB plugged in or not, the output is the same!

I have disables serial for shell, but enabled it for hardware in the rasp-config, is that wrong?