Metric collectors used in RPI

[ylin30]

In desktop/servers, there are lots of data collectors frameworks (such as TCollector by Opentsdb, Telegraf by InfluxDB etc) to collect data and send to TimeSeriesDBs like Influxdb, TickTockDB. But these kind of collectors are heavyweight to RPIs.

I am curious what kind of data collectors people normally used in RPIs? Do they just write their own Python/C programs to collect data and send to TSDBs?

I know MQTT is popular in RPIs. But you still need to write your own codes and send them to MQTT. Why don't you send metrics directly to TSDBs?

Any inputs would be appreciated and helpful for us to serve RPIs better.

Disclaimer: I am coauthor of TickTockDB, a lightweight TSDB ideal to RPIs. Please see perf evaluation on RPI-0-w here.

[W. H. Heydt]

Sounds like a solution in search of a problem.

[ylin30]

Honestly I don't know how people collects metrics in RPIs usually. I do think metrics are essential to monitoring in IoT and have to be collected and visualized. Collector frameworks are good in taking care of sending in batches and failure etc. But based on my own experiments, e.g., TCollector (developed by OpenTsdb) eats up 10% of CPU in RPI-0-w by collecting OS metrics itself.

Writing you own codes is definitely a solution. Or maybe peoples don't mind of losing 10% cpu in PI to enjoy benefits of a framework. I am new to RPIs and really want to hear how RPIs geeks think. Thanks!

[MiscBits]

Sounds like a solution in search of a problem.

Think so given the OP is a TSDB creator

As for the question:

MQTT is the lowest common denominator for data transfer - my microcontrollers to high level languages support it and have many published examples I can work from.
MQTT has service level and 'last Will' services to allow me to monitor and control connections and message flow.
MQTT topic structure and it's support of wildcards is handy for visualisation and debugging.

Performance within the database is not really an issue given the number of data points I am handling. If it takes five seconds to show a graph compared to one I'm relaxed.

A delay on reacting to a switch change in MQTT though is different as they are noticeable to the user. I get an application call back for MQTT rather than having to poll a database for record changes.

Not all of my data needs to be in a database let alone being time based. Some of it is for reference only (e.g. switch state) and can sit in a topic till it changes. OK I could use a record but overhead in coding is less as the MQTT library handles the 'record locking' and update notification etc. (Plus I'm lazy so like transitory IoT data in one place).

Not all of my data is time dependant and I would not use an extra field for time if the data record did not need it.

Familiarity breeds contempt - I know my databases and MQTT reasonably well and this is a hobby so I have no real incentive to learn something without a good reason.

[ylin30]

...
Not all of my data is time dependant and I would not use an extra field for time if the data record did not need it.

Familiarity breeds contempt - I know my databases and MQTT reasonably well and this is a hobby so I have no real incentive to learn something without a good reason.

@MiscBits Thanks for the inputs.

[jahboater]

MQTT is the lowest common denominator for data transfer

I would say that just plain sockets would be lowest common denominator that is normally used.
Light weight, fast, and very portable.
MQTT is a layer above, providing other features.

[ylin30]

MQTT is the lowest common denominator for data transfer

I would say that just plain sockets would be lowest common denominator that is normally used.
Light weight, fast, and very portable.
MQTT is a layer above, providing other features.

My understanding is that MQTT is just a pub-sub system. It is good fit for events but may not be suitable for time series data continually coming.

[MiscBits]

MQTT is the lowest common denominator for data transfer

I would say that just plain sockets would be lowest common denominator that is normally used.
Light weight, fast, and very portable.
MQTT is a layer above, providing other features.

Apologies - it's the lowest level of data transfer I will use that's supported by the boards I have. Sockets involve a lot more coding for the MQTT functions (topics / last will) esp. if you are looking at multiple subscribers to a given topic or want to handle wildcards. Being lazy I do not see the need to replicate something that's already done for me

...
My understanding is that MQTT is just a pub-sub system. It is good fit for events but may not be suitable for time series data continually coming.

True but I do not use it to store historic data, only to pass formatted data around (normally JSON) and if the time is to be reported then I would pass that in the data as well. Closest second is way beyond any accuracy I need for home and in most cases even the minute too fine...