Hi All,
First, let me say that although involved in the computer industry for way too many years, I am very new to embedded systems and eagerly learning out of personal interest. Thank you to everyone who has contributed to these forums because I have learned so much from reading everything and I hope to return something useful as I develop my knowledge too!
I am working on designing my sensor network, using the moteino's of course, and in the process of planning while I wait for my nifty new mega's to arrive. Based on all the valuable contributions in these forums and sample code, I believe I will have no problems getting a couple sensor nodes and gateway running and talking to each other. I'm very optimistic and likely way to over confident ;)
One area that I'm concerned about, and perhaps I've missed good info somewhere, is the area of gathering and processing the sensor data. I will have a moteino connected to a linux box (likely a pi) via usb as my gateway. I see great examples of gateway sketch's that will send the received packets out the serial to the linux host, but then...
There needs to be a program/script on the linux host to gather those strings, do some parsing/formatting, and then, ideally, I would like to stuff that data into an sql table for access by other systems for display/analysis/notifications/etc.
My thinking is, and it could be totally wrong, is that there needs to be multiple processes on the linux host...i.e. given that there could possibly be many messages arriving very quickly from a large or busy collection of sensor nodes, it's necessary for the first processes to accept the raw messages very quickly, do something with it, and go back to waiting for the next message in order not to potentially lose more incoming messages. My thinking is that this first process, call it the "collector/dispatcher" just takes the raw message and then sends it off to another process, call it the "parser", which does the parsing/formatting/etc, which could take a relatively long processing time, and stuffs the result into the database. My thinking is that these "parser" processes need to be spawned by the "collector" on demand for each new message received, and die when finished, again to ensure that incoming messages to the collector don't get lost.
My question is, does any "middleware" like this already exist in some form, i.e. python, that provides this kind of speed/buffering functionality, or something similar, or do I need to roll my own. I'm assuming you folks are using SOMETHING on the host end to capture and utilize your data without losing any but I can't seem to find any references to anything here.
Any info or leads or references would be greatly appreciated! As fun as programming is, I don't want to reinvent the wheel if a good round one already exists ;)
Thanks
Brian
Hey Brian,
Welcome, you are probably overly concerned at this point. I think depending on your traffic there will be 2 bottlenecks in your setup: the gateway moteino itself, and the linux script you mentioned.
At the gateway there is some leverage provided by the retry/ACK (if you use sendWithRetry()) and that will make sure the packets are received, it can still fail but it most likely will get through after a few retries, in case gateway is busy or there are collisions or noise etc.
The linux script I think can be pretty fast, I've used python and node.js on a RaspberryPi and haven't had problems with a single thread script that was logging to mysql and also maintaining a websocket open. So i'd KISS it for now and handle multi threading later if it's really a necessity.
Examples of python scripts used: https://github.com/LowPowerLab/SumpPumpAlert/blob/master/Gateway.py
And nodejs: https://github.com/LowPowerLab/RaspberryPi-Gateway
Thanks Felix...appreciate the quick reply!
Ok, probably over concerned with message contention at this point. I've been watching a couple threads on here which seemed to indicate that as a potential problem area so I wanted to avoid those problem from the beginning and not fix them later ;)
In your examples there is a lot of code related to your specific application and/or parsing needs. Obviously, everyone's application will be different and, no doubt, everyone will develop their own packet protocols and formatting too. I get there isn't ONE gateway application that will work for everyone and every application. I will closely examine the examples and pick out the parts that are specific to just getting the packet from the gateway moteino and sending parsed/formatted data out to a database. Obviously, my parsing parts will be different in the middle. But, it's good to know that I don't need to worry too much about processing speed at this point.
Thanks again!
Alternative is to have your gateway be a TCP client that posts to an HTTP server hosted on your Linux platform. The gateway has already done some amount of 'parsing' based on the source of the information and can simply post to the proper url on your server.
In my case most of my data gets logged to appropriate tables in a MySQL DB and then gets viewed/processed from there with various distributed apps. My gateway mostly posts the data to the same place but, because of the way Motes are bound to the gateway, there wouldn't be a problem having a totally unique url for each type of Mote.
In this way Gateways and Motes are totally decoupled from Server applications.
Just another perspective...
Tom
crossposted from another Raspberry Pi thread:
You could also hook up the RFM69 module directly to the RasPi as in
https://github.com/abouillot/HomeAutomation/tree/master/piGateway (https://github.com/abouillot/HomeAutomation/tree/master/piGateway)
I've been running that gateway for about a week now without any problems, so far very stable. Only issue is that ACKs don't work on the RasPi side, so you cannot rely on sendwithRetry() on the Moteino nodes.
The provided sample piGateway code posts the received data to a local MQTT server, and anything else can simply subscribe to the relevant MQTT topics: FHEM, mqttwarn, etc.
Thanks for the reply guys..these are awesome suggestions to examine.
Cheers,
Quote from: BrianB on January 20, 2015, 12:55:19 PM
My question is, does any "middleware" like this already exist in some form, i.e. python, that provides this kind of speed/buffering functionality, or something similar, or do I need to roll my own. I'm assuming you folks are using SOMETHING on the host end to capture and utilize your data without losing any but I can't seem to find any references to anything here.
Any info or leads or references would be greatly appreciated! As fun as programming is, I don't want to reinvent the wheel if a good round one already exists ;)
Thanks
Brian
My current setup is as follows;
Nodes:
1. Battery operated remote sensors sending infrequent packets.
Gateway:
2. Gateways is a Rapsberry Pi, with Motino attached via USB.
3. Moteino is running a gateway sketch, that formats JSON and output over serial port.
4. Raspberry PI is running Node-Red, with very simple code that reads serial data from Moteino, and sends messages to an MQTT topic. It is very simple just a traffic relay.
Backend:
5. Backend is a Linux (Ubuntu Server) VM running on Proxmox
6. Installed is Mosquito MQTT broker, which acts as main hub.
7. Has Node-Red installed, which acts as main co-ordinator, with lots of little things I am building. Main thing is:
8. Has "emoncms" installed to provide data logging and visualisation.
Will install other apps, on the Backend, or just write simple things in node as needed.
Kiwi
Kiwi (this is getting to be a habit ;) )
I like your setup very much. Regarding our posts in the MQTT topic, in this example, if your device was formatting and sending MQTT messages, then your gateway(Rpi) would not require Node-Red to do any translating...it could just simply collect and publish your MQTT topic to your broker correct? Your Ubuntu server, acting as your presentation/management platform, would just need to subscribe/publish to the appropriate topics and get/receive data and format it for human consumption as you are already doing.
Your architecture is very, very similar to what I have in mind, and building, with some exceptions in terms of what functions are performed where and what software/tools are used.