Moteino - Raspi3 serial port flow control

[koenvanvaerenbergh]

Hi,

I've been writing my own Moteino based home automation system for a while, with a moteino connected to a Raspberry Pi 3 as gateway.
The Pi runs a web server from which messages can be sent to and received from the nodes. Raspi3 and Moteino are connected via a serial connection like Felix' gateway on https://lowpowerlab.com/guide/gateway/setup-hardware/ and communication works in all directions.
However: when I send the packets from Raspi too fast to the Moteino, the data gets lost (or overwritten). I assume the Moteino does not process the packets fast enough (it has to handle radio etc too), but we must accept moteino is busy from time to time. So: It seems to me serial port flow control (to let the Raspi pause sending data) is missing.

How should I solve this?
And how does Felix deal with this in his code, because I don't see any special precautions.
And is there no problem when the Moteino gateway sends an incoming message to the Raspi just while the Raspi is sending something to the Moteino gateway too?

I deliberately sent no code since it is quite long and hope to learn about a more generic solution.
My data packets are binary packets (unlike the one-charachter messages Felix uses) and are only processed by the sketch the moment they are received entirely in the serial buffer. The Raspberry Pi 3 runs Windows IoT core with an entire C# 4.0 stack doing also the serial communication from that side.
Receiving data from the nodes is no problem (since the Raspi processes them faster than the Moteino can feed them so no buffer overflow there), sending data from the Raspi to the nodes is the problem.

Best,
Koen

[Felix]

However: when I send the packets from Raspi too fast to the Moteino, the data gets lost (or overwritten)

Koen,
The Moteino serial buffer is quite small compared to a RaspberryPi.
You cannot expect to send too much serial data serial data and have everything work. At one point it just overflows.
In normal use and even with many nodes that transmit *reasonably occasionally*, this should not happen. There is the serial buffer limitation, and also the radio spectrum limitation, each packet needs to be sent out (and ACK takes even longer), if serial comes in faster than packets going out/ACK'd, then there's a problem. In my code there is no special handling. I have not had this need, nor have users who use the same setup and have potentially dozens of nodes. If this is a big problem for you, it's possible to implement a protocol to avoid bottlenecking the serial port of the Moteino, then you send serial and wait until flagged ready to send again.

Another development is using the serial port at much higher speeds, see this thread by LukaQ. That would be a relatively simple change of baud to 250/500kbps and see how it works (just the sketch change would be OK, bootloader can still work at 115200). I have yet to experiment with this.

[koenvanvaerenbergh]

Hi Felix,

Thanks for your info.
Yes I assume the serial buffer is only 64 bytes.

My setup requires more traffic (e.g. nodes requesting a dhcp alike address and also all nodes have the same software and their behaviour is sent to them via a set of configuration records).
Also, the nodes are able to react on several messages such as diagnostic ones or to execute an action. When a user clicks a button too fast messages would start overwriting each other in the serial buffer resulting in partial messages etc that could end up in wrong behavior.

For now, I implemented some flow control for messages from Raspi to Moteino by letting moteino ACK them before sending the next.
Increasing serial speed would not improve the problem since the overwriting problem depends on the speed the serial message is being processed, not being sent (because while sending, the other messages wait in a queue).

Thanks for your answer, that made me conclude that I had quite a special case here so I had to look for an own solution 

Best,
Koen