dropped transmissions [solved]

greg · May 13, 2016, 02:59:19 PM

Hi. I'm sorry if this info is repeated elsewhere. I have searched and looked through the examples.
I have 9 moteinos and more coming online soon. I am dropping transmissions between moteinos.
I am not finding much information about using radio.send vs radio.sendwithretry (and am using all
send with retry). also can't find anything on why you would use radio.sleep....
But, I will attach two of my sketches. I have one gateway getting input from a serial connection
from a linux host. The serial input is the node to send to and the command to send. I will also post
the log file of what is happening.
The node sketch is a moteino that merely lights up LEDs based on the status of other moteinos.
i.e. garagemote has door open, light an LED.
In the node sketch search for the transmission of "SOS" that should never happen. How could that happen?
Also when it does not send back "SOS", it should (must) send back "18 input: whatever" but sometimes
the gateway prints neither. Please point me in some direction about how to not drop transmissions as I have
more and more moteinos coming online and my airwaves getting busier.
Thanks!!!!!

TomWS · May 13, 2016, 07:11:14 PM

What makes you think it's 'dropped transmission' vs 'lost reception'?

Typically the problem is lost packets due to the receiver being busy doing something else or, overwriting the incoming data.

Tom

greg · May 18, 2016, 10:10:50 AM

Dropped transmission vs lost reception seems like semantics to me. Something was sent and not received.
I have been playing with the sketches, and still want to know more about sendwithretry vs just send. also
what is the purpose of radio.sleep? Am I completely missing something in how you program the radios???
Current status is that I am using all sendwithretry. I have two nodes sending back and forth. the gateway node sends
a status command to the remote node. It replies back with 3 lines of text. I put the process into a loop, and
don't have to wait to long for a failure (this should be easy for anyone to replicate). The remote node is also counting
the number of lines it sends, the tries it takes to send those lines and complete send failures (zero).
The gateway sends the status command to the remote node. it believes it sent it ok, based on sendwithretry.
The remote node never saw the transmission.... SO.. how did the gateway get the ACK???
Is it possible to code the transmissions to guarantee no dropped messages?
Again sketches are attached.
Log file error looks like this (larger version attached):
PLEASE PLEASE HELP!!!!

5/18/16-08:46 1 serial input: 18stat. gateway gets stat command from linux for node 18
5/18/16-08:46 1 message sent OK to node 18 in 1 tries send with retry says it sent OK in 1 try
5/18/16-08:46 18 LED 3:sec:OFF:6:GWRK:ON:7:GWiFi:OFF:10:GOUT:OFF remote node 18 replies back with first of three lines
5/18/16-08:46 18 LED 12:CWRK:OFF:13:CWiFi:OFF:14:COUT:OFF:15:GAR:OFF remote node's 2nd line of text
5/18/16-08:46 18 Fail:0,send=30,try=30 last line, no send failures, 30 lines of text sent in 30 tries.

5/18/16-08:46 1 serial input: 18stat. gateway sends stat command to node 18
5/18/16-08:46 1 message sent OK to node 18 in 1 tries sent OK in one try
RIGHT HERE SHOULD BE STATUS LINES COMING BACK
5/18/16-08:46 1 serial input: 18stat. gateway sends stat command to node 18
5/18/16-08:46 1 message sent OK to node 18 in 1 tries sent OK 1 try
5/18/16-08:46 18 LED 3:sec:OFF:6:GWRK:ON:7:GWiFi:OFF:10:GOUT:OFF 1st line of text back
5/18/16-08:46 18 LED 12:CWRK:OFF:13:CWiFi:OFF:14:COUT:OFF:15:GAR:OFF 2nd line of text back
5/18/16-08:46 18 Fail:0,send=33,try=33 still no send failures, only up to 33 sent lines and 33 tries, so like the
acknowledged transmission was not seen by 18.

TomWS · May 18, 2016, 01:01:43 PM

How often does the linux device send commands to your gateway. The timestamp isn't changing at all, implying that it is quite often.

From the gateway code, it appears that there is a risk of the gateway missing the response from the node and receiving a new command before the gateway checks for a response again. If the command uses send you'll wipe out the receive buffer. Why your node thinks it's getting an ACK is odd. Perhaps all that is happening is that your gateway's serial print buffer is being overrun. Put a Serial.flush() at the end of your receiveDone check block.

Also, for this series of tests, comment out the CheckForWirelessHEX() calls.

Tom

greg · May 18, 2016, 02:22:02 PM

Thank you very much for the reply.
PLEASE TAKE THE TIME TO READ THROUGH THIS LONG DIATRIBE
AND OFFER MORE ADVICE!!!!!

Continuing to pound on this, I simplified my testing.
I am using my gateway and the moteino closest to it physically (a node attached to a winders
machine to do the wireless programming).
I added very minimal code to the wireless programming sketch available on the web.
What I am seeing is that the time required to run through a loop in the code can vary dramatically.
If I manually send a command through the gateway, get a response back from the remote node and print it,
normally the round trip takes about 1 second.
If I put that process into a loop, I can run into errors if I send commands as often as every 7 seconds.
Under "normal" situations I do not send commands very often. However occasionally I run an automated
process that checks the status of each moteino in the system to ensure everything is in sync. When I do that,
I send a status request to one garagemote, upon getting its response that the door is open/closed, the serial
interface on the linux server automatically sends commands to two other moteinos telling them to light/unlight
led bulbs to warn the door is open (left open... kids). Then a status command is sent to the next garage mote.
Again the serial interface automatically tells the other two moteinos to light/unlight LEDs. I only use one LED in
two different locations to notify the door is open. So, I send commands to the LED control moteinos much more often
than the "safe" every 7 seconds. I intend to 1. check the first garage mote early in the automated process and the
second later in the process which will greatly spread out the quick sends while the other moteinos in the system are
checked. and 2. only send the LED controllers on/off commands if they are actually out of sync with the door. Currently
the interface tells them to light/unlight every time the door is checked even if the status is already correct.

Note that I can still get into trouble. my wife could use my mobile app to tell a moteino to do something at the same time
I tell it to which may result in a lost communication but the probability is very low, and if we are both talking to the same
moteino, we are both probably telling it the same thing and one of the transmissions will certainly work.

Bottom line, I currently believe I need to limit transmissions to any given moteino to no more often than every 10 seconds.
Please let me know if you disagree with that. Thank you very much!!!!

TomWS · May 18, 2016, 05:52:03 PM

Quote from: greg on May 18, 2016, 02:22:02 PM
<...snip>
Bottom line, I currently believe I need to limit transmissions to any given moteino to no more often than every 10 seconds.
Please let me know if you disagree with that. Thank you very much!!!!

It sounds like you've got race conditions that are causing collisions and missed data. There are a couple ways around this:

1. As you say, limit the rate at which you send so that you try to minimize collisions.
2. Modify your code so that it is tolerant of the collisions.
3. Create interlocks that will avoid the collisions.

1. is easy.
2. a little more difficult, but not too bad.
3. Much trickier and potentially fatal due to unanticipated lockouts...

You need to decide how much effort you want to invest in this.

As a point of reference, I currently have something in the order of 35 motes running various monitoring/control functions at this site and I don't do anything special to meter traffic. I just try to make sure they're tolerant of the inevitable collisions. Also, a single two way exchange between two moteinos shouldn't take more than 80 milliseconds.

Tom

greg · May 19, 2016, 12:29:05 PM

I am posting this just as information for anyone that has been watching this thread.
I have changed my code again..and again... and again since my last post and things are working very well right now.
I will post the sketches so you have them in their entirety (as I have all along (and you can compare if you want)).

Basic changes:
1. I was doing the radio.sendwithretry inside a loop on both the gateway node and the remote node.
I changed that to be more like Felix's garagemote where the remote node just does one sendwithretry
without even checking the status. I kept the gateway radio.sendwithretry inside a loop and will see
occurrences where multiple sends were required.

2. I changed all code to spend as much time at the radio.receivedone as I could. 99.999 percent of the time
the things loops, it will not be doing anything, and I was running through if this then if that then if this etc.
Now I have one check, in essence "do I have a command to process", if so, run through the ifs, if not don't.

3. All my moteinos have a command called "command" to list commands available on that particular node (my memory isn't very good).
Now you have to run the "command" command to flip a switch where the node will start listening for checkforwirelsshex
else it bypasses that check (10 does not have the check. local connect, upload directly.. cant send to itself anyway).

4. Finally after all that and things were pretty running OK with an occasional error, to emulate Felix as much as I could, I added
a blink of the LED back in (which is silly, you can't see any of my nodes directly). BUT After doing that, I HAVE NO ERRORS!!!
Praise the Lord!

Felix · May 19, 2016, 01:16:40 PM

Thanks for the updates!
I marked this as solved.