Author Topic: Frequent retransmits using sendWithRetry  (Read 3808 times)

joelucid

  • Hero Member
  • *****
  • Posts: 868
Frequent retransmits using sendWithRetry
« on: May 26, 2015, 07:05:43 AM »
Here's something that's been puzzling me for a while:

I have a raspberry 2 with 433mhz moteino as gateway and multiple moteinos communicating sensor data to the gateway. The sensors typically sleep and at defined intervals send updates to the gateway.

The sensors regulate their transmit power based on packet loss according to the following scheme: if a packet doesn't get acknowledged by the gateway three times powerLevel gets incremented by one. Every 5 updates powerLevel gets reduced by one. Since I'm using RFM69HW's almost all Moteinos converge to a powerLevel of 0 with this scheme. Since the gateway is AC powered it always sends acks to the clients at full power.

I used to have lots of trouble with interference from the raspberry, but since switching to a model 2 everything is much smoother. I do however frequently receive the same packet multiple times at the gateway. This means that the clients do not receive the ack from the gateway and resend even though the gateway did receive and ack the message. This is especially surprising since the sensors send at powerLevel 0 and the gateway acks at 31.

I used to think this might still be interference from the Pi. But since I developed my own wireless boot system I know that the Pi can send packets to the clients even at 200kbps using my own radio code almost flawlessly (I rarely get more than 1% retransmits using wireless boot). As an example check this out:

Code: [Select]
05/26 10:17:51: !boot,nd:13,c:26631,ac:54924
Boot request from 13, checksum 26631
Starting to boot 13
Bootloader CRC matches
Installing new app
Serving Client
New profile: 1, timeout: 30
New profile: 0, timeout: 10
Got ack in: 2534, drop rate: 8 empty: 0
Boot client done
05/26 10:17:53: [13] [RX_RSSI:-56]
05/26 10:17:53: >nd:13,nr:0,pw:0,vc:341,it:24,l:1,rs:8,h:376,t:197,ls:0
05/26 10:18:05: [3] [RX_RSSI:-28]
05/26 10:18:05: >nd:3,nr:21,pw:0,vc:332,it:29,l:1,hp:1,mc:0,h:314,t:209
05/26 10:18:13: [13] [RX_RSSI:-56]
05/26 10:18:13: >nd:13,nr:1,pw:0,vc:340,it:24,l:1,h:373,t:197,ls:0
05/26 10:18:14: [5] [RX_RSSI:-82]
05/26 10:18:14: >nd:5,nr:527,pw:0,vc:330,it:16,h:415,t:173
05/26 10:18:28: [4] [RX_RSSI:-67]
05/26 10:18:28: >nd:4,nr:528,pw:0,vc:329,it:12,h:429,t:152
05/26 10:18:28: [4] [RX_RSSI:-53]
05/26 10:18:28: >nd:4,nr:528,pw:0,vc:329,it:12,h:429,t:152
05/26 10:18:33: [13] [RX_RSSI:-56]
05/26 10:18:33: >nd:13,nr:2,pw:0,vc:340,it:24,l:1,h:372,t:197,ls:0
05/26 10:18:45: [3] [RX_RSSI:-28]
05/26 10:18:45: >nd:3,nr:22,pw:0,vc:332,it:29,l:1,hp:1,mc:0,h:315,t:209
05/26 10:18:53: [13] [RX_RSSI:-57]
05/26 10:18:53: >nd:13,nr:3,pw:0,vc:341,it:24,l:1,h:373,t:197,ls:0
05/26 10:18:53: [13] [RX_RSSI:-56]
05/26 10:18:53: >nd:13,nr:3,pw:0,vc:341,it:24,l:1,h:373,t:197,ls:0

Here I'm wirelessly booting node 13 in 2.5s. I get a drop rate of 8 which means 0.8% packet loss. Further down you see that the same nd:13 transmits update nr:3 twice at stock 55kbps, not seeing the ack from the server - something that should not happen given how stable the connection is at 200kpbs. nd:4 also sends a duplicate update.

I'm using 40ms timeouts which should be sufficient and I've experimented with adding a delay before the ack on the server - but it makes no difference.

I remember from when I developed my wireless boot code that I made some changes to the radio code that brought significant benefits in terms of retransmits. But before I start digging that deep again I wanted to see what others' experiences where like rg retransmits with the RFM69 lib. What kind of drop rates are you seeing and do you see double packets due to missed acks?

Any other ideas about what might be going on?

Joe

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Frequent retransmits using sendWithRetry
« Reply #1 on: May 26, 2015, 07:41:10 AM »
I do see repeats in my own setup but usually they do get ACKd by the second or third try. So in general I haven't run into situations where repeats get stuck and the gateway can't reach a node. If anyone has anything more to share about this please do.

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Frequent retransmits using sendWithRetry
« Reply #2 on: May 26, 2015, 10:09:17 AM »
I do see repeats in my own setup but usually they do get ACKd by the second or third try. So in general I haven't run into situations where repeats get stuck and the gateway can't reach a node. If anyone has anything more to share about this please do.
In cases where I see multiple retransmits, it's usually due to collisions with other motes also trying to send to the gateway.  In these cases it often means a three way collision - the original mote, the new mote getting into the mix, and the gateway's Ack to the original's first transmission. 

And this is why we use sendWithRetry() and also why I don't send 'mission critical' data in a single sample.

I've toyed with the idea of developing an adaptive retry delay, but the issue isn't enough of a problem for me to work on it.

Tom

joelucid

  • Hero Member
  • *****
  • Posts: 868
Re: Frequent retransmits using sendWithRetry
« Reply #3 on: May 26, 2015, 11:24:00 AM »
I found the problem:

I've written some code that uses the RFM69HW module on the gateway using OOK modulation to control 433mhz remote controlled power outlets. This works really well and is a cheap way to control AC plug devices. You can buy 4 sockets for 9.90 Euro here in Germany, very cheap and with the RFM69HW the range is much larger than that of the remote that comes with the sockets.

My code sets the preamble length to 0. Unfortunately the RFM69 lib doesn't initialize that setting so my gateway used a preamble of length 0 when ack'ing client packets after a OOK command was sent. Pretty amazing actually that it worked at all ...

I'm now resetting the preamble length to 3 after using the radio to control sockets. Everything's smooth now. Too bad the radio module can't be reset into a known state via software to avoid these types of issues (or tell me if it can be).

If you're interested how it works check out the attached. Unfortunately sharing the radio requires some dirty hacks as one might suspect.

Joe
« Last Edit: May 26, 2015, 11:55:45 AM by joelucid »

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Frequent retransmits using sendWithRetry
« Reply #4 on: May 26, 2015, 11:47:42 AM »
I found the problem:

I've written some code that uses the RFM69HW module on the gateway using OOK modulation to control 433mhz remote controlled power outlets. This works really well and is a cheap way to control AC plug devices. You can buy 4 sockets for 9.90 Euro here in Germany, very cheap and with the RFM69HW the range is much larger than that of the remote that comes with the sockets.

My code sets the preamble length to 0. Unfortunately the RFM69 lib doesn't initialize that setting so my gateway used a preamble of length 0 when ack'ing client packets after a OOK command was sent. Pretty amazing actually that it worked at all ... Too bad the radio module can't be reset into a known state via software (or tell me if it can be).

If you're interested how it works check out the attached. Unfortunately sharing the radio requires some dirty hacks as one might suspect.

Joe
Cool!  Too bad we can't use those AC sockets on this side of the pond!

I'll look over the code later, thanks for posting!  And I'm glad you found your problem!

Tom

joelucid

  • Hero Member
  • *****
  • Posts: 868
Re: Frequent retransmits using sendWithRetry
« Reply #5 on: May 26, 2015, 12:03:29 PM »
Maybe you can use the same approach with other AC sockets in the US. I've looked through the protocol code in http://www.pilight.org/ and found most 433mhz socket protocols were almost identical. I ported this protocol to the Moteino and it was all very straight forward.

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Frequent retransmits using sendWithRetry
« Reply #6 on: May 26, 2015, 01:41:01 PM »
Maybe you can use the same approach with other AC sockets in the US. I've looked through the protocol code in http://www.pilight.org/ and found most 433mhz socket protocols were almost identical. I ported this protocol to the Moteino and it was all very straight forward.
Thanks!  I've ordered some sockets and a remote (so I can confirm the protocol).  I probably won't get to it until later this Summer, but this should be a useful project.

Tom

WhiteHare

  • Hero Member
  • *****
  • Posts: 1300
  • Country: us
Re: Frequent retransmits using sendWithRetry
« Reply #7 on: December 14, 2015, 10:39:23 AM »
...very cheap and with the RFM69HW the range is much larger than that of the remote that comes with the sockets.

Good to know!  I recently picked up a similar set (http://www.etekcity.com/product/100068.html) for controlling Xmas lights around the house using a cheap 50 cent 433Mhz transmitter and the rcswitch library.  The range has been somewhat mediocre, both with the 50 cent transmitter and with the hand-held remotes (which are driven by small 12v batteries), though a line-of-sight link seems to work pretty well.
« Last Edit: December 14, 2015, 11:23:49 AM by WhiteHare »