Author Topic: 1% packet loss [Solution: 0% loss using sendWithRetry()]  (Read 5519 times)

pj

  • NewMember
  • *
  • Posts: 3
1% packet loss [Solution: 0% loss using sendWithRetry()]
« on: March 01, 2019, 05:17:49 PM »
Hi,

I'm using LowPoweLab 1.2.0 library with RFM69W 868MHz connected to arduino pro mini or pro micro 3.3V.

On 1 of the node, I'm sending a packet (13 bytes) using radio.send every 20ms:
Code: [Select]
radio.send(RECEIVER, (const void*)&rcdata, sizeof(rcdata), true);

On the other node, I send ACKs (10 bytes) like this:
Code: [Select]
if (radio.ACKRequested()) {
      radio.sendACK((const void*)&ackdata, sizeof(ackdata));
}

I'm seeing quite consistent ~1% packet loss in the SENDER > RECEIVER direction.

Whereas the ACKs in the opposite direction show only ~0.1% packet loss. The ACKs uses its own packet counter, so the 0.1% packet loss represents only the lost ACKs.

RSSI is around -50/-75 for each node respectively.

The 1% packet loss is more or less consistent even if I set the sending interval to 40ms, 100ms or even 1s.

Setting the requestACK to false in radio.send does not make any difference.

Enabling / Disabling "#define ENABLE_ATC" does not make any difference either.

Power source does not seem to be an issue. I tried powering the setups with USB cable from a PC (pro micros), FTDI (pro minis) or even with quite strong table power supply.

I have tried swapping sending/receiving RFM69 units as well as the arduinos, even swapped pro mini with pro micro. No change in behavior.

Same results whether I place the sender / receiver 0.5m or 15meters far from each other.

It is always the SENDER > RECEIVER direction showing ~1% packet loss.

Interesting enough, I found several posts where other people reports 1% packet loss too. Not necessarily considering it to be an issue:

https://lowpowerlab.com/forum/moteino/frequent-retransmits-using-sendwithretry/msg7210/#msg7210
https://andrehessling.de/2015/02/07/figuring-out-the-power-level-settings-of-hoperfs-rfm69-hwhcw-modules/#comment-1032
https://lowpowerlab.com/forum/moteino/is-packet-loss-common/msg19011/#msg19011
https://openenergymonitor.org/forum-archive/node/10900.html (Submitted by Ian Davies on Thu, 30/07/2015 - 23:00.)
https://lowpowerlab.com/forum/projects/another-alarm-system/msg20231/#msg20231

Just wondering if somebody else is seeing this or maybe if it is a known behavior.

Thanks,
Peter
« Last Edit: March 11, 2019, 10:02:29 AM by Felix »

KrisK

  • NewMember
  • *
  • Posts: 26
  • Country: us
Re: 1% packet loss
« Reply #1 on: March 08, 2019, 12:32:22 PM »
I have recently gone through round trip testing of the Moteino to verify the life of the transmitter as well as functionality between 3AAA batteries and 9V batteries in my enclosure.

In summary I have not seen any packet loss or transmit errors.  I am at nearly 1 million packets from my controller to my receiver with ACK back from the receiver.    I have tweaked the attached code over the last 5 weeks and have settled on a 6 byte structure and using sendWithRetry set to 6.   The second sendWithRetry was only called when battery started to give out on the receiver.  My testing is from second floor computer room to finished basement far corner.  I have also tested outside in my truck.    I totally believe the sendWithretry was key along with making sure you verify the data as soon as you get it else the sender moves on.   Because of a previous code issue and issue with the laptop I downloaded all new library's and AVR as of Feb 5th.   ATC isn't used and promiscuous is off with ACK flag set to 30 ms.  Everything else is out of box from the Examples Felix has on struc_send struc_receive    The antenna is a PCB sticky back antenna for cell  https://www.adafruit.com/product/1991   It is working like a champ.   I also have used the DIPOL and simple stick antenna. 

I have attached the code snippets I used.   Below are some stats I discovered.   Basically I had a checkdigit function with variable as part of the data packet.  I was so happy I never got a packet botched I removed the checkdigit function and variables giving me 85% more efficiency.    The chart below shows what happen when I removed it.  Then I added in a simple check of datalen and theNODEid from packet header as my check digit poor mans packet QC.

I hope this helps.   At least this is what I am going with on a product I am building.

Kris

                                               Large Packet 8 bytes           Large Packet 8 bytes                         Small Packet 6 bytes with
                                              Check-Digit and Parse          Receive with ACK only                  Packet header and Data Len check
Round Trip time MS                          63.22                                    9.46                                                         9.66

1st Tier round trip time                     61.94                                    9.27                                                         9.27                                   
% of 1st Tier to Total                        98.41%                               99.69%                                                   99.12%

2nd Tier Round Trip time                 134.58                                   52.44                                                       52.51
% of 2nd  Tier to Total                       1.5%                                      .24%                                                      .86%

3rd  Tier Round Trip time                  270.8                                  135.72                                                       98.06
% of 3rd  Tier Total                            .09%                                      .07%                                                      .02%
« Last Edit: March 08, 2019, 12:35:47 PM by KrisK »

pj

  • NewMember
  • *
  • Posts: 3
Re: 1% packet loss [SOLVED?: 0% loss]
« Reply #2 on: March 08, 2019, 06:34:03 PM »
Hi KrisK,

Thanks for testing and sharing your test data.

Yes, with sendWithRetry you probably achieve ~0% message loss. But sendWithRetry basically means that you take packet loss into account. sendWithRetry actually does re-transmissions for you in case a packet is not ACKed. If you would be sure that every SENT packet is RECEIVED by the receiver, you do not need to use sendWithRetry. You would use plain radio.send function which does not do any re-transmissions.

That's why I use radio.send. With radio.send on the SENDER side, I just send a packet with increasing packet counter in it. On the RECEIVER side, I check the packet number received in the packet sent by the SENDER. If there is a gap between the previously received packet # and the current packet #, I know that the previous packet from SENDER did not make it to the RECEIVER (or possibly was silently dropped by the receiver). And this is where I see the 1% packet loss.

In you send your packets with relatively big delays between each packet, let's say 50ms or more, you can afford using sendWithRetry. But if you want to send a packet every 20ms (like I do), you cannot use sendWithRetry. Simply just because the round-trip-time is around 8ms. Timeout waiting for an ACK would take some additional time (10ms at minimum). So, basically, you are at 18ms, which is around the time you would send a new packet anyway (I do it every 20ms).

Let's say you lost 1 packet out of 100 (1%). If you would re-transmit (with retries set to 1) in case a packet is not ACKed by the sender; and we assume that every 1% of the re-transmitted packets is lost too, you will effectively need to send 100 x 100 packets to see 1% of not delivered messages. In your case, you use sendWithRetry with 6 retries. It means you theoretically need to send 100 x 100 x 100 x 100 x 100 x 100 x 100 = 10^14 packets to see 1% of not delivered messages.

It would be interesting to see the packet loss if you would use plain radio.send function instead of sendWithRetry. You would see ~1% packet loss, I believe. You don't even need to send any ACKs back to the SENDER. Just check for a gap in the packet numbers on the RECEIVER side.

It is interesting that I see this behavior with any packet frequency (20ms, 100ms, 1s), even with RSSI being ~-30 on each side. And even more interestingly, ACK packets are lost only in about 0.1% cases.

Thanks,
Peter
« Last Edit: March 08, 2019, 07:16:03 PM by pj »

KrisK

  • NewMember
  • *
  • Posts: 26
  • Country: us
Re: 1% packet loss [SOLVED: 0% loss using sendWithRetry()]
« Reply #3 on: March 09, 2019, 02:57:48 AM »
PJ
I guess I luck out.   My controller is operated by humans.   I am happy with even 500milliseconds.   Your points are valid.   I have more end to end and environmental testing to work through, in the next couple of weeks.   

However, I will put this on my KANBAN bucket list to test the "NO" retry and see what I can see.      I actually have a "non human" planned controller that has environmental triggers that might need tighter constraints like you mention.    More than likely it will be 3 triggers sending data simultaneous to the receiver.    3 controllers with sendWithRetry (6) could generate a lot of simultaneous retry noise.

yeeee haaaa..   If it was easy we wouldn't be having fun.
Kris
 

pj

  • NewMember
  • *
  • Posts: 3
Re: 1% packet loss [SOLVED: 0% loss using sendWithRetry()]
« Reply #4 on: March 09, 2019, 06:31:32 AM »
Kris,

I appreciate you will put this test on your bucket.

I look forward for the results.

Thanks,
Peter

vc1234

  • NewMember
  • *
  • Posts: 5
Re: 1% packet loss [Solution: 0% loss using sendWithRetry()]
« Reply #5 on: March 19, 2019, 09:39:04 AM »
I've experienced similar un-retried packet loss.

In my setup, I sent packets between two RFM69 nodes and watched the traffic from the third RFM69 node.  Usually, the sender was sending a packet but the monitoring node did not see it in the slight majority of packet loss situations, occasionally the monitor saw a corrupted packet and occasionally the echo node failed to receive.

From sheer curiosity, I tried using the same LPL RF settings with a CC1101 module as a sender.  The situation was much better.  There was no sent packet loss (out of about 20K packets) with occasional echo server (an RFM69 module) failing to receive.

What is interesting is that an RFM69 module seems to receive more noise than the CC1101 with presumably the same sensitivity (RSSI reading were pretty close usually to a digit).  I saw a similar behaviour in my experiments with receiving zwave packets where a RFM69 module was more sensitive to RF noise, especially when connected to an RPI or an ESP8266.  It is not the collision detection logic that caused sent packet loss because the module notified the MCU via the interrupt pin that the packet was allegedly out of the fifo.

It's hard to extrapolate my results to your environment though because RF noise is tricky thing, and one needs an expensive spectrum analyzer to understand why the thing behave the way they do.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: 1% packet loss [Solution: 0% loss using sendWithRetry()]
« Reply #6 on: March 19, 2019, 10:12:17 AM »
From sheer curiosity, I tried using the same LPL RF settings with a CC1101 module as a sender.
Thanks for the report, but how was that done exactly?
Did you still keep an RFM69 as a receiver?
What was the lib/code on the sender?
It's also important to note how the hardware was setup and how well the antenna was matched. In such comparison you really have to get as much apple-to-apple comparison in all aspects.

FWIW - ESP and Pi are very noisy, so no wonder at all that the RFM69 will pick up noise when directly grounded/powered from those devices.

vc1234

  • NewMember
  • *
  • Posts: 5
Re: 1% packet loss [Solution: 0% loss using sendWithRetry()]
« Reply #7 on: March 19, 2019, 01:17:08 PM »
Case 1:
Two moteinos (RFM69). One sends a small packet containing just an incremented seq number, and the other node echos it back.  Both use the vanilla LPL library.  The third non-moteino but still an RFM69 node listens to all packets using the same LPL lib.  In this case about 1% or slightly less packets are lost mainly due to the sender packet not seen by either the receiver or the third monitoring node. Perhaps the packet is transmitted but not shaped quite right or distorted by ambiant noise.  It's hard to capture the packet without proper equipment.

Case 2:
The sender is a CC1101 module using my own simple library and the LPL settings (mod/bit rate/dev, etc).  The monitor node is the same RFM69. Packet loss is dramatically lower and probably exclusively due to the echo server not receiving/transmitting about 0.05% cases (one in two thousand). E.g. the CC1100 sends three packets  with the same seq number (including two retries), the monitor node sees all of them and the echo server response to the last one, eventually.  I did not check whether the echo server was unable to receive or transmit properly in this scenario.

In both cases, I disabled encryption and CRC-16 because they are not compatible between RFM69 and CC1101 The latter does not support encryption and its CRC-16 is different bitwise.