Author Topic: No ACK received in case of heavy message traffic  (Read 2489 times)

koenvanvaerenbergh

  • NewMember
  • *
  • Posts: 7
No ACK received in case of heavy message traffic
« on: October 09, 2018, 08:32:52 AM »
Hi,

Under heavy stress ACKs seem to get lost always.
My implementation uses the standard RFM69 library. It has one base moteino and several nodes (all moteinos with RFM69HCW). Messaging them is conversation style:
* the base sends a command via sendwithretry and waits for ACK
* the node sends an answer back via sendwithretry

Sometimes the base gets a timeout but gets the answer anyway. Under heavy stress (when messages are sent very frequently) the message gets through (thanks to the retry loop) but the ACK not. Awaiting for the answer message is not always the solution since it is not certain it will come and this can also take quite some time.

My code does not contain any delay statements in order to prevent the Moteino to be 'stuck' for short periods.

Question: is there an alternative way ACKs may be received in a more reliable way?

Thanks,
Koen

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: No ACK received in case of heavy message traffic
« Reply #1 on: October 09, 2018, 09:12:13 AM »
Hi,

Under heavy stress ACKs seem to get lost always.
My implementation uses the standard RFM69 library. It has one base moteino and several nodes (all moteinos with RFM69HCW). Messaging them is conversation style:
* the base sends a command via sendwithretry and waits for ACK
* the node sends an answer back via sendwithretry
ACKs should be sent with sendACK(), not sendWithRetry() and should be sent as soon as possible after (in your case) the base's sendWithRetry().  Note that, soon after the sendACK() is called, your base could receive another signal and, in this case the original receive data, senderid, etc., could be lost.  So, as a rule, after successful return from sendWithRetry() always save all pertinent information (DATA, DATALEN, SENDERID, etc) and then sendACK() before doing anything else.  This makes for the most reliable connections.

koenvanvaerenbergh

  • NewMember
  • *
  • Posts: 7
Re: No ACK received in case of heavy message traffic
« Reply #2 on: October 09, 2018, 10:11:13 AM »
Hi Tom,

I was unclear: of course after reception, firstly all data is copied locally (SENDERID, RSSI etc) and a SendAck() is sent immediately before any other action. After that, the message is processed and a response (containing values) is sent.
So I'm respecting those rules. However the ACK sent via SendAck() does not always arrive, especially under high stress.
Any suggestions?

thx,
Koen

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: No ACK received in case of heavy message traffic
« Reply #3 on: October 09, 2018, 11:00:32 AM »
However the ACK sent via SendAck() does not always arrive, especially under high stress.
From what you've described, with the 'base' initiating transactions, how a 'high stress' condition could even occur.  If the base controls pacing, how could there even be overlapping messages.  Something's missing in your description of the sequencing.

koenvanvaerenbergh

  • NewMember
  • *
  • Posts: 7
Re: No ACK received in case of heavy message traffic
« Reply #4 on: October 10, 2018, 04:15:19 AM »
You're right on that level of communication, however there's more:
the nodes have sensors, so upon events (or timer) they send measurement data to the base. Depending on the type of sensor and the amount of events, you can get a whole lot of sensor messages (100s per sec), and when the ACKs are missing it retries even more (resulting in the base being flooded with copies of the same message).
And in that case, the entire network goes down (all nodes are impacted) and the base can even not request the sending node to stop sending. The only thing I can do is manually powering down that node, and then the network (slowly) recovers.

Of course, I can try to bypass the problem by limiting the amount of messages per sec a node can send, but that is only a patch and it does not solve the fundamental flaw of the network protocol. Moreover, when the amount of nodes increases, limiting might not be sufficient.

So the problem with one node retrying sending many messages and bringing the communications down remains.

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: No ACK received in case of heavy message traffic
« Reply #5 on: October 10, 2018, 07:54:41 AM »
So the problem with one node retrying sending many messages and bringing the communications down remains.
And all messages are vital?  In other words, perhaps send without ACK on all nodes will simply lose any collided packets, but it won't 'bring down the network'.  If the sampled data is continuous and you use sequence numbers for each packet you might be able to interpolate lost packets.

In general, interlocks are good until they become the burden.  In this case, far more sophisticated protocols are necessary to avoid potential deadlocks and the complexity increases substantially.

koenvanvaerenbergh

  • NewMember
  • *
  • Posts: 7
Re: No ACK received in case of heavy message traffic
« Reply #6 on: October 10, 2018, 08:05:26 AM »
Indeed, in this case they are not all vital. But it's not the issue of losing them that worries me, but the fact that they congest the network, and it seems that the "weakest" packets fail first: the ACKs (which are sent only once in the standard RFM69 lib). And that makes the lib retry sending packets even more.
ACKing a vital packet does not work at a certain load.

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: No ACK received in case of heavy message traffic
« Reply #7 on: October 10, 2018, 10:45:09 AM »
Indeed, in this case they are not all vital. But it's not the issue of losing them that worries me, but the fact that they congest the network, and it seems that the "weakest" packets fail first: the ACKs (which are sent only once in the standard RFM69 lib). And that makes the lib retry sending packets even more.
ACKing a vital packet does not work at a certain load.
Ok, one more time.  If you are consuming on the air time retrying non-vital packets, then you're creating a 'wall' preventing vital packets getting through.  If you simply use send() for non-vital packets and not burden the other end to send ACK then you'll maximize BW at both ends and success rate of vital packets.

koenvanvaerenbergh

  • NewMember
  • *
  • Posts: 7
Re: No ACK received in case of heavy message traffic
« Reply #8 on: October 10, 2018, 11:38:00 AM »
Ok thx