AFC Not Working/Flaky in RF69HCW

[jwketchum]

I am having a problem getting AFC to work.
Here is what I am trying to do:
I am building a multi-node outdoor sensor network, using RF69HCW nodes, that needs to operate at seasonal temperature extremes with an indoor gateway.  By temperature extremes, I mean at least 0F to 100+F, which is typical of my area in coastal eastern New England.  Range is also important.
My initial prototype was operated indoors using data rate of 4800 and Fdev of 4800. Frequency is 904020000 (somewhat arbitrary, but I will operate in the 915 ISM band.) Also turned on ATC. I left other radio parameters pretty much default as they are set in the RFM69 library,  This worked nearly flawlessly over indoor ranges, until I got a new device, and tried taking one of the nodes outdoors, at which point the links got unreliable or failed completely.  This is clearly due to temperature drift and also XO variability. 
Next step was to go to a higher data rate, with associated wider Fdev, and increase RxBW to accommodate the higher data rate and 20 kHz of frequency offset:

Code Select Expand

  
  radio.writeReg(REG_BITRATEMSB, RF_BITRATEMSB_19200);
  radio.writeReg(REG_BITRATELSB, RF_BITRATELSB_19200);
  radio.writeReg(REG_FDEVMSB, RF_FDEVMSB_20000);
  radio.writeReg(REG_FDEVLSB, RF_FDEVLSB_20000);

  // RxBW is one sided bandwidth, so RF bandwidth = 2*RxBW
  // set the RxBW to 42 kHz, so RF BW = 84 to match 2*FDEV + BITRATE + Max frequency error =~ 80 kHz
  //  These parameters (Mantissa = 24, Exponent = 3, get us 32e6/24/32 ~= 83kHz
  radio.writeReg(REG_RXBW, (RF_RXBW_DCCFREQ_010 | RF_RXBW_MANT_24 | RF_RXBW_EXP_3));
  // Set RSSI Threshold to D0 ==> -104 dBm
  radio.writeReg(REG_RSSITHRESH, 0xD0);
  // increase preamble length to 6 bytes, and RF sync length to 4 bytes
  radio.writeReg(REG_PREAMBLEMSB, 0x00);
  radio.writeReg(REG_PREAMBLELSB, 0x06);
  radio.writeReg(REG_SYNCCONFIG, RF_SYNC_ON | RF_SYNC_FIFOFILL_AUTO | RF_SYNC_SIZE_4 | RF_SYNC_TOL_0);
  //sync words are 31-ML sequence with a 0-bit tacked on
  radio.writeReg(REG_SYNCVALUE1, 0xFA);
  radio.writeReg(REG_SYNCVALUE2, 0x25);
  radio.writeReg(REG_SYNCVALUE3, 0x61);
  radio.writeReg(REG_SYNCVALUE4, 0xCC);

This so far works fine, but anticipating issues with aging, greater variability from device to device, and greater temperature offsets than I am currently experiency, I want to get AFC working.  However, when I try to enable AFC on my devices, links that work error free without it become either unreliable, or fail completely.  This happens even at short range (-40 to -60 dBm).

Following advice in the forum, and particularly in the NXP AN4983 document, I have tried to use recommended configuration for AFC, but for the moment stick with the 19200 BPS data rate and 20kHz Fdev.  Here are the settings I use,

Code Select Expand

// Set RSSI Threshold to D0 ==> -104 dBm
  radio.writeReg(REG_RSSITHRESH, 0xD0);
  // increase preamble length to 6 bytes, and RF sync length to 4 bytes
  radio.writeReg(REG_PREAMBLEMSB, 0x00);
  radio.writeReg(REG_PREAMBLELSB, 0x06);
  radio.writeReg(REG_SYNCCONFIG, RF_SYNC_ON | RF_SYNC_FIFOFILL_AUTO | RF_SYNC_SIZE_4 | RF_SYNC_TOL_0);
  //sync words are 31-ML sequence with a 0-bit tacked on
  radio.writeReg(REG_SYNCVALUE1, 0xFA);
  radio.writeReg(REG_SYNCVALUE2, 0x25);
  radio.writeReg(REG_SYNCVALUE3, 0x61);
  radio.writeReg(REG_SYNCVALUE4, 0xCC);

  radio.writeReg(REG_AFCFEI, radio.readReg(REG_AFCFEI) | RF_AFCFEI_AFCAUTO_ON | RF_AFCFEI_AFCAUTOCLEAR_ON);

  // Make the AFC BW a bit wider (Mantissa = 16, Exponent = 3 gets us 32e6/16/32 = 62.5kHz
  radio.writeReg(REG_AFCBW, (RF_AFCBW_DCCFREQAFC_010 | RF_AFCBW_MANTAFC_16 | RF_AFCBW_EXPAFC_3));

//  Turn on " optimized AFC " for low bandwidth signal
  radio.writeReg(REG_AFCCTRL, RF_AFCCTRL_LOWBETA_ON);
  radio.writeReg(0x71, 0x07);

I have tried this with and without the "optimized AFC" settings, but it doesn't make a difference.  It is unclear whether this is a good idea when Fdev ~= BR.   Also note that I am treating RxBW and AFCBW as single-sided values, so that the effective RF bandwidth values are twice that.

With AFC running, the gateway node reports frequency offsets in the range from a few 100 Hz to more than 15 kHz, when it actually receives a packet.  These values do not seem at all stable, they vary pretty wildly from packet to packet.

Also complicating matters, I am having trouble with changing the AFC settings:  If I run the nodes for a few minutes with the settings above, then edit the code to turn off AFC, and upload the edited, newly compiled code, the new register values (specifically REG_AFCFEI) don't get reset.  I typically have to remove power and let the devices sit without power for a few minutes, then upload the code again, to get AFC really turned off.  This happens on at least two devices, so I don't think that it is a device-specific hardware issue.

Also, the prevalent failure mode is for the gateway node to receive packets from the sensor node more reliably than the sensor node receives ACKs, so that all uplink packets go un-acked from the POV of the sensor node, but the gateway node sometimes gets three copies of packets from the sensor node, due to retries.

Apologies for the long-winded post, thank you in advance for reading it and any input you might be able to provide in helping me get AFC up and running.

--John

[perky]

Your settings look OK, but AFC won't start until the RSSI threshold is reached. If the threshold is set too low it'll trigger on noise and you'll get invalid results. Try raising the RSSI threshold, say to -80dB. If that works you'll need to drop it until you start to get false triggers, then raise it a bit to reduce their rate of occurance. You can think of the RSSI threshold as a filter, the higher the threshold is set the less likely to get a false trigger, but you will get one sometime.

I note you've read the excellent NXP app note on the AFC. I would recommend you follow this for eliminating false trigger events because they can mess things up with AFC, you need to be able to quickly detect a false trigger event and restart the receiver to restart the AFC process. That can come later though, to get it working initally just set it high so no false triggers are received.

Mark.

[jwketchum]

Thanks for your input Mark. 

I have tried raising the RSSI threshold and continued to have problems.  After getting your note, I tried again, and still problems.  After some extensive debugging, I have found that at least part of the problem I am having has to do with a delay in the sendACK function, associated with setting the RSSI Threshold at -80 dBm.   sendACK won't actually send an ACK until it senses the channel as clear, or a timer expires.  Timer expiration takes one second.  Channel clear means receive RSSI is below a hard-coded value of -90 dBm.   For reasons that I don't understand, when I increase RSSI Threshold above -104 dBm, to -80 dBm, the receive  RSSI value reported in the wait loop goes above the channel clear  threshold, even though nothing else has changed.  As a result, sendACK never senses the channel clear, so waits a full second for the timer to expire before sending the ACK

Since the timeout on send ack takes a second, my node wasn't getting the ack in time. and continued to send specified number of retries, even though the first attempt was succeeding.

I can't see why changing the value in REG_RSSITHRESH should affect the received RSSI value reported in the sendACK wait loop.   Any thoughts?

[perky]

I had real trouble gettting dynamic RSSI measurements to work, there's a long thread about this with some apparent workarounds (especially some results on page 4). Some people have problems with this too causing the canSend() routine to time out, but a lot don't which is odd:
https://lowpowerlab.com/forum/rf-range-antennas-rfm69-library/real-time-rssi-measurement-broken-rfm69cw/

In my workaround I drop the threshold to below the noise floor to guarantee a trigger (e.g. set to 0xFF), wait for an RSSI (or RXREADY) interrupt, read the RSSI register, and restart if PayloadReady happens to go active (by reading the FIFO with autoRestartRx set was the quickest way I found). If PayloadReady does go active during this sampling time the RSSI register will freeze, which is why you need to unfreeze it if it happens. However it appears that the RSSI register does seem to update continuously otherwise. Maybe this is why some code freezes up?

Mark.

[jwketchum]

Thanks for the input, Mark.  That is helpful.  Looks like there isn't a quick solution, but I will continue to pursue your leads, and share any positive results that I find.

[perky]

OK, I'd be interested in your results BTW at the time I wasn't using AFC, so probably a good idea to turn that off during the RSSI sampling as I'm not sure what effect that might have on the results.

Mark.

[jwketchum]

Yes, I have the problem with RSSI sampling while sending an ACK, with AFC off.   Trying to get that working before enabling AFC.

[jwketchum]

Mark -- on further consideration of your reply April 18, and further reading in the RFM69HW manual, on page 28 there is a note:  "RssiValue can only be read when it exceeds RssiThreshold."  This is consistent with your workaround of dropping the threshold before initiating an RSSI measurement.  It appears that reading RegRssiValue in this case returns the last value measured, which was when the threshold was exceeded.  Setting a high threshold means that the read won't happen unless there is actual signal there.  Clearly this won't work when you are trying to operate in CSMA mode to send an ACK.  So probably need to modify the sendACK function to drop the threshold before sending the ACK.  Perhaps this is what you have already done.  I will take stab at that when I get a chance.

[perky]

OK, that sounds reasonable.

[jwketchum]

So, I first tried to just modify the canSend() function Felix's code to set theRSSI Threshold to 0xFF, check RSSI, then reset the threshold to the previous setting, and that didn't seem to change much once AFC was turned on.  The gateway receiver would still show large frequency errors, and the readRSSI function was continuing to return RSSI from the previously received packet.

Then I tried adapting the first solution that you posted on page 4 of the thread that you linked to, involving dropping RSSI threshold,  setting to RX_MODE, waiting for RSSI interrupt, then restarting the receiver and restoring RSSI threshold.  I implemented this in the canSend function.  This worked great ..... for exactly one packet, then the receiver stopped working.  However, if I simply eliminate the Rx Restart step, things work much better.  I get reasonable RSSI measurements, mostly in the range of -100 to -110 dBm, clustering around -104 or -105, occasionally up to higher than -100.  Much of the cited thread dwells on getting very rapid RSSI measurements, which I am not so concerned about.  The whole sendACK process is now taking 12-15 msec, which is fine for my purposes

The other curious thing is that once I had this in place, reported frequency errors are consistently much lower, almost always less than +/-1 kHz, where before they had been all over the place, usually well over 1 kHz, often between 10 and 20 kHz.  Not clear to me why this would change.

However, when I let this run for a while, the receiver eventually hangs, simply stops receiving packets.   Is it reasonable to assume that this is due to a noise spike, which leads to a random frequency error that somehow sends the receiver off into some unrecoverable state?  It is unclear to me why this would be, since AFC is supposed to restart upon entering RX_MODE.

FYI, operating parameters:
Bit Rate: 19,200
Fdev: 20kHz
Receive bandwidth (2*RxBW ~= 83 kHz
AFC bandwidth = 125 kHz

AFC bandwidth should be the relevant BW for determining the noise floor, since I believe RSSI is measured at the beginning of the preamble, which is when AFC is operating, so noise floor here should be -174 + 10log(125000) + 7 dB NF = ~ -116 dBm.  I understand that there is a lot of stray RF causing problems, but is it reasonable that noise spikes above -90 dBm, for example, would put the receiver into some unrecoverable state?

I havene't yet tried the second solution of resetting the receive buffer instead explicitly restarting the receiver, but will get to that.  Not sure why that would make a difference, though.

[perky]

OK, sounds like progress 

The another routine I don't issue a restart. Only if I get a PAYLOADREADY do I restart it, but by reading the FIFO until empty. The reason was issuing a restart by setting RESTARTRX added strange delays of a couple of ms sometimes before it restarted, whereas the FIFO reading approach seemed to restart within 100us consistently. I don't know why as both are supposed to work.

You might now be hitting another bug I think I found where going from TX into STANDBY and then to RX did not completely reset the receiver for the next RX. It was odd, an RSSI interrupt for that receive happened but no address match. The solution was to always go into SLEEP mode after a transmit, or always go from SLEEP to RX and never from STANDBY to RX. Here's a thread that describes what I found:

https://lowpowerlab.com/forum/moteino/communication-only-works-one-way-regardless-of-device/msg13120/#msg13120

[jwketchum]

OK, trying kolaf's method of setting sleep mode in receiveBegin.  So far so good.  I will let it run overnight; if it is still running in the morning, we should be good. (Rx threshold at -90 dBm).

Hopefully I can get on to noise optimization and figuring out if my range requirements are going to require a battery-powered gateway buffer node to address noise issues.

BTW, Mark, you and others, like WhiteHare, have done lots of great work chasing down bugs and finding ways to make things work.  I am still trying to digest all of it and understand the details better.  Thank you so much for sharing your work on this forum. 

John

[perky]

You're welcome, I have found this forum invaluable from other people's contributions too, it's a great resource.

Hopefully this has fixed your issues, let us know what happens

Mark.

[jwketchum]

Didn't have to wait until morning.  Stopped working again after 10 or 15 minutes.  I think I need to insert some debug code into receiveBeg in order to see what is happening when receive stops working.

[perky]

You are doing this RSSI reading with AFC turned off aren't you? maybe that's messing it up.

Mark.