Author Topic: AFC Not Working/Flaky in RF69HCW  (Read 15673 times)

jwketchum

  • NewMember
  • *
  • Posts: 13
AFC Not Working/Flaky in RF69HCW
« on: April 16, 2018, 03:36:41 PM »
I am having a problem getting AFC to work.
Here is what I am trying to do:
I am building a multi-node outdoor sensor network, using RF69HCW nodes, that needs to operate at seasonal temperature extremes with an indoor gateway.  By temperature extremes, I mean at least 0F to 100+F, which is typical of my area in coastal eastern New England.  Range is also important.
My initial prototype was operated indoors using data rate of 4800 and Fdev of 4800. Frequency is 904020000 (somewhat arbitrary, but I will operate in the 915 ISM band.) Also turned on ATC. I left other radio parameters pretty much default as they are set in the RFM69 library,  This worked nearly flawlessly over indoor ranges, until I got a new device, and tried taking one of the nodes outdoors, at which point the links got unreliable or failed completely.  This is clearly due to temperature drift and also XO variability. 
Next step was to go to a higher data rate, with associated wider Fdev, and increase RxBW to accommodate the higher data rate and 20 kHz of frequency offset:

Code: [Select]
  
  radio.writeReg(REG_BITRATEMSB, RF_BITRATEMSB_19200);
  radio.writeReg(REG_BITRATELSB, RF_BITRATELSB_19200);
  radio.writeReg(REG_FDEVMSB, RF_FDEVMSB_20000);
  radio.writeReg(REG_FDEVLSB, RF_FDEVLSB_20000);

  // RxBW is one sided bandwidth, so RF bandwidth = 2*RxBW
  // set the RxBW to 42 kHz, so RF BW = 84 to match 2*FDEV + BITRATE + Max frequency error =~ 80 kHz
  //  These parameters (Mantissa = 24, Exponent = 3, get us 32e6/24/32 ~= 83kHz
  radio.writeReg(REG_RXBW, (RF_RXBW_DCCFREQ_010 | RF_RXBW_MANT_24 | RF_RXBW_EXP_3));
  // Set RSSI Threshold to D0 ==> -104 dBm
  radio.writeReg(REG_RSSITHRESH, 0xD0);
  // increase preamble length to 6 bytes, and RF sync length to 4 bytes
  radio.writeReg(REG_PREAMBLEMSB, 0x00);
  radio.writeReg(REG_PREAMBLELSB, 0x06);
  radio.writeReg(REG_SYNCCONFIG, RF_SYNC_ON | RF_SYNC_FIFOFILL_AUTO | RF_SYNC_SIZE_4 | RF_SYNC_TOL_0);
  //sync words are 31-ML sequence with a 0-bit tacked on
  radio.writeReg(REG_SYNCVALUE1, 0xFA);
  radio.writeReg(REG_SYNCVALUE2, 0x25);
  radio.writeReg(REG_SYNCVALUE3, 0x61);
  radio.writeReg(REG_SYNCVALUE4, 0xCC);

This so far works fine, but anticipating issues with aging, greater variability from device to device, and greater temperature offsets than I am currently experiency, I want to get AFC working.  However, when I try to enable AFC on my devices, links that work error free without it become either unreliable, or fail completely.  This happens even at short range (-40 to -60 dBm).

Following advice in the forum, and particularly in the NXP AN4983 document, I have tried to use recommended configuration for AFC, but for the moment stick with the 19200 BPS data rate and 20kHz Fdev.  Here are the settings I use,

Code: [Select]
 // Set RSSI Threshold to D0 ==> -104 dBm
  radio.writeReg(REG_RSSITHRESH, 0xD0);
  // increase preamble length to 6 bytes, and RF sync length to 4 bytes
  radio.writeReg(REG_PREAMBLEMSB, 0x00);
  radio.writeReg(REG_PREAMBLELSB, 0x06);
  radio.writeReg(REG_SYNCCONFIG, RF_SYNC_ON | RF_SYNC_FIFOFILL_AUTO | RF_SYNC_SIZE_4 | RF_SYNC_TOL_0);
  //sync words are 31-ML sequence with a 0-bit tacked on
  radio.writeReg(REG_SYNCVALUE1, 0xFA);
  radio.writeReg(REG_SYNCVALUE2, 0x25);
  radio.writeReg(REG_SYNCVALUE3, 0x61);
  radio.writeReg(REG_SYNCVALUE4, 0xCC);

  radio.writeReg(REG_AFCFEI, radio.readReg(REG_AFCFEI) | RF_AFCFEI_AFCAUTO_ON | RF_AFCFEI_AFCAUTOCLEAR_ON);

  // Make the AFC BW a bit wider (Mantissa = 16, Exponent = 3 gets us 32e6/16/32 = 62.5kHz
  radio.writeReg(REG_AFCBW, (RF_AFCBW_DCCFREQAFC_010 | RF_AFCBW_MANTAFC_16 | RF_AFCBW_EXPAFC_3));

//  Turn on " optimized AFC " for low bandwidth signal
  radio.writeReg(REG_AFCCTRL, RF_AFCCTRL_LOWBETA_ON);
  radio.writeReg(0x71, 0x07);

I have tried this with and without the "optimized AFC" settings, but it doesn't make a difference.  It is unclear whether this is a good idea when Fdev ~= BR.   Also note that I am treating RxBW and AFCBW as single-sided values, so that the effective RF bandwidth values are twice that.

With AFC running, the gateway node reports frequency offsets in the range from a few 100 Hz to more than 15 kHz, when it actually receives a packet.  These values do not seem at all stable, they vary pretty wildly from packet to packet.

Also complicating matters, I am having trouble with changing the AFC settings:  If I run the nodes for a few minutes with the settings above, then edit the code to turn off AFC, and upload the edited, newly compiled code, the new register values (specifically REG_AFCFEI) don't get reset.  I typically have to remove power and let the devices sit without power for a few minutes, then upload the code again, to get AFC really turned off.  This happens on at least two devices, so I don't think that it is a device-specific hardware issue.

Also, the prevalent failure mode is for the gateway node to receive packets from the sensor node more reliably than the sensor node receives ACKs, so that all uplink packets go un-acked from the POV of the sensor node, but the gateway node sometimes gets three copies of packets from the sensor node, due to retries.

Apologies for the long-winded post, thank you in advance for reading it and any input you might be able to provide in helping me get AFC up and running.

--John

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #1 on: April 16, 2018, 09:47:53 PM »
Your settings look OK, but AFC won't start until the RSSI threshold is reached. If the threshold is set too low it'll trigger on noise and you'll get invalid results. Try raising the RSSI threshold, say to -80dB. If that works you'll need to drop it until you start to get false triggers, then raise it a bit to reduce their rate of occurance. You can think of the RSSI threshold as a filter, the higher the threshold is set the less likely to get a false trigger, but you will get one sometime.

I note you've read the excellent NXP app note on the AFC. I would recommend you follow this for eliminating false trigger events because they can mess things up with AFC, you need to be able to quickly detect a false trigger event and restart the receiver to restart the AFC process. That can come later though, to get it working initally just set it high so no false triggers are received.

Mark.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #2 on: April 18, 2018, 03:06:01 PM »
Thanks for your input Mark. 

I have tried raising the RSSI threshold and continued to have problems.  After getting your note, I tried again, and still problems.  After some extensive debugging, I have found that at least part of the problem I am having has to do with a delay in the sendACK function, associated with setting the RSSI Threshold at -80 dBm.   sendACK won't actually send an ACK until it senses the channel as clear, or a timer expires.  Timer expiration takes one second.  Channel clear means receive RSSI is below a hard-coded value of -90 dBm.   For reasons that I don't understand, when I increase RSSI Threshold above -104 dBm, to -80 dBm, the receive  RSSI value reported in the wait loop goes above the channel clear  threshold, even though nothing else has changed.  As a result, sendACK never senses the channel clear, so waits a full second for the timer to expire before sending the ACK

Since the timeout on send ack takes a second, my node wasn't getting the ack in time. and continued to send specified number of retries, even though the first attempt was succeeding.

I can't see why changing the value in REG_RSSITHRESH should affect the received RSSI value reported in the sendACK wait loop.   Any thoughts?

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #3 on: April 18, 2018, 08:09:16 PM »
I had real trouble gettting dynamic RSSI measurements to work, there's a long thread about this with some apparent workarounds (especially some results on page 4). Some people have problems with this too causing the canSend() routine to time out, but a lot don't which is odd:
https://lowpowerlab.com/forum/rf-range-antennas-rfm69-library/real-time-rssi-measurement-broken-rfm69cw/

In my workaround I drop the threshold to below the noise floor to guarantee a trigger (e.g. set to 0xFF), wait for an RSSI (or RXREADY) interrupt, read the RSSI register, and restart if PayloadReady happens to go active (by reading the FIFO with autoRestartRx set was the quickest way I found). If PayloadReady does go active during this sampling time the RSSI register will freeze, which is why you need to unfreeze it if it happens. However it appears that the RSSI register does seem to update continuously otherwise. Maybe this is why some code freezes up?

Mark.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #4 on: April 18, 2018, 10:23:39 PM »
Thanks for the input, Mark.  That is helpful.  Looks like there isn't a quick solution, but I will continue to pursue your leads, and share any positive results that I find.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #5 on: April 19, 2018, 08:16:49 AM »
OK, I'd be interested in your results ;) BTW at the time I wasn't using AFC, so probably a good idea to turn that off during the RSSI sampling as I'm not sure what effect that might have on the results.

Mark.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #6 on: April 19, 2018, 11:51:43 AM »
Yes, I have the problem with RSSI sampling while sending an ACK, with AFC off.   Trying to get that working before enabling AFC.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #7 on: April 27, 2018, 01:31:05 PM »
Mark -- on further consideration of your reply April 18, and further reading in the RFM69HW manual, on page 28 there is a note:  "RssiValue can only be read when it exceeds RssiThreshold."  This is consistent with your workaround of dropping the threshold before initiating an RSSI measurement.  It appears that reading RegRssiValue in this case returns the last value measured, which was when the threshold was exceeded.  Setting a high threshold means that the read won't happen unless there is actual signal there.  Clearly this won't work when you are trying to operate in CSMA mode to send an ACK.  So probably need to modify the sendACK function to drop the threshold before sending the ACK.  Perhaps this is what you have already done.  I will take stab at that when I get a chance.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #8 on: April 28, 2018, 11:00:11 PM »
OK, that sounds reasonable.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #9 on: May 05, 2018, 04:17:12 PM »
So, I first tried to just modify the canSend() function Felix's code to set theRSSI Threshold to 0xFF, check RSSI, then reset the threshold to the previous setting, and that didn't seem to change much once AFC was turned on.  The gateway receiver would still show large frequency errors, and the readRSSI function was continuing to return RSSI from the previously received packet.

Then I tried adapting the first solution that you posted on page 4 of the thread that you linked to, involving dropping RSSI threshold,  setting to RX_MODE, waiting for RSSI interrupt, then restarting the receiver and restoring RSSI threshold.  I implemented this in the canSend function.  This worked great ..... for exactly one packet, then the receiver stopped working.  However, if I simply eliminate the Rx Restart step, things work much better.  I get reasonable RSSI measurements, mostly in the range of -100 to -110 dBm, clustering around -104 or -105, occasionally up to higher than -100.  Much of the cited thread dwells on getting very rapid RSSI measurements, which I am not so concerned about.  The whole sendACK process is now taking 12-15 msec, which is fine for my purposes

The other curious thing is that once I had this in place, reported frequency errors are consistently much lower, almost always less than +/-1 kHz, where before they had been all over the place, usually well over 1 kHz, often between 10 and 20 kHz.  Not clear to me why this would change.

However, when I let this run for a while, the receiver eventually hangs, simply stops receiving packets.   Is it reasonable to assume that this is due to a noise spike, which leads to a random frequency error that somehow sends the receiver off into some unrecoverable state?  It is unclear to me why this would be, since AFC is supposed to restart upon entering RX_MODE.

FYI, operating parameters:
Bit Rate: 19,200
Fdev: 20kHz
Receive bandwidth (2*RxBW ~= 83 kHz
AFC bandwidth = 125 kHz

AFC bandwidth should be the relevant BW for determining the noise floor, since I believe RSSI is measured at the beginning of the preamble, which is when AFC is operating, so noise floor here should be -174 + 10log(125000) + 7 dB NF = ~ -116 dBm.  I understand that there is a lot of stray RF causing problems, but is it reasonable that noise spikes above -90 dBm, for example, would put the receiver into some unrecoverable state?

I havene't yet tried the second solution of resetting the receive buffer instead explicitly restarting the receiver, but will get to that.  Not sure why that would make a difference, though.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #10 on: May 05, 2018, 05:29:03 PM »
OK, sounds like progress  ;)

The another routine I don't issue a restart. Only if I get a PAYLOADREADY do I restart it, but by reading the FIFO until empty. The reason was issuing a restart by setting RESTARTRX added strange delays of a couple of ms sometimes before it restarted, whereas the FIFO reading approach seemed to restart within 100us consistently. I don't know why as both are supposed to work.

You might now be hitting another bug I think I found where going from TX into STANDBY and then to RX did not completely reset the receiver for the next RX. It was odd, an RSSI interrupt for that receive happened but no address match. The solution was to always go into SLEEP mode after a transmit, or always go from SLEEP to RX and never from STANDBY to RX. Here's a thread that describes what I found:

https://lowpowerlab.com/forum/moteino/communication-only-works-one-way-regardless-of-device/msg13120/#msg13120
« Last Edit: May 05, 2018, 05:31:59 PM by perky »

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #11 on: May 05, 2018, 10:15:40 PM »
OK, trying kolaf's method of setting sleep mode in receiveBegin.  So far so good.  I will let it run overnight; if it is still running in the morning, we should be good. (Rx threshold at -90 dBm).

Hopefully I can get on to noise optimization and figuring out if my range requirements are going to require a battery-powered gateway buffer node to address noise issues.

BTW, Mark, you and others, like WhiteHare, have done lots of great work chasing down bugs and finding ways to make things work.  I am still trying to digest all of it and understand the details better.  Thank you so much for sharing your work on this forum. 

John

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #12 on: May 05, 2018, 10:32:08 PM »
You're welcome, I have found this forum invaluable from other people's contributions too, it's a great resource.

Hopefully this has fixed your issues, let us know what happens ;)

Mark.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #13 on: May 05, 2018, 10:44:40 PM »
Didn't have to wait until morning.  Stopped working again after 10 or 15 minutes.  I think I need to insert some debug code into receiveBeg in order to see what is happening when receive stops working.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #14 on: May 06, 2018, 11:04:23 AM »
You are doing this RSSI reading with AFC turned off aren't you? maybe that's messing it up.

Mark.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #15 on: May 06, 2018, 03:03:36 PM »
I have AFC turned on.  Are you saying that having it turned off would mess things up?

Started inserting print statements to try to track down the issues; added a print statement at the end of the interrupt handler, and couldn't get receive to fail anymore.  One thing is clear, looking at the code in RFM69.cpp is that receiver gets set to standby all over the place and it isn't clear to me that setting sleep mode just at the beginning of receiveBegin is going to have the desired effect.

The other suggestion that you had in the thread with kolaf was to redefine standby mode to sleep mode; he reported that didn't have any effect, but just setting to sleep mode in receiveBegin did have the desired effect..  That seems strange to me.  I am going to try making that change and see what happens.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #16 on: May 06, 2018, 03:19:33 PM »
Another data point.  When I inserted kolaf's code at the beginning of setMode, to set sleep instead of standby, RSSI of -128 is returned for received packets (should be in the range of -40 to -60),as kolaf reported.  However, when getting RSSI in canSend while checking for clear channel to send ACK, appropriate RSSI values are returned, i.e., between -100 and -110.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #17 on: May 06, 2018, 03:22:43 PM »
OK. AFC could mess it up I think if you have it set to not set the AfcAutoclearOn and AfcAutoOn bits in RegAfcFei. I would turn AFC off while sampling RSSI and the turn it back on before receiving anything just in case.

BTW it's important to read the RSSI immediately you get a PAYLOADREADY as the very first thing you do and not after the FIFO has emptied or any mode change.

Mark.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #18 on: May 07, 2018, 08:55:58 AM »
Currently, the link has been running continuously for at least 12 hours without a failure. 

I think part of what makes debugging difficult here is that for whatever reason reprogramming configuration registers on the RF chip seems to be unreliable, so it is never entirely clear what the actual operating configuration is, unless I dump the entire register set.  I found that when I was initially trying to get AFC working, I had to power cycle the chip in order to disable AFC, after having it running.  These smaller changes may be taking a couple of upload cycles to actually take effect.

I am using the interrupt routine in RFM69.cpp.  There is a commented-out line there reading RSSI at the beginning, and instead he is doing it at the very end, after the FIFO has been emptied into the receive buffer.  Doesn't seem to be causing any problems, but I will make the change that you suggest, and see what happens.

Not sure I get your point about AFC messing things up.   What will get messed up by having AFC on while doing RSSI read when checking if the channel is clear?   Currently AFCAuto and AFC Autoclear are both set.  Are you suggesting that I should clear these before readRSSI when checking for clear channel?

Thanks,
John

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #19 on: May 07, 2018, 11:15:43 AM »
Another data point.  When I inserted kolaf's code at the beginning of setMode, to set sleep instead of standby, RSSI of -128 is returned for received packets (should be in the range of -40 to -60),as kolaf reported.  However, when getting RSSI in canSend while checking for clear channel to send ACK, appropriate RSSI values are returned, i.e., between -100 and -110.

....

I found that when I was initially trying to get AFC working, I had to power cycle the chip in order to disable AFC, after having it running.  These smaller changes may be taking a couple of upload cycles to actually take effect.

I am using the interrupt routine in RFM69.cpp.  There is a commented-out line there reading RSSI at the beginning, and instead he is doing it at the very end, after the FIFO has been emptied into the receive buffer.  Doesn't seem to be causing any problems, but I will make the change that you suggest, and see what happens.

Not sure I get your point about AFC messing things up.   What will get messed up by having AFC on while doing RSSI read when checking if the channel is clear?   Currently AFCAuto and AFC Autoclear are both set.  Are you suggesting that I should clear these before readRSSI when checking for clear channel?


If the rado is being put into sleep before the RSSI is read then potentially it might reset the register, which is what will happen if RSSI is read at the end rather than at the beginning. So reading the RSSI immediately you get a PAYLOADREADY should stop that, you should find with kolaf's code that the RSSI should give good values. Also if rx is restarted when the fifo empties then the restart mechanism could potentially unfreeze the RSSI register.

If you're trying to measure the RSSI to determine whether the channel is free then the centre frequency should remain stable, the danger is that AFC will change it and potentially on every restart of RSSI measurement. If the AFCAuto and AFC Autoclear are both set you should be OK as this will clear the FEI on each measurement (this is not the radio power up default BTW), if they weren't you'd possibly face the problem of noise changing the centre frequency and not resetting it which could mean the measurements are done over the wrong frequency.

Mark.
« Last Edit: May 07, 2018, 11:26:01 AM by perky »

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #20 on: May 07, 2018, 01:16:20 PM »
OK, got it. Thanks for the clarification.

jwketchum

  • NewMember
  • *
  • Posts: 13
Re: AFC Not Working/Flaky in RF69HCW
« Reply #21 on: May 07, 2018, 04:32:31 PM »
And yes, you are right, moving the readRSSI to the beginning of the interrupt routine now enables wholesale replacement of idle mode with sleep mode.

Big progress here, with your help.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: AFC Not Working/Flaky in RF69HCW
« Reply #22 on: May 07, 2018, 07:26:01 PM »
Nice! Let's hope it continues working ;

Mark.