Author Topic: CRC vs FEC  (Read 12378 times)

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
CRC vs FEC
« on: December 09, 2016, 07:02:31 PM »
Lots of libraries seem to rely on the radio payload CRC verification, meaning that if the CRC fails the packet is discarded, which on the RFM69 that seems to be long before the receiver sensitivity limit (-120dBm).

LORA relies on FEC to improve packet reconstruction before the CRC is applied, so why do people not do this for transceivers like the RFM69 . . . ?

Anybody played in this area ?

Pre-encoding the payload using something like Hamming 7,4 should allow packet recovery with bit errors where a normal CRC would cause the packet to be discarded by the radio . . . transmitting without the radio CRC check should, therefore, increase sensitivity as packets which contain bit corruption should be recoverable . . . penalty is an increase in the number of bits needing transmitted . . . but in many cases that's not the real issue . . .

Or am I missing something ?

Fd


perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #1 on: December 09, 2016, 07:40:10 PM »
Well radio is inherently unreliable, so assuming you have mechanisms to counter that (like retries) there doesn't seem much point IMO in over-engineering an EDC mechanism, just as long as you can reliably detect a corrupted packet (either by no packet being received, or a packet content based CRC if one happens to get through with multiple bit errors). Also data whitening will scramble the transmitted data, so a single bit error in that will result in a multiple cascade of bit errors when de-whitened. People do try to keep the number if bytes transmitted to a minimum though especially in battery operated systems. There's the issue of variable packet lengths too, if you turn CRC off you might get a corrupted packet length byte.

Pesronally I think the combination of sync word filtering, data whitening, CRC16 of the radio and some parameter checking of the content of the packet is sufficient to detect a missed packet or an errored packet, and then the retry mechanism kicks in.

Mark.



ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #2 on: December 09, 2016, 09:18:45 PM »
Nah, retrying doesn't improve range, it may improve reliability, nothing more. If you're at a range which causes bit errors in the RF datastream then the CRC will always fail and that is the end of the game. If, however, you can reconstruct packets which would otherwise fail the radio CRC you can increase range and reliability . . . and effectively receiver sensitivity.

I work with some RF experts who I have been learning from . . .

Will produce some sample code in the next few weeks . . . it's not too complex . . . but needs some maths . . .

Fd


ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #3 on: December 09, 2016, 09:37:32 PM »
Also . . .

You're thinking about the wrong layer.

I'm thinking about a layer above the RF transport. where the payload length is also encoded in a redundant manner.

This is how people get extreme performance from RF systems . . . specifically radios like the RF69 . . .

the LORA/RF95 system relies heavily on FEC to give the increased range (and probably I would not buy another RF69 having played with RP95's) . . . but I think we can do much better with RF69's . . .

Fd

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #4 on: December 09, 2016, 11:18:42 PM »
Nah, retrying doesn't improve range, it may improve reliability, nothing more. If you're at a range which causes bit errors in the RF datastream then the CRC will always fail and that is the end of the game.

That doesn't ring true.  As you approach the limit of the transmission range, some packets may still get through, even if a lot of them don't or get corrupted and filtered out because they fail the CRC check.
« Last Edit: December 10, 2016, 03:54:22 AM by WhiteHare »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #5 on: December 10, 2016, 03:27:11 AM »
Quote
LORA relies on FEC to improve packet reconstruction before the CRC is applied, so why do people not do this for transceivers like the RFM69 . . . ?

This is a good idea. I have thought about it but haven't tried it yet. I see two issues:

1. Sync word detection starts the packet rx process. You can set the number of tolerated bit errors at the rfm69 though so the chip allows unreliable connections at that stage. I think it's also possible to receive with sync word length of 0 but haven't tried it.

2. The radio uses a bit synchronizer when receiving to generate a clean sequence of 1's and 0's. It'll be interesting to see how good of a signal one needs for that to work. LoRa is at an advantage here because it can use the known chip sequence to find a signal in the noise.

My take is that you can't get quite to the level of LoRa but you should be able to get a very significant improvement. Can't wait to see what you come up with!

Joe

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #6 on: December 10, 2016, 03:31:30 AM »
Quote
Nah, retrying doesn't improve range, it may improve reliability, nothing more. If you're at a range which causes bit errors in the RF datastream then the CRC will always fail and that is the end of the game.

Retrying helps you cope when you have bit errors just as coding does. So it's also a mechanism to increase range but much less effective than what you're trying to do: with retries you need at least one error free transmission which coding doesn't require.

joe

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #7 on: December 10, 2016, 03:36:31 AM »
Forgot a third issue:

Since you'll be receiving close to the noise floor you need to set rssithresh to 255 so the radio continuously looks for the sync word instead of waiting for rssi detection to trigger reception.

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #8 on: December 10, 2016, 08:16:48 AM »
Retrying helps you cope when you have bit errors just as coding does. So it's also a mechanism to increase range but much less effective than what you're trying to do: with retries you need at least one error free transmission which coding doesn't require.

joe

Good point.  Sounds worthwhile.  I hear that LDPC is pretty good.
« Last Edit: December 10, 2016, 08:19:54 AM by WhiteHare »

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #9 on: December 10, 2016, 09:12:27 AM »
OK, I see. Use coding gain by sending EDC codes to lower the S/N needed to achieve a certain overall BER. I'm not sure how RFM69 devices would work in this scenario, how for example AGC and AFC would be used, unless it demands wideband operation so these aren't used which would reduce range but compensated by coding gain. How does frequency offset of the transmitter to the receiver actually affect BER? The sync words can accept bit errors so can be tolerant to a higher BER, but the bit synchronizer might require a decent BER to work in the first place. It would be interesting for people to have a go at this though in case it actually is possible.

Mark.
« Last Edit: December 10, 2016, 02:58:37 PM by perky »

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #10 on: December 10, 2016, 01:34:59 PM »
Seems like there should already be a library somewhere where you just hand it a string and some parameters and it hands you back a string with all the FEC built in.  Then you just Tx that as usual.  Likewise, on the receiving end, you hand it a string of what you received and it hands you back the error corrected string.

If there is such a library (I haven't looked), then this might be fairly easy to try.   ;D

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #11 on: December 10, 2016, 02:02:57 PM »
There are implementations, from simple and inefficient to complex and more efficient. In this case the efficiency is the number of extra bits required to allow reconstruction.

You're correct to mention preamble and sync, which is critical to starting packet detection . . . something else I'm researching.

At its simplest a simple bit tripler allows 2/3 bit voting FEC which allows a 1 OTA bit corruption out of every 3 and still allows full packet reconstruction . . . penalty is it triples the OTA payload, inefficient but robust, but trivial to implement.

If that can be shown to work then you can use something more clever, Hamming 7/4 for example . . . I found the basis of an implementation kicking about on the net somewhere which could be adapted.

This is a very old idea, ADSL/VDSL/DVD/CD all use FEC, and so do serious radio people . . . ;-)

Fd


WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #12 on: December 10, 2016, 06:08:23 PM »
I've heard it's fairly typical for a particular coding scheme to be expressed as equivalent to "x db improvement" in link budget.  Anyone know how much improvement in link budget the triple bit FEC scheme would be equivalent to?

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #13 on: December 11, 2016, 09:41:00 AM »
Forgot a third issue:

Since you'll be receiving close to the noise floor you need to set rssithresh to 255 so the radio continuously looks for the sync word instead of waiting for rssi detection to trigger reception.

I've always thought of FEC as being implemented in the PHY layer, and maybe your comment explains why.  Maybe an MCU just isn't fast enough to detect and lock the signal as often as you might possibly want.  In fact, maybe that's why the bitrate is relatively low on the LoRa radios, since that problem would seem to become even more slippery at faster bitrates.


ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #14 on: December 11, 2016, 12:50:58 PM »
So, sync detection on the 69 is bit error tolerant (up to 7 bit errors in 8 to 64 bits of sync), and there are operating modes where a preamble and sync followed by an unprocessed payload is possible, no CRC required or checked on reception. So it should be possible to move all the packet encoding, decoding and validation into the micro and out of the radio. What this means is that the radio detects packets which would not pass a CRC and provides them to the micro, the micro then performs error recovery on the data then verifies a post error recovery CRC check in software . . . any addressing/application level data in the payload can only be interpreted after this is done . . . if after error recovery the CRC still fails - it's beyond recovery . . .

It may also be worthwhile using manchester encoding for the OTA transmission . . . 1/2 the effective bitrate but more robust.

That the sync detection can deal with bit errors is a clue as to another way of using the radio ;-)




ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #15 on: December 11, 2016, 12:53:10 PM »
I've always thought of FEC as being implemented in the PHY layer, and maybe your comment explains why.  Maybe an MCU just isn't fast enough to detect and lock the signal as often as you might possibly want.  In fact, maybe that's why the bitrate is relatively low on the LoRa radios, since that problem would seem to become even more slippery at faster bitrates.

It is on LORA, it isn't in 69's . . . doesn't really matter either way. LORA builds FEC into it's bit spreading, which is fundamentally part of a different way of transmitting the data. That doesn't stop you doing it elsewhere . . . or make it any less effective.

Fd

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #16 on: December 11, 2016, 01:22:01 PM »
Just to continue to wet appetites:

Quote
Additionally, coding gain could improve drastically the budget link. For example as low BR application, a 1024 bits coding could improve the budget link by 30 dB if BR is decreased by the same factor. The Semtech XE1203F incorporates an 11-bit Barker Encoder / Decoder, which provides for 10 dB of processing gain under the above conditions.

This is from http://www.semtech.com/images/datasheet/fcc_digital_modulation_systems_semtech.pdf which also helps understand why this helps from a compliance perspective as well.

joe

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #17 on: December 11, 2016, 01:22:39 PM »
layers - layering approach.
triples - childishly simply FEC.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #18 on: December 11, 2016, 02:31:17 PM »
Yes, but sync word detection relies on the bit synchronizer working properly too, we have no idea how tolerant that is to noise. It may need a complete noise-free preamble to trigger the reception process.
Mark.

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #19 on: December 11, 2016, 04:17:44 PM »
Why would the sync be error tolerant in that case ?

Fd

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #20 on: December 11, 2016, 04:46:13 PM »
We don't know. Certainly the sync word comparison logic can be programmed to trigger with multiple bit errors which is decoded from output from the bit synchronizer, but the bit syncronizer will be key in decoding bits. If that doesn't work it's likely to have so many bit errors as to make it useless.
Mark.

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #21 on: December 11, 2016, 07:25:10 PM »
Just to continue to wet appetites:

This is from http://www.semtech.com/images/datasheet/fcc_digital_modulation_systems_semtech.pdf which also helps understand why this helps from a compliance perspective as well.

joe
Thanks!  So, from that, I can possibly answer my own question:  the improvement in link budget from Ziplockk's 3 bit FEC would be 4.8db.  Meh, not worth the bother.

However, that 30db improvement from the 1024bit coding.  Yeah, that's pretty compelling.   ;D

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #22 on: December 11, 2016, 08:17:34 PM »
Oh dear . . .


joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #23 on: December 12, 2016, 03:09:24 AM »
I've found a good overview presentation on the subject: atlantarf.com/Error_Control.php

A couple of insights:

  • You have to specify the coding gain at a fixed BER rate. Coding gain increases as BER decreases.
  • At BER's typical in home automation (say 10^-2) only few FEC schemes actually have a positive coding gain. The best achieve around 4 dB coding gain at 10^-2.
  • ACK / Retransmits are actually a very effective form of improvement. Consider a BER of 10^-2 with two retransmits. That gets you down to a BER of 10^-6 with a traffic increase of ~1%. Not bad at all compared to FEC.
  • The most interesting applications for FEC are where you need very low BER and don't have a back channel to ACK or large latency. Like sending from a space mission.

All this argues against FEC's for home automation. However on the plus side:

  • With FEC's you can use a wide-band high bitrate signal to transmit low bitrate information. This allows you to be FCC compliant with higher TX power settings and without frequency hopping (see quoted semtech paper in earlier post).
  • Since you'd use a wider bandwidth you don't need to do AFC to compensate for poor XTALS.
  • You could use multiple coding schemes depending on link quality on top of the same radio settings, effectively using different bitrates depending on link quality per client without TDMA.


And some more implementation thoughts:

  • If you use packet mode you'd have to use fixed packet length since the length field might get corrupted in transmission.
  • Your maximum payload length per packet goes way down with code overhead. Eg if you do use 1024 bits per bit you'd get half a byte into a 64 byte packet  :o.
  • So realistically speaking you'd probably want to do this in continuous mode.

Quote
Thanks!  So, from that, I can possibly answer my own question:  the improvement in link budget from Ziplockk's 3 bit FEC would be 4.8db.  Meh, not worth the bother.

However, that 30db improvement from the 1024bit coding.  Yeah, that's pretty compelling.   ;D

Well it's a 30db (1000x) improvement since you use 1024 bit per bit. That's not the net coding gain. In reality if this was one of those very advanced codes you might get 33db for a net coding gain of 3db.

But still this shows the beauty of it: at 300kbit you can legally transmit at 12dBm or so. Now using 1024 bit per bit you get a bitrate of 300 baud without temp compensation of AFC. And another closer client might transmit to the same gw at ~300kbit net.

Of course now you have to implement encryption and coding in software. A hearty ARM processor would be better at this than our poor little 328p  ;)

Joe

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #24 on: December 12, 2016, 03:46:53 AM »
Quote
Yes, but sync word detection relies on the bit synchronizer working properly too, we have no idea how tolerant that is to noise. It may need a complete noise-free preamble to trigger the reception process.
Mark.

From the datasheet:

Quote
To ensure correct operation of the Bit Synchronizer, the following conditions have to be satisfied:
  A preamble (0x55 or 0xAA) of 12 bits is required for synchronization (from the RxReady interrupt)
  The subsequent payload bit stream must have at least one transition form '0' to '1' or '1' to '0 every 16 bits during data transmission
  The bit rate matching between the transmitter and the receiver must be better than 6.5 %.
Notes
- If the Bit Rates of transmitter and receiver are known to be the same, the RFM69HW will be able to receive an infinite unbalanced sequence (all 0s or all 1s) with no restriction.

But yeah I agree this is the weak spot. The synchronizer needs to get in phase to work properly even if it's not used to align bitrates. That likely means the signal needs to be at least a bit higher than the noise floor.

Joe

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #25 on: December 12, 2016, 03:58:12 AM »
It depends how clever they have been with the bit synchronizer. If it's like a PLL it might be tolerant to noise, if it's more like a DLL then maybe not ;-)
Mark.

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #26 on: December 12, 2016, 07:30:44 AM »
@joe
I guess by "net coding gain," you are discounting that part of the gain which comes from concurrently using the much lower bitrate?  i.e. compared to if you had already reduced your bitrate by 1000x, and then from there added the 1000x bits of coding, the coding part only brings you an extra 3db?  Wow.  Considering the really high overhead (1000x more bits), that's not much to get excited about.

Anyhow, as far as home automation goes, if there's a range issue, it seems you're better off keeping the high bitrate (to maintain high battery life on the mote) and just deploying another mains powered, wi-fi+rfm69 gateway node in the vicinity.  Am I right?
« Last Edit: December 12, 2016, 08:10:19 AM by WhiteHare »

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #27 on: December 12, 2016, 08:28:31 AM »
It all depends on how the BER changes with S/N ratio in the radio. The lower the S/N the higher the BER. What coding gain does is introduce error correcting codes so that the overall BER in the decoded data, i.e. after error correcting, is lower than the BER of the encoded data before the error correction is applied, hence the S/N ratio can be lower to achieve a certain overall BER.The actual net gain is going to be highly dependent on how the BER is related to S/N.
Mark.

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #28 on: December 13, 2016, 08:58:35 PM »
Anyhow, as far as home automation goes, if there's a range issue, it seems you're better off keeping the high bitrate (to maintain high battery life on the mote) and just deploying another mains powered, wi-fi+rfm69 gateway node in the vicinity.  Am I right?

Somewhat misses the point . . . why not just run cables . . . or move your garage closer to your living room . . . use pushrods . . .


WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #29 on: December 14, 2016, 06:36:45 AM »
Somewhat misses the point . . . why not just run cables . . . or move your garage closer to your living room . . . use pushrods . . .

You're right.  I'll retract what I said.

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #30 on: December 16, 2016, 07:15:25 AM »
I've been making some progress on verifying a new radio configuration ready for testing this concept.

Hope to have a configuration which will allow me to determine whether preamble and sync detection behaviour will support the concept, and if so how bit errors manifest themselves on the margin of range.

The FEC part is simple.

Have been having some general unreliability issues at low bitrates which I also wanted to understand, which has been consuming my time.

Seem to have found an interesting bug in the system (radio misbehaviour - not sure why or how it can happen) which is only apparent when you look at the RF traffic on a spectrum analyser . . . (radio transmits crazy long transmission (10x the length of what it's being asked to transmit) every now and again, problem disappears if you put the radio to sleep after every transmission) . . . repeatable. Causes the receiver to not receive anything (not surprisingly). Without a spectrum analyser you would assume it's a simple dropped packet . . .

Fd

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #31 on: December 16, 2016, 08:16:32 AM »
Interesting, I wonder if it's related to the behaviour I found when transitioning from STANDBY to RX didn't fully reset the receiver (causing it to not receive) but going from SLEEP to RX did...
Mark.

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #32 on: December 16, 2016, 08:24:10 AM »
Dunno, but this is completely repeatable and seems to be apparent in higher bitrate configurations also, just more difficult to see.

The correct SPI sequence (so far as I can understand) is being used to feed the radio, but the radio itself is going bonkers . . . very occasionally.

Could be a coding issue with corrupt data in a variable I suppose, but it doesn't square up at the moment.

Certainly the 25 quid SDR I bought is proving very useful as it's difficult to argue with the facts on the screen . . .


perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #33 on: December 16, 2016, 08:57:29 AM »
Could it be a corrupted packet length byte? Do you have the correct SPI mode selected?
Mark.

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #34 on: December 16, 2016, 09:10:06 AM »
Quote
Seem to have found an interesting bug in the system (radio misbehaviour - not sure why or how it can happen) which is only apparent when you look at the RF traffic on a spectrum analyser . . . (radio transmits crazy long transmission (10x the length of what it's being asked to transmit) every now and again, problem disappears if you put the radio to sleep after every transmission) . . . repeatable. Causes the receiver to not receive anything (not surprisingly). Without a spectrum analyser you would assume it's a simple dropped packet . . .

What surprised me initially is that the radio just keeps sending preamble after the packet is transmitted unless you actively set it to rx, standby or sleep. Do you have a code path that doesn't do that?

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #35 on: December 16, 2016, 10:08:27 AM »
Do you have the sequencer enabled?  Sounds like it's in "manual" mode.

[Edit: From the DS: "By default, when switching from a mode to another one, the sub-blocks are woken up according to a pre-defined and optimized sequence. Alternatively, these operating modes can be selected directly by disabling the automatic sequencer (SequencerOff in RegOpMode = 1). " ]
« Last Edit: December 16, 2016, 06:04:43 PM by WhiteHare »

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #36 on: December 20, 2016, 06:47:09 AM »
Logic error on my part, interestingly the radio behaviour is quite inconsistent and somewhat random, hence the confusion.

Anyway, I have a build running without any CRC protection in the radio, low bitrate, 5 dropped packets per 1000 transmissions reliability.

Now need to profile the way in which the link fails as rssi drops which means a walk test to get some range effect.

I'll get round to it and update.

Fd


joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #37 on: December 20, 2016, 06:58:54 AM »
Quote
which means a walk test to get some range effect

Enjoy it! These are times when we all get some exercise  ;)

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #38 on: December 20, 2016, 11:01:40 AM »
There's also another way you could probably squeeze out some extra range/reliability if you wanted to.  It assumes the sender and receiver have closely synchronized RTC's--maybe using Joe's method or another one--and there are pre-defined time blocks when the receiver can expect a transmission from the sender (if there is one, that is).  Basically, record all the high-low and low-high transitions, along with time stamps, into an array during that time internval.  Then, afterward, you have the MCU inspect the array and purge problems that might have caused the radio's PHY to either miss or reject the packet (mainly because it can't approach the problem in as leisurely a way).  It would be the ghetto version of "cognitive radio."

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #39 on: December 26, 2016, 12:10:01 PM »
Got a little time to progress things, still refining the basic radio configuration, no point in building on something that doesn't work perfectly. or isn't perfectly understood.

Low bitrates should produce better range (my objective), but rc osc accuracy causes poor performance with stock library settings . . . generally because at low bitrates it seems to be common to reduce the receiver bandwidth (per the data sheets), making the rc osc accuracy even more critical. So if you increase the receiver bandwidth you can get low bitrate transmission to be as reliable as the stock library settings (actually better in my testing) . . . rather than tweaking the tx/rx frequencies to compensate. Penalty is you may be more susceptible to noise . . . however the radio is fairly noise resilient for packet detection if correctly configured.

I've gone from a few percent packet error rate to zero percent after a number of hours running with this new configuration, all without CRC protection at the radio level (but at the application level).

RF comms are only unreliable if there's a good reason for that to be the case . . . some of the stock radio configurations are not reliable under optimal conditions . . . I'll share some settings to see if they produce reproducible reliability once I'm happy . . . and that will be with very low failure rates under optimal conditions . . .

The FEC side of things is the icing on the cake given my testing so far . . .

Fd

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #40 on: December 27, 2016, 07:42:31 AM »
Ran overnight with 10's of thousands of transmissions, zero failures.

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #41 on: December 27, 2016, 08:19:50 AM »
Which of the stock settings did you find problematic? Sounds like you got an improvement from increasing the rxbw. I also do that at very low bitrates where the frequency offset is large relative to fdev. Did you get a benefit at 55kbit? Were the nodes at dramatically different temperatures? Anything else?

The nice part about a highly redundant Fec scheme is that you can run at high bitrates so the freq offset won't matter much. Narrow-band transmission is kind of the opposite of what I thought was your original plan.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #42 on: December 27, 2016, 11:31:13 AM »
@ziplockk As a matter of interest, which CRC algorithm did you use? Was it CRC16_CCITT?
Mark.
« Last Edit: December 27, 2016, 11:33:30 AM by perky »

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #43 on: December 28, 2016, 09:49:54 AM »
Which of the stock settings did you find problematic? Sounds like you got an improvement from increasing the rxbw. I also do that at very low bitrates where the frequency offset is large relative to fdev. Did you get a benefit at 55kbit? Were the nodes at dramatically different temperatures? Anything else?

I suspect that the rxbw 'stock' settings are a little tight across the board, I've had general minor unreliability issues which, like many others, I've been assuming were RF interference issues. I'm now largely convinced that this is not the case . . . increasing rxbw has made a dramatic difference to reliability. To be clear I'm talking 0.5-2% errors . . . but that is symptomatic of an issue . . . increasing rxbw seems to remove the errors.

Secondly, I was trying to use a cluster if RFM69HW's at low OTA bitrate and ran into random issues with one node being able to receive from its peers but its peers not being able to receive it's response . . . this is the classic frequency mismatch issue I think - but all nodes were at the 'same' temp in the 'same' environment, but that environment changed temp during the day . . . in the afternoons when it was warm, problems appeared, when cooler, everything worked. So I think 'different' temps is not the only issue . . . this change has solved that problem (so far) also. I've also incorporated reasonably frequent RC osc calibration into the general scheme of things at both ends . . .

The nice part about a highly redundant Fec scheme is that you can run at high bitrates so the freq offset won't matter much. Narrow-band transmission is kind of the opposite of what I thought was your original plan.

It seems that the issue is most prevalant at low OTA bitrates, I'm looking at low bitrates for range, and FEC for additional range, my application doesn't have large payloads, trying to squeeze everything into 16bytes so it can be AES256 encrypted in one block . . .

I think minimising the rxbw is a good thing, but not so much as it causes problems . . . I have a feeling that the data sheet figures are optimal and the real world needs a little more slop ;-)

ziplockk

  • Newbie
  • *
  • Posts: 28
  • Country: gb
Re: CRC vs FEC
« Reply #44 on: December 28, 2016, 09:57:58 AM »
@ziplockk As a matter of interest, which CRC algorithm did you use? Was it CRC16_CCITT?
Mark.

Something reliable I have in my code toolkit . . . I'm sending known (or calculatable) payloads so I know precisely what the error pattern looks like . . . so I don't technically need the CRC . . .

Fd

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #45 on: December 28, 2016, 10:38:27 AM »
I assume you've seen this thread, it gives a set of rules you need to follow that relate Fdev, BR and RxBw. There are a couple of default configurations which I don't believe meet these rules:
https://lowpowerlab.com/forum/rf-range-antennas-rfm69-library/definition-of-rxbw-with-rfm69/

Also, frequency offset of transmitter to receiver is a real killer with sensistivity and AFC is therefore a very good idea to use, here's a really useful app note from Freescale on AFC for their MC12311 and MKW01 sub-GHz transceivers which are actually based on the same silicon as the sx1231. Note Figure 8 (in section 12):
www.nxp.com/assets/documents/data/en/application-notes/AN4983.pdf

Mark.

Edit: This app note also neatly explains what low beta offset is all about. Basically with low beta (i.e. low modultation index) most of the signal energy is concentrated at the centre frequency, but this is exactly where the inherent notch on the receiver bandwidth is. So you need to offset the signal slightly so it is out of that notch.
« Last Edit: December 28, 2016, 08:24:58 PM by perky »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #46 on: December 28, 2016, 04:32:27 PM »
Mark, this is interesting data, thanks! I wonder how much the sensitivity to frequency offset in their measurements is due to the low modulation index of 1 they use. I'll need to run some tests on my settings.

Quote
Also, frequency offset of transmitter to receiver is a real killer with sensistivity and AFC is therefore a very good idea to use

The problem I have with AFC is that it requires a whole lot of additional logic to work right: you have to dynamically determine the RSSI threshold - noise levels vary tremendously from install to install and even due to time of day. Plus you need to actively abort noise rx activations if you don't rx your sync word rapidly after RSSI to reset rx and bring frequency back where it belongs.

Joe

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #47 on: January 04, 2017, 06:44:58 AM »
Quote
I'll need to run some tests on my settings.

Wow am I happy that I just ran these tests. Once your frequency offset is larger than about 80% of fdev no packet reception is possible at all. Based on http://jeelabs.net/boards/6/topics/4624 it looks like the xtal has a tolerance of 10 ppm, which calculates to 9150hz at 915mhz. At 1.2k using a 5000 hz fdev even normal tolerances between modules could make it impossible to communicate.

Then throw in the temperature coefficient of the xtal (see https://lowpowerlab.com/forum/moteino/freq-vs-temperature-update/) and it becomes clear that one has to correct for the frequency offset for narrow-band comms. A wider RxBw doesn't help much.

Joe

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #48 on: January 04, 2017, 08:48:31 AM »
Good work Joe. I'm not surprised at the result though  ;)

Another thing about AFC is once it has corrected the RxBw can be the smallest it needs to be, that effectively increases the fade margin while receiving the packet itself.

Mark.

Edit: I'm not sure how strong the signal was above sensitivity level you used for this test. I suspect for very weak signals this offset dependence would be somwhat worse, a graph of required signal strenth to receive a packet v. percentage offset from Fdev would be handy ;)
« Last Edit: January 04, 2017, 09:04:58 AM by perky »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #49 on: January 04, 2017, 10:26:50 AM »
Quote
Edit: I'm not sure how strong the signal was above sensitivity level you used for this test. I suspect for very weak signals this offset dependence would be somwhat worse, a graph of required signal strenth to receive a packet v. percentage offset from Fdev would be handy ;)

:-)

Well for me this just means I do have to do AFC so I don't feel a need to dig into this much further. Doing AFC intelligently is more involved than one would think:

1.) Accessing SPI while receiving is a no-go since that measurably increases noise even on a Moteino. So actively restarting RX if SyncAddress doesn't timely follow RSSI doesn't work since you'd have to change DIO mapping while receiving to then catch PayloadReady. Therefore actively managing the RSSI threshold and continuously using AFC doesn't work.

2.) Additionally I'm concerned that at the edge of your range with these ultra narrow settings you might get so close to the noise floor that there's no room to increase the AfcRxBw and still successfully do AFC.   

I think I'll just measure the frequency offset once when first pairing a device to the gw which you do close to the gw so 2.) isn't yet an issue. That takes care of the manufacturing tolerance. Then I'll periodically do an AFC using the same AfcRxBw as RxBw to adjust for temperature and aging - say once per hour.

For all other packets I'll just use the measured frequency offset and set RSSIThresh to 255 as I usually do, not using AFC or AGC.

Joe
« Last Edit: January 04, 2017, 10:30:53 AM by joelucid »

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #50 on: January 04, 2017, 11:56:31 AM »
Mmm. I've not seen that much noise induced by SPI. I do though use series terminating resistors at the source of SPI signals to take out the high frequency content of the square waves they produce (i.e. slow edges down), and to prevent any possibility of under and overshoot which can cause momentary current spikes if they are clamped by the substrate diodes. That means 56R or 100R series resistors close to the MCU for SCK and MOSI, and 56R or 100R close to the MISO of the radio.

At low signal levels when permanently receiving though you are going to need timeouts from RSSI to address match, and from address match to payload ready, and reset the receiver accordingly.

BTW I did a quick experiment to check this, I had a transmitter sending alternate packets. The first packet was at the correct frequency and correct sync word, the other simulated noise by offsetting the frequency by 20kHz and flipped one of the address word bits so the receiver got an RSSI but no address match. Result without timeouts was 85% packet loss (AFC kicked in for the noise packet and didn't get reset), but with the timeouts version the packet error rate was 0.5%. That's pretty conclusive!

Mark.


joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #51 on: January 04, 2017, 12:14:17 PM »
Quote
Mmm. I've not seen that much noise induced by SPI.

I have a test where I set RSSI thresh to a certain value, then go into RX and see if I get an RSSI interrupt in a short period of time. Using this iteratively with a sweep of RSSI thresh values I can measure the noise floor very reliably. If instead of sleeping I poll a radio register I get a significantly elevated noise floor even with Moteino. You're right though - this could definitely be improved with HW.

Quote
At low signal levels when permanently receiving though you are going to need timeouts from RSSI to address match, and from address match to payload ready, and reset the receiver accordingly.

True if you use AFC or AGC, but not true otherwise. That's why for normal packets I won't use AFC nor AGC.

When I do AFC I'll first send a ping to the gw and establish RSSI from the ACK. Then I'll set the RSSIThresh immediately below that RSSI, send a second packet and do AFC during reception of the ACK. I can use regular ACK timeouts for that packet since I know the timing.

Quote
BTW I did a quick experiment to check this, I had a transmitter sending alternate packets. The first packet was at the correct frequency and correct sync word, the other simulated noise by offsetting the frequency by 20kHz and flipped one of the address word bits so the receiver got an RSSI but no address match. Result without timeouts was 85% packet loss (AFC kicked in for the noise packet and didn't get reset), but with the timeouts version the packet error rate was 0.5%. That's pretty conclusive!

Yeah - that same problem hits you with stock RFM69 and AGC: the low RSSI Thresh below noise floor causes AGC to select min gain immediately and a strong packet can then overload the receiver. Or if you immediately receive loud noise you won't pick up a weak packet afterwards.

Joe
« Last Edit: January 04, 2017, 01:09:40 PM by joelucid »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #52 on: January 04, 2017, 05:47:06 PM »
After a couple of experiments here's another upside of this approach: I notice that the accuracy of AFC is very much dependent on fdev. Used at high bitrates the average error in AFC easily exceeds 5khz and the average manufacturing tolerance. At very tight fdev's however AFC is very precise.

Since I'm doing AFC independently of payload transmission I can do it at a low fdev, yielding some benefit even at the high bitrates. Not sure how useful but it's a theoretical upside.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #53 on: January 06, 2017, 08:18:59 PM »
After a couple of experiments here's another upside of this approach: I notice that the accuracy of AFC is very much dependent on fdev. Used at high bitrates the average error in AFC easily exceeds 5khz and the average manufacturing tolerance. At very tight fdev's however AFC is very precise. Since I'm doing AFC independently of payload transmission I can do it at a low fdev, yielding some benefit even at the high bitrates. Not sure how useful but it's a theoretical upside.
I'm guessing this means the AFC is quantized in steps, and the step size is scaled with Fdev. It also points to the possibility that the S/N is dependent on the ratio of offset to Fdev, so you might not actually gain that much in using a small Fdev for the AFC to lock close to the real frequency while using a much larger Fdev for payload.
Mark.

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #54 on: January 07, 2017, 04:16:10 AM »
Quote
so you might not actually gain that much in using a small Fdev for the AFC to lock close to the real frequency while using a much larger Fdev for payload.

Yes, my thought exactly.

BTW, how do you manage AFC and what bitrate do you use? It seems like you set a RSSI threshold and use AFC on every packet, correct? If yes, how do you determine the threshold? Do you dynamically update it? I've had nothing but trouble trying to get AFC to work reliably in dynamic environment with very weak signals because the threshold was almost always either too high or too low.

The scheme I proposed works well to maintain freq sync. But then how do you send the first packet and be heard?

I'm currently looking at temp compensation along the lines of https://lowpowerlab.com/forum/moteino/freq-vs-temperature-update/ but I'm finding that every module has a different freq offset / T curve. The potential saving grace: they all seem to be fairly linear in the range from 10-30 celsius, so two point calibration would give you enough accuracy to talk to each other at room temperature. And if you take a node into the snow it could learn via AFC what the curve looks like outside of that range.

Joe

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #55 on: January 07, 2017, 09:29:04 AM »
BTW, how do you manage AFC and what bitrate do you use? It seems like you set a RSSI threshold and use AFC on every packet, correct? If yes, how do you determine the threshold? Do you dynamically update it? I've had nothing but trouble trying to get AFC to work reliably in dynamic environment with very weak signals because the threshold was almost always either too high or too low.

There are two sources of noise, internal and external. Assuming there are no radio transmitters around at all, the background RF noise would be below the sensitivity level, so the question is what is the nature of this external RF noise? I think it's actually quite peaky, there's an average but in reality there are gaps where that noise level is below the sensitivity level. So if your receiver only wakes up and receives for a few ms before the real packet is transmitted you've created a very small window that ignores all the other noise peaks outside of it. If you were permanently receiving it get's more difficult, you'd need the timeouts previously discussed and your 'window' increases in size. So I set my threshold to be about 5dB above the sensitivity level (which doesn't change). I also use frequency hopping with randomized timing of packet transmission to prevent the possibility of synchronizing with another source.
Mark.
« Last Edit: January 07, 2017, 09:45:21 AM by perky »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #56 on: January 07, 2017, 10:22:30 AM »
Quote
so the question is what is the nature of this external RF noise?

Exactly. And that's where I see the problem. Recall that one of the FAQ's here in the forum is why Moteinos don't send and then it turns out they always sense a carrier. My noise measurement code measures 2ms to see if a given RSSI level is hit. And just by moving the radio onto my desk with its 5 monitors and 2 computers RSSI is hit a couple of dBm higher for every single 2 ms period. So yes there are spikes, but theres also a background buzz which varies by location and depending on what's on close to the radio.

Even to avoid the CSMA problem one needs to manage the threshold. I do that for listen mode for example - otherwise it just doesn't work well. Whenever I get phantom activations I increase the threshold. And over time when I don't I gradually bring it down. This works but I can construct all kinds of scenarios where it would be fatal.

Quote
So I set my threshold to be about 5dB above the sensitivity level (which doesn't change).

I see - so you give up 5db in sensitivity for AFC and AGC (which I'm trying to avoid). And if you do get a 5dB background buzz your screwed, correct? You might end up having to do the same workarounds I use for listen mode.

Joe

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #57 on: January 07, 2017, 11:51:39 AM »
I see - so you give up 5db in sensitivity for AFC and AGC (which I'm trying to avoid). And if you do get a 5dB background buzz your screwed, correct? You might end up having to do the same workarounds I use for listen mode.

Yes, but I also need a fade margin. In other words the signal has to be somewhat above sensitivity level anyway, and as soon as AFC kicks in a lower RxBw is selected which increases the fade margin further.

I can understand putting things into noisy RF environments with constant wideband noise might be problematical though, and I may yet have to pull the same tricks as you.

BTW a lot of these sensing carrier problems using CSMA I believe has a lot to do with how RSSI is read, which I don't believe actually works properly. There are several threads on this. I don't believe that just reading the RSSI register is sufficient and the manual triggering mechanism is broken in the chip (at least I've never got that to work).

Mark.

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #58 on: January 07, 2017, 01:46:02 PM »
Quote
Yes, but I also need a fade margin. In other words the signal has to be somewhat above sensitivity level anyway, and as soon as AFC kicks in a lower RxBw is selected which increases the fade margin further.

If you have the rssi thresh 5 db above receiver sensitivity you can't count those in your fade margin, can you? Signals weaker than this will not be received so any margin would need to be on top of that.

Quote
BTW a lot of these sensing carrier problems using CSMA I believe has a lot to do with how RSSI is read, which I don't believe actually works properly.

Yeah I'm quite aware. It took me a lot of time to figure out a way to measure RSSI reliably and I did go through all of these threads for inspiration.

Joe

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #59 on: January 07, 2017, 04:09:06 PM »
If you have the rssi thresh 5 db above receiver sensitivity you can't count those in your fade margin, can you? Signals weaker than this will not be received so any margin would need to be on top of that.

You're right, technically I can't. I've chosen a half way house by putting a criteria on initial packet recognition that is below my normal phase margin, but when AFC kicks in I know that the fade margin during actual packet reception is greater due to lower RxBw. Multipath fading is mitigated somewhat by using frequency hopping. I admit it's a bit of a compromise.
Mark.

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #60 on: January 07, 2017, 09:21:46 PM »
And if you do get a 5dB background buzz your screwed, correct?

I may possibly see a way around that.  Upon detection of a >=5dB signal, you start a timer and then continuously resample RSSI, looking for an opening.  If the source of the signal is a legitimate, intentional transmitter (i.e. not  merely the RF pollution of elevated background noise from your collection of 5 monitors), then it must obey the regulatory duty cycle rules.  And that means the signal must stop within a predictable timeframe from the time of first detection.  If it does, then you start transmission because you have what appears to be an opening.  If it doesn't, then you transmit anyway, because you can infer that it's not an intentional transmitter

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #61 on: January 08, 2017, 10:11:44 AM »
Quote
Upon detection of a >=5dB signal, you start a timer and then continuously resample RSSI, looking for an opening.

Remember it's the receiver we're talking about. You need to set an RSSi threshold and as soon as that gets hit AFC starts, potentially detuning your receiver. You need wait for sync and reset the receiver if it doesn't come as quickly as it should so you can RX packets again. It takes like 6-8 bytes of preamble plus the 2 sync bytes until you can make the decision - quite a bit of time.

Set the threshold too low and you'll be latching onto noise all the time, never ready to read a signal. Set it too high and you won't receive weak signals.

Because of differences in background noise you need to dynamically adjust the threshold. But any such algorithm has the problem of differentiating between background noise and signals based on a pretty irregular sampling of unreceivable RSSI threshold triggerings.

You want to raise the threshold quickly if there's noise so you don't miss many packets. And you want to lower it quickly so you don't miss weak signals once the noise is gone. But you don't want to trigger on the noise floor often again missing packets. You need to avoid raising the threshold on periodic signals from other radios etc.

It just seems like a pretty difficult problem to solve for all corner cases. Which are sure to hit you somewhere in a commercial product.

Joe

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #62 on: January 08, 2017, 10:22:30 AM »
Quote
I've chosen a half way house by putting a criteria on initial packet recognition that is below my normal phase margin, but when AFC kicks in I know that the fade margin during actual packet reception is greater due to lower RxBw.

I think it's quite possible that AFC works below the sensitivity of the receiver. So maybe you can set the threshold right at the sensitivity level and still be good. Modulo the threshold management problem.

AFC seems like a great mechanism for low beta, mid bitrate applications (like the example in the app note you quoted) where you need quite a bit of signal to noise to receive anyway, but still don't have a large fdev. In that scenario threshold mgmt is easier and AFC might work quite a bit below RX sensitivity, allowing the thresh to be at sensitivity even while using a larger afcrxbw.

But with ultra narrow-band applications I'm skeptical.

Joe

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #63 on: January 08, 2017, 12:58:56 PM »
Yeah, it's a bit of a pain.

There is one possibility that might help. Currently I open my RX window a few ms before the packet is transmitted. If there's noise that's above the threshold at that point it'll trigger an RSSI and AFC, and as you say that can take some time to complete. If my real packet comes along a few ms later which has a higher signal strength it might not be seen properly as it has already been triggered. If instead I extend the preamble of the packet and start the transmission before the RX starts then it may see the real packet immediately and lock on to that instead, and because the signal strength is higher it should over-ride the noise.

BTW my PER is currently about 1.2% but that is with an 868MHz burglar alarm with 5 transmission sources in the house as well. If I take the sources outside away from the burglar alarm it drops to about 0.7%.

Mark.
« Last Edit: January 08, 2017, 01:00:44 PM by perky »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #64 on: January 14, 2017, 12:31:07 PM »
Quote
BTW a lot of these sensing carrier problems using CSMA I believe has a lot to do with how RSSI is read, which I don't believe actually works properly.

Turns out with my approach of setting RSSI Thresh to 255 and always being in RX RSSI can be read fine over time. The problems only seem to exist when your waiting for RSSI thresh to be hit.

One big problem RFM69 and also my lib has/had is RSSI is read after PayloadAvailable hits. This is after decoding of the packet. Depending on the speed of the sender TX might have ramped down by then and you get inconsistent values. I saw that with 16 Mhz Moteinos, but not with 8 Mhz ones.

The solution is to read RSSI continuously during reception until CRC hits. That way I finally get consistent readings regardless of clock speed settings.

Given that I now sample RSSI all the time I just keep the last 1024 readings (I store one every 5 ms) and can produce nice little charts like the one attached (a 1200 baud packet coming in).

Cool eh?

Joe

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #65 on: January 14, 2017, 12:54:45 PM »
Now I can show you what I mean with the rssi thresh adjustment problem. Attached is a chart showing what happens when I plug in a 433 Mhz AC Plug 1m from my GW.

Switched on at ~3300 ms.

Joe

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #66 on: January 14, 2017, 02:21:29 PM »
That's really good work Joe.

I found that once a packet starts to be received it is frozen at its final value at the point payloadReady happens, but is unfrozen again once the FIFO has been read and payloadReady de-asserts (if you have AutoRxRestartOn set to 1, although I didn't try with it set to 0). So I always read RSSI after a payloadReady asserts but before reading the packet data out of the FIFO, i.e. before any decoding of the packet. I get consistent results with this.

I'm currently trying to solve the RSSI threshold problem with AFC. I think once the receiver has locked onto a transmission sequence it can turn on only during the time the transmitter is sending its extended preamble (similary when sending a reply, the sender would enable its receiver while the preamble of the reply is active). I think that would allow setting RSSI threshold to 255 as it will trigger immediately since to receive the packet it has to be above noise anyway.

The problem with the above is inital locking. Currently I scan the channel for a valid packet, and then lock on to the sequence putting the receiver to sleep and waking it up on the next calculated timeslot. So how do I receive the initial packet to lock to without setting a threshold?

It's clear that the higher the threshold the lower the frequency of noise triggers, and a noise trigger cause the receiver to temporarily not receive until the address match timeout whereby it will be re-enabled, so there are windows where it can't detect a real packet. The frequency of that depends on the threshold and on the amount of noise. Set the threshold too high and you lose range, set it too low and your packet error rate goes up. Mmm.

Mark.

Edit: There may be a way out of this. It's only initial locking where I need to have some kind of dynamic RSSI threshold, after that I think the above scheme would mean I don't need to bother with it. So maybe I try to set the RSSI threshold to one that has a low enough frequency of noise triggers to allow me to lock within a decent timeframe. Any chance in posting your RSSI measuring code? ;-)
« Last Edit: January 14, 2017, 02:31:41 PM by perky »

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #67 on: January 14, 2017, 04:20:47 PM »
Well I've just done an experiment, I had a receiver set at -95dB threshold and turning the receiver on 4ms before the packet was transmitted gave me about 1.2% PER (I've got a burglar alarm on the same frequencies with 5 sources of transmissions). Changing it to turn on the receiver when the transmitter is sending preambles has dropped that to 0%. Still set to -95dB, so need to drop it to below noise floor, but that's encouraging....
Mark.

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #68 on: January 14, 2017, 05:30:54 PM »
Mark, i would not like to have a sketchier connection particularly during locking which requires 25 packets on average. In fact I don't want to require any AFC for usual traffic at all in case it impacts on sensitivity.

Thats why I have an initial pairing in proximity where boh nodes are set to a wide band seting so both can talk. When they're locked the gw starts a calibration session where it heats up its radio and derives its temp correction coefficients by pinging the sensor and receiving the acks at narrow band settings with AFC while cooling down. Here the timing is clear and the nodes are close so either your scheme or a high threshold work. The acks are sent using the narrow band profile where AFC is most effective.

Now both nodes switch to narrow band. The client can now periodically ping the gw with an afc request and learn its temp correction table by applying afc on the acks. Again the timing is clear so your method would work. I've been sending a ping first, and have taken the rssi of the ack - 1 as thresh.

I could also use the heating method for the sensor - but these may be rfm69w's so they aren't capable of self heating. So instead the devices are considered paired when the gw is calibrated and the remainder happens on the go.

The beauty is when I lose connection I can just lock again without any AFC involved. So locking is possible at the most marginal link strength.

Did my first range test today while taking a walk with the family. Gw with dipole in the house, Moteino at sub zero temps outside with usual wire antenna. At 1200 baud I had reception at about 1km away - pretty good considering the gw was indoors. Lowest packet RSSI was -121, packets around -115 were common. Overall I consider it a success.

Joe

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #69 on: January 14, 2017, 06:04:37 PM »
Quote
I found that once a packet starts to be received it is frozen at its final value at the point payloadReady happens, but is unfrozen again once the FIFO has been read and payloadReady de-asserts (if you have AutoRxRestartOn set to 1, although I didn't try with it set to 0). So I always read RSSI after a payloadReady asserts but before reading the packet data out of the FIFO, i.e. before any decoding of the packet. I get consistent results with this.

That's weird - I don't use any additional buffer, so I don't clear the FIFO quickly and I retrieved RSSI in the PayloadAvailable irq. Still I saw the inconsistencies and I could make the stop by adding a delay(1) before putting the sender out of TX.

My sense was that RSSI is locked when you put the radio into opmode standby, but that happens too late in this case.

Quote
Any chance in posting your RSSI measuring code? ;-)

Well the point is there's nothing special about it anymore. I just avoid updating RSSI when CRC has been detected. Here's the section:

Code: [Select]
bool SX1231Driver::available() 
{
if( !isTransmitComplete() ) return false;

if( _mode != SX1231_OPMODE_RECEIVER ) {
setMode(SX1231_OPMODE_RECEIVER);
}
else {
if( !(readReg( SX1231_REG_IRQFLAGS2 ) & SX1231_IRQFLAGS2_CRCOK ))
_rssi = readRssiValue();
return readDIO0();
}
return false;
}

Note that I'm starting to get more comfortable accessing SPI during RX. The reason is I've finally figured out when that hurts and what to do about it: with my new GW's with dipoles at somewhat of a distance I don't have that problem. If I put the dipole close to the board it starts to happen.

The old gw's with wire antenna's all have the problem. So I think it's the ESP8266 radiating into the antenna. Now I only need to stop this on boards with chip antenna. That should be a bit of a challenge.

The more involved RSSI detection code managed RSSI detection without accessing the SPI - still useful to see how much of an impact that has.

Joe

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #70 on: January 14, 2017, 06:19:19 PM »
Quote
(if you have AutoRxRestartOn set to 1, although I didn't try with it set to 0).

Ah that might be it. I have autorxrestart off because that allows faster packet bursts for the bootloader. Will have to try if that fixes it. Would be nice to keep the no spi option open.

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #71 on: January 14, 2017, 06:35:30 PM »
Quote
The beauty is when I lose connection I can just lock again without any AFC involved. So locking is possible at the most marginal link strength.

And if wasn't clear - since the gw never does AFC or AGC other than during pairing it can always use the 255 rssi thresh.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #72 on: January 14, 2017, 07:08:48 PM »
Thanks Joe, some neat solutions there.

Ah that might be it. I have autorxrestart off because that allows faster packet bursts for the bootloader. Will have to try if that fixes it. Would be nice to keep the no spi option open.
That makes sense. The RSSI is probably latched after a payloadReady until a RX reset condition, which with autorxrestart on will happen when the FIFO is emptied.

My system is different to yours it seems. Everything is battery operated, and the status back from the nodes is low bandwidth data, so I use a polling method where the controller requests a response from a specific node which are all locked to the controller's packet sequence. That means the controller only transmits with low duty cycle, and it also needs to be narrow band to allow multiple channels in the small sub-band of 868MHz I have available. The nodes don't send requests to the controller. If I had to have a receiver permanently on in my controller I'd be in the same boat as you! Having said that, it's the locking that's the thorn because that does need to be permanently on during the scanning part for a lock. I also want the system to be easily installed by a user so any node can be made to talk to any controller in a simple configuration process, so I don't really want any calibration (even though it appears this has solved your problems).

So what I'm left with now is trying to come up with a reliable algorithm for setting the threshold for locking, but that ain't easy. There could be other radios on channel for up to 3.6 seconds at a time with only 1.8 seconds between each one (staying within their 1% duty cycle over an hour) which might cause me to raise the threshold too much because that's really a transient reception that will go away. And there could be several devices doing this just at that time. It may be I'd need to do some kind of RSSI scanning for quite some time to establish a reasonable threshold.

It might be that dropping the threshold until the radio permanently triggers then raising it by a fixed amount might do the trick, but If you can think of an algorithm that might work I'm all ears ;)

Mark.
« Last Edit: January 14, 2017, 07:18:14 PM by perky »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #73 on: January 15, 2017, 01:37:09 AM »
Quote
It might be that dropping the threshold until the radio permanently triggers then raising it by a fixed amount might do the trick, but If you can think of an algorithm that might work I'm all ears ;)

How about setting thresh to 255 and rxbw to your afcrxbw and continuously monitoring RSSI. Then when you see something that might be a packet you set thresh to your observed floor and kick off reception with AFC. That might work given your longer preambles.

I do think it will hurt though - depending on your settings. If you use a RxBW of 5000 hz you'd need to use at least 4x that to do AFC reliably. That's 6 dB more noise. And it doesn't seem like you need more than 6 dB signal to noise with the low bitrates to receive.

So don't toss temp calibration out the window too quickly. You could do it during manufacturing to avoid the pairing step and allow every node to talk to every controller. Or you could use TCXO's - HopeRF will do custom modules starting at quantities of 3000 I believe.

Edit: actually you can just keep the noise floor at 255 during afc. But during rssi proving you have afc off.

Joe
« Last Edit: January 15, 2017, 03:47:18 AM by joelucid »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #74 on: January 15, 2017, 02:50:04 AM »
Quote
Ah that might be it. I have autorxrestart off because that allows faster packet bursts for the bootloader. Will have to try if that fixes it. Would be nice to keep the no spi option open.

Yeah - that was it. But I'll stick with the new method for now. So nice to auto-generate these charts.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #75 on: January 15, 2017, 08:52:23 AM »
OK, so given my preamble is now extended (it'll be an additional 6ms plus the original 5 bytes worth of the real packet, that 6ms deals with +/-3ms of LO offset drift so receivers still remain locked even if they miss several packets in a row), I think you're suggesting sample RSSI in ultra-fast non AFC mode and look for a set of consecutive RSSI values that are relatively constant for 6ms to indicate a packet might be being transmitted, then I could trigger AFC and try to receive it. If I can measure RSSI fast enough, do the maths and programme into AFC mode within that 6ms timescale then it might work. Thanks for that!

I might need to look for a step change at the beginning given it will need a certain S/N above ambient noise to receive anyway, have you a feel for that value? would 6dB do it?

Mark.

Edit: BTW I've run the test with the receiver put into RX mode while the transmitter is sending preambles now for over 12 hours with a packet drop (two way) of less than 1 in 1000. So that definitely works well.
« Last Edit: January 15, 2017, 09:59:32 AM by perky »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #76 on: January 16, 2017, 09:22:20 AM »
Quote
I think you're suggesting sample RSSI in ultra-fast non AFC mode and look for a set of consecutive RSSI values that are relatively constant for 6ms to indicate a packet might be being transmitted, then I could trigger AFC and try to receive it.

Yeah exactly. You'll need to experiment with how to detect a packet. Normally a jump in RSSI that holds for a little bit would be a good criterion. But you want to get as close as possible to the noise floor so maybe you start trying if you find consecutive measurements that are atypically clustered at ceiling of the noise floor.

Quote
I might need to look for a step change at the beginning given it will need a certain S/N above ambient noise to receive anyway, have you a feel for that value? would 6dB do it?

Very dependent on bitrate. At 1200 baud it seems like its maybe 3-4 dB from this weekends field data. Which you likely couldn't detect because the larger RxBW adds more in noise. The higher the bitrate the more S/N you'll need.

Joe

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #77 on: January 16, 2017, 09:40:01 AM »
OK, I'm actually running at 12k bps (needed for low battery life). That means I'll need a greater jump, I'll start with 10dB. If this works well then I might even be able to not send the full extended preamble at all (or at least a much shortened version of it) and use the same technique when locked, that will help battery life. I'm going to give this a go today.
Mark.
Edit: It may not even be needed to detect a plateau after the jump, just a jump might be a good indication that a transmission has started. I'll try that first.
« Last Edit: January 16, 2017, 09:49:32 AM by perky »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #78 on: January 16, 2017, 10:56:40 AM »
Quote
I'll start with 10dB

10 dB above RxBW noise floor sounds more than plenty. Now your AfcRxBw will likely be maybe 5x that. So that's around 7 dB more noise. So you're likely looking for 3 dB peaks at AfcRxBw.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #79 on: January 20, 2017, 06:57:53 PM »
I had a go at this with no luck. It sort of worked, but RSSI readings were jumping around too much to make it usable. However I have changed the sending and receiving packets to use the scheme of only going into rx when the tx is already sending a preamble and it's rock solid, very low PER (less than 1 in a thousand), and RSSI threshold is set to 255 for those.
Mark.

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: CRC vs FEC
« Reply #80 on: January 25, 2017, 03:09:48 AM »
Quote
However I have changed the sending and receiving packets to use the scheme of only going into rx when the tx is already sending a preamble and it's rock solid, very low PER (less than 1 in a thousand), and RSSI threshold is set to 255 for those.

Yeah that was a really good idea of yours. I now also switched to this approach for the AFC response. Previously I had done a ping first and used the RSSI of the response plus a few as thresh, but that never worked really well. It's finally coming together!

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #81 on: January 25, 2017, 06:08:44 AM »
It does appear to work remarkably well, and eliminates the RSSI threshold problem completely once locked. You do need tight absolute timing control of course, but you've invariably got a mechanism for that already especially if you're doing frequency hopping (I use the AVR's RTC counter driven by a 10ppm watch crystal for that).
Mark.

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #82 on: January 25, 2017, 06:38:42 AM »
I wonder whether using a longer preamble would have a nearly equivalent effect in terms of reducing packet error rate?  It comes with the penalty of increased transmission time, but it would allow for some CSMA induced variability (if you're using CSMA, that is).

Also, I wonder if it's necessary to go to the extreme of setting the threshold all the way to 255, or whether setting a more conservative lower bound on RSSI (but still lower than anything expected) wouldn't be more of a happy medium.  Granted, in some cases that lower bound might still turn out to be 255, but not in the general case. Not sure  how that more conservative lower bound should be calculated, but I'm just floating the notion to see if it might be worth figuring out.  Perhaps it is yet another way to avoid false triggers and thereby reduce packet error rate still further.

[Edit:  The other nice thing about your method of not turning on the Rx until the preamble is expected is that it eliminates the 16ma of Rx current that would otherwise be wasted listening to nothing useful while the transmitting node is gearing up to respond with its next packet.  So, that's great icing on the cake, and possibly another good reason to do it. ]

Thanks for sharing! :)
« Last Edit: January 25, 2017, 06:57:54 AM by WhiteHare »

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #83 on: January 25, 2017, 07:29:29 AM »
Unfortunately this is not a particularly good scheme for low power. From the receiver's perspective it will still be receiving waiting for the packet proper even if it's just a preamble and not waiting for an RSSI threshold, but the worst bit is when sending a reply because it needs to also send an extended preamble so that the other end can time its turning on of the receiver to coincide with that preamble. What you gain is a lack of dependence on RSSI threshold which means those transfers can be done right down to the noise floor plus the S/N needed to receive reliably, and there's no false triggering if a signal happened to come along which is above the threshold but below the real packet because the real packet would override it.

As for setting the RSSI threshold at 255, there's no point in not doing that if a preamble is already being transmitted. If you did you'd still need to choose a threshold that is at the noise floor to get the best out of it so nothing really gained by doing that.

Horses for courses, you don't get things for free in this life ;-)

Mark.

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: CRC vs FEC
« Reply #84 on: January 25, 2017, 07:39:34 AM »
and there's no false triggering if a signal happened to come along which is above the threshold but below the real packet because the real packet would override it.

Ahhh...  Now I see the beauty in it.  Thanks.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: CRC vs FEC
« Reply #85 on: January 25, 2017, 08:00:38 AM »
Yeah, in fact this I think is the primary reason I went from 1.2% two-way PER down to < 0.1%, I was getting false triggers due to a weak interferer which just happened to be above the RSSI threshold I'd set but below my real signal level.
Mark.