I have an application where I need to eliminate if possible the possibility of receiving false packets from other systems of the same type and from other servers within my own system. I wonder what other people's strategy is for this? I have a system ID and a server ID which yields a unique combination for each server. Currently I encode the system ID (and also the channel number to prevent the possibility of cross channel triggers) in the SYNC word, and I add the system ID and server ID to the packet headers which are protected by the radio CRC16.
Is that enough? Is it worth sacrificing another byte and adding a separate CRC8 to the packet?
My thoughts are that the SYNC word could be subject to bit errors and could every now and then produce a false positive, so that's why the system ID is in the headers as well. I'm trying to work out whether the radio's CRC16 might also give false positives and whether the additional CRC8 in the packet would protect against that.
Any views?
Assuming you mean accidental false positives rather than a nefarious attacker, I think you could simply include a known "really big number" in the payload of every packet you send. For simplicity, it could be the same really big number each time. Expressed as binary, try to avoid too many consecutive 1's or 0's in a row, and for extra overkill you could try to make it very different from any sync words you're likely to encounter. Then you only accept the packet as genuine if its of the expected length and the exact "really big number" is both in it and where it's expected. The bigger the number, the lower the chance that random bit flips could produce it.
Alternately, if the data integrity of the rest of the payload is also important, then use a hash function that produces a really large hash value, and include that hash value in the packet. Then you kill two birds with one stone. If it were me, I'd probably do it that way.
Quote from: perky on June 09, 2016, 06:57:24 PM
I have an application where I need to eliminate if possible the possibility of receiving false packets from other systems of the same type and from other servers within my own system. I wonder what other people's strategy is for this? I have a system ID and a server ID which yields a unique combination for each server. Currently I encode the system ID (and also the channel number to prevent the possibility of cross channel triggers) in the SYNC word, and I add the system ID and server ID to the packet headers which are protected by the radio CRC16.
Is that enough? Is it worth sacrificing another byte and adding a separate CRC8 to the packet?
My thoughts are that the SYNC word could be subject to bit errors and could every now and then produce a false positive, so that's why the system ID is in the headers as well. I'm trying to work out whether the radio's CRC16 might also give false positives and whether the additional CRC8 in the packet would protect against that.
Any views?
ISTM that if you have a known content, destination IDs and a few other meaningful elements (valid command codes, etc) to take a random example, AND you encrypt with a 128 bit AES key (such as with radio.encrypt) then accidental packets are virtually impossible - each packet still has to pass the CRC check to be flagged by the radio. All this doesn't come together unless the packet is valid.
I suppose it's mathematically possible, but I haven't seen it. I've lost packets, but I can't say I've seen a packet where I said 'open the garage door' and the sprinkler valve comes on. And I've got a bunch of rather diverse traffic on my premises.
Tom
QuoteMy thoughts are that the SYNC word could be subject to bit errors and could every now and then produce a false positive, so that's why the system ID is in the headers as well. I'm trying to work out whether the radio's CRC16 might also give false positives and whether the additional CRC8 in the packet would protect against that.
I used to see erroneous packets from time to time when I still used the Pi Gw but I don't see them now. Not sure why.
The rfm69 lib is more prone to these issues than necessary because it uses an RSSI threshold below the noise floor. That means the radio looks for the sync word all the time rather than when a signal is sent. If you want to reduce the likelihood set the threshold higher or increase the length of the sync word.
Thanks for the replies.
I'm trying to keep the packet size as small as possible because it's a battery system so I've had to reduce bit rate for distance and therefore can't afford the transmitter being on for a long time. The data payload is actually only 2 bytes, so I'm loathed to add extra bytes if they don't give any benefit.
The conclusion I've come to so far is putting the system ID and server ID in the SYNC word will filter a large proportion of unwanted packets from other servers (i.e. other systems, and other servers within the same system), but to be sure that I don't decode those packets as legitimate due to bit errors in just the SYNC word I need to include the system ID and the server ID in the packet itself. I'm actually using data whitening too which scrambles the packet before transmission but not the SYNC word, so the SYNC word is subject to normal bit errors but any bit errors in the scrambled packet presumably have a catastrophic effect on the descrambled packet data and would also mostly screw up the CRC16, even if one were to get through that the packet data would be completely mangled and the system ID and server ID check would still have to pass. Adding another CRC8 to the packet could tighten it up even further, but it might not be worth the extra byte and also it might have limited use due to multi-bit errors that the descrambling will introduce.
It's possibly be an idea to disable the CRC16 in the radio, and calculate a new CRC32 - but not on raw packet data but instead on what will be the transmitted whitened data (which presumably requires a software whitener to calculate it and check it on reception). To me that stands a better chance of catching bit errors during reception.
Mark.
Quote from: perky on June 10, 2016, 06:39:53 AM
Thanks for the replies.
I'm trying to keep the packet size as small as possible because it's a battery system so I've had to reduce bit rate for distance and therefore can't afford the transmitter being on for a long time. The data payload is actually only 2 bytes, so I'm loathed to add extra bytes if they don't give any benefit.
The conclusion I've come to so far is putting the system ID and server ID in the SYNC word will filter a large proportion of unwanted packets from other servers (i.e. other systems, and other servers within the same system), but to be sure that I don't decode those packets as legitimate due to bit errors in just the SYNC word I need to include the system ID and the server ID in the packet itself. I'm actually using data whitening too which scrambles the packet before transmission but not the SYNC word, so the SYNC word is subject to normal bit errors but any bit errors in the scrambled packet presumably have a catastrophic effect on the descrambled packet data and would also mostly screw up the CRC16, even if one were to get through that the packet data would be completely mangled and the system ID and server ID check would still have to pass. Adding another CRC8 to the packet could tighten it up even further, but it might not be worth the extra byte and also it might have limited use due to multi-bit errors that the descrambling will introduce.
It's possibly be an idea to disable the CRC16 in the radio, and calculate a new CRC32 - but not on raw packet data but instead on what will be the transmitted whitened data (which presumably requires a software whitener to calculate it and check it on reception). To me that stands a better chance of catching bit errors during reception.
Mark.
Not sure if you've been following some of the recent antenna discussion, but if you improve the antenna on the gateway (e.g. large improvements are easily possible with a dipole), you could use a higher bitrate, and/or, as Joe was suggesting, raise your RSSI threahhold. Presumably you're already using narrow windows of reception given the scenario you described, so that should also help cut down on the potential for false positives. In fact, depending on how much latency you can tolerate, maybe you could have your gateway *always* send a packet during the listen window, since it costs you the same to receive a packet as it does to listen to dead air. That way you could use almost "free" duplicate transmissions and/or chop large payloads into a series of shorter payloads (which your receiver ultimately stitches back together to make the desired longer payload) as yet another safeguard at almost no extra cost.
Quote from: WhiteHare on June 10, 2016, 09:03:18 AM
Not sure if you've been following some of the recent antenna discussion, but if you improve the antenna on the gateway (e.g. large improvements are easily possible with a dipole), you could use a higher bitrate, and/or, as Joe was suggesting, raise your RSSI threahhold. Presumably you're already using narrow windows of reception given the scenario you described, so that should also help cut down on the potential for false positives. In fact, depending on how much latency you can tolerate, maybe you could have your gateway *always* send a packet during the listen window, since it costs you the same to receive a packet as it does to listen to dead air. That way you could use almost "free" duplicate transmissions and/or chop large payloads into a series of shorter payloads (which your receiver ultimately stitches back together to make the desired longer payload) as yet another safeguard at almost no extra cost.
In my application I send a few bytes of common status from a central server to any number of clients, not the other way around. The clients can be some distance away so have low RSSI thresholds and these can in theory pick up packets from other servers if I'm not careful, I also have repeaters that relay these packets to extend the network. The servers send regular status info as a broadcast, so it's OK for a client to miss several of them. I'm trying to harden up the clients so they don't accidentally get the wrong info (mainly from other systems of the same type that might be sending different status). In this case having a large antenna on the servers doesn't really help as the problem is at the client end. The entire system, including repeaters, is battery operated. It's being challenging!
QuoteIt's possibly be an idea to disable the CRC16 in the radio, and calculate a new CRC32 - but not on raw packet data but instead on what will be the transmitted whitened data (which presumably requires a software whitener to calculate it and check it on reception). To me that stands a better chance of catching bit errors during reception.
I've always wondered whether it's possible to ditch packet mode and CRC altogether and use a sw implemented forward error correction mechanism instead. If you use a good coding scheme you'd get both better error identification and automatic fixing of transmission errors via redundant coding. You could then use a much higher chip rate that your normal bitrate, using a much larger fdev, so you could use fairly high tx power and still be fcc compliant. The coding scheme would give you a nice coding gain so overall you should get better results. You'd then have all info inside the payload and could throw out the packets that aren't yours.
The sx1231 can work in continuous mode without sync word, but preamble is necessary to sync the bit synchronizer. So I guess the chip rate limit here is the bitrate to which the bit synchronizer can still attach.
Probably not a very practical idea - just in case you care to make it "interesting". Sort of a DYI LoRa.
I think you probably could. The problem is the design of the radio front end means RSSI signals above the threshold will trigger a start of reception and that's the real problem, congestion with other radios on the same frequency will impact the packet error rate probably more than the radio's inherent bit error rate. LoRa is I think somewhat different and can extract signals from below the ambient noise.
Mark.
I definitely don't mean this as a criticism, but isn't it a bit odd that with so many potential non-overlapping channels to choose from, your system just happens to be sharing the same channel (well, maybe not precisely the same but close enough) and the same bitrate and the same packet length and the same parity scheme as other systems you want to avoid receiving packets from?
Er, the other systems are mine! Different system ID (might be in a different building but overlapping in RF area). Not much of a coincidence there ;)
Edit: Before you ask why I don't just put them on different channels, the system uses frequency hopping across the entire sub-band. I've also got repeaters within the same system, but that's not too much of a problem because the status data is the same.
QuoteThe problem is the design of the radio front end means RSSI signals above the threshold will trigger a start of reception and that's the real problem, congestion with other radios on the same frequency will impact the packet error rate probably more than the radio's inherent bit error rate.
Ah ok. Still forward error correction would help you identify and correct these congestion issues better. Plus you said you had to reduce bitrate so apparently you did have some bit error problems prior to that.
QuoteLoRa is I think somewhat different and can extract signals from below the ambient noise.
LoRa just uses highly redundant chip coding. That way the ambient noise averages out over several bits, but the signal is visible. The same principle could be used with the rfm69, but the bit synchronizer would limit how low one can go.
I suppose one could try it without synchronizer and manually sync to some preamble pattern - but then it starts getting complicated. I remember reading about a couple german guys doing something similar with some very simple 433mhz transmitters, but they used a PC to correlate the received pattern. Probably too much for a 328p.