Hi guys,
I have been using moteinos as sensors using the MySensors library for some time. However, I have lately experienced that the nodes have become a bit unstable. In order to test this thoroughly I decided to jump out of the sensor library and use the Gateway and node example sketches provided on the main moteino site.
I have one R4 moteino usb, one r5 moteino usb, and one regular r4 moteino that I am testing with. All of them are equipped with the RFM69HW radio at 868 MHz. I downloaded the latest IDE (1.6.7), added the URL to the board manager and select the moteino board in the IDE.
The very first time I installed the Gateway sketch on one node and the node sketch on another node things appeared to go quite well. It looked like a few packets were dropped, but otherwise messages seem to reach the Gateway, and acknowledgements seem to reach the node. However, every subsequent time I have tried this it has failed. The messages from the node to the Gateway seem to consistently reach the Gateway, but acknowledgements I never received by the node. I have tried to switch which device is the node and which is the gateway between all three devices, and the results are consistently the same. I have included a brief log from the node and Gateway device. I have tried both with and without the HW and ATC switches in the sketch code at both ends in every combination. As far as I understand this should be the correct configurations for me:
#define FREQUENCY RF69_868MHZ
#define ENCRYPTKEY "sampleEncryptKey" //exactly the same 16 characters/bytes on all nodes!
#define IS_RFM69HW //uncomment only for RFM69HW! Leave out if you have RFM69W!
#define ENABLE_ATC //comment out this line to disable AUTO TRANSMISSION CONTROL
Node:
Transmitting at 868 Mhz...
SPI Flash Init OK ... UniqueID (MAC): D7 63 18 62 27 40 9E 2F
RFM69_ATC Enabled (Auto Transmission Control)
nothing...
Sending[1]: 1 nothing...
Sending[2]: 12 nothing...
Sending[3]: 123 nothing...
Sending[4]: 123 nothing...
Sending[5]: 123 A nothing...
Sending[6]: 123 AB nothing...
Sending[7]: 123 ABC nothing...
Sending[8]: 123 ABCD nothing...
Sending[9]: 123 ABCDE nothing...
Sending[10]: 123 ABCDEF nothing...
Sending[11]: 123 ABCDEFG nothing...
Sending[12]: 123 ABCDEFGH nothing...
Sending[13]: 123 ABCDEFGHI nothing...
Sending[14]: 123 ABCDEFGHIJ nothing...
Sending[15]: 123 ABCDEFGHIJK nothing...
Sending[16]: 123 ABCDEFGHIJKL nothing...
Sending[17]: 123 ABCDEFGHIJKLM nothing...
Sending[18]: 123 ABCDEFGHIJKLMN nothing...
Sending[19]: 123 ABCDEFGHIJKLMNO
Gateway:
Listening at 868 Mhz...
SPI Flash Init OK. Unique MAC = [D7:64:90:B4:C7:33:8B:2E:]
RFM69_ATC Enabled (Auto Transmission Control)
#[1][2] FLASH_MEM_ID:0xEF30 [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[2][2] FLASH_MEM_ID:0xEF30 [RX_RSSI:-23] - ACK sent.
#[3][2] 1 [RX_RSSI:-25] - ACK sent.
#[4][2] 1 [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[5][2] 12 [RX_RSSI:-24] - ACK sent.
#[6][2] 12 [RX_RSSI:-25] - ACK sent.
#[7][2] 12 [RX_RSSI:-29] - ACK sent. Pinging node 2 - ACK...nothing
#[8][2] 123 [RX_RSSI:-23] - ACK sent.
#[9][2] 123 [RX_RSSI:-25] - ACK sent.
#[10][2] 123 [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[11][2] 123 [RX_RSSI:-23] - ACK sent.
#[12][2] 123 A [RX_RSSI:-24] - ACK sent.
#[13][2] 123 A [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[14][2] 123 AB [RX_RSSI:-23] - ACK sent.
#[15][2] 123 AB [RX_RSSI:-24] - ACK sent.
#[16][2] 123 AB [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[17][2] 123 ABC [RX_RSSI:-23] - ACK sent.
#[18][2] 123 ABC [RX_RSSI:-25] - ACK sent.
#[19][2] 123 ABCD [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[20][2] 123 ABCD [RX_RSSI:-23] - ACK sent.
#[21][2] 123 ABCDE [RX_RSSI:-25] - ACK sent.
#[22][2] 123 ABCDE [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[23][2] 123 ABCDEF [RX_RSSI:-23] - ACK sent.
#[24][2] 123 ABCDEF [RX_RSSI:-24] - ACK sent.
#[25][2] 123 ABCDEF [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[26][2] 123 ABCDEFG [RX_RSSI:-23] - ACK sent.
#[27][2] 123 ABCDEFG [RX_RSSI:-23] - ACK sent.
#[28][2] 123 ABCDEFGH [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[29][2] 123 ABCDEFGH [RX_RSSI:-23] - ACK sent.
#[30][2] 123 ABCDEFGHI [RX_RSSI:-24] - ACK sent.
#[31][2] 123 ABCDEFGHI [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[32][2] 123 ABCDEFGHIJ [RX_RSSI:-23] - ACK sent.
#[33][2] 123 ABCDEFGHIJ [RX_RSSI:-24] - ACK sent.
#[34][2] 123 ABCDEFGHIJ [RX_RSSI:-25] - ACK sent. Pinging node 2 - ACK...nothing
#[35][2] 123 ABCDEFGHIJK [RX_RSSI:-23] - ACK sent.
#[36][2] 123 ABCDEFGHIJK [RX_RSSI:-24] - ACK sent.
#[37][2] 123 ABCDEFGHIJKL [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[38][2] 123 ABCDEFGHIJKL [RX_RSSI:-23] - ACK sent.
#[39][2] 123 ABCDEFGHIJKLM [RX_RSSI:-24] - ACK sent.
#[40][2] 123 ABCDEFGHIJKLM [RX_RSSI:-24] - ACK sent. Pinging node 2 - ACK...nothing
#[41][2] 123 ABCDEFGHIJKLMN [RX_RSSI:-23] - ACK sent.
#[42][2] 123 ABCDEFGHIJKLMN [RX_RSSI:-24] - ACK sent.
#[43][2] 123 ABCDEFGHIJKLMN [RX_RSSI:-23] - ACK sent. Pinging node 2 - ACK...nothing
My guess is that I have to be doing something wrong, but I have no idea what.
I've encountered the "ACK...nothing" issue, and it relates to: https://lowpowerlab.com/forum/index.php?topic=1750.0 (which unfortunately no one ever answered). At least in my case, increasing the delay to 4ms solved that issue.
Also, things seem to run a little differently if compiled and uploaded using some of the later IDE releases. They seem to break some things that worked OK using earlier IDE's.
Thanks for the suggestions. I have now tried both increasing the delay on the Gateway and rolling back to the 1.0.6 release of the IDE. This did not affect my problem in anyway, it seems.
However, I did notice something else. When I moved the device (moteino USB as both node and Gateway), sometimes the acknowledgement would get through. If I took hold of the USB cable and quickly waived (shook) the device back and forth the node reported "ok" for a lot of it's messages, but not all. This sometimes happen when I waved either the node or the Gateway device. At the same time, quite often the node started sending its messages much more rapidly while waving, even if it reported "nothing". I guess this has something to do with some kind of interrupt being triggered, something weird happening with the timer?
Since I'm using the USB devices there are no loose connections (apart from the antenna which I have verified does not interfere with any other connections on the board when I wave). I have therefore failed to find any physical reason (apart from the USB connection itself) that should be affected by the waving.
I also tried operating either node or the Gateway from a battery source instead of a USB port, but the results are the same.
In case it is relevant (since there is another thread on the subject), I should also mention that all my devices have flash memories installed.
Have you tried a different USB cable? On he face of it, it sounds like a loose or faulty connection in your wiring.
Yes, and the serial communication to the serial port monitor works flawless on every cable, so I assume the power supply also does.
I have done some more testing. I hacked the node example to add a third device in promiscuous mode to capture everything that goes on between the node and the Gateway. The output of this is included below. The first two numbers is the source and destination. After this comes the payload of the data captured, and whether an acknowledgement is requested or if the packet is an acknowledgement. It is clear from the trace below that the gateway actually sends the acknowledgement. So the problem is in the receiving end for the node.
Promiscuous at 868 Mhz...
SPI Flash Init OK ... UniqueID (MAC): D7 64 90 B4 C7 33 8B 2E
RFM69_ATC Enabled (Auto Transmission Control)
[2, 1] FLASH_MEM_ID:0xEF30 [RX_RSSI:-9] - ACK Requested
[1, 2] [RX_RSSI:-17] is ACK!
[2, 1] FLASH_MEM_ID:0xEF30 [RX_RSSI:-11] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-17] - ACK Requested
[2, 1] FLASH_MEM_ID:0xEF30 [RX_RSSI:-10] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-18] - ACK Requested
[2, 1] 1 [RX_RSSI:-9] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-18] - ACK Requested
[2, 1] 1 [RX_RSSI:-9] - ACK Requested
[1, 2] [RX_RSSI:-17] is ACK!
[2, 1] 1 [RX_RSSI:-10] - ACK Requested
[1, 2] [RX_RSSI:-18] is ACK!
[2, 1] 12 [RX_RSSI:-10] - ACK Requested
[1, 2] [RX_RSSI:-18] is ACK!
[2, 1] 12 [RX_RSSI:-10] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-17] - ACK Requested
[2, 1] 12 [RX_RSSI:-10] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-17] - ACK Requested
[2, 1] 123 [RX_RSSI:-10] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-17] - ACK Requested
[2, 1] 123 [RX_RSSI:-10] - ACK Requested
[1, 2] [RX_RSSI:-17] is ACK!
[2, 1] 123 [RX_RSSI:-10] - ACK Requested
[1, 2] [RX_RSSI:-18] is ACK!
[2, 1] 123 [RX_RSSI:-10] - ACK Requested
[1, 2] [RX_RSSI:-18] is ACK!
[2, 1] 123 [RX_RSSI:-11] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-17] - ACK Requested
[2, 1] 123 [RX_RSSI:-11] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-17] - ACK Requested
[2, 1] 123 A [RX_RSSI:-10] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-19] - ACK Requested
[2, 1] 123 A [RX_RSSI:-17] - ACK Requested
[1, 2] [RX_RSSI:-18] is ACK!
[2, 1] 123 A [RX_RSSI:-12] - ACK Requested
[1, 2] [RX_RSSI:-17] is ACK!
[2, 1] 123 AB [RX_RSSI:-12] - ACK Requested
[1, 2] [RX_RSSI:-17] is ACK!
[2, 1] 123 AB [RX_RSSI:-11] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-17] - ACK Requested
[2, 1] 123 AB [RX_RSSI:-11] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-18] - ACK Requested
[2, 1] 123 ABC [RX_RSSI:-11] - ACK Requested
[1, 2] ACK TEST [RX_RSSI:-18] - ACK Requested
[2, 1] 123 ABC [RX_RSSI:-11] - ACK Requested
[1, 2] [RX_RSSI:-17] is ACK!
Edit: I suddenly got it to work quite well by putting in 200 as the retry value for the node. Suddenly every packet started going through at the first try. However, this was not the case when switching gateway and node devices. Also, I was not able to reproduce this when switching the devices back again. I've done quite a little playing around with timing values as well, without success.
You're RSSI numbers look extremely strong. Is your gear positioned incredibly close together or something?
Yes, they are connected to the same computer, within 50 cm of each other. I tried reducing output power with setPower(1), but the numbers were still pretty high.
Ah, well that opens a whole different can of worms. If you have only one computer, try running just two nodes, but running one of them off a battery positioned at a more normal distance (let's say a minimum of 5 meters for testing purposes just to be sure). Also, reboot the one that's connected to the computer, so that the ATC is reset. What happens then?
I did as you suggested. I move them around 5 m apart (the other device in another room, inside the fuse cabinet with the door open), and reduce the power level to 0 (which is five dBm). The reported RSSI is now around -53 for the gateway. I tried moving both of the devices after each other, but the results were consistently the same. I still get no acknowledgements received.
Uh, why put it inside a fuse cabinet? ??? That sounds like another potential snakepit of electrical noise that you'd want to avoid.
Heh, it was the easiest available power supply at the time, and I need a sensor in there to read the power meter ☺️
Anyway, I have tested them further apart outside of the fuse cabinet (a distance of about 10 m) with different power settings (0 and 15) without significant improvements. I get some acknowledgements on some messages, but now we also loose a number of messages on the way from the node to the Gateway.I guess I can increase power, but that sort of negates the increase in distance :-)
If you have no loose connections, and *if* you have removed your RFM69 gear from sources of electrical noise (it sounds as though you may have some rather contrarian ideas about how to accomplish that ::)), then the only reason I can think of as to why frantically waving your gear around would improve receipt of packets is doppler shift. That would suggest your Tx and Rx frequencies aren't adequately overlapped. Do you have access to a spectrum analyzer (even a cheap SDR one) to check that? If not, try setting your RFM69's to use a much larger FDEV and much wider Rx bandwidth to match it, and see if that improves your percentage of received packets. Does doing that make any difference?
Thanks for the suggestion. Unfortunately, I do not have access to any kind of spectrum analyser, although this might be the perfect opportunity to get a cheap one :-). I tried to go from RF_FDEVMSB_50000 to RF_FDEVMSB_85000 both devices, but it did not seem to change anything. Changing the RX bandwidth does not seem as easy (i.e. I don't know how), so I haven't tried that.
Edit: I do not think we should focus too much on the waving. My guess is that this is more related to small interruptions or disturbances in the power supply or something.
Have you tried changing the RSSI threshold of your receiver when it's supposed to be receiving the ACK? A possible problem is that this is set way too low and is triggering on noise. If that happens the AFC and AGC values latch onto the noise source which may not be at exactly the right frequency and power, and the receiver fails to receive a legitimate packet. Try setting the threshold to something a few 10s of dB below the expected RSSI stength to start with.
Mark.
I will be happy to try that once I figure out how, but I am not sure how this fits the main observation. The gateway is always able to receive messages from the node, while the node almost never receive anything from the Gateway (either acknowledgements or regular messages). This is the case regardless of which device are used as gateway or node, without physically touching them in between uploading the sketches.
Using a third node as a promiscuous listener I can confirm that the Gateway is actually sending the acknowledgements. I think this rules out any frequency issues and leaves me with a library issue, or a hardware issue (although I have tried this with four different devices). I even tried running two nodes talking to each other without success, and I also added the transmission part of the buffer from the node to a separate gateway to see if this works. I actually got a few more messages through, but this may just be coincidence.
Could it be a memory issue of some sort?
For what it's worth I tried changing the RSSI sensitivity by changing the value in the following line to 120. I saw no change in behaviour.
/* 0x29 */ { REG_RSSITHRESH, 220 }, // must be set to dBm = (-Sensitivity / 2), default is 0xE4 = 228 so -114dBm
I'm going to throw something out there, I noticed on my setup (my own code, I don't use any libraries) that the receiver would refuse to decode the first packet it sees after transitioning from standby mode to RX mode. It was strange because it saw an RSSI interrupt of the correct value and at the correct time, but did not detect an address match or get a packet received for the first packet. Subsequent packets were fine. I changed my code to go into sleep mode rather than standby and the problem went away.
Somewhere in the library code there's a function that maps idle to either standy or sleep, could you change it to sleep and see what happens? My theory is that transitioning from standy mode to RX mode doesn't fully reset the receiver, but transitioning from sleep to RX mode does. After the first packet has started to be received (which doesn't complete) the receiver is then able to receive subsequent packets. This may be what you're seeing.
Mark.
I like the theory. I couldn't find the standby versus sleep function you talked about, so as a first attempt I simply sent to acknowledgements right after each other for every received packet. The first time I tried this I got a success rate of about 35% of acks received. Sadly, after playing around with the setMode function to try to map all the standbys to sleep I was not able to reproduce this. Not even after removing all my changes to the library and just going back to the two acknowledgements was I able to get the same results again. I guess this breaks the theory a bit.
Also, for every third packet the Gateway receives from the node, it sends a packet in return waiting for an acknowledgement. If your theory was correct, then the node should always receive this packet since it was sent just after the acknowledgement (with a delay of 4). It almost never receives this packet. Is there an easy way to see whether the radio is receiving anything? It would be nice to see whether it sees the ACK signal at all and is just unable to capture the full packet as you indicate.
If you look in RF_RF69.h you'll see a reference to function
void setIdleMode(uint8_t idleMode);
This function sets the idle state. If you add the following call during intialization you'll set the idle mode to sleep:
setIdleMode(RH_RF69_OPMODE_MODE_SLEEP);
From that point onwards, when the radio is neither transmitting or receiving it'll be in sleep instead of its default state of standy. This is the only change you need to make, I'd recommend going right back to the beginning and reloading a fresh system and make just that change on both the server and the node (it's important to do it on both).
Mark.
I see no declaration of this function in the file you mention. Does this mean that I have the wrong version of the library? I downloaded it from https://github.com/LowPowerLab/RFM69 only yesterday.
Oh, I've been looking at RadioHead! :-( In that there's an idle mode defined which can be either sleep of standby. Felix's libray has simplified things, actually looks much cleaner.
Can I suggest then that the #define of RF69_MODE_STANDBY be changed to the same value as RF69_MODE_SLEEP as a temporary hack? Whenever it's put into standby it should instead be put into sleep. This may have unforeseen consequences but I'm hoping the sequencer will deal with it properly.
Mark.
I already tried that, replacing the reference to STANDBY with SLEEP in setMode. It did not have any appreciable effect.
OK, that probably puts that theory to bed then. Pity, because what I saw was very similar to what you are seeing. It could still have been a red herring due to something I was doing wrong, but I had a scope on it and pretty sure it was failing to see an address match even though it got an RSSI interrupt at the right time and right value. Looks like you'll have to dig deeper.
Mark.
Okay, now just going crazy.
Just for laughs I tried again to change from standby to sleep mode. At the beginning of the setMode function I entered the following code block.
if (newMode == RF69_MODE_STANDBY) {
newMode = RF69_MODE_SLEEP;
}
This time sending messages from the node to the Gateway and receiving acknowledgements seems to work almost every time. The only time it does not work is when the Gateway sends a message in return after every third message it receives from the node. I think this is some kind of race condition were both devices wait for acknowledgement from the other one. This means that the node usually does not receive the message being sent from the gateway.
Still, I call this progress :-)
Edit: I think some of the problems with the messages from the gateway was that I had turned on retries for the message from the gateway to the node. I remove this, and I get more successful messages from the gateway to the node. I also get rid of every third message from the node to the Gateway being unacknowledged. now, if I only could get more reliable messaging from the gateway to the node, everything would be perfect :-)
OK, that's progress. You have done this hack to both the gateway and the node right? Anyhow, can you confirm that just taking out that hack causes it to fail again, because if you can this represents a significant bug. Two of us will have seen it, and Felix should be made aware of this.
Any change in posting the code that does the sending and receiving on both the gateway and node? I'm curious about the race condition and whether there's any timing issues there.
Mark.
Edit: And just to confirm it's not the little delay that your code added maybe put a small delay in where you put the hack, if that also fixes it we're chasing ghosts.
Okay, I have tested the following combinations:
- Without the hack to setMode: Acknowledgements are never received, nor any other message from the gateway to the node.
- With a delay of 1 or 5 in setMode in place of the hack: No change, no acknowledgements are received.
- With the hack only at the node: Some acknowledgements are received by the node, but still less than half.
- With the hack on both devices: all messages from the node to the Gateway are acknowledged, but the messages from the gateway to the node are seldom acknowledged.
You can find the sketches for the node and gateway in the examples folder in the RFM69 library. I have not altered the sketches other than setting the correct frequency and high power mode. The fix consists of changing void RFM69::setMode(uint8_t newMode) in RFM69.cpp by inserting
if (newMode == RF69_MODE_STANDBY) {
newMode = RF69_MODE_SLEEP;
}at line 144. The resulting full function is included below.
It would be great if you could replicate my results.
I'm a bit confused by why this does not solve the problem with the messages that are sent from the gateway to the node. The message is sent 4 ms after the acknowledgement to the last message, so they should arrive in order at the node, and before the node has time to send a new message to the Gateway.
void RFM69::setMode(uint8_t newMode)
{
if (newMode == RF69_MODE_STANDBY) {
newMode = RF69_MODE_SLEEP;
}
// delay(5);
if (newMode == _mode)
return;
switch (newMode) {
case RF69_MODE_TX:
writeReg(REG_OPMODE, (readReg(REG_OPMODE) & 0xE3) | RF_OPMODE_TRANSMITTER);
if (_isRFM69HW) setHighPowerRegs(true);
break;
case RF69_MODE_RX:
writeReg(REG_OPMODE, (readReg(REG_OPMODE) & 0xE3) | RF_OPMODE_RECEIVER);
if (_isRFM69HW) setHighPowerRegs(false);
break;
case RF69_MODE_SYNTH:
writeReg(REG_OPMODE, (readReg(REG_OPMODE) & 0xE3) | RF_OPMODE_SYNTHESIZER);
break;
case RF69_MODE_STANDBY:
writeReg(REG_OPMODE, (readReg(REG_OPMODE) & 0xE3) | RF_OPMODE_STANDBY);
break;
case RF69_MODE_SLEEP:
writeReg(REG_OPMODE, (readReg(REG_OPMODE) & 0xE3) | RF_OPMODE_SLEEP);
break;
default:
return;
}
I'll take a look at the sketch. One difference though is the set mode routine waits for mode ready only when coming out of sleep. Given we're now sleeping now rather than using standby we're going to be waiting for mode ready more often. My code though always waited for mode ready when changing modes regardless of previous mode and I still saw the issue, so I suspect it isn't relevant but worth pointing out.
Mark.
It seems there are some issues with the going to sleep instead of standby. RSSI at the receiver is always reported at -128, which means that the ATC feature will most probably not work.
I'm also having some problems with range, but I am not yet sure if this is related to the fix. I have the HW versions at 868 MHz, but only get around 100 m of range from a node mounted on the first floor in a wooden house. I use the simple wire antenna with a length of just under 9 cm. It might be that I should switch to some better antennas. I will report when I figure out if the range issue is related to the fix at all.
Is there another way we can "reset" the radio so that it doesn't lose the first packet it receives?
I managed to get the RX RSSI reported correctly by changing where we go into sleep. Instead of doing it globally in setMode, I put into sleep, than standby in the beginning of receiveBegin. Like this:
setMode(RF69_MODE_SLEEP);
setMode(RF69_MODE_STANDBY);
This solves both the reception of acknowledgements problem and reports the correct signal strength.
On a happy note I don't think the range issue is related to the receiver sleeping, but I have posted a separate thread on this issue to see if someone can help me out.
Cool, you've solved the acknowledgements in both directions now? Looks like we have a smoking gun, there's definitely a bug with the standby to RX mode transition not fully resetting the receiver.
Mark.
Edit: You may find that call to set standby mode after the call to go to sleep is redundant, since it is now transitioning from sleep to RX the sequencer will deal with that and the set mode call will be waiting for mode ready.
Thanks, I will remove the extra standby and see what happens.
It still has problems with receiving messages at the node from the gateway. While it gets an acknowledgement more or less on the first try of every message from the node to the Gateway, the messages from the gateway to the node are still seldom received. If it is received, the acknowledgement back to the Gateway seems to work fine. Note that the Gateway has zero retries on the messages it sends to the node, but still, most of them should have gotten through.
I remove the extra _SLEEP as you suggested, and as you expected, things continued to work fine.
I also figured out the problem with the node never receiving messages from the gateway. It turns out that the cycle from sending a message from the node to receiving the acknowledgement from the gateway (I have reduced the bit rate slightly, but I don't think that is the major factor) takes around one second. The node transmits messages to the Gateway every 150 to 1000 ms, which means that it transmits immediately after receiving the acknowledgement. I guess it will then discard any message it gets from the radio that is not the acknowledgement it is waiting for in sendWithRetry. Increasing the transmission interval allowed the node to receive everything from the gateway.
I'm not sure what is the cause of this long message/acknowledgement round-trip, maybe it is caused by the sleep? I thought the node will only wait a few hundred milliseconds for an acknowledgement, so I guess the delay is on the receiving side?
Not sure, but RF69_CSMA_LIMIT_MS is set to 1000ms, and in the sendACK routine there's this:
while (!canSend() && millis() - now < RF69_CSMA_LIMIT_MS) receiveDone();
which waits until it can send but with a 1s timeout after which it sends anyway. This could mean that canSend() is not returning true. You could try changing RF69_CSMA_LIMIT_MS to something much smaller to check.
Edit: This may be down to RSSI reading, I had terrible trouble with this and so did WhiteHare. WhiteHare tested a solution that works well and there's a huge thread about it somewhere (basically you need to set the RSSI threshold to below the noise floor to trigger a read, then manually reset the receiver once it triggers to start another read. The mechanism for triggering a read by writing to the RSSI_START bit does not work. There's an additional problem if it happens to start to receive a legitimate packet as well, but it's all covered in the thread).
Mark.
Thanks for the info. Before reading your edit I have managed to increase the RSSI threshold after noticing that the irises I reported while waiting was -67. Setting the threshold at -60 allowed it to send its packet immediately without waiting a second.
I will read your edit once more and see if I can figure out what you're saying.
I think I found the thread (https://lowpowerlab.com/forum/index.php/topic,1468.15.html). However, I think my RSSI readings are correct. A plug my node into my laptop and walked around the house and I saw values varying between -60 and -85. Apparently I have a very noisy RF environment. It could be because I have a z-wave network throughout the house. Is it possible/legal/recommended to nudge the moteino frequency up or down a bit?
Have you tried a different power supply? Better yet run both moteinos from batteries and check if that changes anything. If your noise level is above the csma level that will cause huge problems.
I've generally found the csma code to be more trouble than useful and I often just turn it off.
You can certainly detune the radio. But I doubt that an active Rf signal would regularly cause 1s csma failures. I think it's likely noise that comes in via the power supply.
Thanks for the suggestion, but how will I see what happens if I do not connect them to my computers? I connected one to my laptop without external power, but that didn't help. I'm using the USB moteino as my PC gateway which means that it has to be powered by the computer, or can I do something to cut the power connection? I guess the same goes for the FTDI adapter I use, I could cut the power coming through it, but will that really help as long as the serial lines are connected?
QuoteThanks for the suggestion, but how will I see what happens if I do not connect them to my computers?
Tricky, isn't it? Here's what I do: I run tests between two battery powered Moteinos and report the results back to a third Moteino that's connected to a computer. Well actually I send the results to my esp8266 based gw which forwards it to a syslog daemon on one of my servers - but that's a story for another day.
It's unlikely that noise is happening at that kind of level for a full second with no gaps IMO. There are serious issues with the mechanism for reading the RSSI using RSSI_START (it's even been commented out of the RadioHead library with a comment stating the author couldn't get it to work properly), it simply doesn't work reliably (sometimes it gets frozen and reads the same value over and over again for example). That thread you saw (https://lowpowerlab.com/forum/index.php/topic,1468.15.html) has the details, and a fix.
Mark.
@Kolaf You can also write the results to the winbond flash memory and review them sometime later. I did that when taking RSSI noise measurements inside a sealed metal box.
Sorry I didn't remember this earlier, but would another option for you to consider/investigate be to go into FS mode and shift from there back into Rx mode instead of first going into either idle mode or sleep mode and then (whichever one you pick) from there back to Rx mode? Moving from FS mode back into Rx mode should make for a faster transition, because the latter two modes (idle or sleep) require traversing FS mode before getting back to Rx mode anyway (the sequencer just does it automatically, so it may not be glaringly obvious that's what's actually happening): https://lowpowerlab.com/forum/index.php/topic,1468.msg11711.html#msg11711
Instead of going into sleep mode, perhaps that would more efficiently work around the smoking gun bug you may have discovered? I was tempted to consider it anyway, just because it seems like it would be faster.
Given it appears going from standby into RX doesn't seem to work correctly and that would transition through the FS state in the sequencer, while going from sleep to RX mode does I don't think it would work. I tried lots of things, including issuing a restart, with no effect (can't remember if I explicitly tried that one). I'm not sure what the failing mechanism is but it looks like the firing up of the oscillator and subsequent deep reset of internal logic has something to do with it. Anyway, it would be a good thing to reduce this to a very simple failing scenario and find out if there's any other solution than sleeping.
Mark.
@kolaf How long ago did you acquire the RFM69HW modules?
For the noise problem I configure the node to attach the sensed RSSI directly before sending the packet to the packet so that the Gateway will display it. I then powered the node from a battery to see if the values changed. They did not. I get the same values when using the battery as I do when powering it from a USB port, around -65 to -75. The value varies depending on how I positioned the antenna and where my body is in relation to the node and other equipment in my room. I'm therefore guessing that this is actual background noise. Based on your comments I assume my best bet is to simply raise or remove the CSMA limit.
@WhiteHare all my moteinos a few years old, I think. I have never gotten any RFM69HW module separately, although I have some moteino clones as well.
Wow, -65 dBm - that is VERY noisy in particular if it's continuous. This will impact your range very significantly so it's worth trying to move the frequency a little. Which you can easily do - just look at rfm69.cpp and how it sets the three frequency registers.
I need to ask: you do put the radio into rx while you measure RSSI, correct? Otherwise this might be the RSSI from the last ack.
Yes, it is in receive mode. I am reporting the value as is seen in the canSend function.
That calls readRSSI() which seems to just read the RSSI register assuming the DAGC is continually updating it, but I'm really not sure that happens until it starts receiving something, and even then it seems flaky. I couldn't get it to work properly and I think WhiteHare had the same issue. If you replace that with one of the routines in that RSSI thread you may have more joy.
Something like this (untested):
uint8_t getRssi(void)
{
uint8_t oldRssiThreshold;
uint8_t rssiVal;
oldRssiThreshold = rfm69RdSpi(REG_RSSITHRESH);
rfm69WrSpi(REG_RSSITHRESH, 0xFF); // permanently trigger
while((rfm69RdSpi(REG_IRQFLAGS1) & IRQFLAGS1_RSSI)==0); // wait for RSSI
rssiVal = rfm69RdSpi(REG_RSSIVALUE); // read RSSI
// empty the FIFO if a packet received to restart receiver,
// otherwise RSSI value freezes
if(rfm69RdSpi(REG_IRQFLAGS2) & IRQFLAGS2_PAYLOADREADY)
{
while(rfm69RdSpi(REG_IRQFLAGS2) & IRQFLAGS2_FIFONOTEMPTY)
{
rfm69RdSpi(REG_FIFO);
}
}
else
{
// restart to force a new reading
rfm69WrSpi(REG_PACKETCONFIG2, (PACKETCONFIG2_RESTARTRX | PACKETCONFIG2_AUTORXRESTARTON));
}
rfm69WrSpi(REG_RSSITHRESH, oldRssiThreshold); // restore RSSI threshold
return(rssiVal);
}
Thanks. I guess this is what I should have been able to glean from the other thread we talked about? Apparently I didn't read it carefully enough :-).
Anyway, I implemented the function and fixed it up a bit to make it work, but in the end it seems to report the exact same values as the other function does. I still get between -64 and -70 located around 1 m away from my computer equipment.
The working function looks like this:
int16_t RFM69::getRssi(void)
{
uint8_t oldRssiThreshold;
int16_t rssiVal;
oldRssiThreshold = readReg(REG_RSSITHRESH);
writeReg(REG_RSSITHRESH, 0xFF); // permanently trigger
while((readReg(REG_IRQFLAGS1) & RF_IRQFLAGS1_RSSI)==0); // wait for RSSI
rssiVal = -readReg(REG_RSSIVALUE); // read RSSI
rssiVal >>= 1;
// empty the FIFO if a packet received to restart receiver,
// otherwise RSSI value freezes
if(readReg(REG_IRQFLAGS2) & RF_IRQFLAGS2_PAYLOADREADY)
{
while(readReg(REG_IRQFLAGS2) & RF_IRQFLAGS2_FIFONOTEMPTY)
{
readReg(REG_FIFO);
}
}
else
{
// restart to force a new reading
writeReg(REG_PACKETCONFIG2, (RF_PACKET2_AUTORXRESTART_ON | RF_PACKET2_AUTORXRESTART_ON));
}
writeReg(REG_RSSITHRESH, oldRssiThreshold); // restore RSSI threshold
return(rssiVal);
}
Well, that seems to have eliminated the RSSI reading function and it looks like that's the real noise level. Have you been able to get any lower readings, like remove the antenna, battery operate and stick inside a metal box? And maybe temorarily tack the antenna pin to ground while you do it (don't transmit while you do that)?
I bumped the frequency up to 869 and this dropped the rssi down to -95/-100. I guess the reason for the high RSSI is interference with my zwave network at 868.47 MHz. Still, it is a bit weird says the zwave network is packet-based and does not transmit that often.
Edit: Unfortunately I cannot get the communication to work either on 869 or 867 MHz. I try to change the frequency simply by replacing the three bytes in the radio initialisation so that it looks like this:
/* 0x07 */ { REG_FRFMSB, (uint8_t) (freqBand==RF69_315MHZ ? RF_FRFMSB_315 : (freqBand==RF69_433MHZ ? RF_FRFMSB_433 : (freqBand==RF69_868MHZ ? RF_FRFMSB_867 : RF_FRFMSB_915))) },
/* 0x08 */ { REG_FRFMID, (uint8_t) (freqBand==RF69_315MHZ ? RF_FRFMID_315 : (freqBand==RF69_433MHZ ? RF_FRFMID_433 : (freqBand==RF69_868MHZ ? RF_FRFMID_867 : RF_FRFMID_915))) },
/* 0x09 */ { REG_FRFLSB, (uint8_t) (freqBand==RF69_315MHZ ? RF_FRFLSB_315 : (freqBand==RF69_433MHZ ? RF_FRFLSB_433 : (freqBand==RF69_868MHZ ? RF_FRFLSB_867 : RF_FRFLSB_915))) },
Note the reference to 867. It seems to read the RSSI correctly, but the gateway is not able to receive anything that is sent by the node. I'm not sure if this is because sending fails or if reception fails. Is there any other failsafe that needs to be changed to allow it to transmit at a different frequency, perhaps?
Edit: I am afraid I broke something. I am now not able to receive anything at the gateway. I can verify that the rssi recorded by the gateway increases from -65 to around -17 the short period the node transmits, but it does not seem to be able to detect the packet. I have reverted all changes I did to the frequency shifting. Are there some kind of registers I need to explicitly reset or something to get the frequency to stick?
Edit again: it turns out that calling the getRssi function repeatedly before sending cause something to fail. As soon as I removed this, communication between the two devices was restored as normal, even with the new frequency. I'm happy to report that collocation with the sensor inside my barn (50 m away) has been restored after changing the frequency, probably because a got away from the noise near the gateway which was higher than the signal from the barn node (-77).
So to summarise, Fixing the receiver sleep/standby functionality, reducing the bit rate, and changing the frequency was what it took for me to get a (for now) stable network :-). Thanks for the help, guys.
Well, that's worrying. You're in Norway right? I was under the impression the 868.0MHz to 868.6MHz band has been harmonized over Europe and is restricted to 25mW erp and 1% duty cycle, unless you use LBT+AFA but the effect is still to reduce the duty cycle on any particular frequency. So if your RSSI readings are correct it means the z-wave system (or something) is permanently broadcasting! That's not good.
Anyway, -95 to -100 is much better, although I would expect a little lower. At least you can move forward now ;-)
Mark.
Quote from: perky on April 19, 2016, 04:01:19 PM
I'm going to throw something out there, I noticed on my setup (my own code, I don't use any libraries) that the receiver would refuse to decode the first packet it sees after transitioning from standby mode to RX mode. It was strange because it saw an RSSI interrupt of the correct value and at the correct time, but did not detect an address match or get a packet received for the first packet.
@perky I'm willing to tack a crack at reproducing the failure sequence you're experiencing, but before I do, just a point of clarification. I'm assuming your interrupt handling routine remains active when transitioning from standby into RxMode using the sequencer (or are you waiting until ModeReady before attaching the ISR and/or enabling interrupts?), and that you're in packet mode (obviously), with the interrupt tied to DIO0, and DIOx mapping is 11b. In this case, you're getting two interrupts: the first occuring when PLL Locks and then the second when RSSI goes HIGH. So, was transmission of the packet (the one which you're saying the RFM69 refused to decode) started immediately after ModeReady went high? The reason I ask is that if it was started too much earlier, then the RSSI might still go HIGH (i.e. the second interrupt) and even appear to have the right RSSI value based on energy received from bits transmitted after the preamble, even though at that point the packet was officially missed. So, I guess I'm wondering whether you've ruled out that possibility, and if so, how, so that I can attempt to reproduce your finding using your same method.
As an aside, it appears as though the RFM69 is going to keep the Rx settings it picked during the RSSI samplings (which might have been based on garbage frame fragments or junk transmissions from some other kind of radio or simply spurious noise) basically forever until it finally does actually receive a valid packet, but which must be received
using those possibly erroneous settings (or unless you manually program a time-out via the mcu to restart Rx-mode after the RSSI flag goes HIGH if no packet has been received within a particular timeframe). Other than Listen-Mode, I don't remember reading anywhere that the RFM69 will ever time itself out and restart Rx-mode on its own, that is unless after a packet it deems valid is received and/or its FIFO is subsequently emptied. Is that right? Also, if RxBw were narrow and if AFC happened to screw up for some reason and pull the RFM69 too far from the proper Rx frequency, I suppose it might be lost that way forever unless the mcu was programmed to eventually timeout waiting for a packet to be received and instead restart the Rx-mode. It does look as though the RFM69 has a timeout function (described in 3.4.18 of the SX1231h datasheet), but it appears it only (?) is "used to warn the companion processor to shut down the receiver and return to a lower power mode."
OK, first thing is I am not using any libraries. The code is actually quite simple, I'm polling the SPI for flags like RSSI, address match and payload ready bits so don't use interrupts. The RSSI status happens at exactly the correct time (seen on a scope by setting an output when the SPI reads an RSSI status), and the register reads the expected RSSI, but no address match (and subsequently no payload ready) when putting into RX mode from standby. A simple change to put into sleep rather than standby fixes it. The datasheet says that entering RX mode from any state should reset the receiver, including from standby, but this appears not to reset it fully. I put it into RX mode about 5ms before the packet is sent from the server, I see that exact timing on the scope so it's not immediately triggering on noise. I see exactly the same timing when using sleep instead, but this time I get an address match. It really does look like the internals are not being reset properly.
I hope you can re-create it, it'll add more weight to the argument.
Mark.
Hi guys,
Any progress on pinpointing the issue and creating a better fix for it?
My system has been running quite stably for a time now after switching from standby to sleep, so I'm quite happy.
I just thought I would give a short update on my background noise issue. I bought a cheap sdr radio which I plugged into my android phone to use a is a cheap frequency analyser. I have attached a screenshot where it is tuned to the 868 MHz band. As you can see, it is quite noisy, so I clearly have some kind of transmitter that is very active on this band. It's quite constant everywhere in the house, and even outside several hundred metres from my house. I have no idea what it is yet, but I'm planning to do some strangulation to see if I can figure it out :-).
https://goo.gl/photos/vqo41ZZ3W8H1HRQY6
I can't help you with your background noise issue obviously. But your posts were the ones that finally pushed me to get a cheap sdr myself. I can only recommend it to everybody. $20 or so and you can finally see what the radios are doing in real time. No more wondering whether you have a bug on the rx or tx side. Very useful, thanks!