Author Topic: Stuck in listen mode - Listen Mode bug with workaround  (Read 9301 times)

joelucid

  • Hero Member
  • *****
  • Posts: 867
Stuck in listen mode - Listen Mode bug with workaround
« on: December 28, 2016, 05:46:09 AM »
I've initially often had problems with my listen mode low power motes where some nodes would not react to wakeup bursts after some time. This wasn't solvable by reboot - you actually had to reset the radio for things to work again. Since I had found a mode of operation where this problem never occurred I kind of forgot about it.

Recently though I've changed how most of my nodes operate from a listen mode only to listen mode plus regular updates approach. This triggered the behavior again and today I finally figured out what's going on:

After some unfortunate rhythm of starting and stopping listen mode the radio gets into a state where in listen mode it doesn't keep the radio in RX once a packet is detected. Rather it always switches RX off once the rx period is over, even if RSSI has been detected.

The workaround is to listen to the RSSI interrupt during sleep and when it's detected immediately abort listen mode while staying in RX, thus manually preventing listen mode from pulling the rug from the receiver.

Joe

TomWS

  • Hero Member
  • *****
  • Posts: 1924
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #1 on: December 28, 2016, 12:44:49 PM »
GOOD WORK!  This has annoyed me for a long time but I didn't have the wherewithal (or priority) to look into it.  Thanks for solving!

Tom

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #2 on: December 28, 2016, 04:03:36 PM »
Quote
GOOD WORK!  This has annoyed me for a long time but I didn't have the wherewithal (or priority) to look into it.  Thanks for solving!

Oh man - apparently not good enough: as I rolled the change out to all my nodes some of them started to get stuck in listen mode even earlier than before. Turns out the RFM69 also doesn't like it if listen mode is aborted too early during the listen mode RX period. This was a problem primarily with nodes that had fuses set for very quick wakeup.

So far it seems a
Code: [Select]
delayMicroseconds( 50 )

before

Code: [Select]
		SX1231Driver::writeReg( SX1231_REG_OPMODE, SX1231_OPMODE_SEQUENCER_ON | SX1231_OPMODE_LISTENABORT | SX1231_OPMODE_RECEIVER );
SX1231Driver::writeReg( SX1231_REG_OPMODE, SX1231_OPMODE_SEQUENCER_ON | SX1231_OPMODE_RECEIVER );

does the trick. Even more fragile than a si7021 at >80% humidity  :D

Joe

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #3 on: December 30, 2016, 05:43:41 AM »
I'm sure I'll regret this - but things seem to run smoothly now and it's good to claim victory at times. So here is the definite list of listen mode bugs and how to work around them:

1.) No end of RX period after noise RSSI detection

In many cases if RSSI is detected but no valid packet follows listen mode does not time out (no matter what the end cycle options). Instead of just waiting for SX1231_IRQFLAGS2_PAYLOADREADY you have to use SX1231_DIOMAPPING1_DIO0_11 and wake on RSSI detected instead. Then wait for a packet until it is received or timeout. Shut down listen mode in both cases and restart. Use false RSSI detections to dynamically tune the RSSI threshold.

Note that you'll also receive IRQs for SX1231_IRQFLAGS1_PLLLOCK before RSSI with this mapping. This is useful: you can use these irqs as timer for regular updates without powering the WDT.

2.) Don't shut down listen mode immediately after SX1231_IRQFLAGS1_PLLLOCK hits

Since you regularly get PLLLOCK its tempting to use that as timer source e.g. for regular updates. That's fine if it's only used for that purpose (RSSIThresh = 0). However if you have a RSSIThresh that allows packet reception and decide to end listen mode due to PLLLOCK irq you must wait for some microseconds so you don't disable listen mode during beginning the RX period. I'm using 300us currently.

3.) End listen mode during reception

This is the original issue described in this post. After RSSI detected allow an extra 50-100uS and then end listen mode to work around the case that listen mode ends the RX cycle even though RSSI was detected. Don't do it immediately when RSSI hits since that will sometimes completely kill listen mode (beyond the workaround of 3.) until radio reset.

Overall a big part of the story seems to be not to end listen mode early in the RX phase. Easy to do if your wakeup timer is PLLLOCK. If you use the WDT you'll have to at least record PLLLOCK times and avoid ending listen mode immediately afterwards.

Keeping fingers crossed that we'll actually end 2016 with a stable way to deal with listen mode.

Joe
« Last Edit: December 30, 2016, 06:13:37 AM by joelucid »

joelucid

  • Hero Member
  • *****
  • Posts: 867
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #4 on: December 30, 2016, 06:36:52 AM »
Quote
That's fine if it's only used for that purpose (RSSIThresh = 0)

BTW, it you use plllock only you can get by with a RX period of 128 uS. I use that for delays or when operating without the capability to wake up via listen mode as wakeup timer. If you wake up once per minute the 128uS per 60s at ~20mA cost you around 43nA on top of the sleep current of 1.2uA. Even with a clock crystal attached you won't get much lower than that. Of course an AB18X5 will put these numbers to shame.

Joe

ChemE

  • Sr. Member
  • ****
  • Posts: 417
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #5 on: December 30, 2016, 08:36:32 AM »
Of course an AB18X5 will put these numbers to shame.

Joe

Sure wish Atmel would replace their WDT logic with whatever magic is happening in the AB18X5.  4,000nA vs. 18nA is not just a landslide, it is a solar-system-slide.

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #6 on: December 30, 2016, 08:56:12 AM »
I'm sure I'll regret this - but things seem to run smoothly now and it's good to claim victory at times. So here is the definite list of listen mode bugs and how to work around them:

1.) No end of RX period after noise RSSI detection

In many cases if RSSI is detected but no valid packet follows listen mode does not time out (no matter what the end cycle options). Instead of just waiting for SX1231_IRQFLAGS2_PAYLOADREADY you have to use SX1231_DIOMAPPING1_DIO0_11 and wake on RSSI detected instead. Then wait for a packet until it is received or timeout. Shut down listen mode in both cases and restart. Use false RSSI detections to dynamically tune the RSSI threshold.

Note that you'll also receive IRQs for SX1231_IRQFLAGS1_PLLLOCK before RSSI with this mapping. This is useful: you can use these irqs as timer for regular updates without powering the WDT.

2.) Don't shut down listen mode immediately after SX1231_IRQFLAGS1_PLLLOCK hits

Since you regularly get PLLLOCK its tempting to use that as timer source e.g. for regular updates. That's fine if it's only used for that purpose (RSSIThresh = 0). However if you have a RSSIThresh that allows packet reception and decide to end listen mode due to PLLLOCK irq you must wait for some microseconds so you don't disable listen mode during beginning the RX period. I'm using 300us currently.

3.) End listen mode during reception

This is the original issue described in this post. After RSSI detected allow an extra 50-100uS and then end listen mode to work around the case that listen mode ends the RX cycle even though RSSI was detected. Don't do it immediately when RSSI hits since that will sometimes completely kill listen mode (beyond the workaround of 3.) until radio reset.

Overall a big part of the story seems to be not to end listen mode early in the RX phase. Easy to do if your wakeup timer is PLLLOCK. If you use the WDT you'll have to at least record PLLLOCK times and avoid ending listen mode immediately afterwards.

Keeping fingers crossed that we'll actually end 2016 with a stable way to deal with listen mode.

Joe

This is gold!  Somebody with powers should sticky this.

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #7 on: December 30, 2016, 09:24:30 AM »
Perhaps the following is an alternate workaround?  Just use very short listen Rx window (and a maximum Rx threshhold so that RSSI never triggers) and let the radio go back to sleep (which equals standby mode).  Then, if you want to turn off listen mode after having just been woken up by it, maybe use the radio's transition to sleep/standby as the flag to safely exit listen mode?  Sorry if my memory of the details has hazed over, but I thought I'd put that notion out there.

For a more radical approach, maybe you could try turning off the sequencer?  Then you should be in direct control and can order the radio into any state you want.  Afterward you could turn the sequencer on again.  Presumably the sequencer is what's handling listen mode, so perhaps its a conflict of some sort with the sequencer that hangs things.  Meh, it's an idea.  Not sure if it would work, but maybe it's worth a shot.
« Last Edit: December 30, 2016, 09:47:02 AM by WhiteHare »

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #8 on: December 30, 2016, 09:58:46 AM »
Sure wish Atmel would replace their WDT logic with whatever magic is happening in the AB18X5.  4,000nA vs. 18nA is not just a landslide, it is a solar-system-slide.

Would you be interested in a breakout board for the AB18X5?  I've lately had success in hand soldering the BQ25504, which is unrelated except I believe it's of similar size and has a similar land pattern.  Last time I checked there were no breakout boards for the AB18X5, but I've since learned how to do PCB's and could make a breakout sooner rather than later for the AB18X5 if others had interest too.  That might help expand the base of users so that there could be a decent library for it.
« Last Edit: December 30, 2016, 10:07:20 AM by WhiteHare »

ChemE

  • Sr. Member
  • ****
  • Posts: 417
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #9 on: December 30, 2016, 11:01:16 AM »
Would you be interested in a breakout board for the AB18X5?  I've lately had success in hand soldering the BQ25504, which is unrelated except I believe it's of similar size and has a similar land pattern.  Last time I checked there were no breakout boards for the AB18X5, but I've since learned how to do PCB's and could make a breakout sooner rather than later for the AB18X5 if others had interest too.  That might help expand the base of users so that there could be a decent library for it.

I just recently taught myself Eagle and one of the things I planned on fooling with was an AB18X5.  Yeah, if you or someone here could sell me a teeny board with an AB18X5 I would buy a few for sure.  I like to play with chips on a breadboard before I consider building them into larger projects.

EDIT: I believe the 1815 would be superior to the 1805 since it uses SPI instead of I2C.  Conversations can be faster and shorter via SPI I believe.  If we are going to do this, might as well shave every clock cycle we can!

EDIT2: Yeah SPI maxes as 2MHz while I2C is limited to 400kHz.  SPI uses more power to read and write but since both chips have to be awake to have the conversation this power increase is more than made up for by getting back to sleep faster.  Surely the 1815 is better suited to our needs than is the 1805.
« Last Edit: December 30, 2016, 11:20:15 AM by ChemE »

TomWS

  • Hero Member
  • *****
  • Posts: 1924
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #10 on: December 30, 2016, 11:08:16 AM »
I recommend using a stainless steel stencil for the AB18X5 and similar footprints.  The polyimide stencils don't have crisp enough edge walls for these small pads.  It also might help to use a T5 paste (depending on what else you have on the board).

Osh Stencils is now selling SS stencils for an affordable price.

Tom

ChemE

  • Sr. Member
  • ****
  • Posts: 417
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #11 on: December 30, 2016, 11:30:09 AM »
AB1815 Manual In past searches I've always found a cut down datasheet on Mouser which does not get into any useful specifics of registers and commands.  This seems to have all the details needed to write a library.
« Last Edit: December 30, 2016, 01:54:26 PM by ChemE »

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #12 on: December 30, 2016, 12:52:34 PM »
I could do one or the other of the "typical application circuits" on page 4 of the manual.    Is there any point to having a push button on it?  The upper example circuit has it, but not the lower one. 

As to the platform, I think it would make sense to have it be a Moteino, as that is something we all share.  I'd do it open source, so you could just send the files to OSH PARK or your favorite fab.  In addition, if Felix wants to make a buck off it by making and selling it, I'd have no objection.  With a chip this tiny, I think most people would rather have somebody else do the soldering anyway.  I know I would.   ;D

Unless someone recommends otherwise, I'd probably use the "Recommended tuning fork crystal is ABS07-120-32.768kHz-T."  Is that what others here are using?

The supercap is an interesting option, given how little current this thing uses.  Which one should it be?  Or should I just make it run off the Moteino voltage source and call it a day?


ChemE

  • Sr. Member
  • ****
  • Posts: 417
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #13 on: December 30, 2016, 02:11:27 PM »
How about the platform being a Moteino with no LDO and with a landing pad for a si7021?  And maybe change out the 69W for a CW to save some space/weight?

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #14 on: December 30, 2016, 03:30:35 PM »
Just a moment, Gonzales.  I meant this to be merely a shield for the Moteino to try out this exotic RTC, not as a replacement for the Moteino.  So, whether your Moteino has an LDO or not is really up to whether you remove it or not.  Likewise with your RFM69 choice.

The alternative, which I personally favor, is just a small breakout board that one can plug into a breadboard.  Then you can wire into it however you want with whatever platform you want.  It's really just a way to easily test drive the RTC chip.  ie. it's a learning tool.   So, I think that's what I'll make, as that way the PCB will cost less because it would be smaller than a Moteino shield, as well as taking much less time to design.

It will give some feel for whether interfacing with this RTC is a hassle or easy to do.  I'd much rather have a library for it before embarking on this.  There are some homemade libraries for the AB18x5 on github, but without a way to try them, it's hard to know if they're any good.
« Last Edit: December 30, 2016, 03:38:03 PM by WhiteHare »

ChemE

  • Sr. Member
  • ****
  • Posts: 417
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #15 on: December 30, 2016, 04:17:58 PM »
Just a moment, Gonzales.  I meant this to be merely a shield for the Moteino to try out this exotic RTC, not as a replacement for the Moteino.

My apologies.  I misunderstood what you meant when you said platform.

--Gonzales

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #16 on: December 30, 2016, 06:08:37 PM »
No worries.  Maybe Felix will do one like you said.  I guess time will tell.

For now, I think a quick and dirty test board will be a good way to see how easy/hard the AB18x5 is to solder to a PCB before investing too much time into it.  I know Joe and Tom seem comfortable with soldering it, but it looks challenging to me.  If I hadn't had some recent success soldering the BQ25504, I might have waited longer before attempting the AB18x5.

BTW, is Chip Quik a good brand for T5 paste?  http://www.digikey.com/en/product-highlight/c/chip-quik/type5-solder-paste
Or maybe I should just ask: what is a good brand for T5 paste?
« Last Edit: December 30, 2016, 06:17:59 PM by WhiteHare »

WhiteHare

  • Hero Member
  • *****
  • Posts: 1296
  • Country: us
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #17 on: January 08, 2017, 01:45:13 PM »
I started a new thread topic for the AB18x5 breakout board:  https://lowpowerlab.com/forum/projects/ab18x5-breakout-board/

Jim

  • Newbie
  • *
  • Posts: 9
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #18 on: May 03, 2020, 04:28:35 PM »
Hi with joy and excitment I read this thread.
Quote
Keeping fingers crossed that we'll actually end 2016 with a stable way to deal with listen mode.
I'm also looking for a RX solution, battery powered and a fast response time of approx 2ms.
Would you mind sharing your code how you tackled the issue with listen mode and a very low power consumption?  That would be awesome. Thanks a lot!!

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6206
  • Country: us
    • LowPowerLab
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #19 on: May 03, 2020, 08:56:03 PM »
Jim,
FYI joelucid has been inactive for the past few years.
The native ListenMode in RFM69 has been declared buggy and not adopted, although there are remnants of code in the forum as well as the RFM69 library.
I recommend just implementing your own "listen mode" on top of the regular sleep/transmit/ack stack. It's reliable and you have full control. You won't get a 2ms response back in any case, it takes at least that long to send a very fast packet  ;)
But just keeping the scheme simple will get you farther IMO.

TomWS

  • Hero Member
  • *****
  • Posts: 1924
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #20 on: May 04, 2020, 09:46:30 PM »
I'm also looking for a RX solution, battery powered and a fast response time of approx 2ms.
So, what do you mean "fast response time of approx 2ms"?   I don't know of a fully powered solution using RFM69 & ATMega328P that has this (depending on what you mean).

From my limited experience, using RFM69 & ATmega328P, you can easily get over two years of battery operation without too much difficulty - if you design your logic to deal with the realities of the HW & SW.  Or, you can try to squeeze more out of it and, if you never have a failure or any other devices within RF range, you might get more.  However, if you're within a 'noisy' RF environment (ie, there is more than one transmitting node within your network), your best bet is to structure your network to be 'node' driven - ie, the node determines when and if it exchanges data with the GW and, given this, design in whatever 'hooks' you might want to add for OTA SW updates, overriding commands, reconfiguration, etc. 

FWIW, Listen Mode on the RFM69 is good for one thing - providing a very reliable, lower power wakeup timer than the 328P WDT.  It is worse than useless for long term power savings due to the risks of hangups that drain your battery.   Once I learned this, I've switched to low power HW watchdog timers and use Listen Mode as a short term timer as needed. 


Jim

  • Newbie
  • *
  • Posts: 9
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #21 on: May 16, 2020, 08:57:29 AM »
Hi Felix and Tom,

>So, what do you mean "fast response time of approx 2ms"

I mean the time between a packet is send and the packet is received/decoded.
I want to build a camera flash transmitter/receiver. The commercial one's I measured has a response time of 1.8ms between pressing the button on the transmitter and receiving the signal and triggering the light on the other side. 2ms delay would be great, maximum would be around 5ms if it's technically possible

> you can easily get over two years of battery operation without too much difficulty

are you speaking of the RX part?? I couldn find any working code that demonstrate that unfortunately.

TomWS

  • Hero Member
  • *****
  • Posts: 1924
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #22 on: May 16, 2020, 05:13:50 PM »
>So, what do you mean "fast response time of approx 2ms"

I mean the time between a packet is send and the packet is received/decoded.
I want to build a camera flash transmitter/receiver. The commercial one's I measured has a response time of 1.8ms between pressing the button on the transmitter and receiving the signal and triggering the light on the other side. 2ms delay would be great, maximum would be around 5ms if it's technically possible
Unfortunately, I don't have any data for this kind of application, but, ISTM, if you can take a hybrid approach, you can use something like Listen Mode to 'wake' up your Camera shutter system, use OOK to trigger the shutter as much as you want, and then either command or program your device to go back to sleep if it's idle for some period of time.

Quote
> you can easily get over two years of battery operation without too much difficulty

are you speaking of the RX part?? I couldn find any working code that demonstrate that unfortunately.
No, other than the attempts to get Listen Mode working there are no LP (ie two year battery life) RX solutions that I know of.

It may be if your camera is in the 'wild' where there is very little RF noise, you might be able to get a reliable Listen Mode working.  However, even Listen Mode wake up would be a lot slower than your sub-2mS goal, but it would work with the hybrid approach.

Jim

  • Newbie
  • *
  • Posts: 9
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #23 on: May 17, 2020, 12:10:18 PM »
Hi Tom,

thanks for your idea, I think it's possible to use such a hybrid approach as long as the first delay is not that much maybe around 100-200ms and the final trigger can be quick so that if used more flashes all get triggered at the same time.
What would you suggest me how to start? Do you know of any reliable working code examples for listening mode of the rfm69 or rfm95?

Quote
From my limited experience, using RFM69 & ATmega328P, you can easily get over two years of battery operation without too much difficulty

That gives me a little hope again after all the different chips/modules I tried and bought

PS: I even bought the TPL5010 which you suggested in a thread here

TomWS

  • Hero Member
  • *****
  • Posts: 1924
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #24 on: May 18, 2020, 10:01:36 AM »
I think it's possible to use such a hybrid approach as long as the first delay is not that much maybe around 100-200ms and the final trigger can be quick so that if used more flashes all get triggered at the same time.
What would you suggest me how to start?
Look at the RFM69 library Listen Mode functions and see how the burst wake wakes up the node.  Since you want faster response times, you'll have to adjust the Listen Mode parameters to get a much shorter Listen Mode cycle time.  The current code was optimized for the lowest possible duty cycle timer (giving long term battery life), but the result is that the interval between RX operation could be almost 3 seconds - not good for your use case.  This was a long time ago and I haven't looked at it for a long long time.  Joe Lucid was the expert who came up with the method, but, as Felix has said, Joe is off doing other things now...
Quote
PS: I even bought the TPL5010 which you suggested in a thread here
LOL, that device probably won't help in this case, although it would be a good watchdog to catch the case where Listen Mode hangs.  I currently have a project where I'm on track to yield over a 1.5 year life on a ESP8266 running on a single CR123a.  That uses the TPL5110 timer.

Jim

  • Newbie
  • *
  • Posts: 9
Re: Stuck in listen mode - Listen Mode bug with workaround
« Reply #25 on: May 18, 2020, 03:00:02 PM »
Ok, I will try to understand that - I guess you mean the lib https://github.com/LowPowerLab/RFM69 ?
Is there an example included to start with? I'm soldering an adapter for my two RF69CW modules now

EDIT:
In the meanwhile I tried using the listenmode https://github.com/LowPowerLab/RFM69/tree/ListenMode but constantly failing to send a wake up message. My sdr shows activity but the receiver example doesn't wake up.
The power supply is stable and the units are working with an example sketch from the radiohead rf69 library. There I can send and receive an ack.

(Attachment shows the transmission when selecting "burst")
« Last Edit: May 20, 2020, 07:22:49 AM by Jim »