LowPowerLab Forum

Hardware support => Moteino => Topic started by: G550_Pilot on March 30, 2018, 01:12:47 PM

Title: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on March 30, 2018, 01:12:47 PM
I have a bunch of the Moteinos. I have had fantastic luck with them and use them all over the place.

But I have two that just seem to "blink out" after awhile. They will work fine for several weeks and then for no apparent reason they just stop working and I have to power cycle them to get them back up and running. It is never at the same time, and it is just two (out of maybe 15 or so in use).

I am wondering what I can do to try and troubleshoot, or if this behavior is indicative of a failing unit and I should just save the time and replace them.

Looking for thoughts....
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: Felix on March 30, 2018, 01:59:11 PM
I would begin by asking what is different about those?
Wiring, power, code, environment, etc etc. Ie what could possibly influence those to perhaps fail.
Then there's the question of what kind of fail it truly is.

I would hook up a logic analyzer to the SPI bus to see what data is on it, if any.

This is kind of hit and miss but:
- you could use the LED for debugging - poor man's debug tool but simple blinks at strategic places in code could reveal a problem
- you could add some debug code to spit out some serial during normal cycling, and perhaps see where it last logs.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on March 30, 2018, 02:05:44 PM
Thank Felix -

The two in question has been in operation in the exact same place with the exact same code for about 1.5 years now and are just now starting to show signs where they go offline. They are also in different physical locations. I think in the interest of time I will swap them out (just ordered three new R6s) and then I can troubleshoot these offline!

Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: Felix on March 31, 2018, 11:04:17 AM
Its really hard to guesstimate when everything is about "the same"  :-\
If some single component is somehow failing, it could cause the symptoms. Replacing a part is not hard, just finding the root problem.
Sometimes even the same part coming from different manufacturers can make a difference.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on April 01, 2018, 06:21:35 PM
Understood. FOr the low cost of your wonderful Motes, I am just going to swap it out and see if the problem goes away :-)
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on April 18, 2018, 05:32:36 PM
Felex, et al -

Looking for some more help. I have a Moteino referenced in this thread that happened to be blinking out after some time and required me to unplug it (hence unpowering it) and plugging it back in to get it to work. I spent some time trying to troubleshoot the issue, but could not come up with any good ideas as to why it might be happening.

I replaced it with a brand new R6 and now after about a week or running ok, it just locked up again (for lack of a better description) requiring me again to unplug it and plug it back in to get it to work again.

I am at a loss as to what could be the problem. It is connected to power and to a smart water meter (reed switch) via a Max6816 for debouncing.

At this point, I am left with the thought that maybe I have something screwed up in the code and that is why the R6 "locks up" over time.

I have attached the sketch that I am using here, can someone give me some ideas as to where to look next for the cause of the problem?

Many Thanks

Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: perky on April 18, 2018, 08:31:32 PM
Maybe it's going to sleep but the condition that is supposed to wake it is not happening. Before putting to sleep you should disable global interrupts, check all potential waking interrupt sources to determine if any are active, and if they are don't put to sleep but instead just re-enable global interrupts. Look very carefully around the sleeping code to convince yourself there is no small window where the waking interrupt could sneak in, and therefore be executed and cleared, just prior to executing the sleep instruction. If there is a small window it may take a very long time for it to happen, weeks, months even years depending on the window size and timing.

Mark.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on April 18, 2018, 08:52:01 PM
Mark -

Thanks. I thought of that but I am using this same sketch on all three of my water meters and with the exception of changing the node number and some node comments it is identical which is what led me down the road of thinking I had a bad R5 since my other two R5 on my other meters seem to be working just fine going on over 1.5 years now.

But as you say, maybe it takes a long time for that bug to show up.

Since this Mote is powered 24 x 7 by street power, I am going to disable all power savings to see if that is the problem with this particular mote.

Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 02, 2018, 04:22:05 PM
Well, I am still tackling this problem.

All three of my smart water meters run the exact same code with the exception of the node identifying information. I replaced what I thought was a bad R5 with an R6 and I am still getting the issue where the Mote goes offline and has to have the power recycled to it to get it to come back online.

After looking at everything I think maybe the Mote simply won't wake back up if it has been asleep for a given amount of time. I cannot think of any other reason this would continue to happen.

My two other water meters run daily as they run my irrigation and my home water. The pool water meter might not run for two or more weeks depending on the water level in the pool. So I think it goes to sleep and then never wakes up after a certain amount of sleep time. Putting one of my other motes running the other water meters ends up with the same issue. After a week or so the mote goes offline and requires a power reset. When I moving it back to the main or irrigation water meter the problem goes away.

I have removed all power saving options in the code as I am running the mote on dedicated power, so power savings is not necessary. That did not help, I still lose the mote after about a week or so (give or take).

I am looking for any other ideas to try and keep it awake during the long periods of time when the water meter is not in use.

Thanks

Code:
https://create.arduino.cc/editor/MD500_Pilot/40aa90d7-a429-427e-9303-fb4f3ead4a18/preview
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: TomWS on October 02, 2018, 06:19:21 PM
Given that the issue occurs at that one location I would try a power line filter, maybe even a surge suppressor on that outlet.  I've had some line powered motes that are prone to randomly reset and I suspect power glitches due to some heavy switching load on the same line/vicinity.

Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 02, 2018, 06:30:08 PM
Thanks, Tom -

I actually did that already, although for another reason entirely.

Early on I was having issues with some relays and it was suggested that maybe I did not have clean power. So I installed a power conditioner on the AC circuit that feeds my entire project including a Pi3, wifi adaptor and three Moteinos (Pool filter pressure, temperatures, and the water meter). The Pi and Motes actually run off a battery backup that is charged via the AC adaptor so everything stays running if I lose power.

If it were something with my power, I would assume that it would equally affect all three Motes.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: TomWS on October 02, 2018, 06:49:33 PM
Dang!  Another great idea bites the dust!

Well, if ALL that's been changed between those nodes is the node id, Occam's razor says, change the node ID on the node that's failing (or swap with one that's working).

Of course this is totally silly and can't possibly be the problem...  (I wish I had the time back for all the times I've said that)

Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 02, 2018, 08:38:00 PM
And it was a great idea too, Tom.  :P

I have already swapped a "known good" mote for this one and when I do, this mote works perfectly and the "known good" fails after some random amount of time.

I think what I am going to try next to further troubleshoot is to set my code to run some water into the pool every day and see if having activity on the mote changes the behavior.

I am just running out of ideas....
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: perky on October 02, 2018, 09:25:27 PM
So replacing the Moteino with a known good Moteino from another system still fails, and the one that was failing then works in the other system? This would tend to eliminate the Moteino hardware, assuming you were using exactly the same code and the same external hardware.

Something to try, what network ID are you using? All zeros are forbidden in each byte of the address match word (and probably all 1's). Try swapping this value between the systems.

Mark.

Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 02, 2018, 09:42:33 PM
Hi Perky -

The ONLY difference in the code is the nodeID, otherwise, the code on all three of my water meter motes is identical.

Here is the network information. Network ID is the same on all Motes (210):

//*********************************************************************************************
//************  Here is all the radio stuff, change to meet your configuration!! **************
//*********************************************************************************************
RFM69 radio;
#define RF_freq RF12_433MHZ               // Frequency of RF12B module can be RF12_433MHZ, RF12_868MHZ or RF12_915MHZ. You should use the one matching the module you have.
#define IS_HIGHPOWER   true               // True only for RFM69HW High Power RFM69
int nodeID = 28;                          // Node ID - should be unique on network - I use 30 for testing
const int networkGroup = 210;             // Network group                                                                           
//*********************************************************************************************
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: damonb on October 02, 2018, 11:25:45 PM
Have you looked at how much free RAM is available to your code? I had similar symptoms once that turned out to be insufficient stack + heap space. It only failed under a very rare combination of conditions... very unpredictable and difficult to replicate.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 03, 2018, 11:25:43 AM
Damon -

Good idea but we are not even close to the memory limit, as in by a factor of 10 or 20!
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: Felix on October 03, 2018, 11:32:44 AM
The way they fail indicate it's probably something in code, or maybe maybe as others suggested some power line/transient/noise issue.
If all else fails, save some time by implementing the 328's watchdog to reset the MCU.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 03, 2018, 01:05:24 PM
Really???  :)

I never knew the 328 had a watchdog timer, I would have tried that a LONG time ago...

Always learning something new here....thanks Felix...
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: TomWS on October 03, 2018, 04:42:39 PM
Quote from: G550_Pilot on October 02, 2018, 09:42:33 PM
RFM69 radio;
#define RF_freq RF12_433MHZ               // Frequency of RF12B module can be RF12_433MHZ, RF12_868MHZ or RF12_915MHZ. You should use the one matching the module you have.
Is the RF12_433MHz code point the same as RF69_433MHZ code point?

What else is baggage from RF12?
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 03, 2018, 08:25:47 PM
Not Sure Tom -

I'm using the RF69 libraries so I assume there is no RF12 code or libraries involved:

#define RF69_COMPAT 1                                                 // Set to 1 if using RFM69CW or 0 is using RFM12B
#include <RFM69.h>         //get it here: http://github.com/lowpowerlab/rfm69
RFM69 radio;
#define RF_freq RF12_433MHZ               // Frequency of RF12B module can be RF12_433MHZ, RF12_868MHZ or RF12_915MHZ. You should use the one matching the module you have.
#define IS_HIGHPOWER   true               // True only for RFM69HW High Power RFM69

Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: perky on October 04, 2018, 09:04:41 PM
Quote from: Felix on October 03, 2018, 11:32:44 AM
If all else fails, save some time by implementing the 328's watchdog to reset the MCU.

Is this timer not already being used for sleeping? Are you suggesting using the reset instead of the interrupt?

@G550_Pilot: Could we see the code that puts it to sleep (before and after) and what conditions you are using to wake it up? BTW Tom's pointed out you use the #define RF12_433MHz, which means an RFM12B header file is included somewhere for it to compile. How much more code from that library has been used was his question.

Mark.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 05, 2018, 12:18:42 PM
Mark -

Here is the code:
https://create.arduino.cc/editor/MD500_Pilot/40aa90d7-a429-427e-9303-fb4f3ead4a18/preview (https://create.arduino.cc/editor/MD500_Pilot/40aa90d7-a429-427e-9303-fb4f3ead4a18/preview)

The only #include is for the RFM69.h library which I assume is referencing the RF12. There is no other header included:

//*********************************************************************************************
//***********  Include these libraries:
//*********************************************************************************************
#include <RFM69.h>         //get it here: http://github.com/lowpowerlab/rfm69
#include <SPIFlash.h>      //get it here: http://github.com/lowpowerlab/spiflash
#include <WirelessHEX69.h> //get it here: https://github.com/LowPowerLab/WirelessProgramming
#include <EEPROM.h>
#include <SPI.h>
#include <TimerOne.h>
#include <JeeLib.h>
#include <avr/power.h>
#include <avr/sleep.h>



I think there is some confusion. I NEVER put this mote to sleep as it is powered 24x7 via a wall wart. I have commented out all of the sleep stuff because it was suggested early on that it was something to do with putting it to sleep and it not waking up. So to eliminate that possibility, I never put it to sleep, ever.



Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: perky on October 08, 2018, 08:42:31 PM
OK, you did mention it was like it had gone to sleep and not woken up again so I assumed you were sleeping it. Anyway, why have you got JeeLib.h in there? That includes RF12.h so that's in the code too. It looks like you have a mix of two libraries.

Mark.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 09, 2018, 02:42:54 PM
Hi Mark -

Yes, you are correct. But since I decided to power this full time I removed all of the sleeping and power savings from the code. This eliminated the "going to sleep and not waking up" issue.

On the JeeLib.h I will have to check and see. All of the high power Motes that I am running using this same nomenclature for the radio, I think maybe it had something to do with the HighPower settings being needed and hence the JeeLib.h? Honestly, once I got the radios running under high power a couple of years ago I never messed with the radio part of it again.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 16, 2018, 04:11:04 PM
Quote from: G550_Pilot on October 03, 2018, 01:05:24 PM
Really???  :)

I never knew the 328 had a watchdog timer, I would have tried that a LONG time ago...

Always learning something new here....thanks Felix...

So this was a fantastic idea Felix, right up until the time I realized that I had no idea how to implement it since I am not sleeping AND I am using an interrupt driven event as opposed to a loop.

As I understand the WDT, you set it for a timeout of some sort (say 8 seconds), then you do something in your code and you reset your timeout before the 8 seconds are up and this resets the WDT. But when you have NO loop and it could be a week or so before your interrupt driven event kicks off, how do you make this work?

My first thought would be to simply throw a delay(100) followed by WDT reset in my loop statement (which is currently empty) so that every 100ms (or whatever) the reset happens, but will this screw up my water meter readings when the water meter is running?






Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: perky on October 17, 2018, 11:37:47 AM
Quote from: G550_Pilot on October 16, 2018, 04:11:04 PM
My first thought would be to simply throw a delay(100) followed by WDT reset in my loop statement (which is currently empty) so that every 100ms (or whatever) the reset happens, but will this screw up my water meter readings when the water meter is running?
This isn't really how interrupt driven code should work. There should be minimal processing in ISR, by setting flags or doing quick non-blocking code only, and the main loop should decide what to do based on the flags set.

The general principle for an ISR should be 'get in and get out as quickly as possible', and main loop code written, with state machines if needed, to execute the loop as quickly as possible. The main loop then resets the watchdog somewhere in the loop.

So I wouldn't transmit anything in the ISR personally, or do any sensor measurements. Of course these are general rules and simple systems might get away with it, but bear in mind global interrupts are disabled in ISRs, so things like timers freeze and the millisecond timer counter won't run, and the retry mechanism may depend on it.

Mark
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 17, 2018, 12:02:32 PM
Mark -

Not sure I understand what you are saying. I am using the sketch that Felix wrote for the WaterMote, and all of the sensor measurements are done via an ISR, the radio work is done via a timer. This all works fine on two of the three water motes I run, I just need to figure out where to put the watchdog stuff on the third one that keeps hanging on me. I do not have a main loop in this code, it is completely empty.


attachInterrupt(INTERRUPTPIN, pulseCounterInterrupt, RISING);
Timer1.initialize(XMITPERIOD * 1000L);
Timer1.attachInterrupt(XMIT);
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: Felix on October 17, 2018, 12:11:24 PM
Reset the WDT in your main loop somewhere, as long as you dont hit the 8second mark (or whatever expiration you set) it will be fine and your node won't reset.
You can reset the WDT timer as often as you want, just dont let it expire, that's the whole idea.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 17, 2018, 12:19:33 PM
Felix -

There is no main loop in your WaterMote. It is completely empty. (unless I am misunderstanding something). The loop() is totally empty.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: Felix on October 17, 2018, 12:27:20 PM
Don't you use this sketch?
https://github.com/LowPowerLab/RFM69/blob/master/Examples/PulseMeter/PulseMeter.ino
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 17, 2018, 12:43:21 PM
Nope - this one:

https://github.com/LowPowerLab/WaterMote/blob/master/WaterMote.ino
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: Felix on October 17, 2018, 12:46:20 PM
Ok, that's an older one, up to you but the loop still runs, just does nothing when it's empty.
So you could add your WDT resetting code there.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 17, 2018, 12:55:25 PM
Felix -

Thank You. I realize that I can put something in the loop() since it is empty, but I guess I don't understand fully the process. If the water meter is running and it is an interrupt based measurement of the water meter, does loop still run or will it wait until the water meter quits flowing?

If I simply threw in a WDT_reset(); in the loop would that work or would that be resetting way to fast? If I threw in a delay (say of 5 seconds with the WDT set at 8 seconds) would that delay cause a problem while the water meter is flowing?

And lastly, should I convert to the PulseMeter as opposed to continuing to use the WaterMote?

Thanks
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: Felix on October 17, 2018, 01:29:09 PM
The waterMote sketch was coded back when I used RFM12b, I since switched to RFM69/pulsemeter. If you have it adjusted and it works, I would not bother.
The loop is empty in watermote because it just sits idle when nothing happens. You can add code there without a problem.
Interrupts interrupt normal code flow, and resume normal code when they are done.
Your loop code will be halted when an interrupt occurs.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: perky on October 17, 2018, 01:33:40 PM
Quote from: Felix on October 17, 2018, 12:27:20 PM
Don't you use this sketch?
https://github.com/LowPowerLab/RFM69/blob/master/Examples/PulseMeter/PulseMeter.ino

It looks like you're transmitting in an ISR, which will be with interrupts disabled, and yet sendWithRetry() relies on millis() function which is interrupt driven and won't increment while in the ISR. Have I misunderstood?

Edit: I didn't spot the re-enabling interrupts in XMIT(), so looks OK.

Mark
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 17, 2018, 01:42:31 PM
Quote from: Felix on October 17, 2018, 01:29:09 PM
Your loop code will be halted when an interrupt occurs.

OK so my question is when my water is running and the loop code is halted, won't that by default force the watchdog to reset and reboot the Mote since the loop() is where I will be timing and resetting the WDT??
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 17, 2018, 01:46:19 PM
Quote from: perky on October 17, 2018, 01:33:40 PM
It looks like you're transmitting in an ISR, which will be with interrupts disabled, and yet sendWithRetry() relies on millis() function which is interrupt driven and won't increment while in the ISR. Have I misunderstood?

Mark

I am not using sendWithRetry():


//*********************************************************************************************
//*********** We only transmit if there are changes to send...helps to save power....
//*********************************************************************************************

  if (water.GPM!=GPMlast || water.GAL!=GALlast || water.GLM!=GLMlast)  //Check and see if we used any water.....
  {
    if (debug)
       {
       Serial.println("Time to transmit.......");
       }
   
    radio.send(0, &water, sizeof water);
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: perky on October 17, 2018, 01:58:09 PM
Quote from: G550_Pilot on October 17, 2018, 01:42:31 PM
OK so my question is when my water is running and the loop code is halted, won't that by default force the watchdog to reset and reboot the Mote since the loop() is where I will be timing and resetting the WDT??
The idea is the watchdog timeout is much longer that the time spent in the ISR, so will be continually reset unless a lock-up has happened.

But I think the solution is not a watchdog reset, but to move the XMIT() function which is currently the ISR into the main loop. Use a volatile flag set in the ISR and use that to trigger the transmit routine, clearing the flag when it's done. If I'm right, the sendWithRetry() function might freeze because millis() is frozen in the ISR (assuming the millis() function here is the Arduino interrupt driven one).

Edit: I saw your reply, even the send() routine uses millis(). If the canSend() routine get's stuck (and this has happened before) then there's no timeout and it'll get stuck there. I would move all your transmit code to the main loop as above. I assume this routine is attached to the timer interrupt like Felix's code.

Mark.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 17, 2018, 02:53:24 PM
Mark -

I think I am closer to understanding what you are saying.

I was assuming that since all of the code Felix wrote does, in fact, work the way it is written that having the XMIT function where it is currently located is was not actually a problem. Is that because of the interrupts() here in the XMIT function that re-enables the interrupts so it is not an issue with millis()?:


void XMIT()
{
  noInterrupts();                                   
  PulseCounter = PulseCounterVolatile;
  interrupts();                                            <--------------------
 
  if (millis() - TIMESTAMP_pulse_curr >= 5000)
  {
    ledState = !ledState;
    digitalWrite(LED, ledState);
  }


I appreciate the time you are investing in trying to help me understand. I am working on understanding how the interrupt/noInterrupts works in relation to attachInterrupt. From what I am gathering from you, the transmit function should not work at all if it is in an ISR or at least it's not a good idea to place it there, is that correct?

If that is correct, does the inclusion of the noInterrupts/interrupts functions in the XMIT function alleviate the issues caused by using it there?


On to your recommendation of using a volatile flag/variable. I assume that in my code where the actual transmit happens:


if (water.GPM!=GPMlast || water.GAL!=GALlast || water.GLM!=GLMlast)  //Check and see if we used any water.....
  {
    if (debug)
       {
       Serial.println("Time to transmit.......");
       }
   
    radio.send(0, &water, sizeof water);


I would instead set a variable there indicating that we should transmit and then create a function in my loop that looks at that variable and transmits the information if the variable has been set, then reset the variable (and the WDT). Is that correct?

I guess thanks to my ignorance, I was assuming that as long as my water was running and hence the interrupt driven part of the code was in control, everything stopped until the water stopped, it just stays right there and does not execute anything else (like the loop()) until it is done. Since I think that is not correct, how often (in general) does the loop() run while the water is flowing? Is it once per pulse of the water meter, once per ms or ???



Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: ChemE on October 17, 2018, 04:38:20 PM
An empty loop is just...

while(1) {
}

...which takes 2 clock cycles to branch back to the top.  So your empty loop is executing 8,000,000 times per second.  Crazy waste of energy right?
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 17, 2018, 05:20:08 PM
Quote from: ChemE on October 17, 2018, 04:38:20 PM
An empty loop is just...

while(1) {
}

...which takes 2 clock cycles to branch back to the top.  So your empty loop is executing 8,000,000 times per second.  Crazy waste of energy right?

WOW....I did not realize how many fast it was actually running....
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: perky on October 17, 2018, 09:11:08 PM
@G550_Pilot

Ah, yes. I didn't spot that re-enabling of interrupts in the XMIT() ISR. That would appear to update millis().

As a general programming principle re-enabling interrupts within an ISR can be dangerous, it could be interrupted itself by another interrupt and in cases where there are multiple interrupts can cause stack overflow or retriggering of the same interrupt ISR with non re-entrant code. This could be problematical if the timeout of the send() function is greater than the frequency period of calling XMIT(), but it appears to be OK as XMIT() is called every 5 seconds and the tx timeout is only 1 second.

I don't personally like doing this though, I think it's better to have a static volatile flag that simply gets set by the 5 second timer interrupt instead of running XMIT(), and in the main loop look for the flag being set. When you see it, clear the flag and run XMIT() as a normal function. This minimises the time in the ISR where interrupts are disabled by hardware to just the flag setting part, and millis() will update as it should.

Mark
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: ChemE on October 18, 2018, 08:30:57 AM
As Perky said, it is best practice to keep ISRs as short as humanly possibly.  I try to keep mine to fewer than 100 clock cycles just to stretch myself; most people probably aren't as crazy about efficiency as me.  Sending a packet even at 300kbps and using my uber fast radio code still takes 1ms which is 16,000 clock cycles.  I believe the RFM69 library needs more like 4+ ms.
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 18, 2018, 11:25:13 AM
Quote from: ChemE on October 18, 2018, 08:30:57 AM
As Perky said, it is best practice to keep ISRs as short as humanly possibly.  I try to keep mine to fewer than 100 clock cycles just to stretch myself; most people probably aren't as crazy about efficiency as me.  Sending a packet even at 300kbps and using my uber fast radio code still takes 1ms which is 16,000 clock cycles.  I believe the RFM69 library needs more like 4+ ms.

Are you using a different radio library to get those transmit times down?
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 18, 2018, 11:26:30 AM
Quote from: perky on October 17, 2018, 09:11:08 PM
I don't personally like doing this though, I think it's better to have a static volatile flag that simply gets set by the 1 second timer interrupt instead of running XMIT(), and in the main loop look for the flag being set. When you see it, clear the flag and run XMIT() as a normal function. This minimises the time in the ISR where interrupts are disabled by hardware to just the flag setting part, and millis() will update as it should.

Mark - Thanks for the good info, I will try and poke around altering the code to see if I can make this work!

- Richard
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: ChemE on October 18, 2018, 01:15:30 PM
Quote from: G550_Pilot on October 18, 2018, 11:25:13 AM
Are you using a different radio library to get those transmit times down?

Yes though not a polished or supported one.  It is my own homebrew code that I've developed to interact with the radio. 

Save this in the project as RFM69CW.h

#include "Arduino.h"
#define         OPT_FLAG                                //__attribute__((always_inline))  // Comment this definition out to optimize for size
// ============================== User Definitions ==============================
#define         NETWORKID                               100    //the same on all nodes that talk to each other - 170 is 10101010 DC free value
#define         RECEIVER                                1      // ID of the gateway that all nodes report back to
#define         NODEID                                  2      // ID of this Node
#define SS_PIN                 PB2    // Slave select pin

// ============================== SPI Definitions ==============================
#define SELECT                 noInterrupts(); PORTB &= ~(1<<SS_PIN)
#define UNSELECT                 SS_WRITE_HIGH; interrupts()
#define    SS_WRITE_HIGH                      PORTB |= 1<<SS_PIN
#define         WAIT_WHILE_SPI_BUSY                     asm volatile("nop"); while (!(SPSR & 1<<SPIF))

// ============================== RFM69CW Definitions ==============================
#define         SLEEP_MODE                              B00000000
#define         AUTO_TRANSMITTER                        B01011011    // Enter = FIFO level; Exit = Packet Sent; Intermediate Mode = TX
#define         CHANGE_OP_MODE(mode)                    writeReg(REG_OPMODE, mode)
#define         SET_POWER_LEVEL(level)                  writeReg(REG_PALEVEL, 0x80 | (level & 0x0F));  // Only works for the RFM69CW not the RFM69HCW
#define         REG_FIFO                                0x00
#define         REG_OPMODE                              0x01
#define         REG_BITRATEMSB                          0x03
#define         REG_BITRATELSB                          0x04
#define         REG_FDEVMSB                             0x05
#define         REG_FDEVLSB                             0x06
#define         REG_PALEVEL                             0x11
#define         REG_RXBW                                0x19
#define         REG_RSSITHRESH                          0x29
#define         REG_SYNCCONFIG                          0x2E
#define         REG_SYNCVALUE1                          0x2F
#define         REG_SYNCVALUE2                          0x30
#define         REG_PACKETCONFIG1                       0x37
#define         REG_AUTOMODES                           0x3B
#define         REG_PACKETCONFIG2                       0x3D
#define         RF_BITRATEMSB_300000                    0x00    // Begin 300 kbps auto Tx settings
#define         RF_BITRATELSB_300000                    0x6B
#define         RF_FDEVMSB_300000                       0x13
#define         RF_FDEVLSB_300000                       0x33
#define         RF_RXBW_DCCFREQ_111                     0xE0
#define         RF_RXBW_MANT_16                         0x00
#define         RF_RXBW_EXP_0                           0x00
#define         RF_SYNC_ON                              0x80
#define         RF_SYNC_FIFOFILL_AUTO                   0x00
#define         RF_SYNC_SIZE_2                          0x08
#define         RF_SYNC_TOL_0                           0x00
#define         RF_PACKET1_FORMAT_VARIABLE              0x80
#define         RF_PACKET1_DCFREE_OFF                   0x00
#define         RF_PACKET1_CRC_OFF                      0x00
#define         RF_PACKET1_CRCAUTOCLEAR_OFF             0x08
#define         RF_PACKET1_ADRSFILTERING_OFF            0x00
#define         RF_PACKET2_RXRESTARTDELAY_2BITS         0x10
#define         RF_PACKET2_AUTORXRESTART_ON             0x02
#define         RF_PACKET2_AES_OFF                      0x00    // End 300 kbps auto Tx settings

static inline void SPI_INIT(void) {  // Level 0 code - initialize the SPI bus at fosc/2 (8MHz)
  PORTB |= 1<<SS_PIN;
  DDRB |= _BV(SS_PIN);
  SPCR |= _BV(MSTR) | _BV(SPE);
  SPSR |= (1<<SPI2X);    // Set the SPI bus speed to Fosc/2 = 8MHz at full speed
 
  // No clue why this is neccessary as opposed to DDRB |= 1<<SCK | 1<<MOSI
  volatile uint8_t *reg;
  reg = &DDRB;
  *reg |= 0x28;  // Bit mask of SCK and MOSI
  //DDRB = 0x28;
}

OPT_FLAG uint8_t SPI_XFER(uint8_t data) {    // Level 0 code - move data over the SPI bus
  SPDR = data;
  WAIT_WHILE_SPI_BUSY;
  return SPDR;
}

OPT_FLAG uint8_t readReg(uint8_t addr) {    // Level 1 code - interact with the radio's registers
  SELECT;
  SPDR = ( addr & 0x7F );
  WAIT_WHILE_SPI_BUSY;
  SPDR = ( 0 );
  WAIT_WHILE_SPI_BUSY;
  UNSELECT;
  return SPDR;
}

OPT_FLAG void writeReg(uint8_t addr, uint8_t value) {  // Level 1 code - interact with the radio's registers
  SELECT;
  SPDR = ( addr | 0x80 );
  WAIT_WHILE_SPI_BUSY;
  SPDR = ( value );
  WAIT_WHILE_SPI_BUSY;
  UNSELECT;
}

static inline void SendFrame(uint8_t toAddress, const void* buffer, uint8_t bufferSize) {  // Level 2 code - do useful work
  SELECT;
  SPI_XFER(REG_FIFO | 0x80);   // write to FIFO using SPI burst mode
  SPI_XFER(bufferSize + 3);    // LEN byte
  SPI_XFER(toAddress);         // 1st byte
  SPI_XFER(NODEID);            // 2nd byte
  SPI_XFER(0x00);              // 3rd byte
  for (uint8_t i = 0; i < bufferSize; i++) SPI_XFER(((uint8_t*) buffer)[i]);  // Write 6 more bytes to the FIFO
  UNSELECT;
}

static inline void RadioInit(void) {
    SPI_INIT();
    CHANGE_OP_MODE(SLEEP_MODE);  // Put the radio to sleep ASAP to save power
    writeReg( REG_BITRATEMSB, RF_BITRATEMSB_300000 ); // 0x03
    writeReg( REG_BITRATELSB, RF_BITRATELSB_300000 ); // 0x04
    writeReg( REG_FDEVMSB, RF_FDEVMSB_300000 ); // 0x05
    writeReg( REG_FDEVLSB, RF_FDEVLSB_300000 ); // 0x06
    writeReg( REG_RXBW, RF_RXBW_DCCFREQ_111 | RF_RXBW_MANT_16 | RF_RXBW_EXP_0 ); // 0x19
    writeReg( REG_RSSITHRESH, 220 ); // 0x29
    writeReg( REG_SYNCCONFIG, RF_SYNC_ON | RF_SYNC_FIFOFILL_AUTO | RF_SYNC_SIZE_2 | RF_SYNC_TOL_0 ); // 0x2E - 2 sync bytes
    writeReg( REG_SYNCVALUE1, 0xAA ); // 0x2F
    writeReg( REG_SYNCVALUE2, NETWORKID ); // 0x30
    writeReg( REG_PACKETCONFIG1, RF_PACKET1_FORMAT_VARIABLE | RF_PACKET1_DCFREE_OFF | RF_PACKET1_CRC_OFF | RF_PACKET1_CRCAUTOCLEAR_OFF | RF_PACKET1_ADRSFILTERING_OFF );  // 0x37 // 0x37
    writeReg( REG_AUTOMODES, AUTO_TRANSMITTER );  // 0x3B - Put the radio in automatic mode
    writeReg( REG_PACKETCONFIG2, RF_PACKET2_RXRESTARTDELAY_2BITS | RF_PACKET2_AUTORXRESTART_ON | RF_PACKET2_AES_OFF ); // 0x3D
    SET_POWER_LEVEL(0);
}


And then call it like this

uint8_t data[8];  //16-bit temp, 16-bit RH, 16-bit Vcc, 16-bit packet counter
RadioInit();  // Configure the radio while it is asleep
...populate the bytes of data...
SendFrame(RECEIVER, data, 8);                   // Send the data


Further reading on my quest to speed up the code can be found here: https://lowpowerlab.com/forum/low-power-techniques/speeding-up-the-rfm69-library/
Title: Re: Bad Moteino? R5 seems to "blink out" over time...
Post by: G550_Pilot on October 18, 2018, 01:49:00 PM
Awesome! Thanks