Bad Moteino? R5 seems to "blink out" over time...

[G550_Pilot]

I have a bunch of the Moteinos. I have had fantastic luck with them and use them all over the place.

But I have two that just seem to "blink out" after awhile. They will work fine for several weeks and then for no apparent reason they just stop working and I have to power cycle them to get them back up and running. It is never at the same time, and it is just two (out of maybe 15 or so in use).

I am wondering what I can do to try and troubleshoot, or if this behavior is indicative of a failing unit and I should just save the time and replace them.

Looking for thoughts....

[Felix]

I would begin by asking what is different about those?
Wiring, power, code, environment, etc etc. Ie what could possibly influence those to perhaps fail.
Then there's the question of what kind of fail it truly is.

I would hook up a logic analyzer to the SPI bus to see what data is on it, if any.

This is kind of hit and miss but:
- you could use the LED for debugging - poor man's debug tool but simple blinks at strategic places in code could reveal a problem
- you could add some debug code to spit out some serial during normal cycling, and perhaps see where it last logs.

[G550_Pilot]

Thank Felix -

The two in question has been in operation in the exact same place with the exact same code for about 1.5 years now and are just now starting to show signs where they go offline. They are also in different physical locations. I think in the interest of time I will swap them out (just ordered three new R6s) and then I can troubleshoot these offline!

[Felix]

Its really hard to guesstimate when everything is about "the same" 
If some single component is somehow failing, it could cause the symptoms. Replacing a part is not hard, just finding the root problem.
Sometimes even the same part coming from different manufacturers can make a difference.

[G550_Pilot]

Understood. FOr the low cost of your wonderful Motes, I am just going to swap it out and see if the problem goes away :-)

[G550_Pilot]

Felex, et al -

Looking for some more help. I have a Moteino referenced in this thread that happened to be blinking out after some time and required me to unplug it (hence unpowering it) and plugging it back in to get it to work. I spent some time trying to troubleshoot the issue, but could not come up with any good ideas as to why it might be happening.

I replaced it with a brand new R6 and now after about a week or running ok, it just locked up again (for lack of a better description) requiring me again to unplug it and plug it back in to get it to work again.

I am at a loss as to what could be the problem. It is connected to power and to a smart water meter (reed switch) via a Max6816 for debouncing.

At this point, I am left with the thought that maybe I have something screwed up in the code and that is why the R6 "locks up" over time.

I have attached the sketch that I am using here, can someone give me some ideas as to where to look next for the cause of the problem?

Many Thanks

[perky]

Maybe it's going to sleep but the condition that is supposed to wake it is not happening. Before putting to sleep you should disable global interrupts, check all potential waking interrupt sources to determine if any are active, and if they are don't put to sleep but instead just re-enable global interrupts. Look very carefully around the sleeping code to convince yourself there is no small window where the waking interrupt could sneak in, and therefore be executed and cleared, just prior to executing the sleep instruction. If there is a small window it may take a very long time for it to happen, weeks, months even years depending on the window size and timing.

Mark.

[G550_Pilot]

Mark -

Thanks. I thought of that but I am using this same sketch on all three of my water meters and with the exception of changing the node number and some node comments it is identical which is what led me down the road of thinking I had a bad R5 since my other two R5 on my other meters seem to be working just fine going on over 1.5 years now.

But as you say, maybe it takes a long time for that bug to show up.

Since this Mote is powered 24 x 7 by street power, I am going to disable all power savings to see if that is the problem with this particular mote.

[G550_Pilot]

Well, I am still tackling this problem.

All three of my smart water meters run the exact same code with the exception of the node identifying information. I replaced what I thought was a bad R5 with an R6 and I am still getting the issue where the Mote goes offline and has to have the power recycled to it to get it to come back online.

After looking at everything I think maybe the Mote simply won't wake back up if it has been asleep for a given amount of time. I cannot think of any other reason this would continue to happen.

My two other water meters run daily as they run my irrigation and my home water. The pool water meter might not run for two or more weeks depending on the water level in the pool. So I think it goes to sleep and then never wakes up after a certain amount of sleep time. Putting one of my other motes running the other water meters ends up with the same issue. After a week or so the mote goes offline and requires a power reset. When I moving it back to the main or irrigation water meter the problem goes away.

I have removed all power saving options in the code as I am running the mote on dedicated power, so power savings is not necessary. That did not help, I still lose the mote after about a week or so (give or take).

I am looking for any other ideas to try and keep it awake during the long periods of time when the water meter is not in use.

Thanks

Code:
https://create.arduino.cc/editor/MD500_Pilot/40aa90d7-a429-427e-9303-fb4f3ead4a18/preview

[TomWS]

Given that the issue occurs at that one location I would try a power line filter, maybe even a surge suppressor on that outlet.  I've had some line powered motes that are prone to randomly reset and I suspect power glitches due to some heavy switching load on the same line/vicinity.

[G550_Pilot]

Thanks, Tom -

I actually did that already, although for another reason entirely.

Early on I was having issues with some relays and it was suggested that maybe I did not have clean power. So I installed a power conditioner on the AC circuit that feeds my entire project including a Pi3, wifi adaptor and three Moteinos (Pool filter pressure, temperatures, and the water meter). The Pi and Motes actually run off a battery backup that is charged via the AC adaptor so everything stays running if I lose power.

If it were something with my power, I would assume that it would equally affect all three Motes.

[TomWS]

Dang!  Another great idea bites the dust!

Well, if ALL that's been changed between those nodes is the node id, Occam's razor says, change the node ID on the node that's failing (or swap with one that's working).

Of course this is totally silly and can't possibly be the problem...  (I wish I had the time back for all the times I've said that)

[G550_Pilot]

And it was a great idea too, Tom. 

I have already swapped a "known good" mote for this one and when I do, this mote works perfectly and the "known good" fails after some random amount of time.

I think what I am going to try next to further troubleshoot is to set my code to run some water into the pool every day and see if having activity on the mote changes the behavior.

I am just running out of ideas....

[perky]

So replacing the Moteino with a known good Moteino from another system still fails, and the one that was failing then works in the other system? This would tend to eliminate the Moteino hardware, assuming you were using exactly the same code and the same external hardware.

Something to try, what network ID are you using? All zeros are forbidden in each byte of the address match word (and probably all 1's). Try swapping this value between the systems.

Mark.

[G550_Pilot]

Hi Perky -

The ONLY difference in the code is the nodeID, otherwise, the code on all three of my water meter motes is identical.

Here is the network information. Network ID is the same on all Motes (210):

//*********************************************************************************************
//************  Here is all the radio stuff, change to meet your configuration!! **************
//*********************************************************************************************
RFM69 radio;
#define RF_freq RF12_433MHZ               // Frequency of RF12B module can be RF12_433MHZ, RF12_868MHZ or RF12_915MHZ. You should use the one matching the module you have.
#define IS_HIGHPOWER   true               // True only for RFM69HW High Power RFM69
int nodeID = 28;                          // Node ID - should be unique on network - I use 30 for testing
const int networkGroup = 210;             // Network group                                                                           
//*********************************************************************************************