Gateway MoteinoMEGA freezes after 20 mins [SOLVED]

[K1JOS]

My project is complete and is working quite well (almost) except an odd event where one of my MegMoteino freezes after about 15-20 mins of what should be complete inactivity.   

My base station MegaMoteino has the following functions.

1) Main Loop function sits and waits for an Interrupt on INT1 generated by one of four momentary buttons all tied to INT1 by diodes (and ground on the other end) and then the ISR sets an interrupt flag and calls a debounce routine that reads for the button pin change.  The interrupt flag is cleared after the following routines are completed:

2) the button pressed equates to a specific 'command' (integer) and the base Mega turns associated LEDs on or turn off as needed.

3) following the button "command" selection, the base MegaMoteino updates via I2C an LCD listing the commands,etc. 

4) the button "command" is then sent via the standard RFM69 library routines to a remote MegaMoteino which returns an ACK after receive and the remote interprets the command which activates sensors, motor relays, etc.

5) The Remote Mega, in response to the received "command", sends a sensor reading to the base MegaMoteino who sends an ACK back

6) the base Mega updates the LCD with the sensor data and then waits in idle for another interrupt.

When no new button is sent, there are no interrupts, and the base station Mega does NOTHING but Loops waiting for an interrupt.   The remote will NOT sending anything back to the base Mega unless requested.  The Base Moteino does NOT check for any radio packets unless a command button was first pressed.  Once it receives sensor data from the remote, the base Mega is only in idle for the next interrupt.

It is during this idle time of about 15mins that the base Mega hangs and no longer responds to button presses.  In my preliminary discovery of this problem, I think this maybe is related somehow to a Moteino internal timer because if I press a button and send a new command every few minutes, the freeze occurs much later.

My base MegaMoteino sketch is fairly long and my naming notation is not the clearest.  I certainly don't mind posting the code but maybe something in the problem I described sounds familiar to someone??  Any advice in how to go about debugging this would be appreciated.

[Felix]

Could you perhaps Serial.print() something in the loop every X seconds or so to see at least how long it takes for the "freeze" to happen?

[K1JOS]

Hi Felix.  First congrats on being selected a semi-finalist in the Hackaday Prize !

My problem just got more strange.  I programmed one of the base station Mega buttons to have the Remote Mega continuously send back a sensor data packet (compass heading) every 300 msecs.  After 20 minutes, the base Mega buttons no responded HOWEVER the base Mega is otherwising running correctly -- it is receiving packets of sensor data correctly and updating the LCD and blinking a LED with every received packet... but the buttons don't generate an interrupot anymore !!

Does this make any sense to you (or anyone else)??

[Felix]

That's strange. So it's working except the buttons doesn't interrupt any more?
Which interrupt(s) are you using for the button(s)?

[K1JOS]

I am using pin 11 on the Mega (SER1).  My interrupt code is:

Code Select Expand

void setup();{
  digitalWrite(11,INPUT_PULLUP);
  attachInterrupt(1, debounceInterrupt, FALLING); // on mega INT1 is pin 11
  //more stuff 
}

...
...
...

void debounceInterrupt(){
  if(((long)(micros() - last_micros) >= debouncing_time * 1000)) {
    last_micros = micros();  // I reset last_micros =0 after the interrupt_flag is cleared to false in other subroutine 
    interrupt_flag = true;    //set flag but make sure to reset it later
    Serial.println("INTERRUPT");  // for debugging
   }  
}

I had it running overnight with a Serial.print("Step xx); Serial.println(millis()); at each functional step in my code. 

This morning:

1)  the LCD screen and LED blink were still receiving and updating data packets from the remote Mega
2)  the buttons would NOT generate an interrupt
3) on the serial monitor, all Serial.print statements seemed to be in the correct sequence as I scanned hundreds of them.  I looked carefully at the prints between 10 and 20 minutes timing and at the very end.  The serial monitor ran for 28656225 millis (= 7.9 hours) until the Arduino IDE Java heap memory overflowed at 7.9 hours. 

I know the buttons freeze around 10-20 minutes. 

I don't have any potential infinite loops.. no Do or While loops.  The only 'For' loops are for fixed integer incrementing such as reading or writing a data packet -- and those are working fine.

Is this bizarre?  I am now creating a detailed flowchart of the base Mega code (should have done this from start) to make sure I have every IF statement covered by a debugging serial print. 

My next steps are 1) switch to INT0 (pin 10) and see if that makes a difference.  If not then I will swap my two Mega Moteino's (only have two with RF69 radios) and see if the freezing is occurring with a specific MegaMote and maybe not code.

I was toying with an idea to program a third standard Arrduino to exercise the button pins with programmed pinchange states at intervals but not sure how that info might help me debug this further. 

Any other ideas? 

[K1JOS]

Not sure if it would make sense to remove the pullup and change the attach interrupt from "falling" to "rising" ??

[oric_dan]

The first thing I did when I received my Moteino-Megas was use the gateway and node sketches to send packets for long periods of time, and they worked fine for as long as I tested them - about 5-hours. I did modify the sketches to send constant-length 32-byte packets.

Besides the pull up issue, you should never use Serial.print() statements inside an ISR. This violates the basic 'rule' that ISRs should be short and efficient, and quickly returned from. Similarly, don't use a delay() in there. Eg, you could set a flag inside the ISR, and then use this flag in the main loop to print your msg.

[K1JOS]

Yes, if I program the example Gateway/Node into both Moteinos they both work continously fine but that example code doesnt exercise all the functions of the Amtel chips nor all the interrupts.  My serial print in the ISR was placed there after the problem came up just to see if the interrupt was being called randomly, eg noise on the tristate pin from something.  The freeze occurs with the serial.print removed from the interrupt.  I dont have any delays in the ISR.   All of my subroutines are working fine in the background on my Base Mega ... I2C communications with the LCD, RF packet exchanges between the two Moteino's (including ACK's), LEDs being turned on and off in correct sequence.... all fine EXCEPT no more interrupts on button press.

I changed the interrup from INT1 to INT0 (pin 10) and no improvment.  I will let it go thru another freeze cycle and check if the button pins are still high as they as supposed to be with the pullups.  Maybe something pulls them down so the interrupt has no Falling edge to trigger on.

[Felix]

Not sure if it would make sense to remove the pullup and change the attach interrupt from "falling" to "rising" ??

You can leave internal pullups enabled and use falling interrupts.
Like oric_dan mentioned, if the default sketch is working then I have no reason to worry there is any issue with the MEGAs, and BTW they were intensely tested in the prototype stage to make sure they are stable.

[K1JOS]

Felix and All,

I am not trying to imply that anything is wrong with the Mega.  I am looking for general advice how to debug what appears to be a very odd problem.  If this were two regular Arduino's hardwired I would go to the Arduino forum and ask but given the unique nature (and libraries) of the Moteino's I am hoping you guys can point me in the right direction.

[oric_dan]

1. normally, a debounce period should probably be on the order of 10-20 MILLI-seconds, or more. They are mechanical switches, so you could use millis() here.

2. also, the timing variables should be declared as (unsigned long) rather than (long), although this probably wouldn't matter here.

3. I assume your debouncing_time is declared a long too, and not an int. For my part I always make equivocal statements into unequivocal, ie .... ((unsigned long)debouncing_time * 1000UL), etc, through the use of casts and parentheses. Many people are bit in the butt by deferring to use of precedence tables, and doing it wrong.

4. for reference, the millis() value wraps around after 49.7 days, and micros() after 71.6 minutes, and using a long rather than unsigned long may cause trouble in half that time.
 
5. also, just to see, you might try using an external pullup-R, like 5-10K. The internal pullup is rather high value, so you might have a noise problem, especially if you have long wires going to your switches, although that would probably have the opposite effect to what you're seeing. For my part, I would probably put a small value cap on the interrupt pin.

6. in any case, you need to first bullet-proof ISRs as much as possible: short functionality, no print statement or delay() use, proper casting of variables, low-noise external cktry, and [also] use of 'volatile' type-qualifications if the variables are declared inside the ISR. Then, you have a firm basis for everything else.

There are a lot of things going on here.

[K1JOS]

Thanks for taking the time to offer this solid advice.

The micros() are used because the ISR only sets an interrupt flag that has to be picked up in the Loop and immediately checked for which button was pushed.  I think using millis() instead of micros() would missed the digitalRead on the 4 button pins.  Here is all the code around the interrupt:

Code Select Expand

//declarations
long debouncing_time = 200;   
volatile unsigned long last_micros;
boolean interrupt_flag =  false;

void setup(){
  digitalWrite(10,INPUT_PULLUP);  //INT0 is pin 10 on MegaMote
  attachInterrupt(0, debounceInterrupt, FALLING);
  digitalWrite(18,INPUT_PULLUP);  // these are the 4 buttons
  digitalWrite(19,INPUT_PULLUP);
  digitalWrite(20,INPUT_PULLUP);
  digitalWrite(21, INPUT_PULLUP);
}

void debounceInterrupt(){
  if(((long)(micros() - last_micros) >= debouncing_time * 1000)) {
    last_micros = micros();
    interrupt_flag = true;  
  }  
}

void Loop(){
if (interrupt_flag){    
    button1 = !digitalRead(18);  
    button2 = !digitalRead(19);
    button3 = !digitalRead(20);
    button4 = !digitalRead(21);
    ProcessButtons();  // this does all the work
    interrupt_flag = false;  //reset for next interrupt 
    last_micros = 0; // must reset for next interrupt to prevent overflow
 } 
}

The subroutine ProcessButtons () does all the work (checking for radiopackets, sending packets, updating LCD and LEDS) and this is obviously still working fine when the button interrupts stop working.  I have no infinite loops in ProcessButtons to cause a freeze and if it did freeze the LCD and LED updating would stop as well - which it doesnt.

You can also see that i was declaring my timing variables all as you had suggested (except using micros() instead of millis() as I explained).   Do you think the button digitalRead would occur fast enough if millis() were used in the debounce?  I am less concerned about double pushes and the freezing is occuring without any button pushing.

[oric_dan]

Normally, I would have had the 4 digitalRead() calls inside the ISR, and then process the results in the main loop. They go "fairly" fast, just a few usec I think, and should not bog down the ISR. But I would also rather use direct port/pin reads, rather than Arduino calls, since they're much faster. I believe all you need here, given the way the mote-mega pins are defined is:

inside ISR:
button = PORTC;

outside:
volatile int button;
#define BUTTON1    (button & 0x04)
#define BUTTON2    (button & 0x08)
....

Not completely portable, but very efficient.

Since interrupt_flag, etc, are declared external to the ISR, I'm not completely sure whether they need to volatile or not. ?

It's always difficult to troubleshoot these sorts of things, when you don't have the complete code, and are unable to spend several days working on it.

[Charly86]

K1JOS

I recommend you to disable the interrupt when using var that can be modified in isr AND in the loop

in loop the line

Code Select Expand

last_micros = 0;

is one line, but generated assembler code will be more because it's long var, 4 bytes to reset. So if interrupt is fired in the middle you'll have "unknown state" of your var and unpredictable results.

Interrupt are so powerfull  , but attention need to be at the same level to deal with them 

Would you mind test this way just to see ?

Code Select Expand

void Loop(){
// just a reading bit no problem on this one test
if (interrupt_flag){    
    button1 = !digitalRead(18);  
    button2 = !digitalRead(19);
    button3 = !digitalRead(20);
    button4 = !digitalRead(21);
    ProcessButtons();  // this does all the work
    cli();
    interrupt_flag = false;  //reset for next interrupt 
    last_micros = 0; // must reset for next interrupt to prevent overflow
    sei();
 } 
}

[oric_dan]

I recommend you to disable the interrupt when using var that can be modified in isr

Yes, good thinking. This may be "it".