Author Topic: Moteino WeatherNode won't wake up  (Read 5424 times)

Tony Weil

  • Jr. Member
  • **
  • Posts: 60
  • Country: us
Moteino WeatherNode won't wake up
« on: August 21, 2017, 08:45:48 AM »
I have a Moteino that goes through many sleep/wake cycles and then after a few days never wakes up and needs to be restarted by powering on and off. The battery is always still at least 4.0V

It is a Moteino R6 w/ flash, connected to a Weathershield, and a fully charged 2000mah Lipo battery connected to the Vin terminals with a JST jack. It is running the standard WeatherNode sketch in Github with LowPower.powerDown(SLEEP_8S, ADC_OFF, BOD_OFF) and sleeps 15 times between measurements.

I have determined that it is not waking up by adding a blinking the D9 LED after each wakeup, so that it blinks every 8 Sec.

Thanks.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Moteino WeatherNode won't wake up
« Reply #1 on: August 21, 2017, 10:41:41 AM »
Does it never wake up, not even once?
How is it wired to the Moteino?
Can you post your full sketch in a code block ?

Tony Weil

  • Jr. Member
  • **
  • Posts: 60
  • Country: us
Re: Moteino WeatherNode won't wake up
« Reply #2 on: August 21, 2017, 11:03:34 PM »
For a while, at least, anywhere from 8 hours to several days, it wakes up consistently every 8 seconds and flashes the D9 LED, then posts data to the IOT Gateway every 15 sleep cycles. Then all of a sudden it never wakes up or flashes the LED ever again. The only solution it to disconnect the battery and reconnect to restart. I have checked the LIPO battery and it is usually at 4.0 volts and above. I am running the standard WeatherNode examplefrom the LowPower github with the addition of the LED blink located at https://github.com/LowPowerLab/RFM69/blob/master/Examples/WeatherNode/WeatherNode.ino

Code: [Select]

// **********************************************************************************************************
// WeatherShield R2 (BME280 sensor) sameple sketch that works with Moteinos equipped with RFM69W/RFM69HW
// It sends periodic weather readings (temp, hum, atm pressure) from WeatherShield to the base node Moteino
// For use with MoteinoMEGA you will have to revisit the pin definitions defined below
// http://www.LowPowerLab.com/WeatherShield
// Example setup (with R1): http://lowpowerlab.com/blog/2015/07/24/attic-fan-cooling-tests/
// **********************************************************************************
// Copyright Felix Rusu 2016, http://www.LowPowerLab.com/contact
// **********************************************************************************
// License
// **********************************************************************************
// This program is free software; you can redistribute it
// and/or modify it under the terms of the GNU General   
// Public License as published by the Free Software       
// Foundation; either version 3 of the License, or       
// (at your option) any later version.                   
//                                                       
// This program is distributed in the hope that it will   
// be useful, but WITHOUT ANY WARRANTY; without even the 
// implied warranty of MERCHANTABILITY or FITNESS FOR A   
// PARTICULAR PURPOSE. See the GNU General Public       
// License for more details.                             
//                                                       
// Licence can be viewed at                               
// http://www.gnu.org/licenses/gpl-3.0.txt
//
// Please maintain this license information along with authorship
// and copyright notices in any redistribution of this code
//
// Node 41 is built with an Arduino (MUST BE 3.3V), BME280 and an RFM69HW chip on a breakout.
// Connections are as follows:
//
// RFM69HW    Arduino  (SPI)
// SCK         D13
// MISO        D12
// MOSI        D11
// NSS         D10
// DIO         D2
// 3.3V        3.3V
// GND         GND
//
// BME280      Arduino (I2CAddress = 0x77 default)
// SDA         A4
// SCL         A5
// 3.3V        3.3V
// GND         GND


// **********************************************************************************
#include <RFM69.h>         //get it here: https://github.com/lowpowerlab/rfm69
#include <RFM69_ATC.h>     //get it here: https://github.com/lowpowerlab/rfm69
#include <RFM69_OTA.h>     //get it here: https://github.com/lowpowerlab/rfm69
#include <SPIFlash.h>      //get it here: https://github.com/lowpowerlab/spiflash
#include <SPI.h>           //included in Arduino IDE (www.arduino.cc)
#include <Wire.h>          //included in Arduino IDE (www.arduino.cc)
#include <SparkFunBME280.h>//get it here: https://github.com/sparkfun/SparkFun_BME280_Breakout_Board/tree/master/Libraries/Arduino/src
#include <LowPower.h>      //get it here: https://github.com/lowpowerlab/lowpower
                           //writeup here: http://www.rocketscream.com/blog/2011/07/04/lightweight-low-power-arduino-library/

//*********************************************************************************************
//************ IMPORTANT SETTINGS - YOU MUST CHANGE/CONFIGURE TO FIT YOUR HARDWARE ************
//*********************************************************************************************
#define GATEWAYID   1
#define NODEID      41
#define NETWORKID   100
//#define FREQUENCY     RF69_433MHZ
//#define FREQUENCY     RF69_868MHZ
#define FREQUENCY       RF69_915MHZ //Match this with the version of your Moteino! (others: RF69_433MHZ, RF69_868MHZ)
#define ENCRYPTKEY      "sampleEncryptKey" //has to be same 16 characters/bytes on all nodes, not more not less!
#define IS_RFM69HW_HCW  //uncomment only for RFM69HW/HCW! Leave out if you have RFM69W/CW!
//*********************************************************************************************
#define ENABLE_ATC    //comment out this line to disable AUTO TRANSMISSION CONTROL
#define ATC_RSSI      -75
//*********************************************************************************************
#define SEND_LOOPS   15 //send data this many sleep loops (15 loops of 8sec cycles = 120sec ~ 2 minutes)
#define SLEEP_FASTEST SLEEP_15MS
#define SLEEP_FAST SLEEP_250MS
#define SLEEP_SEC SLEEP_1S
#define SLEEP_LONG SLEEP_2S
#define SLEEP_LONGER SLEEP_4S
#define SLEEP_LONGEST SLEEP_8S
period_t sleepTime = SLEEP_LONGEST; //period_t is an enum type defined in the LowPower library (LowPower.h)
//*********************************************************************************************
#define BATT_MONITOR_EN A3 //enables battery voltage divider to get a reading from a battery, disable it to save power
#define BATT_MONITOR  A7   //through 1Meg+470Kohm and 0.1uF cap from battery VCC - this ratio divides the voltage to bring it below 3.3V where it is scaled to a readable range
#define BATT_CYCLES   2    //read and report battery voltage every this many sleep cycles (ex 30cycles * 8sec sleep = 240sec/4min). For 450 cyclesyou would get ~1 hour intervals
#define BATT_FORMULA(reading) reading * 0.00322 * 1.475  // >>> fine tune this parameter to match your voltage when fully charged
#define BATT_LOW      3.6  //(volts)
#define BATT_READ_LOOPS  SEND_LOOPS*10  // read and report battery voltage every this many sleep cycles (ex 30cycles * 8sec sleep = 240sec/4min). For 450 cycles you would get ~1 hour intervals between readings
//*****************************************************************************************************************************

#ifdef __AVR_ATmega1284P__
  #define LED           15 // Moteino MEGAs have LEDs on D15
  #define FLASH_SS      23 // and FLASH SS on D23
#else
  #define LED           9 // Moteinos have LEDs on D9
  #define FLASH_SS      8 // and FLASH SS on D8
#endif

#define BLINK_EN                 //uncomment to blink LED on every send
#define SERIAL_EN                //comment out if you don't want any serial output

#ifdef SERIAL_EN
  #define SERIAL_BAUD   115200
  #define DEBUG(input)   {Serial.print(input);}
  #define DEBUGln(input) {Serial.println(input);}
  #define SERIALFLUSH() {Serial.flush();}
#else
  #define DEBUG(input);
  #define DEBUGln(input);
  #define SERIALFLUSH();
#endif
//*****************************************************************************************************************************

#ifdef ENABLE_ATC
  RFM69_ATC radio;
#else
  RFM69 radio;
#endif

SPIFlash flash(FLASH_SS, 0xEF30); //WINDBOND 4MBIT flash chip on CS pin D8 (default for Moteino)

BME280 bme280;
char Pstr[10];
char Fstr[10];
char Hstr[10];
char buffer[50];

void setup(void)
{

#ifdef SERIAL_EN
  Serial.begin(SERIAL_BAUD);
  Serial.println("Starting WeatherNode Node 41 ...");
#endif
 
  radio.initialize(FREQUENCY,NODEID,NETWORKID);
#ifdef IS_RFM69HW_HCW
  radio.setHighPower(); //must include this only for RFM69HW/HCW!
#endif
  radio.encrypt(ENCRYPTKEY);

//Auto Transmission Control - dials down transmit power to save battery (-100 is the noise floor, -90 is still pretty good)
//For indoor nodes that are pretty static and at pretty stable temperatures (like a MotionMote) -90dBm is quite safe
//For more variable nodes that can expect to move or experience larger temp drifts a lower margin like -70 to -80 would probably be better
//Always test your ATC mote in the edge cases in your own environment to ensure ATC will perform as you expect
#ifdef ENABLE_ATC
  radio.enableAutoPower(ATC_RSSI);
#endif

  sprintf(buffer, "WeatherMote - transmitting at: %d Mhz...", FREQUENCY==RF69_433MHZ ? 433 : FREQUENCY==RF69_868MHZ ? 868 : 915);
  DEBUGln(buffer);

  //initialize weather shield BME280 sensor
  bme280.settings.commInterface = I2C_MODE;
  bme280.settings.I2CAddress = 0x77;
  bme280.settings.runMode = 3; //Normal mode
  bme280.settings.tStandby = 0;
  bme280.settings.filter = 0;
  bme280.settings.tempOverSample = 1;
  bme280.settings.pressOverSample = 1;
  bme280.settings.humidOverSample = 1;

  radio.sendWithRetry(GATEWAYID, "START", 6);
  Blink(LED, 100);Blink(LED, 100);Blink(LED, 100);

  if (flash.initialize()) flash.sleep();

  for (uint8_t i=0; i<=A5; i++)
  {
    if (i == RF69_SPI_CS) continue;
    if (i == FLASH_SS) continue;
    pinMode(i, OUTPUT);
    digitalWrite(i, LOW);
  }
 
  SERIALFLUSH();
  readBattery();
}

unsigned long doorPulseCount = 0;
char input=0;
double F,P,H;
byte sendLoops=0;
byte battReadLoops=0;
float batteryVolts = 5;
char* BATstr="BAT:5.00v"; //longest battery voltage reading message = 9chars
byte sendLen;

void loop()
{
  if (battReadLoops--<=0) //only read battery every BATT_READ_LOOPS cycles
  {
    readBattery();
    battReadLoops = BATT_READ_LOOPS-1;
  }
 
  if (sendLoops--<=0)   //send readings every SEND_LOOPS
  {
    sendLoops = SEND_LOOPS-1;
   
    //read BME sensor
    bme280.begin();
    P = bme280.readFloatPressure() * 0.0002953; //read Pa and convert to inHg
    F = bme280.readTempF();
    H = bme280.readFloatHumidity();
    bme280.writeRegister(BME280_CTRL_MEAS_REG, 0x00); //sleep the BME280

    dtostrf(F, 3,2, Fstr);
    dtostrf(H, 3,2, Hstr);
    dtostrf(P, 3,2, Pstr);

    sprintf(buffer, "BAT:%sv F:%s H:%s P:%s", BATstr, Fstr, Hstr, Pstr);

    sendLen = strlen(buffer);
    radio.sendWithRetry(GATEWAYID, buffer, sendLen, 1); //retry one time
    DEBUG(buffer); DEBUG(" (packet length:"); DEBUG(sendLen); DEBUGln(")");

    #ifdef BLINK_EN
      Blink(LED, 5);
    #endif
  }
 
  //When this sketch is on a node where you can afford the power to keep the radio awake all the time
  //   you can make it receive messages and also make it wirelessly programmable
  //   otherwise this section can be removed
  if (radio.receiveDone())
  {
    boolean reportStatusRequest=false;
    DEBUG('[');DEBUG(radio.SENDERID);DEBUG("] ");
    for (byte i = 0; i < radio.DATALEN; i++)
      DEBUG((char)radio.DATA[i]);

    flash.wakeup();
    // wireless programming token check - this only works when radio is kept awake to listen for WP tokens
    CheckForWirelessHEX(radio, flash, true);

    //first send any ACK to request
    DEBUG("   [RX_RSSI:");DEBUG(radio.RSSI);DEBUG("]");
    if (radio.ACKRequested())
    {
      radio.sendACK();
      DEBUG(" - ACK sent.");
    }
    DEBUGln();
  }
 
  SERIALFLUSH();
  flash.sleep();
  radio.sleep(); //you can comment out this line if you want this node to listen for wireless programming requests
  LowPower.powerDown(sleepTime, ADC_OFF, BOD_OFF);
  DEBUGln("WAKEUP");
 
  //********************This is my added test to see if it wakes up every 8 sec
  Blink(LED,500);
  //************************************************************************

}

void readBattery()
{
  unsigned int readings=0;
 
  //enable battery monitor on WeatherShield (via mosfet controlled by A3)
  pinMode(BATT_MONITOR_EN, OUTPUT);
  digitalWrite(BATT_MONITOR_EN, LOW);

  for (byte i=0; i<5; i++) //take several samples, and average
    readings+=analogRead(BATT_MONITOR);
 
  //disable battery monitor
  pinMode(BATT_MONITOR_EN, INPUT); //highZ mode will allow p-mosfet to be pulled high and disconnect the voltage divider on the weather shield
   
  batteryVolts = BATT_FORMULA(readings / 5.0);
  dtostrf(batteryVolts,3,2, BATstr); //update the BATStr which gets sent every BATT_CYCLES or along with the MOTION message
  if (batteryVolts <= BATT_LOW) BATstr = "LOW";
}

void Blink(byte PIN, byte DELAY_MS)
{
  pinMode(PIN, OUTPUT);
  digitalWrite(PIN,HIGH);
  delay(DELAY_MS/2);
  digitalWrite(PIN,LOW);
  delay(DELAY_MS/2); 
}



Tony Weil

  • Jr. Member
  • **
  • Posts: 60
  • Country: us
Re: Moteino WeatherNode won't wake up
« Reply #3 on: August 21, 2017, 11:29:14 PM »
You asked for the wiring info. The Moteino is stacked on top of the Weathernode using male headers on the Moteino and female headers on the WeatherShield . The connections are SPI , a couple of pins to monitor the battery and 3.3V power. The Lipo battery is connected to the Moteino using a JST jack soldered to Vin and GND. 
WeatherNode: https://lowpowerlab.com/shop/product/123
There is an attached photo.

***********************************************

Hoping to not confuse the issue, at one point because I ran out of Moteino's and wanted to understand the components and wiring better, I built my own "Moteino" and Weathernode using a 3.3V Arduino Pro Mini, HopeRF69HW, and a Sparkfun BME280 and hooked it up as follows using the same script.

I had NO problems with wakeup.

There is another attached photo.

 //Node 41 is built with an Arduino (MUST BE 3.3V), BME280 and an RFM69HW chip on a breakout.
// Connections are as follows:
//
// RFM69HW    Arduino  (SPI)
// SCK         D13
// MISO        D12
// MOSI        D11
// NSS         D10
// DIO         D2
// 3.3V        3.3V
// GND         GND
//
// BME280      Arduino (I2CAddress = 0x77 default)
// SDA         A4
// SCL         A5
// 3.3V        3.3V
// GND         GND





Tony Weil

  • Jr. Member
  • **
  • Posts: 60
  • Country: us
Re: Moteino WeatherNode won't wake up
« Reply #4 on: August 22, 2017, 12:43:25 AM »
One last thought. I have been troubleshooting this for over a month using different pairs of Moteino's R4, R6 and Weathershields. (Inside and outside). When one fails, sometimes I swap the inside and outside Moteinos. It always seems to be the node that is Outside that fails. The outside node is in a protected location and the temperatures have not been too extreme.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Moteino WeatherNode won't wake up
« Reply #5 on: August 22, 2017, 09:52:59 AM »
Everything seems to be looking good. These types of failure are very hard to guess without real empirical and consistent repeatable behavior.
I also have the same setup and similar code running for a very long time, doing the same thing, without issue.

Without a scope or logic analyzer looking at the I2C and perhaps SPI lines to see if and where the node gets stuck, it's hard to guess why this is happening. Could be wiring or condensation causing the hardware to fail in some way, especially that you mention the outside one fails. Although I have similar setups with mated headers and they have worked untouched for years.

Do you have a scope/logic analyzer to investigate upon lockup?
Or I can attempt to do that if you send me your complete setup, I could take a look to see if I find anything.
Not sure what else to suggest, maybe others have more insight.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: Moteino WeatherNode won't wake up
« Reply #6 on: August 22, 2017, 11:09:16 AM »
Take a good look at the code around when it is put to sleep. This problem can happen if there is a window after determining you need to sleep but before interrrupts are disabled, an interrupt that should have kept it awake can sneak in and be taken before sleep instruction is executed, and that means it is put to sleep and never wakes up. Since this window is very small it can take months, even years, before the window is hit. Make sure global interrupts are disabled before checking conditions to determine whether to sleep.

Mark.
« Last Edit: August 22, 2017, 11:15:19 AM by perky »

Tony Weil

  • Jr. Member
  • **
  • Posts: 60
  • Country: us
Re: Moteino WeatherNode won't wake up
« Reply #7 on: August 22, 2017, 04:02:44 PM »
Thanks Felix. I agree I need to get some good repeatable data on the failures.

Yes, I have a scope. The next time one fails, I can look at the SPI and I2C lines and see what is happening. Given that the LED stops blinking, and that the Node stops transmitting to the Gateway, I am expecting to see nothing because the node is not functioning at all, but I will test it anyway.

My new test setup is 2 identical, brand new, R6 Moteinos connected to R2 Weathershields and powered by Lipo batteries.  One is inside and one is outside in a very protected area. First, for a week or so, I will keep track of which node fails, how long it ran for and then restart it. If it is always the outside node, I will swap the inside and outside nodes and do the test again. Hopefully, this testing with new nodes will shed some light on the problem or at least add some data. I'm headed off on vacation for a couple of weeks, so I'll do this testing when I return and post my results.

Thanks for offering to look at the nodes. I'm just down the way from you in A2. If I can't figure it out, I'll take you up on it.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Moteino WeatherNode won't wake up
« Reply #8 on: August 22, 2017, 04:16:08 PM »
Great, nice that you are open to doing some testing yourself to better understand what's going on, or at least try to.
The scope/analyzer test would provide some insight into whether the node is waking and getting stuck trying to do some I2c (BME280) or SPI (radio, flash) transactions. If so that would point to the libraries.

In any such scenarios, the goal for me is to discover if there's anything wrong with a particular board or schematic. Code is easy to fix, hardware not so much.

I would also suggest doing some logging of the battery via the weatherNode if possible, a full charge should provide a long long run time with the proper sleep code of a Moteino+WShield, and should produce a flat line over just a week of use.

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: Moteino WeatherNode won't wake up
« Reply #9 on: August 22, 2017, 04:45:29 PM »
BTW the reason I point this interrupt disabling thing out is because in the past I had a routine that could be called when interrupts were disabled and I didn't save the state but simply explictly re-enabled them at the end. This was safe in the rest of my code, but I called that routine just before sleeping too, and opened up a small window where a waking interrupt was taken just before the sleep instruction. Four months later it failed to wake up, and a re-examining of the code highlighted the problem.

Mark.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Moteino WeatherNode won't wake up
« Reply #10 on: August 22, 2017, 04:48:13 PM »
Thanks Mark, that's a very valuable clue to look into

Tony Weil

  • Jr. Member
  • **
  • Posts: 60
  • Country: us
Re: Moteino WeatherNode won't wake up
« Reply #11 on: September 23, 2017, 12:44:43 PM »
I have finally been able to get back working on this problem.

After some testing, I still am not able to find the problem. Here are some results:

1) The inside/outside issue was a red herring. Problem happens to both nodes anywhere from 5 minutes to a few days.

2) I switched from batteries to wall power through an FTDI cable. Problem still happens.

3) I hooked up a scope to view activity on the SCL and SDA lines. I also had the LED blink each time data was sent. When the Gateway stops receiving data, there is no more activity on the SCL/SDA lines and the LED stops blinking.

4) I have tried sending data at different rates between every 8 Sec (1 Sleep Period) and 2 minutes (15 sleep periods) with the same results.

I'm about ready to take you up on your offer to let me send the 2 nodes to you. I am quickly getting beyond my tech abilities to troubleshoot.

I am curious about Mark's suggestion about Global Interrupts. Do I just
add the line cli(); to disable global interrupts just before the Sleep Command and add the line sei(); to enable them?

Thanks again,
Tony
 

perky

  • Hero Member
  • *****
  • Posts: 873
  • Country: gb
Re: Moteino WeatherNode won't wake up
« Reply #12 on: September 24, 2017, 02:29:11 PM »
In pseudo code:

Code: [Select]
main_loop:
     do main loop stuff
     disable global interrupts (cli)
     if (conditions OK for sleeping)
          issue sleep instruction and re-enable global interrupts on next instruction (sei)
     else
          re-enable global interrupts (sei)
     goto main_loop

This ensures that the conditions you test are with global interrupts turned off so there's no chance that the condition (e.g. a flag set by an interrupt), might occur inbetween testing for the condition and the sleep instruction with interrupts enabled. If that were to happen there would be a very small window where the interrupt would be actioned just prior to the sleep instruction, and it would appear to just go to sleep and never wake up. Because these things can be asynchronous, and the window is so small, it can take a very long time to happen.

Mark.
« Last Edit: September 25, 2017, 03:27:38 PM by perky »

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Moteino WeatherNode won't wake up
« Reply #13 on: September 25, 2017, 10:39:03 AM »
3) I hooked up a scope to view activity on the SCL and SDA lines. I also had the LED blink each time data was sent. When the Gateway stops receiving data, there is no more activity on the SCL/SDA lines and the LED stops blinking.

I'm about ready to take you up on your offer to let me send the 2 nodes to you. I am quickly getting beyond my tech abilities to troubleshoot.
So it hangs somewhere in code obviously. Try Mark's suggestion. If you want to send them to me I would probably just load them with the stock code, if that won't work then there's a problem to fix.

Tony Weil

  • Jr. Member
  • **
  • Posts: 60
  • Country: us
Re: Moteino WeatherNode won't wake up
« Reply #14 on: September 27, 2017, 11:33:53 AM »
I will try Mark's suggestion. I am just using the standard Weathershield sketch, so I think it would just look like the following. I don't see a need for anything more complicated. The node sleeps on every loop, but only sends data on the 15th loop. (sleepTime = SLEEP_8S)

Code: [Select]
flash.sleep();
radio.sleep();
cli();
LowPower.powerDown(sleepTime, ADC_OFF, BOD_OFF);
(sei();

Assuming the next couple of tests don't work I would like to send the nodes to you. Using the standard Weathershield sketch is all I need or expect.

One last possible data point and red herring: The "problem" nodes are Moteino R6 with RFM69HCW and IS_RFM69HW_HCW is enabled in the sketch. I tried using a Moteino R6 with RFMCW and IS_RFM69HW_HCW disabled. It has been running like a charm so far.

I two other Moteinos I can try:
1)R4 with RFM69HW
2) Homemade "Moteino" and Weathernode using a 3.3V Arduino Pro Mini, HopeRF69HW, and a Sparkfun BME280.

Thanks again, I just want to understand what is going on.
Tony