OTA Target in non working state after failed OTA attempt

Started by Scram, December 17, 2017, 08:44:00 AM

Print
Go Down Pages1
User actions

Scram

December 17, 2017, 08:44:00 AM Last Edit: December 18, 2017, 01:12:54 PM by Scram
Today i tried to solder a flash mem chip (0xEF30 for Winbond 4mbit flash) on some Moteinos and tried the flash memory with the following sketch "SPIFlash_ReadWrite" from the examples. Everthing seemed to work well. I could write and read and erase fine.

Flash:
https://www.dropbox.com/s/tn1a3j9jn4nxbet/Windbond%20Flash%20Mem.jpg?dl=0

Then i tried to use the Windows Program and a gateway Moteino to flash remotely which was successful stated in the windows program (see Log at the end of this post) but now the target Moteino is in a strange state which i can not recover from. The LED was turning to a dimmed state (its like 1/8 of the normal on state brightness) after flashing and now it remains like this no matter what sketch i'm trying to flash via the serial port. It seems like the OTA attempt left it in some undefined state.

How can i fix this? I have no idea anymore. Bootloader was the "DualOptiboot_V5.0_atmega328_BlinkD9.hex".

Here's how it looks now when trying to flash something via serial (Sketch "Blink" which should blink the orange LED on the Moteino):
https://youtu.be/J8LVHWwhBCs

The IDE is happy and states verify was ok but after resetting the Moteino something goes wrong.

Edit: After unsoldering the flash mem chip the Moteino behaves like a normal Moteino again.

Here's the Log from the Windows OTA Prog:
Code Select 
Opening COM3 @ 115200baud ...
SET TARGET: TO:34
Moteino: [TO:34:OK]
TARGET SET OK
FLX?
FLX?
Moteino: [FLX?OK]
HANDSHAKE OK!
TX > FLX:0:100000000C945C000C946E000C946E000C946E0 0CA
Moteino: FLX:0:OK
TX > FLX:1:100010000C946E000C946E000C946E000C946E0 0A8
Moteino: FLX:1:OK
TX > FLX:2:100020000C946E000C946E000C946E000C946E0 098
Moteino: FLX:2:OK
TX > FLX:3:100030000C946E000C946E000C946E000C946E0 088
Moteino: FLX:3:OK
TX > FLX:4:100040000C9415010C946E000C946E000C946E0 0D0
Moteino: FLX:4:OK
TX > FLX:5:100050000C946E000C946E000C946E000C946E0 068
Moteino: FLX:5:OK
TX > FLX:6:100060000C946E000C946E00000000002400270 029
Moteino: FLX:6:OK
TX > FLX:7:100070002A0000000000250028002B000404040 4CE
Moteino: FLX:7:OK
TX > FLX:8:100080000404040402020202020203030303030 342
Moteino: FLX:8:OK
TX > FLX:9:100090000102040810204080010204081020010 21F
Moteino: FLX:9:OK
TX > FLX:10:1000A0000408102000000008000201000003040 7FB
Moteino: FLX:10:OK
TX > FLX:11:1000B000000000000000000011241FBECFEFD8E 0B8
Moteino: FLX:11:OK
TX > FLX:12:1000C000DEBFCDBF21E0A0E0B1E001C01D92A93 0AC
Moteino: FLX:12:OK
TX > FLX:13:1000D000B207E1F70E945F010C94CE010C94000 07E
Moteino: FLX:13:OK
TX > FLX:14:1000E000EDEAF0E02491E9E9F0E09491E5E8F0E 050
Moteino: FLX:14:OK
TX > FLX:15:1000F000E491EE2309F43BC0222339F1233091F 03F
Moteino: FLX:15:OK
TX > FLX:16:1001000038F42130A9F0223001F524B52F7D12C 03A
Moteino: FLX:16:OK
TX > FLX:17:10011000273091F02830A1F02430B9F42091800 0EC
Moteino: FLX:17:OK
TX > FLX:18:100120002F7D03C0209180002F77209380000DC 089
Moteino: FLX:18:OK
TX > FLX:19:1001300024B52F7724BD09C02091B0002F7703C 0CC
Moteino: FLX:19:OK
TX > FLX:20:100140002091B0002F7D2093B000F0E0EE0FFF1 F54
Moteino: FLX:20:OK
TX > FLX:21:10015000EE58FF4FA591B4912FB7F894EC91811 10F
Moteino: FLX:21:OK
TX > FLX:22:1001600003C090959E2301C09E2B9C932FBF089 5A2
Moteino: FLX:22:OK
TX > FLX:23:100170003FB7F8948091050190910601A091070 185
Moteino: FLX:23:OK
TX > FLX:24:10018000B091080126B5A89B05C02F3F19F0019 634
Moteino: FLX:24:OK
TX > FLX:25:10019000A11DB11D3FBFBA2FA92F982F8827820 F0D
Moteino: FLX:25:OK
TX > FLX:26:1001A000911DA11DB11DBC01CD0142E0660F771 F5D
Moteino: FLX:26:OK
TX > FLX:27:1001B000881F991F4A95D1F708958F929F92AF9 209
Moteino: FLX:27:OK
TX > FLX:28:1001C000BF92CF92DF92EF92FF920E94B8004B0 154
Moteino: FLX:28:OK
TX > FLX:29:1001D0005C0188EEC82E83E0D82EE12CF12C0E9 421
Moteino: FLX:29:OK
TX > FLX:30:1001E000B800DC01CB0188199909AA09BB09883 E2E
Moteino: FLX:30:OK
TX > FLX:31:1001F0009340A105B10558F021E0C21AD108E10 8E9
Moteino: FLX:31:OK
TX > FLX:32:10020000F10888EE880E83E0981EA11CB11CC11 471
Moteino: FLX:32:OK
TX > FLX:33:10021000D104E104F10419F7FF90EF90DF90CF9 043
Moteino: [FLX?OK]
EOF
SUCCESS! (time elapsed: 3,52s)
Moteino: FLX:33:OK
TX > FLX:34:10022000BF90AF909F908F9008951F920F920FB 63E
Moteino: FLX:34:OK
TX > FLX:35:100230000F9211242F933F938F939F93AF93BF9 36C
Moteino: FLX:35:OK
TX > FLX:36:100240008091010190910201A0910301B091040 1FC
Moteino: FLX:36:OK
TX > FLX:37:100250003091000123E0230F2D3720F40196A11 DDA
Moteino: FLX:37:OK
TX > FLX:38:10026000B11D05C026E8230F0296A11DB11D209 3E4
Moteino: FLX:38:OK
TX > FLX:39:1002700000018093010190930201A0930301B09 3C8
Moteino: FLX:39:OK
TX > FLX:40:1002800004018091050190910601A0910701B09 1B0
Moteino: FLX:40:OK
TX > FLX:41:1002900008010196A11DB11D809305019093060 1EF
Moteino: FLX:41:OK
TX > FLX:42:1002A000A0930701B0930801BF91AF919F918F9 1E7
Moteino: FLX:42:OK
TX > FLX:43:1002B0003F912F910F900FBE0F901F901895789 43B
Moteino: FLX:43:OK
TX > FLX:44:1002C00084B5826084BD84B5816084BD85B5826 05B
Moteino: FLX:44:OK
TX > FLX:45:1002D00085BD85B5816085BD80916E008160809 30C
Moteino: FLX:45:OK
TX > FLX:46:1002E0006E00109281008091810082608093810 075
Moteino: FLX:46:OK
TX > FLX:47:1002F0008091810081608093810080918000816 085
Moteino: FLX:47:OK
TX > FLX:48:10030000809380008091B10084608093B100809 1DF
Moteino: FLX:48:OK
TX > FLX:49:10031000B00081608093B00080917A008460809 307
Moteino: FLX:49:OK
TX > FLX:50:100320007A0080917A00826080937A0080917A0 0CE
Moteino: FLX:50:OK
TX > FLX:51:10033000816080937A0080917A00806880937A0 04F
Moteino: FLX:51:OK
TX > FLX:52:100340001092C100E9E9F0E02491E5E8F0E0849 141
Moteino: FLX:52:OK
TX > FLX:53:10035000882399F090E0880F991FFC01E859FF4 F1E
Moteino: FLX:53:OK
TX > FLX:54:10036000A591B491FC01EE58FF4F859194918FB 700
Moteino: FLX:54:OK
TX > FLX:55:10037000F894EC91E22BEC938FBFC0E0D0E081E 0E9
Moteino: FLX:55:OK
TX > FLX:56:100380000E9470000E94DD0080E00E9470000E9 4C8
Moteino: FLX:56:OK
TX > FLX:57:10039000DD002097A1F30E940000F1CFF894FFC F79
Moteino: FLX:57:OK


It stops here  ::) doesn't look really finished yet...

Felix

#1
December 18, 2017, 11:58:52 AM
With OTA there are several assumptions which I won't repeat here (see OTA page), but you must ensure your sketch is good and will accept more OTA programming in the future.
You should paste the log in a code block, I see spaces in the HEX which I don't know if they come from you or they got inserted somehow.

But if something went wrong, you should be able to reload your Moteino via FTDI.

In case they are bricked (I can't imagine why they would be since the bootloader section is protected by fuses), just reflash Dualoptiboot with an ISP programmer, and make sure the FLASH is genuine too.

I can't verify if the FLASH is all good though. When you order them from me, they are guaranteed not fakes, and I check them via test sketch.

Scram

#2
December 18, 2017, 01:22:08 PM Last Edit: December 18, 2017, 01:25:16 PM by Scram
These were the used sketches:

Sending OTA node:
Code Select 
#include <RFM69.h>          //get it here: https://github.com/lowpowerlab/RFM69
#include <RFM69_ATC.h>      //get it here: https://github.com/lowpowerlab/RFM69
#include <RFM69_OTA.h>      //get it here: https://github.com/lowpowerlab/RFM69
#include <SPIFlash.h>       //get it here: https://www.github.com/lowpowerlab/spiflash
#include <SPI.h>            //included with Arduino IDE (www.arduino.cc)

//****************************************************************************************************************
//**** IMPORTANT RADIO SETTINGS - YOU MUST CHANGE/CONFIGURE TO MATCH YOUR HARDWARE TRANSCEIVER CONFIGURATION! ****
//****************************************************************************************************************
#define NODEID             254  //this node's ID, should be unique among nodes on this NETWORKID
#define NETWORKID          220  //what network this node is on
//Match frequency to the hardware version of the radio on your Moteino (uncomment one):
#define FREQUENCY         RF69_868MHZ
#define FREQUENCY_EXACT   869000000
#define ENCRYPTKEY        "XXXXXXXXXXXXX" // exactly the same 16 characters/bytes on all nodes!
#define IS_RFM69HW_HCW    //uncomment only for RFM69HW/HCW! Leave out if you have RFM69W/CW!
//*********************************************************************************************
#define ENABLE_ATC               //comment out this line to disable AUTO TRANSMISSION CONTROL
//*********************************************************************************************
#define BR_300KBPS             //run radio at max rate of 300kbps!
//*********************************************************************************************
#define DEBUG_MODE false         //set 'true' to see verbose output from programming sequence

#define SERIAL_BAUD 115200
#define ACK_TIME    50  // # of ms to wait for an ack
#define TIMEOUT     3000

#ifdef __AVR_ATmega1284P__
  #define LED           15 // MoteinoMEGA has LED on D15
#else
  #define LED            9 // Moteino has LED on D9
#endif

#ifdef ENABLE_ATC
  RFM69_ATC radio;
#else
  RFM69 radio;
#endif

char c = 0;
char input[64]; //serial input buffer
byte targetID=0;

void setup(){
  Serial.begin(SERIAL_BAUD);
  radio.initialize(FREQUENCY,NODEID,NETWORKID);
  radio.encrypt(ENCRYPTKEY); //OPTIONAL

#ifdef FREQUENCY_EXACT
  radio.setFrequency(FREQUENCY_EXACT); //set frequency to some custom frequency
#endif

#ifdef IS_RFM69HW_HCW
  radio.setHighPower(); //must include this only for RFM69HW/HCW!
#endif
  Serial.println("Start wireless gateway...");

#ifdef BR_300KBPS
  radio.writeReg(0x03, 0x00);  //REG_BITRATEMSB: 300kbps (0x006B, see DS p20)
  radio.writeReg(0x04, 0x6B);  //REG_BITRATELSB: 300kbps (0x006B, see DS p20)
  radio.writeReg(0x19, 0x40);  //REG_RXBW: 500kHz
  radio.writeReg(0x1A, 0x80);  //REG_AFCBW: 500kHz
  radio.writeReg(0x05, 0x13);  //REG_FDEVMSB: 300khz (0x1333)
  radio.writeReg(0x06, 0x33);  //REG_FDEVLSB: 300khz (0x1333)
  radio.writeReg(0x29, 240);   //set REG_RSSITHRESH to -120dBm
#endif
}

void loop(){
  byte inputLen = readSerialLine(input, 10, 64, 100); //readSerialLine(char* input, char endOfLineChar=10, byte maxLength=64, uint16_t timeout=1000);
  
  if (inputLen==4 && input[0]=='F' && input[1]=='L' && input[2]=='X' && input[3]=='?') {
    if (targetID==0)
      Serial.println("TO?");
    else
      CheckForSerialHEX((byte*)input, inputLen, radio, targetID, TIMEOUT, ACK_TIME, DEBUG_MODE);
  }
  else if (inputLen>3 && inputLen<=6 && input[0]=='T' && input[1]=='O' && input[2]==':')
  {
    byte newTarget=0;
    for (byte i = 3; i<inputLen; i++) //up to 3 characters for target ID
      if (input[i] >=48 && input[i]<=57)
        newTarget = newTarget*10+input[i]-48;
      else
      {
        newTarget=0;
        break;
      }
    if (newTarget>0)
    {
      targetID = newTarget;
      Serial.print("TO:");
      Serial.print(newTarget);
      Serial.println(":OK");
    }
    else
    {
      Serial.print(input);
      Serial.print(":INV");
    }
  }
  else if (inputLen>0) { //just echo back
    Serial.print("SERIAL IN > ");Serial.println(input);
  }

  if (radio.receiveDone())
  {
    for (byte i = 0; i < radio.DATALEN; i++)
      Serial.print((char)radio.DATA[i]);
    
    if (radio.ACK_REQUESTED)
    {
      radio.sendACK();
      Serial.print(" - ACK sent");
    }
    
    Serial.println();
  }
  Blink(LED,5); //heartbeat
}

void Blink(byte PIN, int DELAY_MS)
{
  pinMode(PIN, OUTPUT);
  digitalWrite(PIN,HIGH);
  delay(DELAY_MS);
  digitalWrite(PIN,LOW);
}


Receiving OTA node:
Code Select 
#include <RFM69.h>         //get it here: https://github.com/lowpowerlab/RFM69
#include <RFM69_ATC.h>     //get it here: https://github.com/lowpowerlab/RFM69
#include <RFM69_OTA.h>     //get it here: https://github.com/lowpowerlab/RFM69
#include <SPIFlash.h>      //get it here: https://github.com/lowpowerlab/spiflash
#include <SPI.h>           //included with Arduino IDE install (www.arduino.cc)

//****************************************************************************************************************
//**** IMPORTANT RADIO SETTINGS - YOU MUST CHANGE/CONFIGURE TO MATCH YOUR HARDWARE TRANSCEIVER CONFIGURATION! ****
//****************************************************************************************************************
#define NODEID      123       // node ID used for this unit
#define NETWORKID   220
//Match frequency to the hardware version of the radio on your Moteino (uncomment one):
//#define FREQUENCY   RF69_433MHZ
#define FREQUENCY   RF69_868MHZ
#define FREQUENCY_EXACT 869000000
#define IS_RFM69HW_HCW  //uncomment only for RFM69HW/HCW! Leave out if you have RFM69W/CW!
//*****************************************************************************************************************************
#define ENABLE_ATC    //comment out this line to disable AUTO TRANSMISSION CONTROL
#define ATC_RSSI      -75
//*****************************************************************************************************************************
#define BR_300KBPS             //run radio at max rate of 300kbps!
//*****************************************************************************************************************************
#define SERIAL_BAUD 115200
#define ACK_TIME    30  // # of ms to wait for an ack
#define ENCRYPTKEY        "XXXXXXXXXXXXX" // exactly the same 16 characters/bytes on all nodes!
#define BLINKPERIOD 500

#ifdef __AVR_ATmega1284P__
  #define LED           15 // Moteino MEGAs have LEDs on D15
  #define FLASH_SS      23 // and FLASH SS on D23
#else
  #define LED           9 // Moteinos hsave LEDs on D9
  #define FLASH_SS      8 // and FLASH SS on D8
#endif

#ifdef ENABLE_ATC
  RFM69_ATC radio;
#else
  RFM69 radio;
#endif

char input = 0;
long lastPeriod = -1;

//*****************************************************************************************************************************
// flash(SPI_CS, MANUFACTURER_ID)
// SPI_CS          - CS pin attached to SPI flash chip (8 in case of Moteino)
// MANUFACTURER_ID - OPTIONAL, 0x1F44 for adesto(ex atmel) 4mbit flash
//                             0xEF30 for windbond 4mbit flash
//                             0xEF40 for windbond 16/64mbit flash
//*****************************************************************************************************************************
SPIFlash flash(FLASH_SS, 0xEF30); //EF30 for windbond 4mbit flash

void setup() {
  pinMode(LED, OUTPUT);
  Serial.begin(SERIAL_BAUD);
  radio.initialize(FREQUENCY,NODEID,NETWORKID);
  radio.encrypt(ENCRYPTKEY); //OPTIONAL

#ifdef FREQUENCY_EXACT
  radio.setFrequency(FREQUENCY_EXACT); //set frequency to some custom frequency
#endif

#ifdef ENABLE_ATC
  radio.enableAutoPower(ATC_RSSI);
#endif

#ifdef IS_RFM69HW_HCW
  radio.setHighPower(); //must include this only for RFM69HW/HCW!
#endif
  Serial.print("Start node...");
  Serial.print("Node ID = ");
  Serial.println(NODEID);

  if (flash.initialize())
    Serial.println("SPI Flash Init OK!");
  else
    Serial.println("SPI Flash Init FAIL!");

#ifdef BR_300KBPS
  radio.writeReg(0x03, 0x00);  //REG_BITRATEMSB: 300kbps (0x006B, see DS p20)
  radio.writeReg(0x04, 0x6B);  //REG_BITRATELSB: 300kbps (0x006B, see DS p20)
  radio.writeReg(0x19, 0x40);  //REG_RXBW: 500kHz
  radio.writeReg(0x1A, 0x80);  //REG_AFCBW: 500kHz
  radio.writeReg(0x05, 0x13);  //REG_FDEVMSB: 300khz (0x1333)
  radio.writeReg(0x06, 0x33);  //REG_FDEVLSB: 300khz (0x1333)
  radio.writeReg(0x29, 240);   //set REG_RSSITHRESH to -120dBm
#endif
}

void loop(){
  // This part is optional, useful for some debugging.
  // Handle serial input (to allow basic DEBUGGING of FLASH chip)
  // ie: display first 256 bytes in FLASH, erase chip, write bytes at first 10 positions, etc
  if (Serial.available() > 0) {
    input = Serial.read();
    if (input == 'd') //d=dump first page
    {
      Serial.println("Flash content:");
      int counter = 0;

      while(counter<=256){
        Serial.print(flash.readByte(counter++), HEX);
        Serial.print('.');
      }
      
      Serial.println();
    }
    else if (input == 'e')
    {
      Serial.print("Erasing Flash chip ... ");
      flash.chipErase();
      while(flash.busy());
      Serial.println("DONE");
    }
    else if (input == 'i')
    {
      Serial.print("DeviceID: ");
      Serial.println(flash.readDeviceId(), HEX);
    }
    else if (input == 'r')
    {
      Serial.print("Rebooting");
      resetUsingWatchdog(true);
    }
    else if (input == 'R')
    {
      Serial.print("RFM69 registers:");
      radio.readAllRegs();
    }
    else if (input >= 48 && input <= 57) //0-9
    {
      Serial.print("\nWriteByte("); Serial.print(input); Serial.print(")");
      flash.writeByte(input-48, millis()%2 ? 0xaa : 0xbb);
    }
  }
  
  // Check for existing RF data, potentially for a new sketch wireless upload
  // For this to work this check has to be done often enough to be
  // picked up when a GATEWAY is trying hard to reach this node for a new sketch wireless upload
  if (radio.receiveDone())
  {
    Serial.print("Got [");
    Serial.print(radio.SENDERID);
    Serial.print(':');
    Serial.print(radio.DATALEN);
    Serial.print("] > ");
    for (byte i = 0; i < radio.DATALEN; i++)
      Serial.print((char)radio.DATA[i], HEX);
    Serial.println();
    CheckForWirelessHEX(radio, flash, false);
    Serial.println();
  }
  //else Serial.print('.');
  
  //*****************************************************************************************************************************
  // Real sketch code here, let's blink the onboard LED
  if ((int)(millis()/BLINKPERIOD) > lastPeriod)
  {
    lastPeriod++;
    digitalWrite(LED, lastPeriod%2);
  }
  //*****************************************************************************************************************************
}


The problem is repeatable. Meanwhile i "bricked" 3 Moteinos.

When removing the SPI Flash the Moteino seems ok. So it must be something within the flash memory which prevents proper booting. But strangewise it insist's on booting the faulty code from flash instead of taking a new sketch from FTDI interface when i try to recover it.

If those sketches are ok it can only be faulty Flash right!? They seemed to be working ok when i tested them with the mentioned test sketch. I also tried to write the last 10 Bytes of the 512K to make sure they're not fake ones with less capacity. I'll better dump them and buy new ones  :'(.

Felix

#3
December 18, 2017, 04:48:49 PM
I wish I could help but without the actual devices I can't really tell what's going on.
OTA is a little harder to debug simply because there are so many parts involved (not just hardware).

As long as the chips are genuine and not abused over the max specs, they should not fail, I've never had one fail or stop working.

FWIW I was asking for your OTA log not the sketches...

Scram

#4
December 19, 2017, 09:36:58 AM Last Edit: December 19, 2017, 09:43:14 AM by Scram
Sorry. I did not really store the logs any other than within the first post.

As far as i understand the sketches were ok so if my Moteinos are ok, which i'm pretty sure as i have at least 20 of this type running, it can only be the Flash which is causing trouble here.

The point i did not really understand is why the obviously corrupted download which is CRC verified by the receiving node is taken as valid? Since it never really succeeded in downloading it should never be marked as OK from the receiver side before resetting and starting to copy it into the processor flash memory.

Anyway, lets go forward...

as i could not get them here in Germany i ordered them from China which was probably not the best idea.

The other Type from ATMEL/Adesto AT25DF041A seems to be EOL

Is the DualOptiBoot checking the manufacturer ID 0x1F44? Or would it be ok to substitute with AT25SF041 which would answer with 0x1F84 instead but is having pretty similar specs!? These ones would be still available here.

Felix

#5
December 19, 2017, 10:21:53 AM
Any FLASH that has the same instruction set will work. The SPIFlash library works with the instruction set for the stock Windbond 4mbit flash I provide.
You will need to make sure they do the same thing, by checking the instructions from their datasheet, OR modify/fork the SPIFlash lib to fit yours. However in that case you'd also need to rebuild Dualoptiboot to match those instructions. Quite a hassle but doable if you're motivated. From what i've seen, other flash chips have very similar instructions but at least 1 or 2 instructions sometimes differ, which is unfortunate.

The OTA protocol includes CRC checking. The transfer can stop for a few reasons, including corruption or lost signal, but again - as long as everything is done correctly that should not really happen.
Print
Go Up Pages1
User actions