Is anyone doing over-the-air updates via a Raspberry Pi Gateway?

[thinkpeace]

I've got a project working based loosely on the Pi Gateway project.

The code I'm running on the Pi is C++ rather than Python, and it connects to our web server using MQTT (Paho C client).  It's been working pretty reliably, and now I would like to be able to update the remote Moteino's over-the-air. 

I would like to be able to initiate the update from our web site, and it should be able to update a single device or all remote devices in sequence from a single request.  The website should show the status of the update as it progresses as well.

Eric

[jarrods]

I have, but I am using Python so I was able to leverage Felix's Python code for programming. What I do is have a location that hex is downloaded to on the Pi via wget (Pi only over calls out, no inbound connection for security reasons). Then I have a command queue. The command queue can be set to update 1+ of the nodes. However since my nodes sleep for 5 minutes at a time I wait until the Node calls home then I start the update.

Since your using C++ you will need to rewrite the Wireless Programming Module or you can try to turn it into a c++ module with http://cython.org/. I have tried that for other stuff in the past with varying degrees of success. Given the serial port in play you should try to rewrite it.

NOTE: You will likely need to play with the radio ACK to do a logarithmic retry (I haven't quite figured out the best formula for this yet)otherwise you will loose reports from the other nodes while updates are in progress. This is the issue i am trying to figure out currently. Also if you are updating lots nodes at once it could cause issues with your network as it will take ~20 seconds each depending on range and size of program. With 120 nodes your looking at 40 minutes. So you will likely need to implement something that will queue then and leave time for normal traffic between each update.

- Jarrod

[BenCranston]

I've been working on this as well and took a slightly different approach.  The command structure of the stock SwitchMote code was extended to include functions for receiving firmware updates over a period of time as part of the main loop. Offset + code block, which are written into the flash whenever they are received.  Once the last block with checksum is sent, the magic header is written ("FLX:" + image size) addressing the case of a reboot before complete and validated upload so DualOptiboot won't install partially uploaded firmware.  this way the firmware can be "drizzled" out to the nodes without saturating the wireless network and allowing all nodes to continue to process event traffic, including the nodes which are getting firmware updates.  It's a work in progress... The logic currently does not comprehend updating multiple devices from the same firmware image like a broadcast update.  but should allow for multiple different updates across several nodes simultaneously as a background process. really it's minor code changes in the main loop for the new function and skipping the CheckForWirelessHEX calls which would put the Moteino in a tight loop waiting for the firmware update.  the drawback to this approach is that signaling and firmware are now in the same wireless channel competing for bandwidth, so the firmware update will take a lot longer than the original approach of shifting the radio up 1MHz to get out of the way of the other Moteinos and pushing the firmware in one sitting...  Keeping the original method you'll really need two radios (two Moteinos) on the RPi, one for code push and the other for message traffic.

[TomWS]

I've been working on this as well and took a slightly different approach.  The command structure of the stock SwitchMote code was extended to include functions for receiving firmware updates over a period of time as part of the main loop. Offset + code block, which are written into the flash whenever they are received.  Once the last block with checksum is sent, the magic header is written ("FLX:" + image size) addressing the case of a reboot before complete and validated upload so DualOptiboot won't install partially uploaded firmware.  this way the firmware can be "drizzled" out to the nodes without saturating the wireless network and allowing all nodes to continue to process event traffic, including the nodes which are getting firmware updates.  It's a work in progress... The logic currently does not comprehend updating multiple devices from the same firmware image like a broadcast update.  but should allow for multiple different updates across several nodes simultaneously as a background process. really it's minor code changes in the main loop for the new function and skipping the CheckForWirelessHEX calls which would put the Moteino in a tight loop waiting for the firmware update.  the drawback to this approach is that signaling and firmware are now in the same wireless channel competing for bandwidth, so the firmware update will take a lot longer than the original approach of shifting the radio up 1MHz to get out of the way of the other Moteinos and pushing the firmware in one sitting...  Keeping the original method you'll really need two radios (two Moteinos) on the RPi, one for code push and the other for message traffic.

Yes, this is similar to what I do my gateway (not RPi).  I have an SD card in my gateway, so I download the image to SD Card and, like you say, 'drizzle' it out to Motes using a 'pull' methodology.   The image is stored in 'PIF' (PacketImage - offset and code) format so that it's ready to be sent out without any further parsing. 

Downloading is a multistage process.  Stage 1: Home server pushes the PIF file to the gateway.  Stage 2: Home server tells Gateway to download the file to a set of Motes.  Stage 3:  When a Mote checks in (data or status), the gateway sees that it's to get a new image and tells the Mote in its Ack data.  The Mote then requests the download packets, one at a time.  Like you, each packet has an offset and code block so the code block can be stored in the appropriate place in Flash (even if received out of order).  Once they're all downloaded, also like you, the FLX header is finished and it's ready for reboot. 

In my case, I require an explicit reboot command (after the Mote says download is complete) from the Home Server so that I can verify the download before letting the Mote disappear into the ether...

I generally have battery operated Motes request only a few packets at a time and usually don't increase the connect interval.  It takes a long time to download, but what the hey?  It's all occurring in background and, as you say, I don't need to shift the frequency (not sure how effective that is in preventing side 'channel' interference).  AC operated motes request more packets but they don't swamp the network with downloads.

With SD card, the image can be kept 'on file' for any of the motes the Gateway is responsible for and can be sent as many times as necessary.

I don't use the Wireless Programming Library at all.

Tom

[BenCranston]

Tom,
That's way cool what you're doing.  I really struggled with the "control" of the Motes...  In that who or what was the control of their behavior, Controller or Mote or equal footing, etc.  I opted to have the Motes be subservient to the gateway controller.  Accept command and then report status back.  However, I'm wondering if a re-thinking of that stance might be in order.  where did you divide the line?  I allowed for the SwitchMotes to continue with their inter-mote messaging that Felix has in the stock SwitchMote code, but that was it. and to that point added functions to have the Mote dump it's message table and be able to receive an update from the controller of what should be in that table.

I know I've strayed off topic.  I should open a new thread for a wider discussion.. 

-Ben

[thinkpeace]

Thanks everyone for your ideas.

I had to take a break from this for a while.  I got back to it recently, and I ported Felix's Python code to C, and modified the wireless programming gateway example.

At this point, it starts to download to the node, but the node is hangs up before finishing the watchdog timer times out.

Below is the code which runs on the Raspberry Pi

Code Select Expand

// The following code was translated from Python, https://github.com/LowPowerLab/WirelessProgramming/blob/master/WirelessProgramming.py

bool waitHandshake(bool isEOF)
{
  bool rc;
  char handshake[20];
    if( isEOF )
    {
        strcpy( handshake, "FLX?EOF\n" );
    }
    else
    {
        strcpy( handshake, "FLX?\n" );
    }

   // The radio class is generally used to send and receive messages with the moteino nodes.
   // The send() function is used to address the gateway moteino rather than the remote nodes.

    rc = radio.send((byte *)handshake, strlen(handshake), true);

    if(rc && radio.BUFFER_LENGTH > 0 )
    {
        if( strncmp( (const char*) radio.data, "FLX?OK", 6 ) == 0 )
        {
          return true;
        }
        else if( strncmp( (const char*) radio.data, "FLX?NOK", 7 ) == 0 )
        {
          printf( "HANDSHAKE NOK [IMG REFUSED BY TARGET]\n" );
          return false;
        }
     }
        else printf( "NO REPLY\n" );
  return false;
}

bool  waitForToSet()
{
  char to[20];
  sprintf( to, "TO:%d\n", TARGET );

  bool rc = radio.send( (byte *) to, strlen(to), true );

  if( rc && radio.BUFFER_LENGTH > 0 )
  {
        printf( "Moteino: [%s]:%d:%d\n", (const char*) radio.data, strlen((const char*)radio.data), strlen(to) );

		if( strncmp( (const char*) radio.data, to, strlen(to)-1 ) == 0 )
          return true;
        else
          return false;
  }
  return false;
}


// return 0:timeout, 1:OK!, 2:match but out of synch
int waitForSEQ(int seq)
{
        if( radio.BUFFER_LENGTH > 0 )
        {
                printf( "Moteino: [%s]\n", radio.data );
                if( radio.data[0] != 'F'  || !isdigit( radio.data[4] ))
                        return 0;
                int s;
                s = atoi((const char*)  &radio.data[4] );
                if( s == seq )
                        return 1;
                else
                        return 2;
        }
  return 0;
}

int updateDeviceModule()
{
  char tx[124];

  fp = fopen( deviceModuleUpdateFilename, "r" );

  if( fp == NULL )
  {
        return 2;
  }

  int retries = 2;
  if( waitForToSet() )
      printf( "TARGET SET OK\n" );
  else
  {
      printf( "TARGET SET FAIL, exiting...\n" );
      fclose( fp );
      return 1;
  }


  if( waitHandshake(false) )
  {
      int seq = 0;

      while( !feof(fp) )
      {
        if( fgets ( line, 118, fp ) == NULL )
                break;

        trimLine( line );
        printf( "%s\n", line );
        if( validateHEXData( line + 1, strlen(line + 1)  ) == 0  )
                printf( "INVALID HEX DATA :-o" );

        sprintf( tx, "FLX:%d%s\n", seq, line );

        bool isEOF = ( strncmp( line, ":00000001FF", 11 ) == 0); // this should be the last line in any valid intel HEX file
        int result;

        if( !isEOF )
        {
                printf( "TX > %s", tx );
                radio.send( (byte *) tx, strlen(tx), true );
                result = waitForSEQ(seq);
        }
        else if( waitHandshake(true) )
        {
                printf( "SUCCESS!\n" );
                fclose( fp );
                return 0;
        }
        else
        {
                printf( "FAIL, IMG REFUSED BY TARGET" );
                fclose( fp );
                return 3;
        }
        if( result == 1 )
                seq+=1;
        else if( result == 2 )
                continue; // out of synch, retry
        else
        {
                if( retries > 0 )
                {
                  retries -= 1;
                  printf( "Timeout, retry...\n" );
                  continue;
                }
                else
                {
                  printf( "TIMEOUT, aborting...\n" );
                  fclose( fp );
                  return 4;
                }
        }
      }
      while( 1 )
      {
          if( !radio.receiveDoneRaw() )
                break;
          printf( (const char*) radio.data );
      }
    }
    else
    {
        printf( "No response from Moteino, exiting.\n" );
        fclose( fp );
        return 1;
    }
    fclose( fp );
    return 0;
}

[thinkpeace]

Here's the code on the gateway Moteino.  It's a combination of the Pi Gateway and the wireless programming gateway.  It's using Felix's WirelessHEX69 library.

Code Select Expand

// RFM69 gateway sketch based on the LowPowerLab.com gateway sketch found
// here: https://github.com/LowPowerLab/RFM69/blob/master/Examples/Gateway/Gateway.ino
// Passes through any wireless received messages to the serial port & responds to ACKs

#include <RFM69.h>
#include <SPI.h>
#include <SPIFlash.h>
#include <WirelessHEX69.h>

#define NODEID        1    //unique for each node on same network
#define NETWORKID     33  //the same on all nodes that talk to each other
//Match frequency to the hardware version of the radio on your Moteino (uncomment one):
#define FREQUENCY     RF69_433MHZ
//#define FREQUENCY     RF69_868MHZ
//#define FREQUENCY     RF69_915MHZ
#define ENCRYPTKEY    "kd8w3a0mz8q01gbh" 
#define IS_RFM69HW    
#define ACK_TIME      50 // max # of ms to wait for an ack
#define SERIAL_BAUD   115200

#ifdef __AVR_ATmega1284P__
  #define LED           15 // Moteino MEGAs have LEDs on D15
  #define FLASH_SS      23 // and FLASH SS on D23
#else
  #define LED           9 // Moteinos have LEDs on D9
  #define FLASH_SS      8 // and FLASH SS on D8
#endif

#define TIMEOUT     3000
// #define DEBUG true;

RFM69 radio;
SPIFlash flash(FLASH_SS, 0xEF30); //EF30 for 4mbit  Windbond chip (W25X40CL)
bool promiscuousMode = false; //set to 'true' to sniff all packets on the same network
int targetID = 0;

void Blink();

void setup() {
  pinMode( LED, OUTPUT );
  Serial.begin(SERIAL_BAUD);
  delay(10);
  Serial.println( "scGateway version 1.2.1" );
  delay(10);
  radio.initialize(FREQUENCY,NODEID,NETWORKID);
#ifdef IS_RFM69HW
  radio.setHighPower(); //only for RFM69HW!
#endif
  Serial.println( "Setting EncryptKey" );
  radio.encrypt(ENCRYPTKEY);
  radio.promiscuous(promiscuousMode);
  //radio.setFrequency(919000000);
  char buff[50];
  sprintf(buff, "\nListening at %d Mhz...", FREQUENCY==RF69_433MHZ ? 433 : FREQUENCY==RF69_868MHZ ? 868 : 915);
  Serial.println(buff);
}

byte ackCount=0;
uint32_t packetCount = 0;

char inputBuffer[100];
int inputIndex = 0;

void parseMsgToNode();
void processInputString();

bool serialDataAvailable()
{
  if (Serial.available() > 0)
  {
    char input = Serial.read();
    if( !isspace(input) || inputIndex > 0 )
    {
      if( input=='\n' || input=='\r' )
      {
        inputBuffer[inputIndex] = 0;
        return true;
      }
      else
      {
        inputBuffer[ inputIndex ] = input;      
        inputIndex++;
      }
    }
  }
  return false;
}

void loop() {
  //process any serial input
  if( serialDataAvailable() )
  {
        processInputString();
        inputIndex = 0;
  }
  
  if (radio.receiveDone())
  {
    if (radio.ACK_REQUESTED)
    {
      radio.sendACK();
    }
    Serial.print("#[");
    Serial.print(++packetCount);
    Serial.print(']');
    Serial.print('[');Serial.print(radio.SENDERID, DEC);Serial.print("]");
    for (byte i = 0; i < radio.DATALEN; i++)
      Serial.print((char)radio.DATA[i]);
    Serial.print("[");Serial.print(radio.RSSI);Serial.print("]");
    Serial.println();
    
    
  }
  Blink();
}

void processInputString()
{
  switch( inputBuffer[0] )
    {
      case 'T':
          if( inputBuffer[1]=='O' && inputBuffer[2] == ':' )
          {
              targetID = atoi( (const char*) &inputBuffer[3] );
              Serial.print( "TO:");
              Serial.println( targetID );
             
          }
          break;
      case 'F':
          {
               if( targetID == 0 )
               {
                   Serial.println( "TO?\n" );
               }
               else if( inputBuffer[1] == 'L' )
               {
                    CheckForSerialHEX((byte*)inputBuffer, strlen( inputBuffer ), radio, targetID, TIMEOUT, ACK_TIME, false);
               }
                break;
          }
 
      case '#':
        parseMsgToNode();
        break;
  }
}

void parseMsgToNode()
{
  char *b = &inputBuffer[1];
  char *sendBuffer;
  
    if( *b++ != '[' )
    {
        Serial.println( "missing [1" );
        return;
    }
    
    long targetNode = strtol( b, &b, 10 );  // Parse target node number
    if( *b++ != ']' )
    {
        Serial.println( "missing ]1" );
        return;
    }
    if( *b++ != '[' )
    {
        Serial.println( "missing [2" );
      return;
    }
      
    sendBuffer = b;
    
    while( *b && *b != ']' )
      *b++;

    if( *b != ']' )
    {
        Serial.println( "missing ]2" );
      return;
    }
    *b = 0;
    
    Serial.print( "Sending:" );
    Serial.println( sendBuffer );
    Serial.println( b-sendBuffer );  
    radio.send( targetNode, sendBuffer, b - sendBuffer );    
     
}    


}

[Felix]

thinkpeace,
For the long run I would recommend keeping a separate dedicated node for wireless programming. That allows the rest of the network to be uninterrupted because of a busy gateway.
Also for optimum performance verify you have the latest code and libraries. Start with 2 nodes load them with example sketches and verify those work before you move onto more complex sketches that involve wireless programming. One step at a time makes it easy to understand and isolate any issues.

[thinkpeace]

If it is a busy gateway, wouldn't the radio traffic interfere even if it was a dedicated wireless update note?

[Felix]

No, the WP shifts frequency to another channel to leave the original channel free of the heavier WP traffic.

[thinkpeace]

Does it affect the range when shifting the frequency?

[thinkpeace]

Thank Felix.

I test the wireless programming examples and then worked my way back to my implementation.  It turns out the node moteino was running out of memory. 

I will try the frequency shift to see how that works.

[Felix]

No, range is not really affected, we only switch by a few Mhz (defined by SHIFTCHANNEL):

From WirelessHEX69.cpp:

Code Select Expand

#ifdef SHIFTCHANNEL
boolean HandleWirelessHEXDataWrapper(RFM69 radio, byte remoteID, SPIFlash flash, boolean DEBUG, byte LEDpin) {
  radio.sendACK("FLX?OK",6); //ACK the HANDSHAKE
  radio.setFrequency(radio.getFrequency() + SHIFTCHANNEL); //shift center freq by SHIFTCHANNEL amount
  boolean result = HandleWirelessHEXData(radio, remoteID, flash, DEBUG, LEDpin);
  radio.setFrequency(radio.getFrequency() - SHIFTCHANNEL); //restore center freq
  return result;
}
#endif