Data Structure of OTA Hex Data

[RatTrap]

I've been studying the over the air programming method and it seems pretty straight forward.  The first 10 spaces on the flash memory let the bootloader know if there is a valid program image stored on the SPIflash chip then proceeds to reprogram the MCU flash memory from what is stored on the chip.  What I am not understanding is the process by which the rfm69_OTA library is altering the HEX file made by Arduino, into the data that is being written to the flash chip.  Simply setting the first 10bytes of the flash to   FLXIMG:SizeByte1,SizeByte2:  and then writing the hex file compiled by Arduino definitely does not work, also if it did the OTA programmer would likely be far simpler.  In any event I am studying the code and I can't quite tell what it is doing.  To my amateur eyes it looks like it is casting the HEX file as a char array then doing some sort of reduction on the char array to turn 2 bytes into 1 byte.  I'm just trying to understand this, I am going to have to pass my OTA updates to the nodes via a master node that will get the data from the cellular network.  I can see that most of the programming code is more about data integrity which makes sense, and my current system I can ensure the quality and accuracy of the transmission and that the data reaches the node and is correctly saved to the node.  I just can't quite figure out how to modify the Arduino HEX data into the correct format for the Bootloader to reprogram the node.  Any advice would be awesome.

[Felix]

The compiled hex is in this format: https://en.wikipedia.org/wiki/Intel_HEX
Also, we're not writing hex, it would be very inefficient and waste a lot of space, we're practically writing the bytes that then get written directly in the internal AVR flash.

[RatTrap]

thank you for the link and the help.  One day I am going to stop letting simple things confuse me. lol
Now for the easy part of coding my own software to handle the cellular uploads, downloads, and remote programming from my server.  YAY!!! ;p

Thank you again Felix always there to slap me with just what I need to know.

Charlie 

[jbdww]

Been also trying to understand what gets written to the flash. Here is what I have in 256 bytes of my flash before rebooting.

46.4C.58.49.4D.47.3A.15.E0.3A.31.30.30.30.30.30.3 0.30.30.43.39.34.38.31.30.30.30.43.39.34.41.42.30 .30.30.43.39.34.41.42.30.30.30.43.39.34.41.42.30. 30.45.45.31.30.30.30.31.30.30.30.30.43.39.34.41.4 2.30.30.30.43.39.34.41.42.30.30.30.43.39.34.41.42 .30.30.30.43.39.34.41.42.30.30.42.34.31.30.30.30. 32.30.30.30.30.43.39.34.41.42.30.30.30.43.39.34.4 1.42.30.30.30.43.39.34.41.42.30.30.30.43.39.34.41 .42.30.30.41.34.31.30.30.30.33.30.30.30.30.43.39. 34.41.42.30.30.30.43.39.34.41.42.30.30.30.43.39.3 4.41.42.30.30.30.43.39.34.41.42.30.30.39.34.31.30 .30.30.34.30.30.30.30.43.39.34.41.42.30.30.30.43. 39.34.41.42.30.30.30.43.39.34.44.41.30.32.30.43.3 9.34.41.42.30.30.35.33.31.30.30.30.35.30.30.30.30 .43.39.34.41.34.30.32.30.43.39.34.37.41.30.32.30. 43.39.34.41.42.30.30.30.43.39.34.41.

This is the code I am using to read and print the flash to serial.
            while(counter<=256)
            {
              Serial.print(flash.readByte(counter++), HEX);
              Serial.print('.');
            }

Here is a few lines of my hex file.
:100000000C9481000C94AB000C94AB000C94AB00EE
:100010000C94AB000C94AB000C94AB000C94AB00B4
:100020000C94AB000C94AB000C94AB000C94AB00A4
:100030000C94AB000C94AB000C94AB000C94AB0094
:100040000C94AB000C94AB000C94DA020C94AB0053
:100050000C94A4020C947A020C94AB000C94AB00A8

Once the board reboots it flashes the LED three times, and a steady on.

[Felix]

See my post above.
The first 10 bytes of that hex dump are the signature FLXIMG:xx: where xx is a 2 byte integer representing the # of payload bytes of raw compiled sketch that follows (in your case 15E0) and which should match the number of data bytes in the hex file. The hex file is in a certain format that also contains a lot more than just the payload - headers, addresses, CRC bytes, all of which are stripped before putting it back to back for transmission and storage into the external flash memory. And also the hex file is actually stored as hex ascii rather than raw bytes as in the case of the OTA transmitted and stored bytes. This is why when you Serial.print you need to tell it to interpret the bytes as ascii HEX and not as ascii chars - ie each byte becomes two 0-9,A-F characters.

[jbdww]

Got it all figured out and working. Thank you for the tips. I missed a step.