UPDATE: See Step 2 for actual results.
Well, the title may sound somewhat misleading. But read on. After many days/hours of trial and error I was finally able to validate and transmit an intel HEX flash image to a remote Moteino, which stores it on an external SPI flash chip (soldered to the new SPI Flash footprint available on all Moteino R2s). This is the first step towards wireless programming, and probably the easier step.
I really hate to write bits and pieces of an unfinished solution and keep the readers in suspense/anticipation of what’s coming next (I see there’s a trend for this on some blogs, to increase blog hits and keep users “hooked”), but I think the wireless programming endeavor is rather long and complicated. Yesterday I achieved what I long wanted, to get a new sketch wirelessly transmitted and stored on a remote Moteino, for later interception and reprogramming through a custom bootloader.
For those unfamiliar with a HEX file, it’s the file generated by your Arduino sketch, and then uploaded to your Arduino/Moteino when you click “Upload”. It all happens behind the scenes and through the attached FTDI adapter. It’s a complex protocol that involves a handful of moving parts. The most important is the bootloader which lives on your Arduino/Moteino, which must be able to intercept the “Hey here’s a new program for you!” request and start re-programming the internal flash memory. But for all this you need wires!
Wireless programming of your Arduino/Moteino is useful when you have nodes sitting in trees, on top of your house, in your attic, underwater, underground, inside sealed enclosures, generally where it’s very hard to physically reach once you have a node deployed.
The easy part (of this first step – getting the flash image over the air) is transmitting the bytes. The hard part is doing it all reliably and integrally. A single corrupted, missing or misplaced byte and the remote target might become completely disabled. There are 3 parts:
- A python script runs on a host computer, reads the HEX image, and serially hands it over to the attached (perhaps master/gateway) Moteino, line by line, through a handshake protocol which ensures the file is received in order and integral
- The attached Moteino receives the image, validates the records size and checksums, and begins a similar over-the-air handshake communication with the remote Moteino (the target)
- The remote Moteino intercepts the communication and stores the received flash bytes to the local SPI flash chip. If anything fails during the transmission, it will erase the first 32K of the flash chip (to avoid a false positive or corrupt image). To make this all work I had to write a library for easy access to SPI flash memory chips, a small challenge in itself.
Communication can fail at any time in this chain of events, which will interrupt the image transmission. The remote Moteino has a timeout protocol that keeps track of everything and if messages stop coming in before complete transmission it will timeout and erase the data stored up to that point.
The second step towards achieving wireless programming is having a bootloader that is aware of the flash image and that can read it from the external flash chip and program the internal flash memory with it. UPDATE: this is achieved in a newer post.
The 2boots bootloader allows an Arduino/clone to reprogram itself when a flash image is detected on an attached SD card. Moteinos don’t have room for an SD card, but can accomodate a SOIC SPI Flash chip. This comes in handy for data logging and … wireless programming!
Unfortunately 2boots will not work for this (even though it’s SPI based) and after some research I gave up trying to find a bootloader that could read from an external SPI flash. For several reasons, I’m opting to modify the original Optiboot bootloader to read the image from the external flash chip instead. That is a complicated business since the one single most important and robust mechanism in any embedded system is the bootloader.
Constraints and assumptions for wireless programming:
- the transmitted flash image has to be a valid flash image, that matches the target MCU type. The only part that can be validated in this process is the actual integrity and structure of the HEX file records (based on their own headers and checksums).
- the master/gateway Moteino will need to run a chunk of code dedicated to intercepting a flash image from the host and transmitting it to the target. Hence it does not itself need an SPI flash chip
- the target Moteino will need to run a chunk of code dedicated to this wireless programming protocol, which can “listen” for a valid message and start the reprogramming cycle
- the target Moteino will need to have an SPI flash chip soldered on the dedicated SMD footprint
- the first 32K of this SPI flash memory are dedicated for the temporary storage of a potential wirelessly-received flash image. If data logging is desired, it will need to be done after the first 32K. This space is only reserved when a flash image is received and during re-programming, after which it’s erased.
You may ask why not just directly transmit the flash image to the internal ATMega flash memory? The answer is long and complicated, but the bottom line is:
- once you start erasing the original flash image, there’s no going back, if transmission fails, you’re left with half written or corrupt flash which will completely disable that node
- the bootloader is a very small piece of code, and a complex handshake protocol with supporting libraries will increase its size many times from the original 512Kbytes that Optiboot takes. Also the more code you add to a bootloader, the more potential bugs are introduced, which is not something you want in a reliable system
I’ll post progress about this, hopefully when I have a complete working solution. The next goal is to wirelessly reprogram a remote Moteino with the “Hello World!” of electronics, a blinky sketch. If I can achieve that I will once again feel that same satisfaction when I first blinked an LED!