Author Topic: Wireless Boot in 2.5 Seconds / High-Speed Datastreams  (Read 111346 times)

joelucid

  • Hero Member
  • *****
  • Posts: 868
Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« on: March 19, 2015, 12:31:16 PM »
Hi guys,

I've been using a bunch of Moteinos lately - great platform! With the first 10 in use as various sensors / controllers throughout the house I got annoyed by having to run around to reprogram them. I also found the programming time of  >10s a major turn-off.

So I thought I'd give Felix wireless programming a try and ordered the next 5 Moteinos with flash. Thankfully I live in Germany and any order takes about 14 days to arrive here - so I started to experiment with alternatives.

In the end I put together a wireless bootloader that installs from an Raspberry Pi based install server directly to the atmega328p flash - no extra chip required. I also provide a feature in the bootloader that allows an app to install an updated bootloader - so you really never have to physically get to your nodes again.

Since I wanted it quick I did the following:

- Asynchronous protocol

Latency quickly brings down performance with handshake-heavy protocols. So I'm just streaming over as many packets as fit into the 328p RAM and then wait for one response saying what arrived. That brings you down to ~14 acks for a 20k payload - much better than handshaking each packet

- Higher bandwidth settings

I found a set of RFM69 settings that allow fairly reliable transmission at 200kbit throughout the house. Here are the important parameters:

    /* 0x03 */ { REG_BITRATEMSB, RF_BITRATEMSB_200000},
    /* 0x04 */ { REG_BITRATELSB, RF_BITRATELSB_200000},
    /* 0x05 */ { REG_FDEVMSB, RF_FDEVMSB_300000},
    /* 0x06 */ { REG_FDEVLSB, RF_FDEVLSB_300000},
    /* 0x19 */ { REG_RXBW, RF_RXBW_DCCFREQ_000 | RF_RXBW_MANT_20 | RF_RXBW_EXP_0 },

- Transmit / Receive without Standby Pauses

If you send multiple packets in a row it is much faster to keep the radio in transmit mode and send one packet after the other than to switch off the transmitter between packets. I rewrote the RFM69 library to support these streaming cases. I also eliminated the interrupt handler since it kept causing me problems. I now poll instead.

When you don't go into Standby on the receiver side you also don't need to set RF_PACKET2_AUTORXRESTART which saves additional time each packet.

- RF_PACKET1_DCFREE_WHITENING

This parameter needs to be switched on to enable whitening of the data packets by the radio. Otherwise empty pages (0xff) will cause lots of dropped packets - it seems the radio needs some change in the data stream to work effectively.

- No busy wait in sendFrame

I've eliminated all the busy waits. No canSend, no busy wait for transmission ready.

----

With these changes (and I'm sure some I'm forgetting) I can transfer 20k of code in about 1.5 seconds. Together with the page erasing and flashing in the bootloader I flash 20k of code wirelessly in 2.5 seconds.

As for size this is now around 3.5k for the bootloader. I'm pretty sure I could get it under 2k with effort but I don't need that yet. I also plan to add serial boot capabilities to the bootloader using the same protocol so I can boot the wired gateway Moteinos from the install server, so will probably end as 4k bootloader anyway.

The bootloader transmits a node ID, CRC of itself and of the application on each boot and the bootserver sends updated versions as needed. Apps are sent directly to the bootloader for flashing. If the bootloader needs updating an app that flashes the new bootloader gets sent. Once that's done the app gets installed again.

For robustness the bootloader / bootserver are completely seperate from RFM69. But I'm using standard RFM69 for apps. Therefore protocol changes can be implemented at the app level without impacting the boot system (in fact I reenabled encryption at the app level again yesterday without a glitch for booting). The boot server uses the same Moteino as the gateway and shares the radio. The bootserver receive gets 15 ms every 500 ms - or a 3% duty cycle of the radio, without impact for sensor data transmissions.

All in all this works reasonably well. I hope this will be helpful to others implementing streaming applications or interested in wireless bootloaders. Let me know what you think.

I'm sure the speed could be tuned quite a bit more - eg by using 255 byte data packets rather than the 64 bytes required by encryption. However I'm now happy with what I have and will focus in the immediate future on putting the next 12 Moteinos which have finally arrived to use :)

Joe

joelucid

  • Hero Member
  • *****
  • Posts: 868
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #1 on: March 19, 2015, 12:31:50 PM »
BTW one change request for Felix:

If I reinitialize the RFM69 lib after running my radio code in the bootloader sometimes an interrupt gets triggered in the RFM69 constructor. At that point selfPointer isn't yet set so the application restarts. What helps is to set

RFM69* RFM69::selfPointer = 0;

and then:

void RFM69::isr0() {
    if( selfPointer ) selfPointer->interruptHandler();
}

This probably best since we don't want a virtual function to be called in the ctor anyway. Could you add that so I don't lose it with the next library update?

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #2 on: March 19, 2015, 04:42:02 PM »
Hi guys,

I've been using a bunch of Moteinos lately - great platform! With the first 10 in use as various sensors / controllers throughout the house I got annoyed by having to run around to reprogram them. I also found the programming time of  >10s a major turn-off.

So I thought I'd give Felix wireless programming a try and ordered the next 5 Moteinos with flash. Thankfully I live in Germany and any order takes about 14 days to arrive here - so I started to experiment with alternatives.

In the end I put together a wireless bootloader that installs from an Raspberry Pi based install server directly to the atmega328p flash - no extra chip required. I also provide a feature in the bootloader that allows an app to install an updated bootloader - so you really never have to physically get to your nodes again.

<snip...>
Ok, let's see if I understand what you've done.   Paraphrasing what I 'heard':

You've modified Moteino nodes with a new bootloader.

On reset, the new bootloader does not run from flash, but, instead, contacts your gateway at the RaspPi and downloads an image?  What initiates this decision?

The new bootloader streams its image from the server, stuffs it into flash (presumably at its final address?), and then boots to it once the image is completely downloaded?

I'm sure I've missed something, but that's the general idea?

Tom

joelucid

  • Hero Member
  • *****
  • Posts: 868
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #3 on: March 19, 2015, 05:15:40 PM »
Yeah that's basically it.

Obviously the bootloader does run from flash. But it then contacts the server on every boot to potentially update the application or itself. Whether a new application or bootloader image is required is determined by matching the CRC16's of the images currently installed on the client with the ones set up on the server for the booting node. After application install or timeout the app is run.

Since the bootloader installs the app in the application section it's not required to install it via flash chip to avoid bricking the moteino: if the install fails the checksums will not match on restart and the installation will just be repeated. If not the app but the bootloader is updated that happens by installing an app which will then burn the bootloader. Since the app install is safe the bootloader install also requires no external flash (if the bootloader installation app install fails it will just be repeated).
« Last Edit: March 19, 2015, 05:19:29 PM by joelucid »

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #4 on: March 19, 2015, 05:50:31 PM »
Yeah that's basically it.

Obviously the bootloader does run from flash. But it then contacts the server on every boot to potentially update the application or itself. Whether a new application or bootloader image is required is determined by matching the CRC16's of the images currently installed on the client with the ones set up on the server for the booting node. After application install or timeout the app is run.

Since the bootloader installs the app in the application section it's not required to install it via flash chip to avoid bricking the moteino: if the install fails the checksums will not match on restart and the installation will just be repeated. If not the app but the bootloader is updated that happens by installing an app which will then burn the bootloader. Since the app install is safe the bootloader install also requires no external flash (if the bootloader installation app install fails it will just be repeated).
I don't want to pick on your last statement, because I think it's a second order problem (we wouldn't HAVE to update the bootloader this way), however, I'm trying to understand why you think loading a new bootloader is failsafe.  If the bootloader is being overwritten during download, and doesn't succeed, the app could try to reload it, but, if, for some reason, the system resets before the successful download then isn't the node bricked?  Like I said, this is not something I would lose sleep over, but if I'm missing something I'd like to know.

Re your streaming method, I like this idea for powered nodes. I'm struggling with whether this is energy efficient for battery operated nodes.  I suppose if we can determine that the receiving node is rarely transmitting, then this certainly offsets the sustained awake time on receive.

Where is your code?

Tom

joelucid

  • Hero Member
  • *****
  • Posts: 868
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #5 on: March 20, 2015, 02:46:41 AM »
Tom,

rg the bootloader update: I've written a minimal app that flashes a bootloader. If the server finds that a new bootloader has to be installed it takes that app and merges it with the new bootloader (essentially it just takes the bootloader code which resides at 0x7000-0x7fff and appends it to the app at 0x6000). The resulting app can install the bootloader without any wireless access since it already contains the bootloader code. Therefore the normal failsafe app install process guarantees failsafe transmission of the bootloader.

Rg battery: to minimize battery use you want to minimize send/receive/wake time. No better way to do that than flashing the whole app in 2.5 seconds. And yes, the receiving node transmits very rarely since it doesn't ack each package (only 1 ack every 24 packets at the moment).

The code isn't public yet. If there's enough interest I might make it available.


kobuki

  • Sr. Member
  • ****
  • Posts: 289
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #6 on: March 20, 2015, 06:04:36 AM »
I'm certainly interested. Most sketches are perfectly fine on the available 28k. The ability to program a node remotely without a flash part that's only used for remote flashing twice a year is very nice. I think the transmit rate could be lower to make transmission more stable. I know you use a CRC and it works perfectly within your house but such a high rate limits the programming distance of the nodes, while the most important usage scenario is reprogramming nodes at remote, inconvenient places, out in the field, etc. where they are not as close.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #7 on: March 20, 2015, 08:39:59 AM »
Joe, this all sounds fantastic. However I've read and re-read your post and I may be missing the obvious however I wonder how you'd deal with the fact that you need your packets to arrive in perfect order. Any dropped packets will corrupt the entire image. In my scheme, the external memory is used as temporary storage to ensure the entire image is received as expected, before moving it to its final destination in the AVR flash memory.
And if I understand this right, it's the node that initiates the download (ie gateway must have the image ready when the node wants it), which is exactly the reverse of what my scheme has - the node must be ready when a programmer node (typically not the gateway) wants to reflash it. This may not work in all situations. In my Moteino Framework, one idea was decentralization and autonomy, hence the wireless programming scheme I adopted. A small bootloader that is lean and doesn't depend on a certain radio implementation is also essential for this requirement, again a difference in purpose.

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #8 on: March 20, 2015, 08:40:55 AM »
Tom,

rg the bootloader update: I've written a minimal app that flashes a bootloader. If the server finds that a new bootloader has to be installed it takes that app and merges it with the new bootloader (essentially it just takes the bootloader code which resides at 0x7000-0x7fff and appends it to the app at 0x6000). The resulting app can install the bootloader without any wireless access since it already contains the bootloader code. Therefore the normal failsafe app install process guarantees failsafe transmission of the bootloader.
If I understand correctly, you're saying the new bootloader is appended to an application which, if successfully loaded, will run and copy the bootloader to its final address?  If so, yes, that should work nicely.

Rg battery: to minimize battery use you want to minimize send/receive/wake time. No better way to do that than flashing the whole app in 2.5 seconds. And yes, the receiving node transmits very rarely since it doesn't ack each package (only 1 ack every 24 packets at the moment).
The problem is the internal resistance of some batteries, eg CR2032.  It will work fine with 130mA peaks for 4mS if the proper capacitor is used.  It will not work at all if drained at an average of 16mA for 2.5 seconds.

The code isn't public yet. If there's enough interest I might make it available.
Fascinating, let us know when it's available.

Tom

joelucid

  • Hero Member
  • *****
  • Posts: 868
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #9 on: March 20, 2015, 09:13:40 AM »
@Felix, the protocol works like this: the bootserver sends 24 packets to the client and then waits for an ack which contains exactly which packets where received at the client. The client receives until it doesn't get a packet for 10 ms and then sends the ack telling the server what he got. The server resends any packets that weren't received by the client in the next packet burst.

As you see it's completely fault tolerant. With my own rfm69 code I typically lose less than 1% of the packets if I only transfer. That increases to between 1% and 2% if I program the flash because it takes variable time to program and so I sometimes drop packets.

Rg. the reversed roles of client/server: the server doesn't need to have the image ready at client boot time - in that case the client times out and starts its old version. I do support the server pushing an update to the client at the app level - basically I set up the new image and tell the client to reboot. In general my battery clients can't afford to keep the receiver on so they check in periodically with the server to report data or receive commands. Of course wired clients can keep the receiver on and allow for instantaneous push updates.

Having a simple and tiny bootloader is clearly a strong advantage of your solution. You buy that by putting the wireless programming support on the app level which has a couple of drawbacks: (a) A broken app will brick the moteino, (b) can't update the bootloader wirelessly, (c) standard Arduino Hex files can't be installed since they don't include the wireless programming support (d) flash chip required.

Joe
« Last Edit: March 20, 2015, 09:28:15 AM by joelucid »

joelucid

  • Hero Member
  • *****
  • Posts: 868
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #10 on: March 20, 2015, 09:23:14 AM »
@Tom, I think you've got it.

Interesting point about the internal resistance of some batteries. I haven't had that issue yet. Clearly if you can't take the whole app in one stream a bootloader based solution is no good and the flash chip / trickle download approach is preferable.

@Kobuki, yes, it would be nice to support slower bitrates, too, for more remote clients. A big motivator for me to work on this was to get programming turnaround times as short as possible - so I've tried to get the transfer speed as high as I could. It would be nice if the protocol could gracefully degrade down to lower speeds if RSSI or drop rates indicate probems.

BTW I noticed that drop rates drastically came down as I changed my code to stay in transmit for each packet burst. I haven't tested this in more difficult scenarios yet, but I suspect results could be better than using stock RFM69 for streaming data.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #11 on: March 20, 2015, 11:04:34 AM »
@Felix, the protocol works like this: the bootserver sends 24 packets to the client and then waits for an ack which contains exactly which packets where received at the client. The client receives until it doesn't get a packet for 10 ms and then sends the ack telling the server what he got. The server resends any packets that weren't received by the client in the next packet burst.

As you see it's completely fault tolerant.

So you write the packets as you get them into the flash because there is no temporary storage. The moment you write a single packet, the assumption is that you will get the whole image, there is no going back. If something happens in between and transmission drops the target is bricked. But the assumption is this will not happen in the short few seconds of transmission.

Having a simple and tiny bootloader is clearly a strong advantage of your solution. You buy that by putting the wireless programming support on the app level which has a couple of drawbacks: (a) A broken app will brick the moteino, (b) can't update the bootloader wirelessly, (c) standard Arduino Hex files can't be installed since they don't include the wireless programming support (d) flash chip required.
A potential problem with a custom bootloader is that there is duplication of code for the radio, increases the size multi fold, the masses will complain, more so if the size is considerably larger than the stock optiboot (512b).
I believe the bootloader should be write-protected at all times, this ensures the bootloader section cannot be overwritten by accident. This has also happened in the past when fuses were not set and the result was a bricked unit.

My opinion is that your solution will work great for those who can implement it in their project constraints and need a tight coupling between the application and bootloader, and where the special character of the bootloader will fit their requirements well.

kobuki

  • Sr. Member
  • ****
  • Posts: 289
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #12 on: March 20, 2015, 11:19:35 AM »
@Felix: I'd like to note that the footprint of your OTA flashing offering is quite significant, and it adds to the radio code which is always needed. I think that anyone who can afford using that, can afford the 4k booloader - and this alternative solution doesn't need an additional flash chip. OTOH, if I understood correctly, joelucid ensures that all blocks are written safely before rebooting the Atmega, using that TCP-like ACK algo and by employing CRC16 checking. Of course, there are many ways his solution can be refined. I'd like to see his code so we can dissect it and make suggestions for the improvements already ;)

joelucid

  • Hero Member
  • *****
  • Posts: 868
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #13 on: March 20, 2015, 02:29:21 PM »
@Felix, all pages are verified via CRC's before they are written to flash. So corruption can only occur if the transfer gets interrupted - say because of power failure. if that happens it's no problem: on next boot the app checksum is calculated and will not match the server checksum since the flash process was interrupted. So the bootloader just starts another download.

As for bootloader protection: I have the fuses set similar to what you have. What I do is I provide a method eraseAndProgramPage in the bootloader via an API that's accessible for applications. The app can use that to flash the bootloader - not something that will happen by accident. [Actually it's not quite that simple since you need a way to flash the bootloader page that contains the eraseAndProgramPage code. I've created two implementations of the method on different pages in the bootloader. The app burns all bootloader pages but the one that contains one of the methods. Then as last step it uses one of the two implementation of eraseAndProgramPage in the newly flashed bootloader to burn the last page.]

I've thought about the size issue a lot. It's possible to call bootloader code from an app so I had planned to do that to save space for the RFM69 code - but then as my implementation diverged that became unfeasible. On the other hand I was able to reduce the size of the radio code quite a bit by eliminating what I didn't need, throwing out the Arduino boiler plate code, making all methods static etc. There's really not that much left there.

Note that the bootloader and the app are not tightly coupled at all. They are so independent of each other that you can install regular sample Arduino apps that don't know anything about the bootloader. I'd argue that's less tightly coupled than your solution.

I need to find time to publish the code so you guys can check it out.

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Wireless Boot in 2.5 Seconds / High-Speed Datastreams
« Reply #14 on: March 20, 2015, 03:27:32 PM »
<...snip>
A potential problem with a custom bootloader is that there is duplication of code for the radio, increases the size multi fold, the masses will complain, more so if the size is considerably larger than the stock optiboot (512b).
<snip...>
We should take a poll on how much program storage our apps take.  In my experience, (and I've got some FAT apps  ;) , I don't come anywhere near filling up 32K of program space.  RAM, yes!  I'd really like to see a new version of the 328P with double or more of RAM, but program space, I'm not sure it's an issue.  And, as Joe points out, his slimmed down RFM69 code is probably smaller than the Wireless Programming library.

Does anyone know how to set up a poll?

I'd like to gather (for 328 type procs only - don't care about Mega, they have PLENTY!):
Program Storage: max, min, average
RAM (not including heap): Max, Min, average

Tom