Wireless Boot in 2.5 Seconds / High-Speed Datastreams

[kobuki]

Instead of starting a new thread, start a new poll, using the button at the bottom. I think you can only create simple multiple choices polls. The most important is the largest possible size so something like Up to 5, 10, 15, 25, and "I need all of the RAM dammit" or something similar should be good.

[TomWS]

<...snip>

Note that the bootloader and the app are not tightly coupled at all. They are so independent of each other that you can install regular sample Arduino apps that don't know anything about the bootloader. I'd argue that's less tightly coupled than your solution.

I was thinking about this aspect and see that you did mention it.  So, from a mote's perspective, assuming that you're willing to pull it out of the wall one time and reprogram the bootloader (something not many people have done before), there would be NO change to existing apps other than removing the Wireless Programming library while keeping a snippet of code to reboot on demand.

The real burden would be on the gateway which would have to support two different wireless protocols (assuming you didn't have a dedicated reprogramming Gateway).  Have you looked at a gateway implementation that supports both (current RFM69 library and the code needed to reprogram on boot)?  I can't help thinking these would be mutually exclusive...

I need to find time to publish the code so you guys can check it out.

Yeah, that's a good idea...

Tom

[joelucid]

@Tom, yes, correct on the ease of use of the bootloader approach.

My raspberry uses one Moteino for both boot and regular messaging. The way that works is that the Moteino listens for normal packets 97% of the time and for boot packets the remaining 3%. Since a booting Moteino will persistently ping the server for about a second such a low allocation to boot packet listening works. Apps don't notice it at all - apart from a tiny increase in retransmits.

[TomWS]

@Tom, yes, correct on the ease of use of the bootloader approach.

My raspberry uses one Moteino for both boot and regular messaging. The way that works is that the Moteino listens for normal packets 97% of the time and for boot packets the remaining 3%. Since a booting Moteino will persistently ping the server for about a second such a low allocation to boot packet listening works. Apps don't notice it at all - apart from a tiny increase in retransmits.

Interesting.  What is the interval between switching modes, ie, how long is 97% and how long is 3%?   And you can switch modes of the radio that often without breaking anything?

Also, I was having some more thoughts on having to pull out a mote that's already been installed with the existing library.  Would you be able to load a new app using the current wireless programming library (since that's how the installed motes are configured) and that new app has the new bootloader attached and can load it? 

That load would obviously take longer than what you're talking about, but presumably you'd only have to do it once and you wouldn't have to uninstall an existing mote (a HUGE plus in my book!).

Tom

[joelucid]

@Tom, I currently use 550 : 15 ms. I haven't had a problem with it other than the one with the Irq handler being called in initialize (see second post this thread).

Unfortunately you can't change the bootloader without physical access to the moteino unless specifically enabled by the previous bootloader (or with lax fuse settings which would expose you to the risk of accidentally bricking the device). That really is a pain since it sort of forces you to solder on headers just for the initial boot programming - even if you're just building an otherwise tiny thermometer.

At least with my bootloader you only have to do it once.

[KanyonKris]

joelucid, excellent work! While your system may not work for very low power nodes (like those powered by coin cells, as TomWS pointed out), I can see your fast wireless reprogramming being very useful, especially when the system is under heavy development and needing many changes. I hope you will post your code so we can see the details of what you've done, and so others can use your method.

[ivars211]

Sounds really intriguing. Any chance of seeing the source code?

[joelucid]

I'm planning on making it available - but I need to do some cleanup before.

I used the bootloader in my weekend house this weekend and had significant transfer issues. It's really strange: here in Berlin I live in an apartment that has such massive walls that I need two wifi basestations to penetrate everything. Yet even at Powerlevel 0 my RFM69hw installs at 300kbit without a glitch everywhere.

At the weekend home even at powerlevel 31 more distant nodes don't get a response from the bootserver even though the house is made of wood and wifi has no problem at all.

Anyway I'm implementing graceful bitrate reduction now. On the upside I found a 300kbit setting that's even faster than what I had. Will probably do 300kbit -> 200kbit -> 55555 -> 19200. When that works and I have the code clean I plan to make it public.

[kobuki]

Joelucid, have you ever released the source code for this downloader? Sorry if it escaped my attention before but I haven't been able to find it anywhere yet.

[WhiteHare]

Joelucid, have you ever released the source code for this downloader? Sorry if it escaped my attention before but I haven't been able to find it anywhere yet.

+1

[joelucid]

I've completely rewritten it for use with coin cell motes so no release yet. It's starting to work again, still fixing lots of bugs though. But it definitely works well with coin cells now.

[WhiteHare]

@Felix,

Setting aside the bootloader issue and Joe's alternate paradigm, it would seem you could nonetheless adopt the majority of Joe's suggestions (below) into your existing wireless programming approach (i.e. using extra flash memory) so as to do much faster wireless programming, and you would incur no extra risks by doing so.  Even just by itself, the faster wireless programming speed would be a nice improvement.   Yes?

Since I wanted it quick I did the following:

- Asynchronous protocol

Latency quickly brings down performance with handshake-heavy protocols. So I'm just streaming over as many packets as fit into the 328p RAM and then wait for one response saying what arrived. That brings you down to ~14 acks for a 20k payload - much better than handshaking each packet

- Higher bandwidth settings

I found a set of RFM69 settings that allow fairly reliable transmission at 200kbit throughout the house. Here are the important parameters:

    /* 0x03 */ { REG_BITRATEMSB, RF_BITRATEMSB_200000},
    /* 0x04 */ { REG_BITRATELSB, RF_BITRATELSB_200000},
    /* 0x05 */ { REG_FDEVMSB, RF_FDEVMSB_300000},
    /* 0x06 */ { REG_FDEVLSB, RF_FDEVLSB_300000},
    /* 0x19 */ { REG_RXBW, RF_RXBW_DCCFREQ_000 | RF_RXBW_MANT_20 | RF_RXBW_EXP_0 },

- Transmit / Receive without Standby Pauses

If you send multiple packets in a row it is much faster to keep the radio in transmit mode and send one packet after the other than to switch off the transmitter between packets. I rewrote the RFM69 library to support these streaming cases. I also eliminated the interrupt handler since it kept causing me problems. I now poll instead.

When you don't go into Standby on the receiver side you also don't need to set RF_PACKET2_AUTORXRESTART which saves additional time each packet.

- RF_PACKET1_DCFREE_WHITENING

This parameter needs to be switched on to enable whitening of the data packets by the radio. Otherwise empty pages (0xff) will cause lots of dropped packets - it seems the radio needs some change in the data stream to work effectively.

- No busy wait in sendFrame

I've eliminated all the busy waits. No canSend, no busy wait for transmission ready.

----

With these changes (and I'm sure some I'm forgetting) I can transfer 20k of code in about 1.5 seconds. Together with the page erasing and flashing in the bootloader I flash 20k of code wirelessly in 2.5 seconds.

[Felix]

@Felix,

Setting aside the bootloader issue and Joe's alternate paradigm, it would seem you could nonetheless adopt the majority of Joe's suggestions (below) into your existing wireless programming approach (i.e. using extra flash memory) so as to do much faster wireless programming, and you would incur no extra risks by doing so.  Even just by itself, the faster wireless programming speed would be a nice improvement.   Yes?

Yes, it's something to really consider, however if I put this on my to-do list I will get to it sometime in the near future.

[TomWS]

@Felix,

Setting aside the bootloader issue and Joe's alternate paradigm, it would seem you could nonetheless adopt the majority of Joe's suggestions (below) into your existing wireless programming approach (i.e. using extra flash memory) so as to do much faster wireless programming, and you would incur no extra risks by doing so.  Even just by itself, the faster wireless programming speed would be a nice improvement.   Yes?

Yes, it's something to really consider, however if I put this on my to-do list I will get to it sometime in the near future.

Looks like a golden opportunity for some clever person to step in and provide a solution! 

[executivul]

Hello to all of you,
I'm building the automation for the house of my dreams. In this process I've come across this great site, thanks Felix.
Unfortunately I'm not using Moteinos but Mega 2560 for my project since I need far mor GPIOs than a 328p would offer, I need some 6/8 channel light controls with rotary encoders on each of them, 8 relay board, DHT, light sensor, sound sensor, PIR, and 8+1 RGB LEDs (8 for the rotary encoders which change color, and 1 for some "halo" light arround each panel).
I'm planning to have wireless program capability since getting to each node just to change a the color of a LED for example is quite a nuissance, not to mention getting to the ceiling ones controlling RGB LED strips.

The bootloader Joe created seems great since it does not require an external flash. Which would add cost and complexity to my wire rat's nest nodes.
I tried PMing Joe, but got no answer, maybe I'm not "old" enough on the forum to be able to send PM, the forum reported no error either.
So please Joe can you share your bootloader code with us? I want to use it as a starting point for developing my own and would be a great time saving.

Best regards,
Mircea.

PS. please excuse my English, not my mother's tongue