A bootloader to bootstrap a bootloader

[TomWS]

'Inspired' by a comment in Joe's post: https://lowpowerlab.com/forum/index.php/topic,1347.msg9347.html#msg9347, where he suggests that you could have a self-modifying bootloader, it begs the question whether it's possible to install a simple variant of the original opti bootloader in which an application (AKA 'sketch') can have a replacement bootloader as part of its PROGMEM datablock and, on execution, 'escape' to the 'extra' code in the current bootloader that will, in turn, replace itself with the replacement from the datablock.

I believe the key is to have some 'escape' code in the existing bootloader that will overwrite itself since only a bootloader (or ISP programmer) can write to the upper memory block.  Presumably the replacement bootloader also incorporates the escape code.

This still requires a one time ISP programming, but would be more generic in providing the ability to anyone to 'easily' replace it with whatever flavor bootloader they prefer.

Tom

[joelucid]

Yeah that would work - my bootloader has this functionality. Effectively I have a routine that erases and programs a page exposed at a well known address in the bootloader. You need one trick to program the page that contains this routine since the currently executing page cannot be flashed: the avr machine code contains only relative addresses so it can be executed from any address. I first copy it's page somewhere else, install the new routine from that location and then use it to flash the rest.

Unfortunately there's one big issue: size of the bootloader is set by fuse. So you can only replace similar sized bootloaders. Optiboot is so small that you can't install something that supports OTA instead.

Well actually that's not completely true. One could have a small bootloader which calls into other memory for most of its functionality. You'd just have to tell gcc to put the jump table at the right location. So with some hacking your proposal could actually work. It'd be quite cool if Moteinos came that way since you could install other bootloaders without soldering headers or pogo jigs.

[TomWS]

Well, the bootsize issue could be handled simply by claiming more than the barebones loader needs.  We just need to know the largest program anyone wants to allow. ISTM I once had a poll along these lines... probably had the same responsiveness as WhiteHare's 

To my mind, program memory isn't the constraint with 328Ps.  As Kobuki pointed out, data memory is what is really constrained. 

So the sequence would be,
1) purchase a 'Moteino' with the barebones bootstrap-able loader,
2) play around with it as long as you want using normal IDE sketch load. 
3) When you're ready to deploy it to the field, build (or have prebuilt) and run a sketch that installs your new bootloader.
4) Re-install your device sketch, either with IDE or with your funky little MoteinoUSB wireless loader gizmo that works with the new bootloader you just installed.
5) operate your device for decades on a single coin cell, modifying the sketch code as needed.
6) 25 years later, when you find you need to upgrade your bootloader, wirelessly transfer that new bootloader sketch, reboot, and then begin again...

My question is, would you need to have a prebuilt image of the bootloader to include in the sketch or can that be built during the installer sketch build?

Tom

[joelucid]

Well, the bootsize issue could be handled simply by claiming more than the barebones loader needs.

Of course that would work - but then wired customers would pay a tax for the OTA crowd. Just something to be aware of. I'd prefer to have a 1k core bootloader and install an extension just below it for the Moteino use case. For Tino I'd just always install the OTA bootloader.

So the sequence would be,
1) purchase a 'Moteino' with the barebones bootstrap-able loader,
2) play around with it as long as you want using normal IDE sketch load. 
3) When you're ready to deploy it to the field, build (or have prebuilt) and run a sketch that installs your new bootloader.
4) Re-install your device sketch, either with IDE or with your funky little MoteinoUSB wireless loader gizmo that works with the new bootloader you just installed.
5) operate your device for decades on a single coin cell, modifying the sketch code as needed.
6) 25 years later, when you find you need to upgrade your bootloader, wirelessly transfer that new bootloader sketch, reboot, and then begin again...

My question is, would you need to have a prebuilt image of the bootloader to include in the sketch or can that be built during the installer sketch build?

Yes, the sequence could work. Of course you could also just directly install the OTA bootloader after unboxing and develop your app wirelessly.

I think you'd have to prebuild the bootloader image since it requires a couple of compiler/linker command line switches different from what you would use for the installer app.

In my setup I have a prebuilt version of the installer and if I want to install a new bootloader the server combines it with the installer into a new image on the fly and that then gets wirelessly installed on the mote.

[TomWS]

In my setup I have a prebuilt version of the installer and if I want to install a new bootloader the server combines it with the installer into a new image on the fly...

Is this a 'server' step or a linker step in building the new installer?

Tom

[joelucid]

It's a server step - I just read the two hex files and combine them in memory before sending. The bootloader code I map to an app address - say starting at 0x1000. The installer app then burns those pages into the bootloader pages starting at 0x7000.

[TomWS]

It's a server step - I just read the two hex files and combine them in memory before sending. The bootloader code I map to an app address - say starting at 0x1000. The installer app then burns those pages into the bootloader pages starting at 0x7000.

wakarimashita, arigato gozaimasu!

[Felix]

I resisted to reply because ... i had a crazy day and didn't want to say stupid things, also tried to think about it during the day.
I think it's not just awesome but also very ingenious. Assuming we could figure out how to get past the bootloader/fuse/boot mem locks this could be the next next best thing to sliced bread.
I will follow this thread closely and will let you guys iron out the details, I don't want to mess with your creative genius

[TomWS]

I don't want to mess with your creative genius

Geez, no!  Don't mess with that!  It is FAR too fleeting to deal with any challenges!

I think Joe's notion of going minimalistic on core bootloader size and then loading below it is a very good tradeoff.  The only one's who will complain about this are the one who think some rogue sketch will be able to totally subvert your bootloader - you know, guys like me 

Seriously, I think this is a practical and, for most of humankind, totally acceptable approach.

Now, if I can only figure out how to subvert the bootloader AND set the fuses to lock it in place!  Boo Ha Ha Ha!

Tom

[joelucid]

What I would suggest is to build a special variant of optiboot that does not contain the flash code of DualOptiboot so it all fits into 1k. Then Felix could just provide a binary hex file for download which replaces that with DualOptiboot for people using external flash.

It's really pretty trivial to do this. Here's the code that needs to be added to the bootloader:

Code Select Expand

extern "C" void writePage( uint16_t address, uint8_t* source ) __attribute__ ((section( ".eap1" ) )); 
void writePage( uint16_t address, uint8_t* source )
{
	uint8_t sreg;
	sreg = SREG;
	boot_spm_busy_wait();
	cli();
	boot_page_erase( address );

	uint8_t i;
	uint16_t addrPtr = address;
	i = SPM_PAGESIZE / 2;
	do {
        uint16_t a;
        a = *source++;
        a |= (*source++) << 8;
        boot_page_fill(addrPtr,a);
        addrPtr += 2;
	} while (--i);
	boot_page_write(address);
	boot_spm_busy_wait();
	boot_rww_enable();
	SREG = sreg;
}

Add the following linker flags:

Code Select Expand

-section-start=.eap1=0x7f80,-uwritePage

Then the installer can flash a page to address 0x7000 like this:

   

Code Select Expand

        extern "C" typedef void (*func)( uint16_t, void*);
	
        uint8_t page[ 128 ];
        uint16_t eap1 =  0x7f80 / 2;
	(*(func)eap1)( 0x7000, page );

That should be it really. Maybe writePage would even fit with DualOptiboot into 1k which would be even simpler. The complexity of creating the "extended" OTA bootloader would then be completely independent from the core Moteino bootloaders.

It would be a rule that any bootloader that wants to be replaceable comes with the writePage function at 0x7f80. It would probably be smart to add a signature to the bootloader somewhere well known so that an installer can verify that it's dealing with a replaceable bootloader. I used to have that but gave it up for space reasons.

If there is still space it might make sense to force a two step writePage procedure where you first have to call enableWritePage, then writePage. That would significantly reduce the odds of writePage corrupting your flash when called accidentally by a rogue Moteino.

[TomWS]

Thanks Joe!  This should give us all something to do on a dark and stormy day!

It actually might fit in DualOptiBoot if the current page write code is changed to call the function instead of the current inline code. assuming it can be knitted in at the fixed address and not create any significant gaps. 

UPDATE: It occurred to me that if a jump to the writePage function was placed at the very end of memory (eg 0x7FFC - 0x7FFE is taken with version number) then this would avoid any gaps and would be a well known address to call.

Tom

[WhiteHare]

FWIW, along the lines of what Tom was sayiing earlier, I wouldn't mind giving up flash space if it means simpler setup or usability.  The default should really be OTA programming straight out the box.  It's just plain silly not to. 

[Felix]

The default should really be OTA programming straight out the box.  It's just plain silly not to. 

All these years ... silly me

[joelucid]

UPDATE: It occurred to me that if a jump to the writePage function was placed at the very end of memory (eg 0x7FFC - 0x7FFE is taken with version number) then this would avoid any gaps and would be a well known address to call.

Yes, that's how I had it initially, too. I think I had changed it to absolute addresses on page boundaries when I realized I couldn't flash the page that contained writePage using writePage. My first workaround was to have two versions of writePage on different pages and I would flash everything but the writePage page using writePage, and the writePage page using writePage2. So both functions needed to be on different pages.

Then I came up with the trick of copying the writePage page to a different location and flash the writePage page using that copied code. But I never switched back to the pointer approach. Oh - I think I remember why: it's also nice to know that writePage occupies exactly one page because otherwise copying it elsewhere is a headache. No way to ensure that if its just part of .text.

But of course one could also just copy two adjacent pages and get rid of the headache that way.

Boy, lots of pages 

[joelucid]

The default should really be OTA programming straight out the box.  It's just plain silly not to.

Problem with that is it requires the server side - or that nifty new Moteino USB based device.

There's also something else to consider:

I initially created a completely different network for installs which didn't use encryption. The idea was to have the bootloader be completely independent of the protocol spoken on the sensor network. My server listened on both networks using a very low duty cycle for the boot network. That required a client on boot to send boot requests for a solid 1-2 seconds to ensure the server got the message. But the bootloader didn't need to know anything about the sensor network.

You see where this is going: an absolute no-no for coin cells. So I've modified the bootloader to send the boot request on the normal sensor network. Which requires it to know network id, server id, encryption key, bandwidth settings etc. Now it works great with coin cells, but it's no longer generic.

Two possible solutions for customer deployments: either require the NIFTY DEVICE and still use the dedicated network. Or configure the bootloader prior to using it (which likely means serial - so defaulting to the non OTA bootloader).