Large File Wireless Programming

[adam]

Hey all,

I'm working with a modified version of Wireless Programming and have run into an interesting problem. I am able to successfully load and boot a small blink sketch but am getting a some corrupted data when I try a larger program. Here is some background:

I've been working with Moteinos for a while now. I actually mostly use the Moteino MEGAs because I've got lots of extra hardware hooked in. I'm also using the RFM95 modules for extra range. After a few years of having to keep track of the software running on all the units manually, I finally decided to give OTA programming a try.

As has been discussed before, the old https://github.com/LowPowerLab/WirelessProgramming repo and the RFM69_OTA both require the RFM 69 so I've been trying to work on a RFM95 compatible version. I decided to use the older version as a starting point because there was more transparency into the programmer side. Instead of jumping right into the RFM95 directly, I'm starting by just seeing if I can load a program onto FLASH over Serial and then add back in the PHY after that. I basically combined the HandleSerialHEX and HandleWirelessHEX methods in a single program while making the RFM95 radio portions transparent.

OK so now more details on the problem. I am able to successfully load the Blink.hex file (3.6 KB for me), but when I try to load my actual program file (48.6 KB), I apparently have a corrupted program. I'm working with a MoteinoMEGA with the most recent DualOptiboot bootloader for the 1284P. Even though the hex program I am trying to upload is large (3000 lines on the .hex file), I'm not getting any errors in the validateHEX process and since the Blinky sketch works, I'm not inclined to think that it's the reprogramming code that's the issue.

That leaves a much harder question and I wanted to get others help. Here's the issues I'm looking into:
1) Has anyone uploaded a sketch this large before? I see in the code there are limitations for 64 KB file uploads (though, on looking at it, this catch won't work btw. the bytesFlashed counter is just a uint16_t, so it won't really work properly when trying to count over 65535)
2) Is the DualOptiboot bootloader somehow limited in only taking the first 32 KB of the program?
3) Are there prohibited libraries/methods/macros that cause problems for the bootloader? (I really, really, really hope this isn't the case btw).

Anything else that could be an issue? Thanks guys I appreciate your help here.

[adam]

Update:

I made the Blink.ino sketch I was working with take up a roughly equivalent amount of progmem as my main application code by adding some long strings with the F() macro.

It seemed to continue to work fine. It doesn't seem that file size is the main issue here. Unfortunately (for me), it may mean going through my application method by method to try to figure out where the problem arises.

I wanted to see if anyone had an ideas. You'd think that if anything compiles to the intel hex file format, then it wouldn't really matter what the contents of the program are. Any insight or help would be amazing.

Thanks!

[Felix]

Hmmm, can't think of anything right now.

Back when I developed DualOptiboot there was a revision at one point but I can't remember the details exactly. My impression is it was because I moved to Optiboot 5.0. I recall there was a change to allow MoteinoMEGAs to work beyond the first 32K of the FLASHMEM. Either way the latest DualOptiboot hex should include this change.
It was verified to allow sketches up to 64K on MoteinoMEGA, which you seem to confirm in your second post.

FWIW I have to revisit OTA in general and make it better on the front end, but that's a bit unrelated.

[adam]

Felix,

Thanks for the reply. Good news is that I was finally able to get it to work. Bad news, not yet sure why. Based on my git diff, it looks like a few things changed.

1) I had originally implemented a flash.chipErase() to clear out the FLASH memory in place of the flash.blockErase32K(0) in the example. In the version that worked, I changed this to two 32K block erases, one starting at 0 and the other at 32768. Any idea why this could have been causing an issue? I know that the flash commands aren't blocking operations (right?), so maybe had started to move on to writing the FLXIMG start tag?

2) I re-burned the bootloader. Would there be any way for a bootloader to get corrupted by messing with the flash? It was clearly working at some level.

So far I can tell, all other changes were just adding in extra debug print statements to learn more about the problem. Any ideas you could throw out would be helpful. I'm scared of "phantom" bugs that may re-appear.

[Felix]

Adam,

1) If you look in the latest SPIFlash codebase, the command will wait for any write/erase to finish before issuing another command. So I don't think that's an issue.

2) Corrupting the bootloader is possible if fuses don't protect it. I've been shipping Moteinos with those fuses set for at least the past 2 years or so. But those can easily get reset by the end user.

Honestly OTA has worked very well for me and I haven't had any phantom issues that I could not explain.
But you should always have a backup scheme. OTA is meant to just work, and make it easy to .. OTA .. when non-OTA is a pain to do. But still OTA is not bullet proof because of many factors, and sometimes when things change you will find yourself having to access a node physically to do breaking upgrades.