OTA Handshake fails with any change to target code [SOLVED]

[William Coles]

After quite a bit of fun we have been able to program one Mega (target) with another (programmer) using a mac with the python OTA.py script. However a tiny change to the target program either an addition or a subtraction of one line of code will compile fine but will fail the handshake as shown. Different changes will fail at different places, but a given change fails repeatably.
Our chips are more than a year old and the boot loader has some minor changes, but this error occurs before the loader is exercised. It appears to be a checksum error, but not caused by RF noise as it is repeatable. Has anybody seen anything similar? Its almost like a word alignment problem. In one case we could remove 10 lines, but the 11th caused a crash. Suggestions welcome!

Bill Coles (new to OTA)

TX > FLX:261:3030F000495D31FD4052419302D0A9F7EACFB4E0A6959795879577956795BA95C9F700976105710508959B01AC010A2E0694579506 RX > FLX:261:OK
TX > FLX:262:2A312000479537952795BA95C9F7620F731F841F951FA01D0895F894FFCFF70EF70EF70E476F742075 RX > FLX:INV:254
Programmer reply 'FLX:INV:254' does not match expected pattern: 'FLX:([0-9]*):OK'
Timeout, retry...

[Felix]

It could be something else, if the hex gets compiled with variable record length for instance.
Although I think I coded that into the RFM69_OTA as I made changes to support MoteinoM0 OTA.

So is this at the END of the transmission? At the last packet before handshake?

I would be interested to fix this if there's a bug. Is the sketch you have share-able in any way?
Or can you reproduce with the Target sketch in a way that fails on your end?
As a last resort I would use the Programmer (assuming you use it unchanged) and your Target as a HEX if you can share that.

[Sean]

I seem to have encountered the same/similar issue. I had successfully got OTA working {very very cool!!}, also using ota.py on a mac.  But today, having made a small change to the sketch, I'm unable to send the updated sketch via OTA.

ota.py successfully sends a series of packets, then fails, albeit with a few retries. Below are the last good and first failed packets from the OTA terminal window.

The sketch being updated is very much a work in progress that I'm happy to share a little less publicly if helpful (mail?).

Best, Sean
(New to Moteino and even more so to OTA)

TX > FLX:367:3044CE0000464C583F4F4B00464C58494D473A00202D2041434B2073656E740044657669636520416464726573733A200054656D2E RX > FLX:367:OK
TX > FLX:368:2644FE007020433A20006E616E00696E66006F7666000D0A00464C583F4E4F4B3A4E4F464C41534800006F39 RX > FLX:INV:254
Programmer reply 'FLX:INV:254' does not match expected pattern: 'FLX:([0-9]*):OK' RX > Timeout getting FLASH image from SERIAL, aborting..
Programmer reply 'Timeout getting FLASH image from SERIAL, aborting..' does not match expected pattern: 'FLX:([0-9]*):OK'
Timeout, retry...

[Felix]

Without going into too much detail of your code, can you describe (if not share) what is the actual change which goes from working to breaking OTA?

[Sean]

From memory the changes were:

1. Added an unsigned int to a global struct
2. Added a readVcc function - from the post by TomWS in reply #8 on thread https://lowpowerlab.com/forum/moteino/accurate-moteino-battery-voltage-monitoring/
3. Added a call to readVcc in the loop function, and set the new unsigned int element of the global struct
4. Added some Serial.println statements around the line calling readVCC

[Felix]

Those changes should be insignificant assuming:

- you're not somehow doing any of that during the OTA transfer to interfere with it
- reading VCC should be done very seldomly, during OTA you should expect the Vbat to be affected a little (even if it may mostly recover when OTA is done)

I really don't see why those changes would breaking a working OTA node.
I suggest going back to the working code, and incrementally adding your changes and see where it fails.

[Sean]

OK, I'm trying to start from a known good state...

I'm now using the example programmer and target sketches, with network settings adapted for my environment (network ID, key, nodeid, gateway, frequency definition). I'm using the same hardware for the target. I've uploaded the target sketch to the device over a usb connection. Disconnected the device, and tried installing my sketch OTA using ota.py from the command line as before. 

I also downloaded the SPIFlash and RFM69 libraries from github again and copied the extracted folders (with the -master suffixc removed) into the Arduino libraries folder (having renamed the folders already there).

The error persists.

Seans-MBP:pythonLibs sean$ python ota.py -f FiveSensorsEtapeSend.ino.hex -t 99 -s /dev/tty.usbserial-DN02YR6B

TO:99
Moteino: [TO:99:OK]
TARGET SET OK
File found, passing to Moteino OTA Programmer...
FLX?
FLX?
Moteino: [FLX?OK]
HANDSHAKE OK!
TX > FLX:0:300000000C94EB000C94F1040C94CA040C9413010C9413010C9413010C9418050C9413010C9413010C9413010C9413010C941301E5 RX > FLX:0:OK


<SNIP.....>


TX > FLX:315:303B100018F0495D31FD4052419302D0A9F7EACFB4E0A6959795879577956795BA95C9F700976105710508959B01AC010A2E0694BF RX > FLX:315:OK
TX > FLX:316:2C3B40005795479537952795BA95C9F7620F731F841F951FA01D0895F894FFCF6D026D02FFFFFFFF2C01000038 RX > FLX:INV:254
Programmer reply 'FLX:INV:254' does not match expected pattern: 'FLX:([0-9]*):OK'
Timeout, retry...

TX > FLX:316:2C3B40005795479537952795BA95C9F7620F731F841F951FA01D0895F894FFCF6D026D02FFFFFFFF2C01000038 RX > FLX:INV:254
Programmer reply 'FLX:INV:254' does not match expected pattern: 'FLX:([0-9]*):OK'
Timeout, retry...

TX > FLX:316:2C3B40005795479537952795BA95C9F7620F731F841F951FA01D0895F894FFCF6D026D02FFFFFFFF2C01000038 RX > FLX:INV:254
Programmer reply 'FLX:INV:254' does not match expected pattern: 'FLX:([0-9]*):OK'
FAIL: timeout (are you running the latest OTA libs/sources?)

I have a small collection of Moteinos so I could try a different device, but I'm clutching at straws now.

Sean

[Felix]

I think this is software not hardware. But you could try another if that's a quick thing, just to eliminate assumptions.
Can you try the desktop OTA GUI instead?

Do not rename old libs, you have to REMOVE them from the Arduino folder. Ex: zip them up instead. I suggest using the Arduino IDE native library manager instead. It will also update libs for you as new releases are available.

Use the vanilla OTA examples, if there's a problem there then you have my full attention.

[Sean]

I don't (easily) have access to a Windows machine at the moment. I'll try a different moteino, and if that fails will remove Arduino IDE and reinstall. I might have made a mess there.

[Sean]

Changing the hardware had no effect (as expected).  Reinstalling Arduino IDE, re-downloading and re-installing the various libraries did the trick. The sample code works perfectly now.

Felix,
Thanks for your patience and advice.

Best,
Sean
 

[Felix]

No problem, I appreciate your effort to reinstall everything and follow up.
Makes me feel good I have solid products/libs and some really good users who are willing to put in a little work to solve problems

[Lukapple]

Sorry for resurrecting the old thread - @Sean did you have any more problems/errors after reinstalling IDE and everything else?
I've exactly same problem, which occurs often...
First OTA upload worked perfectly. Then I commented one part of code,
this one:

Code Select Expand

  radio.sendWithRetry(RING_NODE_ID, "RING", 4);

and after that change I get this error:

Code Select Expand

TX > FLX:262:30312000504920466C61736820496E6974204641494C2100546F7765724F66536175726F6E3636360053544152540053746172744B RX > FLX:262:OK
TX > FLX:263:30315000656400476F74205B005D203E2000464C58494D473A00464C583A25753A4F4B00464C583F4E4F4B3A4845583E33316B0036 RX > FLX:263:OK
TX > FLX:264:26318000554B415A2047505221004E455A4E414E4920554B415A2100202D2041434B2073656E742E0000B4F1 RX > FLX:INV:254
Programmer reply 'FLX:INV:254' does not match expected pattern: 'FLX:([0-9]*):OK'
Timeout, retry...

TX > FLX:264:26318000554B415A2047505221004E455A4E414E4920554B415A2100202D2041434B2073656E742E0000B4F1 RX > FLX:INV:254
Programmer reply 'FLX:INV:254' does not match expected pattern: 'FLX:([0-9]*):OK'
Timeout, retry...

TX > FLX:264:26318000554B415A2047505221004E455A4E414E4920554B415A2100202D2041434B2073656E742E0000B4F1 RX > FLX:INV:254
Programmer reply 'FLX:INV:254' does not match expected pattern: 'FLX:([0-9]*):OK' RX > Timeout getting FLASH image from SERIAL, aborting..
Programmer reply 'Timeout getting FLASH image from SERIAL, aborting..' does not match expected pattern: 'FLX:([0-9]*):OK'
FAIL: timeout (are you running the latest OTA libs/sources?)

[Sean]

@Lukapple - Everything started working as expected once I'd reinstalled the IDE and libraries; I suspect that at some point I downloaded an incompatible version of one of the libraries. I only dabble here from time to time (depending on the project to hand) and may have followed an out of date link at some stage and grabbed an older library. In anycase, once I got hold of the current libraries the issue went away. Good luck on your quest, Sean