transparent OTA programming

[WhiteHare]

Anyone here yet devised a setup where from the Arduino IDE the  wireless OTA programming is pretty much entirely transparent?  I'm indefinitely limited to using IDE 1.0.6, and so I have to search  around to find the HEX file in the temp directory, which slows things down.  I'd rather just press the compile+upload button in the Arduino IDE and have all the wireless programming just happen automatically without thinking about it.  I can imagine different ways that might be done, but maybe someone here has already blazed a trail?

[Edit: Presently I'm thinking the easiest way will be to setup (just once) a virtual serial port, which can be selected  from the Windows Arduino IDE just like a regular serial port.  At that point, the IDE (actually AVRDude) will be sending its interactions to an IP address, where a proxy would spoof AVRDude into sending the entire stream, and then turn around and program the atmega328p wirelessly.  I could monitor the success/failure of that on a console window, or by other means.  That may be easier to do in Linux, but I believe there may exist some serial-to-IP freeware packages in windows also.  Anyhow, I'm flexible.  I may soon want to be doing this from AVR Studio as well as just Arduino IDE.  Because of AVRDude timing issues, it may turn out that the proxy has to be either directly attached to the PC or else running on the PC itself.  Not ideal, but that would be a possible fallback.]

[joelucid]

I already suggested a solution to this problem: the Nifty Device, a boot server that connects to USB mimicking optiboot and relaying the packets to the node to be programmed. I'm considering it for my bootloader but it means adding a USB/serial bridge to the boot server. Which is not really required otherwise thus uncecessarily increasing cost.

I'd be interested to hear how important this feature is for others interested in wireless programming. As compromise I could add the feature but require an external ftdi bridge.

[WhiteHare]

Well, yes, that does sound nifty, but I was thinking more about what do I do in the meantime (the near-term or possibly medium-term), since I don't have and can't yet get a nifty device.  If you think you'll be releasing it very soon, I don't mind waiting, whereas if you might be releasing it at the end of the year, I probably want to utilize a stopgap of some kind in the meanwhile.

[joelucid]

sure. I'm just saying instead of writing a proxy connect a moteino to the serial and have it relay the packets. Same complexity as the proxy. But much better usability since no virtual ports needed.

[WhiteHare]

If it turns out no one else has already done it, then I will probably take a low-effort, low-risk approach:
1. Create a VM with the Linux version of Arduino IDE
2. Run the proxy on the same Linux OS. 
3. Have the proxy transmit via IP  to my esp-Moteino_wireless_programming_gateway, which will OTA upload to the target the usual way.  A Raspberry Pi Moteino (MightyHat or equivalent) could probably be used just as well for this purpose, but I haven't yet tried that.
4. Then I just WinSCP to the VM when I want to use the Arduino IDE.

IIRC, the main hitch in getting this to work transparently is that the AVRDude that's bundled into the Arduino IDE is an older release that has "issues" that interfere.  Offhand I don't recall what those issues are, but I do remember it was a sticking point.  The newer AVRDude in regular Linux works just fine.  Maybe that situation has since changed with a newer IDE release.  I need to check on that.  Nonetheless, I'm presently marooned on Arduino IDE 1.0.6, so there's the rub.

[Edit: One of the reasons I'm leaning toward a virtualized serial (IP) connection is that it's unproblematic for a VM manager, whereas USB might not be. However, if push came to shove, I could just dedicate an old machine to this rather than use a VM.  After all, it's just a stopgap until the Nifty Device becomes available. ]

[TomWS]

I'm considering it for my bootloader but it means adding a USB/serial bridge to the boot server. Which is not really required otherwise thus uncecessarily increasing cost.

I think it's safe to assume that anyone messing around with this stuff already has an FTDI cable so you should just need a 6 pin header on your Nifty device to connect to the IDE.  In fact, if you really want to save money just put down pads like Moteino does - I'm sure most users have LPL supplied 6 pin headers lying around - especially if you set this thing up 

Tom

[joelucid]

I think it's safe to assume that anyone messing around with this stuff already has an FTDI cable so you should just need a 6 pin header on your Nifty device to connect to the IDE. 

Yeah makes sense. Now if only that Moteino Mega would finally make it through customs ... Felix, why can Digikey get a package to me in 48 hours when every moteino has to sit in customs for two weeks?

[WhiteHare]

OK, so to keep this as simple as possible, and so that literally anyone can follow along if they want to, I'll start with a just a plain vanilla Linux computer and a USB-FTDI connection to a Moteino (as suggested above by JoeLucid). The other aspects (using VM's and virtualized serial connections and the rest of it) are just nice-to-have extensions and aren't essential.  I may get lucky if:
1. the latest Linux Arduino IDE turns out to use the current AVRDude, and
2. the latest Linux Arduino IDE turns out to correctly generate code (unlike, apparently, the Windows Arduino IDE's that are later than version 1.0.6).

Otherwise, I may need to consider a different approach: since it's just a throw-away stopgap, the amount of effort needs to remain as low as possible.

[Felix]

Now if only that Moteino Mega would finally make it through customs ... Felix, why can Digikey get a package to me in 48 hours when every moteino has to sit in customs for two weeks?

Because:
- you probably paid a lot more for UPS shipping, they have a customs broker for their own packages, just like Fedex/DHL etc. USPS shipping goes through the government german post.
- regular German customs are the worst in the world, i'm not kidding.

[joelucid]

Actually with Digikey I never pay shipping. I just order enough to qualify for free express UPS delivery. My trick is to just fill up the tab with 328p's. You want to never order a penny more or less than the amount that qualifies for free shipping.

[Felix]

Well I'm a little surprised. So apparently $100 worth of parts gets you free shipping if you live in Germany, and I see other EU countries too. Good for you. But I assume you pay on the order of 30% import VATs.

I regularly order thousands of $ in parts from digikey, i never got free shipping, not even an upgrade. So no free shipping in the US, not that I know of.

FWIW It would cost me (a little drop in the bucket business) about $100 to ship a small package (2-3lb) via Fedex to Germany. From what I can tell the shipping companies won't even talk to a small business that doesn't ship thousands of packages per week/month, not worth their time. Digikey probably ships millions, so that's how they probably get those nice perks, I'm sure they don't offer you free shipping out of the goodness of their corporate heart.

[joelucid]

It's about 19% VAT and the prices are about the same. But I really like the free shipping offer. Shipping costs 25 Euro otherwise so this is a BIG discount. And the package arrives 2 days later - I really like them.

[kobuki]

I don't know if I've already described my own method, but I'm running 2 Moteinos attached to my Linux server. They normally run connected to a service (via TCP on ser2net), but when I stop the service, I can access them on my W7 workstation via a serial emulator. The Linux end is using ser2net, and the Windows end is using com0com. There's another emulator on Windows I've successfully used, can't recall its name ATM, but com0com works fine, I never had a problem with remote programming. It's a little cumbersome for service testing, because the connected service must be stopped/restarted/etc, but I tend to test locally and then upload via the mentioned method on the target device.

OTOH, IDK why you're still using 1.06 but later IDE versions feature a pluggable programming interface and you can just swap the programmer toolchain to one of your choosing, for any kind of OTA programmer that can interpret the avrdude parameters, for example. I think this is the correct method for transparent programming (eg. using the pluggable programming interface), instead of proxies, "Nifty Devices" and whatnot.

(You can still do that on 1.06, by replacing avrdude with an executable/wrapper you yourself write. There's a wrapper for that in windows IIRC. In linux it's a matter of writing a script in place of avrdude.)