Use an ESP8266 to OTA program a Moteino?

[WhiteHare]

Given how cheap ESP8266's are, and how seemingly difficult doing OTA programming is in practice, and given how fast ESP8266's are and given their much higher datarates, would it make any sense to tack one on and use it purely for programming a moteino?  Yeah, OK, forget coincell operation.  I can live with that loss.  Yeah, OK, so an ESP8266 isn't as power efficient.  So what? Have the Moteino briefly turn on the ESP8266 if and only if you want to do an OTA firmware update of said moteino, and then turn the ESP8266 off and forget it even exists.  Is it a waste of transistors?  Sure, but who cares?  Those are cheap transistors.     Besides, wouldn't using an ESP8266 be an incredibly FAST way to do the OTA programming?  At the very least, it could receive the programming code much faster, and that would eliminate at least some of the lag.  I mean, you can use the arduino IDE to program an ESP8266, and so this really  should be easy, right?  Arduino's programming arduino's: plenty of examples of how to do that.  So, notionally, why not treat an ESP8266 like an arduino and just brute force an ESP8266 to receive and perform an OTA programming of a Moteino?  It has 80K of user RAM, so it could (potentially) receive the entire Moteino firmware into memory and then dump it into the moteino when it's fully received and validated.

OK, yes, I'm spitballing this, but is there some obvious obstacle I'm overlooking?  If so, what is it?  Yes, I've seen the youtube video on doing the OTA moteino programming using flash memory, and--to be frank--it looked like a *lot* of steps.   So many, in fact, that hooking up the wires as I'm currently doing seems like it would be faster.  Isn't there an EASY, stupid simple way to do this?  I've got to believe that FAST and EASY OTA programming is on everybody's wish list, and yet how many of us are currently doing it?  Maybe I have the wrong perception, but the impression I get is that the number of moteino users currently doing it is very, very few.  I've got Moteino's with flash on them, so I guess I could, in theory, be doing it, but, like I've already said, it looks like a *lot* of steps, so it doesn't currently look like a win to me.  Maybe I'm totally wrong, and if so, I would be very happy to be wrong if, in reality, it turns out I can actually do OTA programming both EASILY and FAST.

So, let's take a poll.  Who's doing it?  Is it easy and fast?  Who's not doing it, but wish they could do it easy and fast?  Please answer by taking the pole here:  https://lowpowerlab.com/forum/index.php/topic,1348.msg9305.html#msg9305

[joelucid]

Using a esp8266 for firmware updates is completely unnecessary. The speed at which you can update a Moteino is not limited by the radio but by the flash on the 328p which will flash at about 30kb per second and chip speed.

Its possible to design a boot loader that updates a Moteino without external flash in a few seconds. Granted it costs you 4kb in flash but the overall solution is much simpler and more cost effective than adding another mcu. And it does work with coin cells 

[WhiteHare]

Using a esp8266 for firmware updates is completely unnecessary. The speed at which you can update a Moteino is not limited by the radio but by the flash on the 328p which will flash at about 30kb per second and chip speed.

Thanks.  Good to know.   

Looks like it's fairly easy for an Uno to program an Uno (https://www.arduino.cc/en/Tutorial/ArduinoISP). 

The sketch comes prebuilt into the Examples of the Arduino IDE.  Not quite on-point with what I need, but it's a possible starting point. 

[joelucid]

The best way to do it is to write a bootloader. That's a special piece of code that gets flashed at the end of the 328p's flash memory replacing optiboot using an ISP like the one you've posted here (Felix likes to point out though that you should really use an official atmel's programmer like the mkii instead - and after frying 2 moteinos with the moteino as ISP approach I agree).

The bootloader then gets invoked when the 328p is reset, can download the app and can use special instructions to self program the flash on the 328p. It can even replace itself that way - so you need the ISP only once. Check out the datasheet it's all explained there.

The challenge then is to make the procedure robust and fast and fit everything in 4K, the largest allowed bootloader. It can be done but it'll take some time.

[WhiteHare]

Found what I needed.  Here's a complete recipe for how to transparently and wirelessly program an arduino/moteino using an esp8266:

http://hackaday.com/2015/09/18/transparent-esp8266-wifi-to-serial-bridge/

Enjoy!

[WhiteHare]

Aliexpress has esp8266's for $1.80, delivery included.

I won't want the esp8266 to be running continuously, but only when it's needed.  According to one of the datasheets (http://wiki.iteadstudio.com/ESP8266_Serial_WIFI_Module), it "typically" consumes 500na when in "Total Shutdown," which would be OK by me, but the same datasheet only promises it to be something less than 10ua.  Notably, the DS also says it can typically draw up to 215ma when doing Wi-Fi, so I obviously can't rely on just a Moteino GPIO pin to power it.   

So...  I'm tentatively thinking I'll power the esp8266 ON as needed for sketch updates using an N-channel mosfet, and then OFF again after the update finishes.  Any favorites for an n-channel mosfet that has ultra low current when turned off? 

[joelucid]

I suspect using the esp8266 as transparent wifi/serial bridge for app installs will be pretty slow. The protocol isn't optimized for high latency environments so you'll get lots of round trips and slow transfer speeds.

But it should work - even if keeping an extra radio around just for installs feels a bit weird.

[jra]

I would use a P-channel MOSFET instead of an N-channel for the reasons outlined in http://forum.arduino.cc/index.php?topic=285286.0 and http://www.onsemi.com/pub/Collateral/AND9093-D.PDF.  The Fairchild FDV304P/FDV304N might be a reasonable choice but I still get the feeling that either the Moteino is going to be redundant or the ESP8266 is going to be redundant.

[WhiteHare]

I would use a P-channel MOSFET instead of an N-channel for the reasons outlined in http://forum.arduino.cc/index.php?topic=285286.0 and http://www.onsemi.com/pub/Collateral/AND9093-D.PDF.  The Fairchild FDV304P/FDV304N might be a reasonable choice...

Thanks for the steerage and the links!  I'll read up on that. 

Because the ESP8266 is such a power hog, I don't think the Moteino will be redundant in a battery operated scenario.  When coupled with listen-mode, the Moteino becomes super energy efficient.  Unless I'm missing something, I don't think the ESP8266 can compete with that.  I don't deny though that on its face their union has earmarks of an ugly kludge.  Doesn't matter:  it's cheap.  My objective is just a low-effort, immediate solution until something better comes along.

Besides, I have hopes that the ESP8266 will be an interesting add-on for doing wireless updates to the regular Arduino's that I have scattered about that run from mains rather than batteries.  I'd love not having to reconnect them by wire every time I need to make some minor update to a sketch.  Now that ESP8266 wi-fi is dirt cheap and (reportedly) works so well, I'd like to put it on everything!  You're right, though, that in those cases the arduino's might be redundant, given that the the ESP8266 MCU outclasses most Arduino's, at least in terms of memory, speed, crunch power, and some other dimensions too.  However, setting aside that technical truth and looking at the bigger picture, for $1.80 a pop for a simple retrofit, it's just irresistible!

[jra]

Did you have a particular application in mind?  For those Arduinos that you have powered from AC it should be straightforward to replace them with ESP8266's using Ivan's ESP8266 Arduino IDE unless they are connected to some particular sensor/display/device for which support is still lacking.  If you are planning on using MQTT there is an OTA update example that could easily be integrated into an existing MQTT environment.  You could MAYBE run it on solar/lipo if the duty cycle was low enough.  If you want to run using a much smaller battery and/or solar recharging is not an option and/or you need the range/penetration of a sub-GHz radio, I would use the Moteino OTA solution that has already been debugged.  I've definitely done more programming on ESP8266's over the past year than I have on AVR's (because they are bigger, faster, cheaper) but I don't run any of them on batteries.

[TomWS]

... but I don't run any of them on batteries.

LOL!!! Really? 

@jra, Seriously, given your experience with the ESP8266, where do you think it 'shines'?  Given the cost point and intrinsic WiFi/TCPIP capability, there has to be a sweet spot for this device.  Probably most nodes that are AC powered, be my guess.

Tom
UPDATE: Fixed typo

[jra]

@tomws  If your application can function within its limitations which are currently:

1) Voracious appetite for power (~300ma peak vs. 45ma peak for an RFM69W and 130ma peak for an RFM69HW)
2) Lack of I/O  (20 pins max including 1 HW UART and 1 ADC) relative to AVR/ARM-based Arduinos

I think the ESP8266 family is currently the cheapest/easiest way to get data from a variety of locations onto your WAN/LAN/Internet.  Although the modules are generally available for $3-4, they're not really usable as-is.  Most of them have castellated pins on 2mm centers so you need to solder it to some other carrier to make it breadboard friendly.  You also need a well regulated 3.3v power source, pull up/down resistors on key pins, etc.  The modules that do provide pins on 0.1" centers tend to have very few pins exposed.  Not a big problem if you plan on using it as an attached processor but you still need to address the power issue etc.  There are a number of  boards available as complete solutions (Adafruit Huzzah, NodeMcu, Electrodragon, Omilex) and even more available on oshpark.com or dirtypcbs.com if you want to solder your own.  Taking all of this into consideration, I think you should plan on the actual cost of a usable solution to be closer to $10 rather than $3-4.  Still inexpensive by Arduino standards unless you count the Pro-Mini/STM32F103 clones.

As an example suppose you want to monitor the temperature and humidity in your basement, 1'st floor and 2'nd floor and make it available via a web page.  One solution could be to buy:

Three Moteno R4's with RFM69W's and three Weather Shields (3x$20 + 3x$20 = $120)
Three 5V power supplies (a few bucks each or reuse the ones you may have lying around or use batteries)
One Raspberry Pi (maybe $50-60 including case, PS, WiFi dongle)

After learning how to use the Arduino IDE and maybe just enough Linux to be dangerous, you deploy the Moteinos to the three locations, connect one via serial to the Pi, program them using the code that Felix has already gratefully provided and you can be up and running in fairly short order for less than $200.

An ESP8266 option might be to buy:

Three ESP8266 boards (~3x$10) and three i2c HDC1008 temperature/humidity sensors (~3x$10) = $60
Three 5V power supplies (a few bucks each or reuse the ones you may have lying around)
One Raspberry Pi (maybe $50-60 including case, PS, WiFi dongle)

The code will be a little more work but all of the pieces are already available as examples. You could use a little finesse and set up an MQTT broker on the Pi along with MQTT clients on the ESP8266's or you could go old school/brute force and make the data available on each of the ESP's via http/port 80 and simply scrape it from the Pi if you are already Linux/Unix savvy.  If you are hesitant to learn Linux  and don't mind putting your data on the cloud, you can even skip the Raspberry Pi entirely by opening up a free account on https://thingspeak.com or https://data.sparkfun.com or https://m2x.att.com and push the data directly from your ESP8266's, also plenty of examples online how to do this.

If you want to add another sensor node using Moteinos (maybe to sense the ambient, freezer and refrigerator temperatures in a garage) I suspect you will need to modify code on both the Moteino receiver (unless you make it generic enough the first time around to listen for senser node ID's that haven't yet been established as well as pass arbitrary structures to the Pi gateway) as well as the Pi.  Since the ESP's in this example don't talk to one another, the only change would be on the Pi.  Or the cloud if you use one of the services mentioned above. Maybe that's a slight advantage.

When the ESP-01 first came out the model was to communicate with it via AT commands (like a modem) and connect it to your existing Arduino/Pic/whatever to access your WiFi network.  When the SDK became available, people started programming it in C/C++ and a variety of custom firmware packages became available including a Lua interpreter.  All of these mechanisms still have their advocates/users but I think the two things that really allowed this device to take off were:

1) The availability of modules with enough exposed pins to make them useful as standalone devices
2) The ability to program them using the Arduino environment

The latter allowed folks to leverage most of their Arduino experience and libraries.  If you were previously using an Arduino + a WiFi sheid to do some processing and get it on the WAN/LAN, you may be able to replace both with a single ESP with very minor changes to the code.  Where the ESP won't help is if you have a lot of analog sensors as there is only a single ADC on the ESP and to be honest I'm not sure how good it is.  You also only have a single HW UART.  This may not be as big a deal since SoftWare serial is available and bit banging a SW UART with an 80MHz processor is not particularly taxing.

This brings up another point.  If you have a sketch that is resource intensive (either RAM or flash or CPU speed), a 1284 AVR processor maxes out at 16K RAM/128KBytes Flash/20MHz CPU.  A Teensy 3.x (chosen as an example of a small footprint ARM-based board with Arduino support) has  64K RAM/256KBytes Flash/72MHz CPU.  An ESP8266 has 64K Instruction RAM (+96K Data RAM)/512K-4M Flash (all external)/80 MHz CPU.  If it doesn't fit on a Moteino Mega, you could maybe run it on an ESP for half the price of a Teensy even if you don't need/use the WiFi.  As an example, I compiled the following Blink sketch on a 328P Pro-Mini clone:

void setup() {
  Serial.begin(9600);
  pinMode(5, OUTPUT);
}

void loop() {
  Serial.println("5 on");
  digitalWrite(5, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(1000);              // wait for a second
  Serial.println("5 off");
  digitalWrite(5, LOW);    // turn the LED off by making the voltage LOW
  delay(1000);              // wait for a second
}

Reported resource usage was as follows:

Sketch uses 2,378 bytes (7%) of program storage space. Maximum is 30,720 bytes.
Global variables use 210 bytes (10%) of dynamic memory, leaving 1,838 bytes for local variables. Maximum is 2,048 bytes.

When compiled on an ESP-12e configured with 1M Flash allocated for sketches plus 3M Flash allocated for SPIFFS filesystem the result was as follows:

Sketch uses 203,652 bytes (19%) of program storage space. Maximum is 1,044,464 bytes.
Global variables use 33,034 bytes (40%) of dynamic memory, leaving 48,886 bytes for local variables. Maximum is 81,920 bytes.

Yeah, a lot of crud gets pulled in by default for WiFi support but you still have a lot left over.  Not sure how you would ever use all of it up in an embedded system byut my experience has been that an excess of resources leads to sloppy coding.  There are some new projects out there now in various stages of development including:

1) A Basic interpreter (maybe appeals to the BasicStamp affectionados)
2) A MicroPython interpreter
3) A Lisp interpreter (maybe someone wants to port emacs to the chip)

Like I said, I've probably spent more time programming ESP's over the past year than Moteinos but a lot of that has to do with the Moteinos I have already doing what I want them to.  An early limitation was lack of a stable programming environment but I believe that one has been largely overcome to the point that it is just as easy to program an ESP today as it is to program any other Arduino.  Bottom line for me is that the sweet spot for this device is any application within WiFi range that doesn't need to run on a battery.

@whitehare
If you are using the Arduino environment you will want to use @lmroy's MQTT port https://github.com/Imroy/pubsubclient.  One of the examples is the OTA update via MQTT https://github.com/Imroy/pubsubclient/blob/master/examples/ESP8266-OTA/ESP8266-OTA.ino.  If you are using @tuanpmt's standalone MQTT firmware https://github.com/tuanpmt/esp_mqtt there is a fork by @nqd that supports OTA https://github.com/nqd/esp8266-dev/.  I've used the Arduino MQTT library but haven't personally tried OTA yet, day job keeps getting in the way.

[TomWS]

@jra, WHOA!  Just when I was beginning to think you didn't care... 

Wow!  Great!  Uh, it's gonna take me a few minutes to work my way through this.  The first thing I'd like to know is HOW did you fit this in the 5K limit this forum has on a post!

EXCELLENT SUMMMARY!  Thank you!

The 2mm castellated pins don't pose an issue for me, I've got some XBee pins and sockets laying around so easily adapted.  20 pins sure beats RFDuino's 6 pins of I/O and I've done some usable stuff with that.   

Big surprise on the amount of code need to support the stack.  I guess this tends to be invisible on some of the Nordic devices, however, so hard to compare.  In any case, thanks again.  I'm sure many people will benefit from you're taking the time to write this!

Tom

[WhiteHare]

@jra

Thank you for your fantastic post!

Not really knowing where to start, I decided to get the Adafruit ESP8266 (http://www.adafruit.com/product/2471), but I'd be curious as to which of the dozen or so different ESP8266 models a savvy guy such as you likes the best.  Supposedly the ESP8266EX is FCC certified.  Also, according to the ESP8266EX datasheet, "Deep sleep power <10uA, Power down leakage current < 5uA," so maybe I can skip the MOSFET after all.   

[jra]

@tomws Didn't even know about the 5K limit, cut/paste | wc says 8K, maybe it's magic?  The 20 pins was for the ESP-12e but I'm not sure all of those are actually usable.  Unlike the AVR's, there is no internal flash on the ESP8266 so all flash is external.  The ESP-12e exposes the 4 SPI pins used to communicate with flash but there is some controversy as to whether you can use them to communicate with other SPI devices without screwing the pooch, see http://www.esp8266.com/viewtopic.php?f=32&t=4094 for details.

@whitehare The Adafruit Huzzah is a good start, it is actually an ESP-12 mounted on a board with all of the necessities and the IDE (at least the latest github version) has a specific entry for the Huzzah.  You will also want a USB-TTL converter that supports DTR and RTS so you can flash the device without having to press the reset/flash switches in the right sequence. My latest purchase (and current favorites) was five ESP-07's and five ESP-12e's from http://stores.ebay.com/tomyuen007/ along with nine https://oshpark.com/shared_projects/9CsYq2PY.  The ESP-07 and ESP-12(e) are nearly the same.  The only difference is the 07 has a chip antenna that can be disabled by unsoldering a coupling capacitor to enable the U-FL connector for an external antenna; the 12's have PCB antennas.  That is so I can put one inside my shed with the antenna outside.  The board is by the same German fellow whose link I pointed you to previously, see also https://oshpark.com/profiles/8n1.  You may be interested in his (still untested) newer boards that utilize LDO's with enable pins. Advantage to that solution is that you can power down the ESP plus the sensors in case they draw any leakage current.  The ESP-12 is FCC certified, don't know if that applies to the 12e as well.