MightyHat R3

A new revision of MightyHat has started to ship out. Most everything is the same. Here is a rundown of the most significant changes:

  • the slide switch that cuts off the battery is replaced with a more robust hooded switch that also has a longer slider which can reach out of a 1/8″ thick acrylic case
  • The RST pin of the LCD is now connected to the RST of the atmega328, (previously wired to A1). This saves the A1 pin and resets the LCD whenever the atmega is reset. This is the only change posted in the RFM69 MightyHat sample sketch and I added a directive setting to make it easy to switch between R2 and R3.
  • enlarged the slot in front of the battery connector for easier wrapping of the battery wiring
  • added a capacitor footprint on the 5V* output rail. This makes it easier to add more capacitance to the boosted voltage when needed.
  • other various cosmetic and silkscreen adjustments

Wireless Programming just got 50% faster!

The Wireless Programming GUI app that I introduced in an earlier post is now at version 1.2 (get exe here). It features the ability to bundle up to 3 HEX file lines per RF packet (instead of the default 1), which yields a significant transfer speed increase. Let’s just do the TL;DR first and gaze at the results of transferring a compiled sketch of 13,858 bytes:

  • running with 1 line per packet (the same as before) yields 866 packets and ~43s OTA time:
  • running with 3 lines per packet yields 290 packets and ~18s OTA time:

That’s an average 58% speed gain, not too bad!
By comparison, the same sketch uploaded directly via FTDI/serial takes about 13s, so much closer to that figure than before. I think a few other techniques could be used to squeeze yet more speed (increasing RF bitrate!) but that’s for another day.

Note that the same exact data is transferred, except in fewer longer packets. I had to make some changes to the WirelessHEX69 library to support these longer packets. Also I fixed some bugs that emerged during this development, see the github commit for all the changes.

To support the new GUI v1.2 speed increase you only need to update your WP programmer Moteino with a new version of the WirelessProgramming_gateway sketch (to refresh the WirelessHEX69 code) to make it support the faster WP speed (ie also update the WirelessHEX69 lib!). Everything else is completely backwards compatible, and you can use the new GUI v1.2 with an older WP programmer Moteino if need be (1 line per packet only), and also all old target nodes that are OTA/WP programmable are not affected at all and can be updated at the new speed. FWIW you can actually use 1 or 2 (not sure why you would) or 3 lines per packet (the new default going forward). Experiment and see what your own speed gains are! Please report any bugs or issues in this forum!

Gateway software update (v8)

There is a new gateway release, “V8”. The sources are at Github. The download link to the latest image is posted here. It contains several updates and new features, bug fixes etc. Most notably:

  • there is a new settings page accessible from the main dashboard. This allows you to edit your settings directly from the GUI instead of manually in the settings.json5 file. I also added a button on the settings page that allows restarting the gateway app, for such cases where this is useful (ie some settings require this, like changing the serial port). There is not a lot of validation done on the settings page, so edit these mindfully of that. The assumption is that you are not trying to break your own server. Also some settings are not exposed, so they will not show up in the GUI, and some are not editable – these will show but you are not supposed to change them. if you still need to, you can manually change the settings.json5 file any way you want on disk, but keep the formatting/nesting of the objects since that format is assumed in the new gateway app release. Here’s a peek at the new UI:
  • I added a new setting called genNodeIfNoMatch (boolean) which can disable new node creation when data is received from a node but no metric is matched to the metrics definition in metrics.js. If you set it to true, a node is added even if no data can be matched to a meaningful metric (node would show in the UI).
  • some new metrics/events were added/improved
  • logs generated by the gateway app are now rotated and archived with logrotate in a new logs subdirectory (if you manually upgrade you need to mkdir /home/pi/gateway/logs). To facilitate this there is a new /etc/logrotate.d/gateway config file that controls this feature. This means the log is kept under a size and history limit to avoid it growing indefinitely.
  • avrdude is now installed on the image so it would support programming MightyHat with a new HEX sketch. Instructions for doing this are on the MightyHat page.

Essentially 4 files have changed: gateway.js, metrics.js, settings.json5, index.html; one file was added: /etc/logrotate.d/gateway. I also posted a MightyHat blueprint DXF case for 1/8″/3mm acrylic (the green areas need to be etched rather than cut).

As always, the image contains pi:raspberry default credentials (change it with passwd) for the main pi login and for the http authentication (make your own .httpasswd file). If you got an ATXRaspi or MightyHat you need to uncomment the line running the shutdowncheck script in /etc/rc.local, to enable power button control. You may also want to create your own secure certificate and setup your wifi. Please let me know if there are any issues or bugs with this release.

MightyHat R2

Revision R2 of MightyHat is now available. Since the changes are not very significant I debated whether this should be really a R1.1 but I don’t like minor revision numbers so I went with R2. Basically it just adds the following main features:

  • Battery power switch – this is a slide switch that can cutoff the battery. During my many hours of testing, it occurred it would be useful to just have an onboard battery switch instead of pulling it in and out. Also if you have the battery tucked tight underneath the board it’s not always really easy to pull the JST connector. Or say if you want to ship a complete Pi+MightyHat+Battery inside an enclosure and don’t want to have it powered up. A side switch is convenient for this purpose:
    If you don’t want the switch then there is a solder jumper behind it which you will need to solder-bridge if you are using a backup battery.
  • Two momentary tactile buttons tied to the remaining available I2C pins which were previously unused (A4 and A5). These will be general purpose buttons to do whatever you like. If you want you can build some menu interaction for the LCD or whatever else. They connect GND to A4 and A5 and the pins should be declared as INPUT_PULLUP. The provided kit buttons have long actuators that rise just above the main power button (to allow pressing them when enclosed), but you could otherwise solder any standard 6mm tactile buttons here.
  • A few power path optimizations and tweaks.
  • I sourced RP-SMA connectors which seem to be a lot more popular than SMA (shown in photo below – note male pin in the center).
  • Besides the new positions/cutouts of the buttons and switch all other physical features remain at the same positions as R1.

I have an enclosure close to finalizing so maybe I will post that when it’s ready for those who’d like to lasercut their own case. I played around with 1/8″ and 1/16″ acrylic and finally settled on 1/8″ because it’s more dimensionally stable and makes for a sturdier case. The thin 1/16″ was light but it was prone to breaking where holes were very close to the edge or between USB/Ethernet connectors. Here are a few snapshots of R2:

Wondering what it takes to put together a kit with all the necessary parts to make a completely functional piece of hardware like this? Consider these facts:

  • The PCB board has some 62 SMD parts on it after picking and placing.
  • Some 13 more through hole parts (and the SMD transceiver) are required to be sourced and soldered separately (some optional like the LCD, buzzer or SMA connector).
  • To make a nice tight fit while avoiding any shorts or touching ontop of the Pi, an optimal height of 17mm is required between the Pi and MightyHat. Standoff combinations were sourced to achieve this height since 17mm is not standard or easy to find stanoff height.
  • Add a  1-cell 3.7v Lithium Polymer battery (optional) to act as a UPS and you are golden. I recommend this 2Ah battery from Adafruit which has decent capacity and attaches nicely with velcro under the Hat.
  • Total height of the Pi+MightyHat with all options installed is just 32mm (about 40mm when enclosed with 1/8″/3mm acrylic).

Finally I need to mention I have made numerous tweaks and improvements to the RFM69 gateway sample sketch for the MightyHat. I added a bunch of defines that make it easy to disable the LCD, wireless programming, or Automatic Transmission Control. Besides making the sketch a lot smaller (LCD has the largest footprint because of fonts and graphics), it allows  running your gateway without an LCD:

Gateway software update (v7)

The Moteino Framework Gateway software stack was updated to V7 and new sources are available at their Github repo, and also a new debian-wheezy image with the changes is released (for the image link see the Gateway image page). This release addresses some bugs and adds some significant refactoring and moving of directories. Most importantly the gateway application is now residing in /home/pi/gateway along with the public www directory for ease and more consistency (previously in /home/pi/moteino). Here are the other changes that go with this:

  • the www directory is moved from /var/www/default to /home/pi/gateway/www
  • the .htpasswd file and ssl certificate files are now in /home/pi/gateway/data/secure
  • the database files and log data are now kept in /home/pi/gateway/data/db
  • nconf is used to manage the settings file (previously known as settings.js, it was using straight nodejs exports to import the settings) which is now named settings.json5 and is a JSON5 based (via json5 package) settings file which can be manually edited as well as maintained through the app (in a future release)
  • there is a new package.json file which includes all the node dependencies. If you’re inclined to manually install the software stack, now all you have to do is cd /home/pi/gateway and npm install to get all these dependencies installed
  • the nedb power-loss database purging bug is fixed with a new nedb 1.5 upgrade
  • all server configuration files are updated to reflect the new paths
  • other misc minor changes and optimizations

As usual, if you detect any bugs in this release please let me know asap. Enjoy!

RFM69_ATC: Automatic Transmission Control

There is now a new extension to the RFM69 library. It’s called RFM69_ATC aka Automatic Transmission Control. Many thanks to Tom Studwell who implemented this and shared it in the forum. The basic idea behind this extension is to allow your nodes to dial down transmission power based on the received signal strength indicator (RSSI). For instance a sleepy battery node like a MotionMote sits still inside the home and usually has a very strong received signal, somewhere in the range of -30 to -40dBm when transmitting at full power. You could manually tune that down using radio.setPowerLevel(..) in code but its tedious and is a static change, if you move the node or conditions change it will not be smart to adjust the power level to the new environment. However with RFM69_ATC this is done automatically for you, you just need to indicate a target RSSI. On each packet sent and ACK received (using sendWithRetry(…) is required), the node analyzes the actual RSSI and continuously adjusts its own transmission power level of the RFM69 transceiver to attempt to match the target RSSI (+ or -). This way that end node is only “loud” enough to be heard by the gateway, not much louder. Close by nodes can “whisper” while nodes farther away or with more obstacles will “speak up” as necessary but avoid that default fixed maximum “screaming” level. This is “polite” in terms of “RF pollution” and efficient in terms of power consumption. Even non-battery nodes where power is no problem should implement this for the sake of the “polite” factor.

I have updated the Node and Gateway examples to have ATC enabled, and also the MotionMote example is now ATC enabled, in this case with a target RSSI of -90dBm. The noise floor is somewhere around -100dBm with the default RFM69 lib settings, so for static nodes that won’t experience a lot of movement or temperature drifts, a -90dBm target is pretty safe. That will keep the transmitter power to a minimum and save power. We know that the greatest toll on a battery powered node are the spikes of current used when the transmitter is active (up to 130mA for RFM69HW). Reducing the transmit power level to the minimum required will exponentially reduce that spike and result in longer battery life and a more “quiet” sensor network that won’t reach across your whole neighborhood. This is really awesome!

Here is a sample transmission using the Gateway and Node examples linked above. Note how the node starts transmitting at full power, then dials down power to match a test RSSI target of -68dBm. This is output from the Gateway end:

ATC

There are a few required changes to a sketch where you want to use ATC:

  • you must #include <RFM69_ATC.h> in addition to #include <RFM69.h>
  • you must use RFM69_ATC radio; instead of RFM69 radio;
  • for the gateway/receiver end the above two changes are sufficient, for the end node that does the power level adjustment you must also do the following:
  • in your setup() function (after all RFM69 initialization is complete) call this function to set your target RSSI: radio.enableAutoPower(targetRSSI);
    • targetRSSI is a negative integer, should be from very strong (-30) to very weak (-95). Usually you would want to be closer to the noise floor end (-100) since you want to reduce transmit power to the bare minimum
    • this is a one time call that enables the dynamic adjustment of the output power on that node
  • you must use radio.sendWithRetry() instead of just radio.send() or a combination of radio.send() and radio.receiveDone() for the radio to be able to receive the important headers from the gateway that tells it how to handle its power, this was done in the MotionMote example linked above

Once these changes are implemented, the node will start to progressively dial down power (assuming it starts transmitting at full power) with each packet sent, until the RSSI meets the target. When the RSSI is below the target power is dialed up again and so on, in an attempt to stay as close to the target as possible.

In the examples that I mentioned there is a pattern that I followed by implementing a define directive (#define ENABLE_ATC) which when left uncommented will enable ATC at compile time, if removed/commented the sketch will run normally without ATC. As always, bug reports, suggestions and contributions are welcome, the forum is the best place for that purpose.

I must mention here that at this time I have only partially implemented his variant with some adjustments to Tom’s implementation (I left the differences commented out in my version). For now I chose to leave my power control resolution as is. He actually went into more detail allowing a finer control of the power control – this is because of the differences between RFM69W and HW which handle output power control differently. If you’d like to try his variant with that extra control check this forum post or his Github repo. Thanks Tom for your great contributions and inspiring this piece of work and also the awesome forum projects you’ve posted!

New products!

I am quite happy to announce a few new products (or new revisions), some of which people have been asking for a long time, now finally available at the shop!
First there is the IOShield which is now available as an assembled PCB with optional “24VAC input package” which allows you to place it in places where you have 24VAC input, like I did in the wireless sprinkler controller project.

Then there is a small BellMote run with OSHPark purple PCBs. Completely automate your doorbell with this kit which allows you to detect, trigger, and disable your doorbell, all wirelessly via Moteino gateway.

Next up is the SonarMote which finally has a new revision (R2) which addresses some bugs, has a simpler BOM and is even lower power – just 10uA average when the Moteino is properly slept – sample sketches here. A wide range of applications are possible, from parking sensor, inventory control, sump pump monitor, and others.

Finally we have the MiniBoost which is a small SIP form boost regulator that yields 5V from an input voltage of 2-4V. Up to about 1A is possible from a charged LiPo battery (4V) and will easily carry a moderate load like a RaspberryPi. It uses the same boost regulator as the MightyBoost but the SIP size makes it very versatile for breadboard projects. It’s perfect for projects where you have a main working voltage of 3.3V but you have some sensor that needs a hefty 5V source. Comes with onboard green power LED. It has is the same pinout and size as the legendary 7805 regulators:

It takes a long time to go through prototype stages and then manufacture electronics of high quality, all as a small operation like I have. I hope to get more time to put more documentation and assembly instructions together for these kits and boards. Some of them have already been introduced as prototypes and have a fair amount of information available. There are also other things I’d like to blog about, just not enough time, stay tuned for more!

Gateway software update (v6)

The Moteino Framework Gateway software stack was updated and new sources are available at their Github repo, and also a new image with the changes is released (for the image link see the Gateway image page). Let’s call this v6, since it’s the 6th gateway image I’m releasing. This release addresses some bugs and adds some new features:

  • Added ability to make HTTP requests from a node using the node request module. This means you could include controls that trigger a request to some IP in the LAN/WAN and does something. This is really cool because it means we can have non-Moteino based nodes that speak HTTP instead of RF, like commercial wifi thermostats. Which leads to …drum roll
  • …the addition of a specific type of open API wifi thermostat (Radio Thermostat CT50) that allows full control of HVAC equipment. I will review it and blog about this addition whenever I get some time. Also added a new node type and icon for the water meter:
    Gateway_ThermostatWaterMeterIcons
  • adding (injecting) a non-moteino node (ex for the thermostat I mentioned above) to the gateway nodes collection can now be done as easily as this (in the terminal screen, type a new node ID and “NEW” in the msg box, click SEND):
    Gateway_NodeInject
  • Graph auto scaling, requested and discussed in this forum topic. The yaxis scale will automatically scale depending on the data viewed, very useful for highly variable or granular metrics, see the thread for screenshots of before/after and specific code changes to allow this.
  • fixed a bug in the new log storage engine where negative data was stored as unsigned 32bit integers, thus being extracted as positive. Negative data should now be welcome.
  • don’t allow graph live incoming data values when the graph was panned/zommed into a specific region
  • migrated from sliders to top bar toggle buttons for node-visibility, metric-pinning, enable-graph. Sliders were a pain to work with.
    Gateway_pinnedGraphed
  • lots of other refactoring and optimizations especially in the metrics.js and UI areas

I hope you enjoy the new release.

Gateway log engine upgrade

So far the Moteino Framework Gateway server application has been using neDB for storing node metadata/settings, and also to store historical metric data logs. Since neDB keeps all data in memory, it gets slower over time as data grows. Metrics with frequent updates could take 10sec to load 1 week worth of data (a few thousand data points). Visually it’s completely pointless to load 10K points on a graph that’s 1000px wide (you can’t see 10 points in 1 screen pixel). So a lot of wasted time and performance to search/move/display all that “invisible” data.

I researched a few options to improve storage/graph loading time. The list is below and if you want more details about each and about this effort see this page.

  • I tried implementing a binary search for neDB to get the graph data, but it wasn’t much better, the core engine of neDB is too slow for this purpose
  • I searched online other databases to store timestamped sensor data. There are expensive enterprise engines and also open source toolchains like Apache Hadoop – this looks very nice but it’s complex and a steep learning curve (Pi support?).
  • There’s mongoDB which is wonderfully capable and similar to neDB (or neDB similar to mongo rather) but still not officially supported on the Pi so it’s a no-go for now.
  • There’s Timestore – a C++ app by Mike Stirling. This is a great standalone engine but I have a requirement: I cannot assume when a data point comes in to be stored, it can be highly variable and I can have a thousand points in 1 hour, then nothing for a week. So I need to store the timestamp.
  • The guys over at OpenEnergyMonitor have done a lot of work based on Timestore. They had the same problems to solve as I did. Their EmonCMS app used MySQL to log sensor data, which was increasingly slow as tables got very deep. So they created PHP storage variations based off Timestore to allow logging variable interval data. Thanks guys for sharing your work and research results!

After considering all these options, I decided to build my own node.js storage engine based on bits and pieces from Timestore and the variable interval engine from OEM. The main reason is that it needed to be all in node.js. Doing it in PHP and then hitting a PHP script from a node.js socket app to query/post data felt very wrong (not sure how performant either).

The result is a much faster binary storage engine that can query and search and aggregate a month of data in about 500ms (on a Pi B Rev1 256RAM!). That’s more than a magnitude faster than neDB. However neDB will still be used to store node metadata, it’s perfect for this purpose and will be very fast and use very little memory.

The Gateway guide has been refactored to include this new engine details along with a few other changes and features (mostly UI):NodeView2

  • vastly improve graph loading times regardless of zoom and span viewed
  • add nice customizable tooltips to data points (customize in metrics.js)
  • add customizable legends on the graps (customize in metrics.js)
  • using upstart for starting/stopping/restarting the gateway.js script. This replaces the init.d way to start it and also ensures the gateway script will be respawned in case it crashes
  • add Font-awesome icon pack (over 580 awesome icons to use with jQuery-mobile) and use some of these new icons for some buttons
  • toggling node/metric visibility and metric graphing is now done via buttons instead of sliders which were not working very well on mobile devices. This also removes clutter on the node and metric pages

Upgrading from old neDB gatewayLog.db data

If you’ve used my Gateway interface before, you probably have log data stored as JSON in gatewayLog.js. No problem. Just save your existing gatewayLog.db, upgrade your Pi to the latest image, copy the gatewayLog.db into /home/pi/moteino, then run this upgrade script in the same directory – it parses your gatewayLog.db data and puts it into binary files in a db subdirectory, ready for the new gateway.js script to use. Otherwise gateway.js will just create logs from scratch as data comes in.

UpgradeGateway

It was a lot of work to get this right, I’ve been tweaking all this for the past few weeks. It’s free to use non-commercially, I hope you like it, I think it’s a major improvement overall, both back-end server and front-end UI. If I missed something in the guide or there’s any bug let me know. There’s more good stuff coming, just not enough time to document and blog it all. Cheers!

DoorBell Mote regulator update

The last BellMote prototype ended up having 2 LDO linear regulators, 12V and 3.3V feeding from the 12v one. I was split on using LDOs going forward. The upside is they are cheap, even 2 of them are under $2. The downside is they are inefficient and run warm/hot depending on the load. I could have gone with a single 5V LM7805 which will run even warmer, the higher the gap between Vin and Vout.

Digikey-ing around, I found these two switching regulators that are drop in SIP replacements for any LM78xx series LDOs: a RECOM 0.5A output with 7-28v input range ($2.8), and a Murata 1.5A 7-36v input range ($4.25). I tried both, and below are the results:

I won’t repeat the captions here but the clear winner is the Murata (plenty stocks at Digikey) it’s quite a bargain given the advantages. I tested it on 24VAC as well and it does not heat up at all either so it will be perfect for some coming interesting projects I have in the works that will feed on 24VAC. The RECOM regulator is just $2.8, still good enough for 16VAC but I prefer the Murata for further 16-24VAC powered projects. That said, I will offer these for sale (2 of them, one RECOM, one Murata, and I will replace the LDOs with a Murata and keep the third one). I may make a kit of this, not sure yet. For now you can grab these two from the first prototype batch, they will be worth millions when LowPowerLab is like Apple and me like Steve Wozniak 30 years from now (-: