Gateway software update (v7)

The Moteino Framework Gateway software stack was updated to V7 and new sources are available at their Github repo, and also a new debian-wheezy image with the changes is released (for the image link see the Gateway image page). This release addresses some bugs and adds some significant refactoring and moving of directories. Most importantly the gateway application is now residing in /home/pi/gateway along with the public www directory for ease and more consistency (previously in /home/pi/moteino). Here are the other changes that go with this:

  • the www directory is moved from /var/www/default to /home/pi/gateway/www
  • the .htpasswd file and ssl certificate files are now in /home/pi/gateway/data/secure
  • the database files and log data are now kept in /home/pi/gateway/data/db
  • nconf is used to manage the settings file (previously known as settings.js, it was using straight nodejs exports to import the settings) which is now named settings.json5 and is a JSON5 based (via json5 package) settings file which can be manually edited as well as maintained through the app (in a future release)
  • there is a new package.json file which includes all the node dependencies. If you’re inclined to manually install the software stack, now all you have to do is cd /home/pi/gateway and npm install to get all these dependencies installed
  • the nedb power-loss database purging bug is fixed with a new nedb 1.5 upgrade
  • all server configuration files are updated to reflect the new paths
  • other misc minor changes and optimizations

As usual, if you detect any bugs in this release please let me know asap. Enjoy!

Gateway software update (v6)

The Moteino Framework Gateway software stack was updated and new sources are available at their Github repo, and also a new image with the changes is released (for the image link see the Gateway image page). Let’s call this v6, since it’s the 6th gateway image I’m releasing. This release addresses some bugs and adds some new features:

  • Added ability to make HTTP requests from a node using the node request module. This means you could include controls that trigger a request to some IP in the LAN/WAN and does something. This is really cool because it means we can have non-Moteino based nodes that speak HTTP instead of RF, like commercial wifi thermostats. Which leads to …drum roll
  • …the addition of a specific type of open API wifi thermostat (Radio Thermostat CT50) that allows full control of HVAC equipment. I will review it and blog about this addition whenever I get some time. Also added a new node type and icon for the water meter:
    Gateway_ThermostatWaterMeterIcons
  • adding (injecting) a non-moteino node (ex for the thermostat I mentioned above) to the gateway nodes collection can now be done as easily as this (in the terminal screen, type a new node ID and “NEW” in the msg box, click SEND):
    Gateway_NodeInject
  • Graph auto scaling, requested and discussed in this forum topic. The yaxis scale will automatically scale depending on the data viewed, very useful for highly variable or granular metrics, see the thread for screenshots of before/after and specific code changes to allow this.
  • fixed a bug in the new log storage engine where negative data was stored as unsigned 32bit integers, thus being extracted as positive. Negative data should now be welcome.
  • don’t allow graph live incoming data values when the graph was panned/zommed into a specific region
  • migrated from sliders to top bar toggle buttons for node-visibility, metric-pinning, enable-graph. Sliders were a pain to work with.
    Gateway_pinnedGraphed
  • lots of other refactoring and optimizations especially in the metrics.js and UI areas

I hope you enjoy the new release.

Gateway log engine upgrade

So far the Moteino Framework Gateway server application has been using neDB for storing node metadata/settings, and also to store historical metric data logs. Since neDB keeps all data in memory, it gets slower over time as data grows. Metrics with frequent updates could take 10sec to load 1 week worth of data (a few thousand data points). Visually it’s completely pointless to load 10K points on a graph that’s 1000px wide (you can’t see 10 points in 1 screen pixel). So a lot of wasted time and performance to search/move/display all that “invisible” data.

I researched a few options to improve storage/graph loading time. The list is below and if you want more details about each and about this effort see this page.

  • I tried implementing a binary search for neDB to get the graph data, but it wasn’t much better, the core engine of neDB is too slow for this purpose
  • I searched online other databases to store timestamped sensor data. There are expensive enterprise engines and also open source toolchains like Apache Hadoop – this looks very nice but it’s complex and a steep learning curve (Pi support?).
  • There’s mongoDB which is wonderfully capable and similar to neDB (or neDB similar to mongo rather) but still not officially supported on the Pi so it’s a no-go for now.
  • There’s Timestore – a C++ app by Mike Stirling. This is a great standalone engine but I have a requirement: I cannot assume when a data point comes in to be stored, it can be highly variable and I can have a thousand points in 1 hour, then nothing for a week. So I need to store the timestamp.
  • The guys over at OpenEnergyMonitor have done a lot of work based on Timestore. They had the same problems to solve as I did. Their EmonCMS app used MySQL to log sensor data, which was increasingly slow as tables got very deep. So they created PHP storage variations based off Timestore to allow logging variable interval data. Thanks guys for sharing your work and research results!

After considering all these options, I decided to build my own node.js storage engine based on bits and pieces from Timestore and the variable interval engine from OEM. The main reason is that it needed to be all in node.js. Doing it in PHP and then hitting a PHP script from a node.js socket app to query/post data felt very wrong (not sure how performant either).

The result is a much faster binary storage engine that can query and search and aggregate a month of data in about 500ms (on a Pi B Rev1 256RAM!). That’s more than a magnitude faster than neDB. However neDB will still be used to store node metadata, it’s perfect for this purpose and will be very fast and use very little memory.

The Gateway guide has been refactored to include this new engine details along with a few other changes and features (mostly UI):NodeView2

  • vastly improve graph loading times regardless of zoom and span viewed
  • add nice customizable tooltips to data points (customize in metrics.js)
  • add customizable legends on the graps (customize in metrics.js)
  • using upstart for starting/stopping/restarting the gateway.js script. This replaces the init.d way to start it and also ensures the gateway script will be respawned in case it crashes
  • add Font-awesome icon pack (over 580 awesome icons to use with jQuery-mobile) and use some of these new icons for some buttons
  • toggling node/metric visibility and metric graphing is now done via buttons instead of sliders which were not working very well on mobile devices. This also removes clutter on the node and metric pages

Upgrading from old neDB gatewayLog.db data

If you’ve used my Gateway interface before, you probably have log data stored as JSON in gatewayLog.js. No problem. Just save your existing gatewayLog.db, upgrade your Pi to the latest image, copy the gatewayLog.db into /home/pi/moteino, then run this upgrade script in the same directory – it parses your gatewayLog.db data and puts it into binary files in a db subdirectory, ready for the new gateway.js script to use. Otherwise gateway.js will just create logs from scratch as data comes in.

UpgradeGateway

It was a lot of work to get this right, I’ve been tweaking all this for the past few weeks. It’s free to use non-commercially, I hope you like it, I think it’s a major improvement overall, both back-end server and front-end UI. If I missed something in the guide or there’s any bug let me know. There’s more good stuff coming, just not enough time to document and blog it all. Cheers!