Gateway v9.1 release

The Pi Gateway software package is continuing to be improved and the latest release is v9.1.0 (see GitHub releases). It contains some breaking changes, new features, and bug fixes. This blog entry serves as a feature review, change log and update guide.

Breaking & potentially breaking changes

  • the NGINX site configuration file is now renamed to gateway instead of default and so are the references to its log files (gateway.access.log and gateway.error.log located in /var/log/nxinx)
  • settings: there is a new interface section that contains a new responsive setting. The uiTitle setting is moved to this section. See below for responsive details.
  • metrics completely reorganized as detailed below

Metrics modules restructured

The metrics.js file is now renamed to core.js and moved into a new metrics folder. Also, all the prior default mote definitions are removed from this file and broken into individual metrics files that contain all the functionality pertaining to that particular mote. They are now called metric modules. You will see a new metrics/_LowPowerLab folder containing all these default metrics modules which used to be all bundled in the former metrics.js file. The userMetrics folder is removed and any examples or other sample metrics modules are moved under the new metrics folder.

Metrics modules loading order

In addition to the metrics reorganization into modules, the main app will load everything in the metrics folder in a specific order as follows:

  • core.js module first – required
  • any other metrics module files in the metrics folder in alphabetical order (case insensitive)
  • any metric modules in metrics/subfolders (1 level deep only) in alphabetical order (both folder and file order – case insensitive)

Globalized metric modules functions & variables

Given the powerful modularity of nodeJS, we can share functions and (string, numeric) variables between metric modules. To define a global variable or function you simply have to define it with the exports prefix, example from core.js:

exports.ONEDAY = 86400000;

exports.isNumeric =  function(n) {
  return !isNaN(parseFloat(n)) && isFinite(n); //http://stackoverflow.com/questions/18082/validate-decimal-numbers-in-javascript-isnumeric/1830844#1830844
}

So now ONEDAY and isNumeric() can be called across the entire application, anywhere. If you wish to make a function local (available only to that metric module) then omit the exports prefix – examples can be seen in the RadioThermostat_CT50.js  module.

Here is a sample of how the app loading the metric modules, along with their globalized members.

The loading order affects overriding of any definitions/variables/functions defined with the same names.
[04-16-20 09:39:31.274] [INFO]   *********************************************************************
[04-16-20 09:39:31.293] [INFO]   ************************* GATEWAY APP START *************************
[04-16-20 09:39:31.296] [INFO]   *********************************************************************
[04-16-20 09:39:31.302] [INFO]   LOADING METRICS MODULE [./metrics/core.js]
[04-16-20 09:39:31.307] [LOG]    |- GLOBALIZING ONEDAY
[04-16-20 09:39:31.308] [LOG]    |- GLOBALIZING isNumeric()
[04-16-20 09:39:31.309] [LOG]    |- GLOBALIZING isValidIP()
[04-16-20 09:39:31.310] [LOG]    |- GLOBALIZING isValidNodeId()
[04-16-20 09:39:31.311] [LOG]    |- GLOBALIZING determineValue()
[04-16-20 09:39:31.312] [LOG]    |- GLOBALIZING determineGraphValue()
[04-16-20 09:39:31.313] [LOG]    |- GLOBALIZING timeoutOffset()
[04-16-20 09:39:31.313] [LOG]    |- GLOBALIZING millisToFutureDate()
[04-16-20 09:39:31.314] [LOG]    |- GLOBALIZING nextSunriseOrSunset()
[04-16-20 09:39:31.330] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/doorbellmote.js]
[04-16-20 09:39:31.340] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/garagemote.js]
[04-16-20 09:39:31.351] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/motionmote.js]
[04-16-20 09:39:31.363] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/RadioThermostat_CT50.js]
[04-16-20 09:39:31.386] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/sonarmote.js]
[04-16-20 09:39:31.396] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/sprinklers.js]
[04-16-20 09:39:31.412] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/switchmote.js]
[04-16-20 09:39:31.446] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/watermeter.js]
[04-16-20 09:39:31.455] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/weathershield.js]
[04-16-20 09:39:31.466] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/examples/_example.js]
[04-16-20 09:39:31.478] [LOG]    |- GLOBALIZING ONEDAYHOURS
[04-16-20 09:39:31.478] [LOG]    |- GLOBALIZING ONEDAY (WARNING:OVERRIDING PREVIOUS VALUE!)
[04-16-20 09:39:31.479] [LOG]    |- GLOBALIZING secondsInOneDay()
[04-16-20 09:39:31.483] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/examples/_garage-auto-close.js]
[04-16-20 09:39:31.492] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/examples/_InternetSpeedTest.js]

Given these major changes, you should rewrite/break any custom metrics in their own definition files following the patterns shown in the default LowPowerLab metrics.

UI changes

New Menu buttons & app info

You can now do a Pi reboot and shutdown straight from the app menu. There is a bunch of additional information shown – versioning, RF gateway information such as frequency and uptime (if available). The main page node list header is removed, and the Show Hidden button now shows how many nodes are hidden in that list (the button toggles their visibility as before).

Metric pinning affects order on dashboard

As requested in this forum thread, the metrics will appear in the node bubble in the order they were pinned. If you unpin+pin a metric, it will become the first in the node’s info bubble.

Responsive UI

The new interface.responsive setting (boolean) determines if the UI becomes a responsive grid when it is viewed on large screens (thresholds are 768px, 1020px).

Log/Terminal & debug messages

A few significant changes happened on the terminal page to make this UI more useful and readable:

  • there is a new “Simulation” set of fields that allows simulating a node message just as if it came from the actual node (through the RF gateway’s serial port)
  • the terminal button is removed on the terminal and settings page – the terminal input fields are now always visible
  • instead of serialized JSON, serial messages from the RF gateway are now pasted in plain text and prepended with the serial port’s name
  • simulated serial messages (such as from this UI or from non-RF nodes like the CT50-Thermostat), the logs are marked as such with (simulated)
  • pressing ENTER in these fields will trigger the Send or Simulate
  • if you want to log debug messages from your RF gateway in this UI, you can simply prefix those messages with DEBUG: and all such messages are sent to all client UI terminal logs as well as in the permanent ~/gateway/logs/gateway.sys.log file (metric matching is skipped on these special DEBUG:something to log  messages). There is also a debug:heyNodeLogThisValue metric (in core.js among other special metrics) which allows you to send a debugging value from an end node, this is treated just like any other metric and is stored along with the other node’s metrics.
  • some special gateway requests can be made to the RF gateway (if it is coded to respond to them like the new PiGateway and MightyHat examples): BEEP, UPTIME, SYSFREQ, FREERAM, ENCRYPTIONKEY

Node Requests

A new experimental feature allowing you to send pending requests to nodes is available. This is a key:value pair that can be sent to a node to ask it to do something – like change a variable like transmission period, change behavior etc. There is a new button [+REQUEST] on the node details page. You can queue requests and the gateway will send them back to nodes in ACKs as they contact the gateway. This feature will require the new PiGateway or MightyHat examples which were updated and also improved with serial buffering. Your node then needs the ability to process all such requests and ACK them back to the RF gateway which in turn will ACK them to the app itself and mark that request as complete. Node requests can also fail, expire, be edited (ie updated with some new value), and repeated.

There are some special commands that the new sketches will process WRT the new node requests feature. You can see the request queue by typing REQUESTQUEUE in the terminal, the new sketches will respond with their content (or VOID if empty). You may also do things like manually VOID a certain request to a certain node (ex. 123:VOID:REQUEST), or void all requests to that node (ex. 123:VOID).

Miscellaneous other changes

  • license is now CC-BY-NC-SA-4.0
  • latest NGINX, PHP7, nodeJS/npm packages, along with app node modules updated
  • settings page Save and Cancel buttons removed – changes are now applied upon leaving the settings page
  • graphs show 1 day of graph data (vs. 8 hours previously)
  • changing the serial port baud setting automatically switches the port to the new baud without the need of restarting the application
  • npm-request replaced with native node http module – this is to handle any http requests (like for the CT50 Thermostat)
  • Fail2ban optionally installed during setup. There is a guide page that goes into some detail about adding this post install or on prior versions.
  • various setup script fixes & updates as well as:
    • ability to install last stable version or latest unreleased code
    • added symlinks to webserver access/error logs in the ~/gateway/logs directory
  • new default icons are generated at 300px to make them look better on the larger responsive grid, the icons_template.cdr is also updated
  • dashboard MENU button hidden if there is no socket connection
  • dropped PiBakery

Updating from an older version

Before you plan to try the upgrade make sure to:

  • copy-backup existing database, settings, any uploaded images and metrics files
  • backup your entire existing SD card

While it’s possible to do an in-place update, the changes are so extensive that it will take a long time to document everything and every file/script/permission that changed, and it’s very likely I or you will miss something. I recommend doing a clean install, on a new (good time to upgrade to a newer faster) SD card, then copy your old data. To do this, follow the setup guide on this page. Once everything is running and you see the gateway app run, you will need to sudo systemctl stop gateway to stop the app from running. Then copy over all your data files, images, merge old settings into the new settings.json5 file, and add any custom metrics you may have had into the new metrics folder – ensure they are formatted after the same pattern of other default files.

Gateway V9 Released

The Pi Gateway software is now at v9.0.0, this release is a major new feature and bug fix release. This blog entry serves as a change log and feature review. Below are the main highlights.

New Node button & sample metric generators (ex: Internet Speed polling event)

Clicking New Node you may add a numeric ID node. This could be a future RF or LAN node, or a dummy node to hold data such as your internet speed over time (add the Internet Speed event and enable it).

For RF nodes, any node that starts sending data through the serial port attached RF gateway, will automatically generate a new node. For RF nodes, node IDs up to 1023 are now possible thanks to 10bit addressing. Below you can also see a node with ID 999 that reports temperature. 

MultiGraphs

A new Multigraph button on the node page allows generating a graph containing multiple metrics right on the node page. This enables you to easily compare the various metrics of a node chronologically. Clicking the far right remove button under the graph allows you to generate another graph with a different combination of metrics. There is a practical limit to how many metrics you can show since each metric can contain a lot of data and generates a separate vertical legend, so be reasonable and avoid pushing limits. Here’s such a graph showing the various metrics of an RF node:

Desktop graph scroll-zoom & double-click zoom

You may now scroll-zoom with your mouse wheel over a graph (works in all graphs) and it will zoom in and out. You can also double-click and it will zoom in. On mobile devices double tapping will zoom-in.

Edit and Delete metric data

Graphs now show lines between data points instead of bars by default. Still, sometimes it can be confusing looking at lines or bars, where the actual data points are. So you can now turn on data points. Additionally, you can edit and remove any single data point (click on specific data point to edit/remove) or range of data points (select range of data points to remove or change to a new value) if you wish by enabling the edit/delete mode using two buttons below the metric graph. Data points are automatically enabled when editing data.

Easy node image change

You can now change a node’s icon very easily to any existing icon (under the /www/images folder), or upload a new image of your choice (uploaded to /www/images/uploads). Icons should be 120x120px, if larger they are resized. Clicking the node icon:

This brings up the node icon change dialog (pick existing or upload new):

Other icon changes:

  • some default icons have changed (ex. CT-50 thermostat)
  • for consistency, all default icon nodes are now prefixed with icon_
  • rssi icons renamed to rssi_n with n=[1, 7]

New HTTPEndpoint for posting data from LAN

You may now submit data to the Gateway from any LAN device, thus enabling the use of WiFi (ESP anyone?) and network connected nodes, and also essentially removing the need of using only RF nodes. For instance:

https://raspberrypi/httpendpoint/?id=1234&MOTION&F=77.77

will generate a new node (if non existent) with ID=1234, and generate two new metrics (MOTION, and temperature). If an ID is not provided, then the IP of that node becomes the ID. The generated node will always have a new _ip property which contains the IP of the sender. Other examples:

  • /httpendpoint/?MOTION&F=12    ->  ID=(valid IP)
  • /httpendpoint/    -> no update, just lastupdated timestamp

The response will be JSON of this form:
response: {"status":"success","message":"SUCCESS!","matchedMetrics":2}

Run in internet isolated LAN

All script files are now loaded locally and there is no longer a dependence on an internet connected RaspberryPi to fetch these scripts. So there is no concern running this completely disconnected or behind firewalls etc.

Title setting & Menu changes

You can set your own custom title from the settings page, don’t forget to click Save to apply changes. The menu now shows the Gateway app version, when the database was last compacted (done once daily automatically), and an option to compact the database on demand.

You can now quickly restart the app via Ctrl+Atl+Shift+R, or from the Menu as before.

Run it as a Mobile app

You can now run the application as a mobile app. A manifest.json definition that loads automatically with the web page allows you to create a shortcut via the Add To Home Screen option in your mobile device’s browser. Then clicking that icon on your home screen opens the web page like a mobile application without the extra browser bars.

New default metrics

  • It’s sometimes desirable to compare the RSSI of an RF node with the transmit level of an RF node. The new X:n metric with n=[0, 31] allows this to work nicely with the RFM69_ATC library extension and you can see the transmit level drop or rise based on the RSSI change.
  • A new TYPE:nodeType which matches the exports.motes definition of a mote in metrics.js allows to automatically pick that mote’s icon and other features like buttons. Example: when a SwitchMote is powered up, it can send a 1 time TYPE:SwitchMote token to indicate it’s a SwitchMote. You can always later override and pick the node type from the node type dropdown

Other misc changes

  • gateway.service is now run as pi:pi, and systemd will keep retrying to restart it if it crashes
  • fixed genNodeIfNoMatch setting
  • the /www/images/uploads directory is now chowned by www-data:pi during setup
  • node name/title added on the metric page – this helps determine which node’s temperature metric you may be looking at (vs. just “Temperature”)
  • ordered Node/Event type dropDowns (A-Z)
  • reworded and simplified licensing terms

Installation and upgrade path

This being such a major change with many new/renamed/deleted files, it’s a good idea to reinstall the application from scratch on a new raspbian image rather than try to do an in-place manual upgrade. This also ensures you will be running the latest and best raspbian, nginx, PHP7, nodeJs & packages, etc.

To transition your data to V9, backup the content of the /home/pi/gateway/data/dbfolder, then make a complete image backup of your currently functional previous version and store that until you are sure V9 runs smoothly on your Pi and all your old data is picked up.

Then follow the Gateway software setup steps in the official guide to install V9 to the latest raspbian image.

Once the V9 is running, to load your old data, you will need to sudo systemctl stop gateway, copy/overwrite the saved data into the new /home/pi/gateway/data/db folder, and then sudo systemctl restart gateway, and you should see all your old nodes and their data.

For any issues and bugs, please use this forum or you may submit a PR to the Github repository. Note that this change log may be edited as any loose ends are tied and before an official release is created.

Gateway app Updated to v8.10

The Pi Gateway software is now at v8.10, this release is mainly a new feature release and also it fixes some issues. You can view a list of all the changes in the official release notes. Here are some quick highlights:

Node/overridable Settings

If you’d like a particular global setting (from settings.json5) to be overridable in a node (for instance minimum voltage for battery powered nodes) you can now do so in metrics.js by using the new settings section under exports.motes, note that the included settings must match the name in settings.json5 or they will be ignored. Example from exports.motes.MotionMote:

imageThen on nodes of that type, the setting can now be set a custom value:
image

Wifi RadioThermostat CT50 IP setting support

Specific code for the CT50 was updated to support a new IP setting in the UI. This allows the user to override and set the IP of a thermostat on the node page, thus enabling having multiple thermostats with different IPs in the app:
image

New Setting Types

This enables more user friendly settings. Supported types are jQuery mobile HTML5 defaults, and examples of how to use these are found in the settings.json5 file:

  • checkbox – for true/false settings
  • number
  • email
  • password – obsoletes setting.password:true/false
  • range – min, max
  • default (no type) is text

A few examples using new types:
image

Email attachments

The sendEmail function in gateway.js has a new parameter where you can pass the URL of an attachment to include in sent emails: global.sendEmail = function(SUBJECT, BODY, **ATTACHMENTS**). Example email with attachment:
image

Scheduled events time remaining & datetime

Scheduled events now show time remaining until they will run, and the datetime when that happens:
image

Sunset/sunrise API for events

Based on suncalc, this node API allows creating events that run at various times during the day based on calculation of the sun position. A few examples are:

  • sunrise
  • sunset
  • solarNoon
  • goldenHour
  • dusk
  • dawn

To calculate these events, the latitude/longitude coordinates can be provided on the general settings page:

image

New All-Events page which shows all scheduled and disabled events:

image

Modified main menu:

  • removed the Exit item (redundant)
  • added app version:
    image

RSSI is now a metric and logged/graphed by default:

imageimage

Low Battery Voltage

Warning icon now blinks in/out over the node icon:
image

As always, when you update:

Controlling AC outlets with SwitchMote

Sometimes you need to remote control an AC outlet that powers something (like a heater or a light or motor etc). There are different IoT solutions out there  like the PowerSwitch Tail which allows hooking up your arduino/IoT device of choice and control 1 outlet.

Naturally, having created the SwitchMote kit, and after being asked in the forums if this could be modded/used to control AC outlets, I made a quick video of how to implement an AC outlet switch-box using a SwitchMote 2x10A, allowing control of 2 independent outlets with local button control as well as visual feedback via the status LEDs of the SwitchMote.

Gateway app updates

There are some notable changes and some new features for the Gateway app – these are published in this Github release. Below is a video overview and a summary of changes:

New settings:

  • serial baud is now 19200 (old: 115200), this is to eliminate any serial corruption when lots of serial data is going in/out to Moteino/MightyHat
  • keepFilesOnDelete (default: false) ensures the binary log files are removed when node/metrics are removed
  • graphMaxPoints – graph data point resolution (total points graphed from raw data)

image

New features

  • List reordering (desktop only) – drag & drop nodes to reorder list & broadcast to all clients:
    image
  • Graph legend div reformat + count of total raw data points that produced a graph are now displayed on top left.
  • Also there is a new Export-Raw-Data button that generates a CSV file from the selected time window. This was requested here.
    image

Motorized blinds with Moteino

James from snorp.net looked into automated blinds but when he saw the eye watering prices he decided to make his own wireless motorized blinds. He posted a great detailed tutorial to do just that. The result is motorized blinds automation device that is wireless, low power and costs just $40 for the DIYer. This is a blast considering commercial devices can cost upwards of 10x or more. Here are the highlights of his project:

  • after experimenting with Bluetooth, he decided to use Moteino a try for it’s ability to use ListenMode and make the project very low power
  • the chosen motor was the 28BYJ-48 stepper motor for its low cost ($2), quiet operation and fine control ability
  • he used Fritzing to design a simple yet very elegant shield to host his motor driver and Moteino and posted the schematic, layout and BOM then fabbed the PCBs at MakerStudio:
  • the whole assembly is pretty compact, it uses a 4xAA pack to yield a consumption of just 46µA while running the motor an average 12s/day, that comes to a theoretical 7 years life on a set of high quality 4xAAs. In real life that should last at least 2 years. Here’s what it looks like assembled and installed:

And a demo of it in operation:

John’s Gateway Deep Dive #2: Utility room node

I’m pleased to share another great video from John’s DIY Playground who has posted a detailed overview of his utility room node that can report the dryer status, detect water leaks and control his water heater. A nice example of not only surveillance but also control from a responsive interface he built on top of the Moteino Gateway stack. It’s quite a thrill to see other people innovate and build new truly useful things using some of the hardware and software building blocks I’ve created. He shares his project’s state diagram, explains his code, shows the KiCad design he made for integrating all the parts in a compact design, then moves along to install details and a demo of it in action. His adapted SMS-when-dryer-is-done event notifies him when the clothes are dry and the dryer ready for a new load, nice touch John.

Gateway software update (v8)

There is a new gateway release, “V8”. The sources are at Github. The download link to the latest image is posted here. It contains several updates and new features, bug fixes etc. Most notably:

  • there is a new settings page accessible from the main dashboard. This allows you to edit your settings directly from the GUI instead of manually in the settings.json5 file. I also added a button on the settings page that allows restarting the gateway app, for such cases where this is useful (ie some settings require this, like changing the serial port). There is not a lot of validation done on the settings page, so edit these mindfully of that. The assumption is that you are not trying to break your own server. Also some settings are not exposed, so they will not show up in the GUI, and some are not editable – these will show but you are not supposed to change them. if you still need to, you can manually change the settings.json5 file any way you want on disk, but keep the formatting/nesting of the objects since that format is assumed in the new gateway app release. Here’s a peek at the new UI:
  • I added a new setting called genNodeIfNoMatch (boolean) which can disable new node creation when data is received from a node but no metric is matched to the metrics definition in metrics.js. If you set it to true, a node is added even if no data can be matched to a meaningful metric (node would show in the UI).
  • some new metrics/events were added/improved
  • logs generated by the gateway app are now rotated and archived with logrotate in a new logs subdirectory (if you manually upgrade you need to mkdir /home/pi/gateway/logs). To facilitate this there is a new /etc/logrotate.d/gateway config file that controls this feature. This means the log is kept under a size and history limit to avoid it growing indefinitely.
  • avrdude is now installed on the image so it would support programming MightyHat with a new HEX sketch. Instructions for doing this are on the MightyHat page.

Essentially 4 files have changed: gateway.js, metrics.js, settings.json5, index.html; one file was added: /etc/logrotate.d/gateway. I also posted a MightyHat blueprint DXF case for 1/8″/3mm acrylic (the green areas need to be etched rather than cut).

As always, the image contains pi:raspberry default credentials (change it with passwd) for the main pi login and for the http authentication (make your own .httpasswd file). If you got an ATXRaspi or MightyHat you need to uncomment the line running the shutdowncheck script in /etc/rc.local, to enable power button control. You may also want to create your own secure certificate and setup your wifi. Please let me know if there are any issues or bugs with this release.

John’s Gateway Deep Dive #1: Configuration

John’s DIY Playground has posted a new video where he goes into more detail about how the gateway main code files work and how to start extending the interface and in specific the nodes/metrics that you see on the dashboard. Great video and definitely worth checking out especially for those who want to better understand the inner workings of the gateway code, thanks John and keep it up, looking forward to see more of your setup and hardware!

Another Gateway overview and example implementation

Check out this video from John (his youtube channel here) who implemented a custom home automation system of his own, based on my gateway solution. He has done a little script editing to augment and add to the functionality (new metrics and events) already given via metrics.js. I am looking forward to see more of the details of his implementation and especially the hardware. He mentioned something cool which I initially envisioned as a use case when I designed the hardware but never implemented this feature on my own SwitchMotes: using the unused status LEDs (and button?) of a SwitchMote to display the status of his GarageMote. He also added a sonar sensor to his garage mote to detect whether there is a car parked or not, this could be a really cool addition to a GarageMote. The video also gives an overview of how the gateway and end nodes interact. If John will create more follow up content I will post it wherever appropriate. Thanks John for sharing this!