The Pi Gateway software package is continuing to be improved and the latest release is v9.1.0 (see GitHub releases). It contains some breaking changes, new features, and bug fixes. This blog entry serves as a feature review, change log and update guide.

Breaking & potentially breaking changes

• the NGINX site configuration file is now renamed to gateway instead of default and so are the references to its log files (gateway.access.log and gateway.error.log located in /var/log/nxinx)
• settings: there is a new interface section that contains a new responsive setting. The uiTitle setting is moved to this section. See below for responsive details.
• metrics completely reorganized as detailed below

Metrics modules restructured

The metrics.js file is now renamed to core.js and moved into a new metrics folder. Also, all the prior default mote definitions are removed from this file and broken into individual metrics files that contain all the functionality pertaining to that particular mote. They are now called metric modules. You will see a new metrics/_LowPowerLab folder containing all these default metrics modules which used to be all bundled in the former metrics.js file. The userMetrics folder is removed and any examples or other sample metrics modules are moved under the new metrics folder.

Metrics modules loading order

In addition to the metrics reorganization into modules, the main app will load everything in the metrics folder in a specific order as follows:

• core.js module first – required
• any other metrics module files in the metrics folder in alphabetical order (case insensitive)
• any metric modules in metrics/subfolders (1 level deep only) in alphabetical order (both folder and file order – case insensitive)

Globalized metric modules functions & variables

Given the powerful modularity of nodeJS, we can share functions and (string, numeric) variables between metric modules. To define a global variable or function you simply have to define it with the exports prefix, example from core.js:

exports.ONEDAY = 86400000;

exports.isNumeric =  function(n) {
  return !isNaN(parseFloat(n)) && isFinite(n); //http://stackoverflow.com/questions/18082/validate-decimal-numbers-in-javascript-isnumeric/1830844#1830844
}

So now ONEDAY and isNumeric() can be called across the entire application, anywhere. If you wish to make a function local (available only to that metric module) then omit the exports prefix – examples can be seen in the RadioThermostat_CT50.js  module.

Here is a sample of how the app loading the metric modules, along with their globalized members.

[04-16-20 09:39:31.274] [INFO]   *********************************************************************
[04-16-20 09:39:31.293] [INFO]   ************************* GATEWAY APP START *************************
[04-16-20 09:39:31.296] [INFO]   *********************************************************************
[04-16-20 09:39:31.302] [INFO]   LOADING METRICS MODULE [./metrics/core.js]
[04-16-20 09:39:31.307] [LOG]    |- GLOBALIZING ONEDAY
[04-16-20 09:39:31.308] [LOG]    |- GLOBALIZING isNumeric()
[04-16-20 09:39:31.309] [LOG]    |- GLOBALIZING isValidIP()
[04-16-20 09:39:31.310] [LOG]    |- GLOBALIZING isValidNodeId()
[04-16-20 09:39:31.311] [LOG]    |- GLOBALIZING determineValue()
[04-16-20 09:39:31.312] [LOG]    |- GLOBALIZING determineGraphValue()
[04-16-20 09:39:31.313] [LOG]    |- GLOBALIZING timeoutOffset()
[04-16-20 09:39:31.313] [LOG]    |- GLOBALIZING millisToFutureDate()
[04-16-20 09:39:31.314] [LOG]    |- GLOBALIZING nextSunriseOrSunset()
[04-16-20 09:39:31.330] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/doorbellmote.js]
[04-16-20 09:39:31.340] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/garagemote.js]
[04-16-20 09:39:31.351] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/motionmote.js]
[04-16-20 09:39:31.363] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/RadioThermostat_CT50.js]
[04-16-20 09:39:31.386] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/sonarmote.js]
[04-16-20 09:39:31.396] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/sprinklers.js]
[04-16-20 09:39:31.412] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/switchmote.js]
[04-16-20 09:39:31.446] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/watermeter.js]
[04-16-20 09:39:31.455] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/_LowPowerLab/weathershield.js]
[04-16-20 09:39:31.466] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/examples/_example.js]
[04-16-20 09:39:31.478] [LOG]    |- GLOBALIZING ONEDAYHOURS
[04-16-20 09:39:31.478] [LOG]    |- GLOBALIZING ONEDAY (WARNING:OVERRIDING PREVIOUS VALUE!)
[04-16-20 09:39:31.479] [LOG]    |- GLOBALIZING secondsInOneDay()
[04-16-20 09:39:31.483] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/examples/_garage-auto-close.js]
[04-16-20 09:39:31.492] [INFO]   LOADING METRICS MODULE [/home/pi/gateway/metrics/examples/_InternetSpeedTest.js]

Given these major changes, you should rewrite/break any custom metrics in their own definition files following the patterns shown in the default LowPowerLab metrics.

UI changes

New Menu buttons & app info

You can now do a Pi reboot and shutdown straight from the app menu. There is a bunch of additional information shown – versioning, RF gateway information such as frequency and uptime (if available). The main page node list header is removed, and the Show Hidden button now shows how many nodes are hidden in that list (the button toggles their visibility as before).

Metric pinning affects order on dashboard

As requested in this forum thread, the metrics will appear in the node bubble in the order they were pinned. If you unpin+pin a metric, it will become the first in the node’s info bubble.

Responsive UI

The new interface.responsive setting (boolean) determines if the UI becomes a responsive grid when it is viewed on large screens (thresholds are 768px, 1020px).

Log/Terminal & debug messages

A few significant changes happened on the terminal page to make this UI more useful and readable:

• there is a new “Simulation” set of fields that allows simulating a node message just as if it came from the actual node (through the RF gateway’s serial port)
• the terminal button is removed on the terminal and settings page – the terminal input fields are now always visible
• instead of serialized JSON, serial messages from the RF gateway are now pasted in plain text and prepended with the serial port’s name
• simulated serial messages (such as from this UI or from non-RF nodes like the CT50-Thermostat), the logs are marked as such with (simulated)
• pressing ENTER in these fields will trigger the Send or Simulate
• if you want to log debug messages from your RF gateway in this UI, you can simply prefix those messages with DEBUG: and all such messages are sent to all client UI terminal logs as well as in the permanent ~/gateway/logs/gateway.sys.log file (metric matching is skipped on these special DEBUG:something to log  messages). There is also a debug:heyNodeLogThisValue metric (in core.js among other special metrics) which allows you to send a debugging value from an end node, this is treated just like any other metric and is stored along with the other node’s metrics.
• some special gateway requests can be made to the RF gateway (if it is coded to respond to them like the new PiGateway and MightyHat examples): BEEP, UPTIME, SYSFREQ, FREERAM, ENCRYPTIONKEY

Node Requests

A new experimental feature allowing you to send pending requests to nodes is available. This is a key:value pair that can be sent to a node to ask it to do something – like change a variable like transmission period, change behavior etc. There is a new button [+REQUEST] on the node details page. You can queue requests and the gateway will send them back to nodes in ACKs as they contact the gateway. This feature will require the new PiGateway or MightyHat examples which were updated and also improved with serial buffering. Your node then needs the ability to process all such requests and ACK them back to the RF gateway which in turn will ACK them to the app itself and mark that request as complete. Node requests can also fail, expire, be edited (ie updated with some new value), and repeated.

There are some special commands that the new sketches will process WRT the new node requests feature. You can see the request queue by typing REQUESTQUEUE in the terminal, the new sketches will respond with their content (or VOID if empty). You may also do things like manually VOID a certain request to a certain node (ex. 123:VOID:REQUEST), or void all requests to that node (ex. 123:VOID).

Miscellaneous other changes

• license is now CC-BY-NC-SA-4.0
• latest NGINX, PHP7, nodeJS/npm packages, along with app node modules updated
• settings page Save and Cancel buttons removed – changes are now applied upon leaving the settings page
• graphs show 1 day of graph data (vs. 8 hours previously)
• changing the serial port baud setting automatically switches the port to the new baud without the need of restarting the application
• npm-request replaced with native node http module – this is to handle any http requests (like for the CT50 Thermostat)
• Fail2ban optionally installed during setup. There is a guide page that goes into some detail about adding this post install or on prior versions.
• various setup script fixes & updates as well as:
  • ability to install last stable version or latest unreleased code
  • added symlinks to webserver access/error logs in the ~/gateway/logs directory
• new default icons are generated at 300px to make them look better on the larger responsive grid, the icons_template.cdr is also updated
• dashboard MENU button hidden if there is no socket connection
• dropped PiBakery

Updating from an older version

Before you plan to try the upgrade make sure to:

• copy-backup existing database, settings, any uploaded images and metrics files
• backup your entire existing SD card

While it’s possible to do an in-place update, the changes are so extensive that it will take a long time to document everything and every file/script/permission that changed, and it’s very likely I or you will miss something. I recommend doing a clean install, on a new (good time to upgrade to a newer faster) SD card, then copy your old data. To do this, follow the setup guide on this page. Once everything is running and you see the gateway app run, you will need to sudo systemctl stop gateway to stop the app from running. Then copy over all your data files, images, merge old settings into the new settings.json5 file, and add any custom metrics you may have had into the new metrics folder – ensure they are formatted after the same pattern of other default files.