Introduction

This page is all about making a RaspberryPi gateway for the Moteino IoT Framework.
More material will be added as it becomes available. Here is an overview video detailing the software stack and a demo of how it works. Below there are some action screenshots as well. Enjoy!

Here’s a custom implementation of this gateway by John from John’s DIY Playground:

System Functional Diagram:

Below is a functional diagram of how this gateway interfaces to the world and how the websocket is proxied to the backend via the HTTP webserver, hence providing a secure transport. An authentication front end wraps everything, so any unathorized access is denied:

Screenshots!

Main dashboard with some default mote types

Responsive UI (boolean setting):

MAILBOX_DSC_7214DOORBELL_DSC_7220
MOTION_DETAILSMOTION
SWITCHMOTE_DETAILSWITCHMOTE_GRAPH
GARAGE_DETAILGARAGE_GRAPH
DOORBELL_DETAILSDOORBELL_GRAPH
WATER_GRAPHSUMP_Graph2

Why bother?

People ask why did I bother with making this when there are so many great IoT frameworks and interfaces out there? It’s because after reviewing some of them trying to decide which to choose, I realize each has some flaw or lack of functionality that I really wanted. So why not? I started coding one day to see how far I could get and pretty soon this software stack has taken shape. I kept refining and realized that instead of spending my time reviewing/learning other frameworks I can just create my own the way I like. I tried to keep in mind that others might want to use it as well for their Moteino IoT stuff so I tried to make it as easy to integrate as possible. I’ve given a lot of sample code and predefined nodes and metrics and events. But if you need custom nodes, events, graphs, appearance you will have to write your own which should be pretty straight forward if you grasp Javascript and are willing to do a little reading on the different APIs of the software stack (like the popular flot for graphing or neDB for storage).

Other reasons why I made my own IoT framework from scratch:

  • I want complete control of my data and don’t want it to sit in someone else’s “free cloud” which the cloud owner could profit from or be hacked into and get exposed/stolen/data-mined etc.
  • I want a fast, responsive, modular and easy to expand or customize, platform independent, easy to configure and understand interface that isn’t a clunky polling 5 button page and doesn’t look like a shuttle dashboard and that any average non-tech savvy person could use

I am perfectly aware some of you might think this software is no good and it lacks features and it doesn’t solve your problem and there are much better things out there etc etc. Please keep in mind I did not create this to solve everyone’s requirements or problems. I created this for use on my own hardware that I create as part of LowPowerLab and in hope that others who use my hardware will find it useful. If you don’t like it don’t use it, always use whatever makes your life the easiest and makes you the most productive.

Setup: hardware

The most basic gateway setup is just a Moteino connected to your RaspberryPi, either through the GPIO (GND, VIN, TX, RX) or through the USB (MoteinoUSB).

A step up from that is adding a power button and power control. Below is a photo of what you need. Included is an ATXRaspi board and power button which allows external shutdown/reboots without needing to login to the Pi, but essentially these are not even needed, they are just convenience add-ons. To the right is a sample laser cut acrylic case I’ve built where all this hardware fits nicely:

If you want to get even fancier, you can get a MightyHat which provides all this functionality plus power backup from an optional LiPo battery and much more. Here’s what that looks like:

For reference, here’s another example open-side enclosure I’ve build for a Pi stack with included LCD that displays various messages. This build includes a MightyBoost for power/reboot control (like ATXRaspi) and also battery backup. For more details on how to make an enclosure case like this please see this blog post. This has long served as my own gateway but has now been replaced by a MightyHat driven Pi.

Setup: software

There is now an easy interactive setup script that allows you to install this Gateway solution on a raw Raspbian image. But first let’s make sure you get the image prepared correctly.

Install raspbian, enable ssh, setup wifi

The recommended distribution to use is the latest raspbian-lite on a Pi3 or better.

If you have a headless Pi (no monitor/keyboard) follow this official guide to enable ssh and setup your wifi credentials (before first boot). Other recommended steps:

  • change default password
  • change your Pi’s hostname to something other than raspberrypi
  • change timezone to where the Pi is located

Run the setup script

Open your Pi’s SSH/terminal and type these commands:

cd /home/pi
sudo wget https://raw.githubusercontent.com/LowPowerLab/RaspberryPi-Gateway/master/.setup/gatewaysetup.sh
sudo bash gatewaysetup.sh && sudo rm gatewaysetup.sh

This script was tested on RaspberryPi2 and 3. It installs everything required to support the Pi-Gateway app and will prompt you at various steps for information/passwords/etc.

Serial port check

By default the Pi-Gateway app is configured to work with the GPIO serial port /dev/ttyAMA0. The setup process automatically enables this port, disables the system console over it, and disables bluetooth (because it wired to it). Full details about the Pi’s UARTs and bluetooth module are found on this official page.

Now if your RF gateway (ex. Moteino, MightyHat) is attached to the GPIO serial you’re all set, the PiGateway app is ready to receive data.

To the GPIO serial port is enabled and bluetooth is disabled run sudo nano /boot/config.txt and ensure you have these lines at the end of this file:

enable_uart=1
dtoverlay=pi3-disable-bt

Also, ensure the console shell over serial is disabled. Run sudo nano /boot/cmdline.txt and make sure the token console=serial0,115200 is removed (only console=tty1 should remain). This can also be achieved within raspi-config (step 5 – interfacing options).

Using USB serial?

If your RF gateway (ex MoteinoUSB) is connected to a USB port, you will need to change the port setting on the UI Settings page once the app is installed. A USB port usually looks like /dev/ttyUSBn. You can use ls /dev/tty* to try to identify which one it is (ie run this command before and after connecting to the USB port).

Serial port debugging

It may be useful to debug the serial port. A lightweight serial terminal is picocom, install it with sudo apt-get install picocom. Next ensure no other application has that port open (ex. run sudo service gateway stop to stop the PiGateway app). Then open the serial port with picocom -b 19200 /dev/ttyAMA0. If your RF gateway is attached, you should start seeing packets coming in from your RF nodes.

FTP-ing into your Pi

It’s very useful to FTP into your Pi and you should install proftpd. The setup script offers to do it for you, but you can also do it manually at any time by running sudo apt-get install proftpd.

By default proftpd will not show hidden files (ex. anything prefixed with a .) so it’s useful to enable hidden files access – by running sudo nano /etc/proftpd/proftpd.conf and change the line ListOptions -l to ListOptions "-a".

Gateway app usage

As soon as the setup is complete and reboot is performed, the PiGateway app should already be running, just point your browser to your Pi’s IP address or hostname (ex http://raspberrypi/).

The Gateway app source code is published in this Github Repository – updates and bug fixes will be posted there. Releases are published here. Please use this forum to post any questions or report issues/bugs.

Here are the main application files:

Starting, stopping, autostart

The Gateway app is configured through a systemd service to run automatically when your Pi boots. If the application ever crashes or stops for some reason, systemd should keep trying to restart it. To manually start/stop/restart the app you can use these commands:

sudo service gateway start
sudo service gateway stop
sudo service gateway restart

You can check if the gatweway app is running by running this command, you should see two lines that contain gateway.js:

ps aux | grep gateway

In the UI main menu there are now available options to:

  • restart the application
  • restart the Pi host computer
  • shutdown the Pi host computer

Gateway node & data database

The uses neDB to store node configuration, this is located in ~/gateway/data/db/gateway.db. Node metric data (data points like temperature and motion) are stored in individual binary files in the same directory. For instance  temperature data for node with ID 123 would be stored in 0123_F.bin. Each data point is 9 bytes: 1 reserved byte, 4 bytes store the unix epoch timestamp (in seconds) of the data point, 4 bytes store the decimal value.

Gateway app logs

The gateway.js produces logs that are permanently stored in ~/gateway/logs/gateway.sys.log. As this log grows in size it is rotated/compressed with logrotate into archives in the same directory. In the same directory there are simbolic links to the webserver logs starting with v9.1.

Serial input/output formats from the Gateway Moteino

While this app is completely hardware agnostic and does not require a Moteino to run, it is typically used with a Moteino or equivalent Arduino, which is attached to the Pi either via the GPIO hardware serial port (eg. Moteino on pins 14,15 which are attached to /dev/ttyAMA0 serial port) or via the USB ports (ex. MoteinoUSB, MoteinoMEGA-USB, Moteino M0) which would produce a serial port named like /dev/ttyUSBn). The serial port needs to be identified and saved to the settings page, and app needs a restart to use a new port.

When you buy a genuine Moteino or LowPowerLab hardware, you not only support the development of this free app, but you are sure to receive hardware that will out of the box.

This Moteino acts as the RF gateway to your Internet Of Things. If this is a Moteino+RFM module, then all the rest of your nodes will have compatible radio modules, and they send data to the Gateway Moteino which then hands off that data to this PiGateway app via the serial port. The RFM69 library has many examples which run on LowPowerLab hardware (WeatherNode, MotionMote, SwitchMote etc), all you need is to ensure each node will report a unique node ID, to generate distinct nodes in the UI.

The sketch to load on your Gateway Moteino is the PiGateway sketch, not the general use Gateway sketch. This sketch is ready to work with the other RFM69 examples and is already abiding to the required input/output data formats.

You could also use this app with any other hardware that conforms to the expected data formats. For input to the Gateway app, data packets can contain multiple metrics (pieces of data, ex: Temperature:value). These available predefined metrics are defined in metrics.js as regular expressions. The expected pattern is very simple:

[NodeID] metric1:value metric2:value ... [RSSI:XX]

For data going from the Gateway towards your hardware Gateway (which then gets sent to a specific node) the pattern is the following, node that there is no rule to what the message needs to be as long as your end node can interpret and take action on that message:

[TargetNodeID] message

Gateway app self-signed SSL certificate

The Gateway app runs with an autogenerated self-signed SSL certificate. This means browsers will not recognize this as signed by a known CA authority. This is OK  and you will be greeted with the SSL warning page, just ignore that, you are safe. Then the auth_basic prompt where you need to enter the credentials that you placed in the .htpasswd file. The youtube video at the top of this post shows how all this works.

Gateway access from the internet

To access the Gateway app from the internet, you need port forwarding setup on your router. You should forward ports 80 and 443 to your Pi’s local IP address, you should then be able to access this home automation interface from any internet computer or mobile device when you’re on the road.

Resources that will help with understanding the source code:

Fail2Ban

Fail2Ban helps secure a webserver by scanning log files like /var/log/auth.log or /var/log/nginx/error.log and bans IP addresses conducting too many failed login attempts. It does this by updating system firewall rules to reject new connections from those IP addresses, for a configurable amount of time. The default is 10minutes. This limits brute force attacks such as dictionary attacks that exploit weak passwords to gain access to the server.

Starting in v9.1.0 the fail2ban package can now be optionally installed during the gateway setup process. Below is a general guide to installing this package manually or for previous versions. See this this link for an in-depth guide to installing fail2ban.

Prerequisites:

Install fail2ban

sudo apt-get update
sudo apt-get install fail2ban
By default fail2ban is configured to only ban failed SSH login attempts. In addition to this we want to secure the Gateway application which is running as a default website under nginx.

We need to create a new “jail” rule file:

sudo nano /etc/fail2ban/jail.local

Paste the following code in this file and save.

[nginx-http-auth]

enabled = true
filter = nginx-http-auth
port = http,https
logpath = /var/log/nginx/gateway.error.log #or default.error.log (v9.0 & prior)
ignoreip = 127.0.0.1/8
maxretry = 2
#bantime = 3600    #10 minutes is the default if not specified
#findtime = 3600   # These lines combine to ban clients that fail
#maxretry = 6      # to authenticate 6 times within a half hour.

Fail2ban should already include the required /etc/fail2ban/filter.d/nginx-http-auth.conf file which contains the rules for matching log file login failures.

Activate fail2ban:

sudo service fail2ban restart

Status

To get the status of all enabled “jail” rules (failed attempts & banned IP table) use this command:

sudo fail2ban-client status

Or for a single specific “jail” rule (ex. nginx-http-auth):

sudo fail2ban-client status nginx-http-auth

Unbanning

You may do your own testing and ban yourself out. To unban a specific IP address call this:

sudo fail2ban-client set nginx-http-auth unbanip 111.111.111.111

Quick logs search for auth errors

Here are some some quick commands to search a specific log for specific keywords:

grep 'mismatch\|htpasswd' /var/log/nginx/default.error.log

Exporting data (CSV)

The application features exporting data from individual node metrics or from the whole node (all metrics merged as separate columns in a CSV file, indexed by time).

This video shows how to export data from a single metric (starts at 4:45):

You may also download the full CSV from the node page by clicking the Export All Data button. Then open the CSV in Excel and add a new column and format it as shown below:

Possible / known issues

Serial GPIO port cannot be opened

Sympton

You are using the GPIO serial port (/dev/serial0 aka /dev/ttyAMA0), and the nodes do not update. If you open the /logs/gateway.sys.log you will see an error message like this:

[02-26-19_13:01:50.144] [ERROR] Cannot open /dev/ttyAMA0

Solution

You need to ensure the hardware GPIO serial port/dev/ttyAMA0 is enabled and not bound by any service. If you run sudo raspi-config and under Interfacing->Serial you should answer No to the “Allow logon over serial console?” question, and Yes to the “Enable GPIO serial port?“. This frees the serial port from being used by the logon service.

Then run sudo nano /boot/config.txt and ensure the GPIO uart is enabled, you should find enable_uart=1 at the end of the file, if not there add it and ensure it is set to “1”.

Also, add this line at the end of the file if not already there: dtoverlay=pi3-disable-bt.

Save and exit editing config.txt with Ctrl-X,Y. Then run this command: sudo systemctl disable hciuart, and finally reboot for all changes to take effect.

To verify the serial port is free for use by the gateway.js software, install and open minicom (assumes 19200 baud on attached Moteino/MightyHat):

sudo apt-get install minicom
sudo minicom -b 19200 -D /dev/ttyAMA0
Note that if the gateway.js app (or anything else using the port) is running, no serial data/activity will appear in minicom – there is no error given by minicom that the port is used (like on Windows).

For reference and more details you may also want to read this helpful blog post that reviews the Pi3’s GPIO serial port and its usage with other hardware. There is also this blog post that discusses serial port issues on the RaspberryPi 3.

I do not receive SMS/emails from events

Symptom

You have enabled events that send text messages or emails, but  you are not receiving them.

Solution

Make sure you visit the settings page where you setup your email, phone number and passwords so that the email application can use your email for this purpose.

Pre-built Pi image

There’s a Raspbian-Wheezy image that includes an older Gateway app. This is only for reference and will be removed eventually. Note: this is an old image with outdated versions of the stack software and you should first try the setup script before you use this image as a last resort. Also this image won’t work on a RaspberryPi 3!

After installing this image you must:

  • edit your settings.json5 to your own email/SMS settings/passwords and other relevant settings
  • edit the default HTTP_AUTH password in ~/gateway/data/secure/.htpasswd (you can use httpasswd tool that comes in the apache2-utils package)
  • change your Pi’s default password with the passwd command
  • setup your keyboard and timezone with raspi-config
  • setup port forwarding on your home router to allow access from anywhere on the internet. You should forward ports 80 and 443 to the internal IP (the Pi’s IP) ports 80 and 443.

The link to the v8.1 pre-built image is here.

Data logging engine

In older versions this application used neDB for storing historical metric data logs. Since neDB keeps all data in memory, it gets slower over time as data grows. Indexes help a lot, but really, this was not a good way to store millions of data points.

I tried implementing a binary search for neDB to get the graph data, but it wasn’t making the cut, the fundamental storage engine of neDB was just too slow for this purpose and the performance was not a lot better, so I ditched that effort. I researched other methods to store timestamped sensor data. There are expensive enterprise engines and also open source toolchains like Apache Hadoop – nice but complex and a steep learning curve (Pi support?). There’s mongoDB which is wonderfully capable and similar to neDB (or neDB similar to mongo rather) but still not officially supported on the Pi so it’s a no-go for now.

Then there’s Timestore, a C++ app by Mike Stirling, which he pointed out to me after he released it and I was very excited that a fast small standalone IoT storage engine exists for the rest of us. It stores data in binary format at fixed intervals (so no timestamps stored, just a starting point, and each data point value). During extraction, each data point will have a calculated timestamp based on the seed time. It uses binary search tree, a fast search algorithm to query and only return a reasonable # of data points. It also supports storing averaged values and has other features. This is all great but I have an important requirement: I cannot assume when a data point comes in to be stored, it can be highly variable and I can have a thousand points in 1 hour, then nothing for a week. So I need to store the timestamp. Turns out the guys over at OpenEnergyMonitor have done a lot of work based on Timestore. They had the same problems to solve, getting a lot of data out of log databases, queried/aggregated and returned as fast as possible. Their EmonCMS app used MySQL to log sensor data, which was increasingly slow as tables got very deep. So they created PHP storage variations based off Timestore to allow logging variable interval data. Thanks guys for sharing your work and research results!

After considering all these options, I decided to build my own node.js storage engine for logging variable time based sensor data based on bits and pieces from Timestore and the variable interval engine from OEM. The main reason is that it needed to be all in node.js. Doing it in PHP and then hitting a PHP script from a node.js socket app to query/post data felt very wrong (not sure how performant either).

The result is a much faster binary storage engine that can query and aggregate weeks worth of data in ~500ms (on a Pi B Rev1 256RAM!) and ~80-200ms on a Pi 2 (quad 900mhz, 1GBRAM), and much faster on Pi3. That’s more than a magnitude faster than neDB which was increasingly slower as data accumulated.

Upgrading from old neDB gatewayLog.db data

If you’ve used my Gateway interface before this new storage engine was implemented, you probably have log data stored as JSON in /home/pi/gateway/gatewayLog.js. To upgrade just save your existing gatewayLog.db, upgrade your Pi to the latest release, copy the gatewayLog.db into /home/pi/gateway/data/db, then run this upgrade script in the same directory – it parses your gatewayLog.db data and puts it into binary files in a db subdirectory, ready for the new gateway.js script to use. Otherwise gateway.js will just create logs from scratch as data comes in.

UpgradeGateway

NodeJS & npm updates

Sometimes there is a need to update nodeJS. This is a quick reference how that might be achieved. This is not as straightforward as updating packages.json and running npm update because some packages require rebuilding from source with the latest node version already in place.

For instance a critical timer bug exists in nodeJS (<=10.8.0) that causes events to stop running after 24.8 days of the app’s run time. An update to 10.9 or later is needed. Run node -v to find out the installed nodeJS version.

If you decide to update packages.json, you should update the whole gateway app and not just bits and pieces, since code dependencies exist between the app and certain nodeJS packages.
As always, it’s a good idea to make an image backup of the entire SD card in case anything goes wrong. Or at the very least back up the /home/pigateway folder that holds your database, uploaded images, custom metrics, settings, logs, etc.

Quick step guide to replacing nodeJS with latest available for the RaspberryPi:

#Stop gateway app and cd to gateway dir:
sudo systemctl stop gateway
cd /home/pi/gateway

#remove old package lock:
sudo rm -rf package-lock.json

#remove (or rename) node_modules directory:
sudo rm -rf node_modules
#Update nodejs (removes old node version):
sudo wget -O - https://raw.githubusercontent.com/LowPowerLab/RaspberryPi-Gateway/master/.setup/Install-Node.sh | sudo bash

#Update npm:
sudo npm install -g npm

#Re-install all packages/build from source:
sudo npm install --unsafe-perm --build-from-source
#and clear caches:
sudo npm cache verify

#Finally:
sudo systemctl start gateway

John's tutorials

999John from John’s DIY Playground has been developing a bunch of interesting projects and node customizations around this gateway software. He has been using a MoteinoUSB as a gateway for his Moteino wireless network.

Here’s his overview of installing the gateway image to his RaspberryP. He also walks through the basic settings setup and interacting with a node for the first time:

In the following video John goes into the details of the configuration files that are behind the gateway software. It’s a good walkthrogh for those that want to learn more about customizing their nodes and understanding more of the data and setting structures. Some features shown are obsolete, for instance settings can now be edited directly in the UI, and the gateway app can be restarted from the settings page also, no need to pkill to restart the gateway.

In this video he shows how to add a simple Moteino node that can turn an LED on/off from the Gateway web interface, and also reports the light level from a light sensor:

One my favorite video’s John ever made shows a completely custom node that controls and monitors his utility room. A perfect example of how to build both a custom hardware node with different attached peripherals, as well as how to setup a new customized gateway node to control and monitor that hardware: