Introduction

This page is all about making a RaspberryPi gateway for the Moteino IoT Framework.
More material will be added as it becomes available. Here is an overview video detailing the software stack and a demo of how it works. Below there are some action screenshots as well. Enjoy!

Here’s a custom implementation of this gateway by John from John’s DIY Playground:

System Functional Diagram:

Below is a functional diagram of how this gateway interfaces to the world and how the websocket is proxied to the backend via the HTTP webserver, hence providing a secure transport. An authentication front end wraps everything, so any unathorized access is denied:

Screenshots!

MAILBOX_DSC_7214DOORBELL_DSC_7220
MOTION_DETAILSMOTION
SWITCHMOTE_DETAILSWITCHMOTE_GRAPH
GARAGE_DETAILGARAGE_GRAPH
DOORBELL_DETAILSDOORBELL_GRAPH
WATER_GRAPHSUMP_Graph2

Why bother?

People ask why did I bother with making this when there are so many great IoT frameworks and interfaces out there? It’s because after reviewing some of them trying to decide which to choose, I realize each has some flaw or lack of functionality that I really wanted. So why not? I started coding one day to see how far I could get and pretty soon this software stack has taken shape. I kept refining and realized that instead of spending my time reviewing/learning other frameworks I can just create my own the way I like. I tried to keep in mind that others might want to use it as well for their Moteino IoT stuff so I tried to make it as easy to integrate as possible. I’ve given a lot of sample code and predefined nodes and metrics and events. But if you need custom nodes, events, graphs, appearance you will have to write your own which should be pretty straight forward if you grasp Javascript and are willing to do a little reading on the different APIs of the software stack (like the popular flot for graphing or neDB for storage).

Other reasons why I made my own IoT framework from scratch:

  • I want complete control of my data and don’t want it to sit in someone else’s “free cloud” which the cloud owner could profit from or be hacked into and get exposed/stolen/data-mined etc.
  • I want a fast, responsive, modular and easy to expand or customize, platform independent, easy to configure and understand interface that isn’t a clunky polling 5 button page and doesn’t look like a shuttle dashboard and that any average non-tech savvy person could use

I am perfectly aware some of you might think this software is no good and it lacks features and it doesn’t solve your problem and there are much better things out there etc etc. Please keep in mind I did not create this to solve everyone’s requirements or problems. I created this for use on my own hardware that I create as part of LowPowerLab and in hope that others who use my hardware will find it useful. If you don’t like it don’t use it, always use whatever makes your life the easiest and makes you the most productive.

Setup: hardware

The most basic gateway setup is just a Moteino connected to your RaspberryPi, either through the GPIO (GND, VIN, TX, RX) or through the USB (MoteinoUSB).

A step up from that is adding a power button and power control. Below is a photo of what you need. Included is an ATXRaspi board and power button which allows external shutdown/reboots without needing to login to the Pi, but essentially these are not even needed, they are just convenience add-ons. To the right is a sample laser cut acrylic case I’ve built where all this hardware fits nicely:

If you want to get even fancier, you can get a MightyHat which provides all this functionality plus power backup from an optional LiPo battery and much more. Here’s what that looks like:

For reference, here’s another example open-side enclosure I’ve build for a Pi stack with included LCD that displays various messages. This build includes a MightyBoost for power/reboot control (like ATXRaspi) and also battery backup. For more details on how to make an enclosure case like this please see this blog post. This has long served as my own gateway but has now been replaced by a MightyHat driven Pi.

Setup: software

There is now an easy interactive setup script that allows you to install this Gateway solution on a raw Raspbian image. But first let’s make sure you get the image prepared correctly.

Preparing your image manually

After you install the latest raspbian image (stretch lite recommended), the first thing you will need is enable SSH so you can remote into your Pi via SSH. This is accomplished via the sudo raspi-config command. If you have a Moteino or MightyHat attached to the GPIO serial (/dev/ttyAMA0 is the hardware GPIO serial port) then also enable hardware serial, and disable the serial console. If you have a MoteinoUSB or another USB device receiving your serial data, then you don’t need this, but you will need to change the serial-port setting once the app in installed (on the Settings page). You may also want to rename the hostname from raspberrypi to something more unique. And while here, if you want to use WiFi instead of a wired LAN connection, then set your Wifi credentials. Reboot to apply all changes.

If you have a Pi3 and you are using the GPIO serial (by default used by built-in Bluetooth) you will need to disable Bluetooth in /boot/config.txt (edit file using sudo nano /boot/config.txt). Go to the end of the file and ensure you have these lines at the end of it to disable the onboard Bluetooth and free the GPIO serial port:

enable_uart=1
dtoverlay=pi3-disable-bt

Other useful packages to install are minicom for serial debugging, samba (publishes hostname on your LAN so you can access it via hostname in your browser) and proftpd for FTP-ing into your Pi with the same credentials you use to SSH:

sudo apt-get install minicom
sudo apt-get install proftpd
sudo apt-get install samba

Install your image and configure your Pi

It’s recommended to use the Raspbian Lite image with this software. After flashing the image, run sudo raspi-config and setup your Wi-Fi, enable SSH etc.

Running the setup script

Open your Pi’s SSH/terminal and type these commands:

cd /home/pi
sudo wget https://raw.githubusercontent.com/LowPowerLab/RaspberryPi-Gateway/master/.setup/gatewaysetup.sh
sudo bash gatewaysetup.sh && sudo rm gatewaysetup.sh

This script was tested on RaspberryPi2 and 3. It installs everything required to support the Gateway and will prompt you for passwords and other input.

Gateway app usage

As soon as the setup step is complete, assuming everything went well, the Gateway app should already be running, just point your browser to your Pi’s IP address or hostname (ex http://raspberrypi/).

The Gateway app source code is published in this Github Repository – updates and bug fixes will be posted there. Releases are published here. Please use this forum to post any questions or report issues/bugs.

Here are the main application files:

Starting, stopping, autostart

The Gateway app is configured through a systemd service to run automatically when your Pi boots. If the application ever crashes or stops for some reason, systemd should restart it. To manually start/stop/restart the app you can use these commands:

sudo systemctl start gateway
sudo systemctl stop gateway
sudo systemctl restart gateway

You can check if the gatweway app is running by running this command, you should see two lines that contain “gateway.js”:

ps aux | grep gateway

You can also restart the application from the UI (the app simply stops itself via an exit call, then systemd restarts it). This self-restart option is available on the main menu. This can be useful if you make code changes in the user metrics or main source code. Note that if you change back-end nodeJS source code (in any of the .js files) and there is a bug, the app will crash upon restart, and you will see the errors in the logs/gateway.sys.log file.

Gateway node & data database

The uses neDB to store node configuration, this is located in ~/gateway/data/db/gateway.db. Node metric data (data points like temperature and motion) are stored in individual binary files in the same directory. For instance  temperature data for node with ID 123 would be stored in 0123_F.bin. Each data point is 9 bytes: 1 reserved byte, 4 bytes store the unix epoch timestamp (in seconds) of the data point, 4 bytes store the decimal value.

Gateway app logs

All stdout and stderror and console output from the gateway.js app will be logged in /home/pi/gateway/logs/gateway.sys.log. As this log grows in size it is rotated/compressed with logrotate into archives in the same directory.

Serial input/output formats from the Gateway Moteino

While this app is completely hardware agnostic and does not require a Moteino to run, it is typically used with a Moteino or equivalent Arduino, which is attached to the Pi either via the GPIO hardware serial port (eg. Moteino or MoteinoMEGA on pins 14,15 which produce a serial port named /dev/ttyAMA0) or via the USB ports (eg. MoteinoUSB, MoteinoMEGA-USB, Moteino M0 which would produce a serial port named like /dev/ttyUSBx). The serial port needs to be identified and saved to the settings page, and app needs a restart to use a new port. By default the app comes with a /dev/ttyAMA0 setting (GPIO serial port).

When you buy a genuine Moteino or LowPowerLab hardware, you support the development of this nice app and other research by LowPowerLab, and it’s a nice way to give back for the free stuff you get here 😉

This Moteino acts as the hardware gateway to your Internet Of Things. If this is a Moteino+RFM module, then all the rest of your nodes will have compatible radio modules, and they send data to the Gateway Moteino which then hands off that data to this Gateway app via the serial port. The RFM69 library has many examples which run on LowPowerLab hardware (WeatherNode, MotionMote, SwitchMote etc), all you need is to ensure each node will report a unique node ID, to generate distinct nodes in the UI.

The sketch to load on your Gateway Moteino is the PiGateway sketch, not the general use Gateway sketch. This sketch is ready to work with the other RFM69 examples and is already abiding to the required input/output data formats.

You could also use this app with any other hardware that conforms to the expected data formats. For input to the Gateway app, data packets can contain multiple metrics (pieces of data, ex: Temperature:value). These available predefined metrics are defined in metrics.js as regular expressions. The expected pattern is very simple:

[NodeID] metric1:value metric2:value ... [RSSI:XX]

For data going from the Gateway towards your hardware Gateway (which then gets sent to a specific node) the pattern is the following, node that there is no rule to what the message needs to be as long as your end node can interpret and take action on that message:

[TargetNodeID] message

Gateway app self-signed SSL certificate

The Gateway app runs with an autogenerated self-signed SSL certificate. This means browsers will not recognize this as signed by a known CA authority. This is OK  and you will be greeted with the SSL warning page, just ignore that, you are safe. Then the auth_basic prompt where you need to enter the credentials that you placed in the .htpasswd file. The youtube video at the top of this post shows how all this works.

Gateway access from the internet

To access the Gateway app from the internet, you need port forwarding setup on your router. You should forward ports 80 and 443 to your Pi’s local IP address, you should then be able to access this home automation interface from any internet computer or mobile device when you’re on the road.

Resources that will help with understanding the source code:

Exporting data (CSV)

The application features exporting data from individual node metrics or from the whole node (all metrics merged as separate columns in a CSV file, indexed by time).

This video shows how to export data from a single metric (starts at 4:45):

You may also download the full CSV from the node page by clicking the Export All Data button. Then open the CSV in Excel and add a new column and format it as shown below:

Possible / known issues

Serial GPIO port cannot be opened

Sympton

You are using the GPIO serial port (/dev/serial0 aka /dev/ttyAMA0), and the nodes do not update. If you open the /logs/gateway.sys.log you will see an error message like this:

[02-26-19_13:01:50.144] [ERROR] Cannot open /dev/ttyAMA0

Solution

You need to ensure the hardware GPIO serial port/dev/ttyAMA0 is enabled and not bound by any service. If you run sudo raspi-config and under Interfacing->Serial you should answer No to the “Allow logon over serial console?” question, and Yes to the “Enable GPIO serial port?“. This frees the serial port from being used by the logon service.

Then run sudo nano /boot/config.txt and ensure the GPIO uart is enabled, you should find enable_uart=1 at the end of the file, if not there add it and ensure it is set to “1”.

Also, add this line at the end of the file if not already there: dtoverlay=pi3-disable-bt.

Save and exit editing config.txt with Ctrl-X,Y. Then run this command: sudo systemctl disable hciuart, and finally reboot for all changes to take effect.

To verify the serial port is free for use by the gateway.js software, install and open minicom (assumes 19200 baud on attached Moteino/MightyHat):

sudo apt-get install minicom
sudo minicom -b 19200 -D /dev/ttyAMA0
Note that if the gateway.js app (or anything else using the port) is running, no serial data/activity will appear in minicom – there is no error given by minicom that the port is used (like on Windows).

For reference and more details you may also want to read this helpful blog post that reviews the Pi3’s GPIO serial port and its usage with other hardware. There is also this blog post that discusses serial port issues on the RaspberryPi 3.

I do not receive SMS/emails from events

Symptom

You have enabled events that send text messages or emails, but  you are not receiving them.

Solution

Make sure you visit the settings page where you setup your email, phone number and passwords so that the email application can use your email for this purpose.

Pre-built Pi image

There’s a Raspbian-Wheezy image that includes an older Gateway app. This is only for reference and will be removed eventually. Note: this is an old image with outdated versions of the stack software and you should first try the setup script before you use this image as a last resort. Also this image won’t work on a RaspberryPi 3!

After installing this image you must:

  • edit your settings.json5 to your own email/SMS settings/passwords and other relevant settings
  • edit the default HTTP_AUTH password in ~/gateway/data/secure/.htpasswd (you can use httpasswd tool that comes in the apache2-utils package)
  • change your Pi’s default password with the passwd command
  • setup your keyboard and timezone with raspi-config
  • setup port forwarding on your home router to allow access from anywhere on the internet. You should forward ports 80 and 443 to the internal IP (the Pi’s IP) ports 80 and 443.

The link to the v8.1 pre-built image is here.

Data logging engine

In older versions this application used neDB for storing historical metric data logs. Since neDB keeps all data in memory, it gets slower over time as data grows. Indexes help a lot, but really, this was not a good way to store millions of data points.

I tried implementing a binary search for neDB to get the graph data, but it wasn’t making the cut, the fundamental storage engine of neDB was just too slow for this purpose and the performance was not a lot better, so I ditched that effort. I researched other methods to store timestamped sensor data. There are expensive enterprise engines and also open source toolchains like Apache Hadoop – nice but complex and a steep learning curve (Pi support?). There’s mongoDB which is wonderfully capable and similar to neDB (or neDB similar to mongo rather) but still not officially supported on the Pi so it’s a no-go for now.

Then there’s Timestore, a C++ app by Mike Stirling, which he pointed out to me after he released it and I was very excited that a fast small standalone IoT storage engine exists for the rest of us. It stores data in binary format at fixed intervals (so no timestamps stored, just a starting point, and each data point value). During extraction, each data point will have a calculated timestamp based on the seed time. It uses binary search tree, a fast search algorithm to query and only return a reasonable # of data points. It also supports storing averaged values and has other features. This is all great but I have an important requirement: I cannot assume when a data point comes in to be stored, it can be highly variable and I can have a thousand points in 1 hour, then nothing for a week. So I need to store the timestamp. Turns out the guys over at OpenEnergyMonitor have done a lot of work based on Timestore. They had the same problems to solve, getting a lot of data out of log databases, queried/aggregated and returned as fast as possible. Their EmonCMS app used MySQL to log sensor data, which was increasingly slow as tables got very deep. So they created PHP storage variations based off Timestore to allow logging variable interval data. Thanks guys for sharing your work and research results!

After considering all these options, I decided to build my own node.js storage engine for logging variable time based sensor data based on bits and pieces from Timestore and the variable interval engine from OEM. The main reason is that it needed to be all in node.js. Doing it in PHP and then hitting a PHP script from a node.js socket app to query/post data felt very wrong (not sure how performant either).

The result is a much faster binary storage engine that can query and aggregate weeks worth of data in ~500ms (on a Pi B Rev1 256RAM!) and ~80-200ms on a Pi 2 (quad 900mhz, 1GBRAM), and much faster on Pi3. That’s more than a magnitude faster than neDB which was increasingly slower as data accumulated.

Upgrading from old neDB gatewayLog.db data

If you’ve used my Gateway interface before this new storage engine was implemented, you probably have log data stored as JSON in /home/pi/gateway/gatewayLog.js. To upgrade just save your existing gatewayLog.db, upgrade your Pi to the latest release, copy the gatewayLog.db into /home/pi/gateway/data/db, then run this upgrade script in the same directory – it parses your gatewayLog.db data and puts it into binary files in a db subdirectory, ready for the new gateway.js script to use. Otherwise gateway.js will just create logs from scratch as data comes in.

UpgradeGateway

John's tutorials

John from John’s DIY Playground has been developing a bunch of interesting projects and node customizations around this gateway software. He has been using a MoteinoUSB as a gateway for his Moteino wireless network.

Here’s his overview of installing the gateway image to his RaspberryP. He also walks through the basic settings setup and interacting with a node for the first time:

In the following video John goes into the details of the configuration files that are behind the gateway software. It’s a good walkthrogh for those that want to learn more about customizing their nodes and understanding more of the data and setting structures. Some features shown are obsolete, for instance settings can now be edited directly in the UI, and the gateway app can be restarted from the settings page also, no need to pkill to restart the gateway.

In this video he shows how to add a simple Moteino node that can turn an LED on/off from the Gateway web interface, and also reports the light level from a light sensor:

One my favorite video’s John ever made shows a completely custom node that controls and monitors his utility room. A perfect example of how to build both a custom hardware node with different attached peripherals, as well as how to setup a new customized gateway node to control and monitor that hardware: