RaspberryPi home automation gateway – Setup

In this third part of the RaspberyPi home automation gateway series I’d like to present the RaspberryPi software stack that will allow a secured realtime webserver-websocket gateway to your home automation project.

As a webserver I decided to try Nginx. I’ve previously used Apache which always felt a bit slugish and had a large disk and memory footprint. There are many bechmarks that contrast webservers performance, here’s a nice one that focuses on RaspberryPi, and it includes Nginx and Apache.

The stack will be a webserver (Nginx) and a websocket server (Socket.io). The websocket will be proxied through Nginx, and everything will be encrypted with SSL and authenticated with auth_basic. A bunch of other things need to be setup and configured to make these work nicely together.

Unfortunately it’s not as easy as “apt-get install”-ing everything. The reason being that both Nginx and Node.js versions are behind on the apt-get repository. The websocket proxy feature is only available in Nginx 1.3.13 and above. Socket.io also needs a fairly recent version of Node.js. So both of those need to be installed manually. Before that, I’ve actually tried to point apt-get to other repositories and download more recent versions but that created a huge mess on my Pi and had to start from scratch. In case you’re wondering, Apache also has a websocket proxy module but after a few hours of trying to get it on my Pi (without disturbing the current EmonCMS setup) I gave up and decided to try Nginx on a fresh SD card and see how it goes. Turns out it went well, and it’s a very solid setup, haven’t seen Nginx or Socket.io fail in the year following this writing.

By the way, I started with a fresh image of the latest Wheezy-Raspbian distribution (2014-09-09 at the time of this install). Here are a few things that are needed or nice to have before you dive into the heavier installs:

First boot setup, in raspi-config:

  • enable SSH (already enabled as of raspbian Sept 9 2014 distro).
  • set GPU memory at 16MB (I only use the terminal on my Pi)
  • 900Mhz overclock (medium)
  • configure locale and keyboard layout (default is UK)
  • set time and time zone (in raspi-config or with sudo dpkg-reconfigure tzdata)
  • install ATXRaspi (if you got one)

Other misc stuff I did:

  • enable serial port for GPIO (I’ve got a gateway Moteino running on the GPIO serial port), here’s a how-to guide for that. You can do this in raspi-config as of raspbian Sept 9 2014 distro.
  • install ProFTPD to enable FTP for easy web files uploading to your Pi: sudo apt-get install proftpd (choose standalone, inetd didn’t work for me)

Also good to update everything before installing new stuff:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

Installing Nginx 1.6.2 from source

sudo apt-get install build-essential -y
sudo apt-get install libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev -y
sudo mkdir ~/sources
cd ~/sources/
sudo wget http://nginx.org/download/nginx-1.6.2.tar.gz
sudo tar zxf nginx-1.6.2.tar.gz
cd nginx-1.6.2
sudo ./configure \
    --prefix=/usr \
    --conf-path=/etc/nginx/nginx.conf \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --pid-path=/var/run/nginx.pid \
    --lock-path=/var/lock/nginx.lock \
    --with-http_ssl_module \
    --user=www-data \
    --group=www-data \
    --with-http_stub_status_module \
    --with-http_gzip_static_module \
    --without-mail_pop3_module \
    --without-mail_imap_module \
    --without-mail_smtp_module
sudo make
sudo make install

Also install PHP, it will be needed for my example code, and if you’re running things like EmonCMS:

sudo apt-get install php5-common php5-cli php5-fpm

Nginx should now be installed in the /etc/nginx folder. Here’s how to configure it to start up when RaspberryPi boots, this command will create a new file and open it in the editor, dump this content and save it:

sudo nano /etc/init.d/nginx
#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
    . /etc/default/nginx
fi

set -e

. /lib/lsb/init-functions

case "$1" in
  start)
    echo -n "Starting $DESC: "
    start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid \
        --exec $DAEMON -- $DAEMON_OPTS || true
    echo "$NAME."
    ;;
  stop)
    echo -n "Stopping $DESC: "
    start-stop-daemon --stop --quiet --pidfile /var/run/$NAME.pid \
        --exec $DAEMON || true
    echo "$NAME."
    ;;
  restart|force-reload)
    echo -n "Restarting $DESC: "
    start-stop-daemon --stop --quiet --pidfile \
        /var/run/$NAME.pid --exec $DAEMON || true
    sleep 1
    start-stop-daemon --start --quiet --pidfile \
        /var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true
    echo "$NAME."
    ;;
  reload)
      echo -n "Reloading $DESC configuration: "
      start-stop-daemon --stop --signal HUP --quiet --pidfile /var/run/$NAME.pid \
          --exec $DAEMON || true
      echo "$NAME."
      ;;
  status)
      status_of_proc -p /var/run/$NAME.pid "$DAEMON" nginx && exit 0 || exit $?
      ;;
  *)
    N=/etc/init.d/$NAME
    echo "Usage: $N {start|stop|restart|reload|force-reload|status}" >&2
    exit 1
    ;;
esac

exit 0

Now make the file executable and register it with update-rc.d:

sudo chmod +x /etc/init.d/nginx
sudo update-rc.d -f nginx defaults

Create the configuration directories for nginx and the web directory for the default website:

sudo mkdir /etc/nginx/{sites-available,sites-enabled}
sudo mkdir /var/www/default -p
sudo chown pi /var/www              #allow FTP-ing files to your www dirs
sudo chown pi /var/www/default      #allow FTP-ing files to your www dirs

Edit the nginx server config file and make it point the additional site-level config files:

sudo nano /etc/nginx/nginx.conf

Dump this configuration in the nginx.config file, and save it:

user  www-data;
worker_processes  2;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/sites-enabled/*;
}

Creating a self signed SSL certificate for use with Nginx

You could use a certified signature SSL certificate which you’ll have to pay for. I prefer my own self signed since I trust my own certificate. This will create a 2048 bit RSA cypher that will expire in 365 days. If the expiration gives you trouble you can omit that parameter. It will ask you for your several details like location, name etc. These will appear in the certificate details when your browser will alert you that you are opening an untrusted SSL pipe. You can always delete it and start over if you want a new one:

sudo mkdir /etc/nginx/ssl
cd /etc/nginx/ssl
sudo openssl req -new -x509 -nodes -days 365 -newkey rsa:2048 -out server.crt -keyout server.key

Enabling auth_basic in NGINX

For authentication I used basic HTTP authentication. The way this works is like this: once you open the site you will be greeted with a generic browser authentication dialog. The credentials will be base64 encoded and transmitted to the server that will compare them to the encrypted and salted credentials stored in the .htpasswd file we’re about to create. Basically the base64 is not encryption, just an encoding that could easily be intercepted and decoded by an attacker. Fortunately, we will be using SSL which will encrypt everything between the browser and the webserver, an essential component of any secure web connection. Here’s how to make the .htpasswd file, (it doesn’t even need to have that name, but as you will see later – we will block any access to any files that start with “.ht”). This command will create a username/password credential. Replace USERNAME, PASSWORD and SALT with your own chosen tokens, then copy the output from the first command and paste it in a new .htpasswd file using sudo nano):

sudo echo -e "USERNAME:`perl -le 'print crypt("PASSWORD","SALT")'`"
sudo nano /var/www/default/.htpasswd

By the way, you could create multiple credentials with this command and place them each on a newline in the .htpassword to allow multiple users with different username/password combinations, if you need to.

Now you’re ready to create the default site config file, enable SSL basic_auth and websocket proxying, given the files created in the previous steps (the SSL certificate .key and .crt files, and the .htpasswd basic authentication credentials file). Pay attention to the line “proxy_pass http://localhost:8080”. This is very important for socket.io, it will redirect all socket traffic from poublic port 80 to internal port 8080 where socket.io is actually listening. So your internal “secret” port 8080 is now hidden from the rest of the internet world, awesome. This port number should be the same port number as the one in the chat.js application below and also in gateway.js later on:

sudo nano /etc/nginx/sites-available/default
#use upstream when you want to do load balancing on your websocket
#upstream backend {
#  server localhost:8080;
#  server localhost:8081;
#}

server{
  listen 80;
  listen 443 ssl;
  server_name default;

  # enable basic authentication and point to the credentials file
  auth_basic "Restricted";
  auth_basic_user_file /var/www/default/.htpasswd;

  # point to the SSL certificate required for SSL
  ssl_certificate /etc/nginx/ssl/server.crt;
  ssl_certificate_key /etc/nginx/ssl/server.key;

  access_log /var/log/nginx/default.access.log;
  error_log /var/log/nginx/default.error.log;

  # Redirect all non-SSL traffic to SSL.
  if ($ssl_protocol = "") {
    rewrite ^ https://$host$request_uri? permanent;
  }

  # point to the www directory of this site, and setup the index pages
  root /var/www/default;
  index index.html index.php index.htm; ## first that is found is matched

  # enable websocket proxying to a backend socket server running on a custom port
  # by default, the endpoint for the socket.io websocket server is “socket.io”
  location /socket.io/ {
    proxy_pass http://localhost:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection upgrade;
  }

  # enable and configure PHP
  location ~* \.php$ {
    fastcgi_split_path_info ^(.+\.php)(.*)$;

    # typically the fpm.sock is used, but check to make sure by using:
    # sudo grep -ri "listen = " /etc/php5/fpm/
    # should get something like: 
    #  /etc/php5/fpm/pool.d/www.conf:listen = /var/run/php5-fpm.sock
    # then you know it's fpm.sock
    fastcgi_pass unix:/var/run/php5-fpm.sock; #uncomment if PHP runs on fpm.sock
    #fastcgi_pass 127.0.0.1:9000; #uncomment if PHP is running on port 9000

    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /var/www/default$fastcgi_script_name;
    fastcgi_param QUERY_STRING $query_string;
    fastcgi_param REQUEST_METHOD $request_method;
    fastcgi_param CONTENT_TYPE $content_type;
    fastcgi_param CONTENT_LENGTH $content_length;
    include fastcgi_params;
  }

  # redirect server error pages to the static page /50x.html
  error_page 500 502 503 504 /50x.html;
  location = /50x.html {
      root html;
  }

  ## Disable viewing .htaccess & .htpassword
  location ~ /\.ht {
      deny all;
  }
}

This will only create the configuration file for the default site. You need to publish it to the sites-available directory before it is picked up by Nginx, which will need to be restarted.

cd /etc/nginx/sites-enabled/
sudo ln -s /etc/nginx/sites-available/default
sudo service nginx restart

To test your NGINX setup, create the following index.php file in the /var/www/default directory:

<?php phpinfo(); ?>

Open a browser on another computer and load your Pi’s IP address, you should be warned about the SSL certificate (which you trust cause you made it right?) and after you take the risk and continue you should be asked for the auth_basic credentials. Once you get past that you should see the PHP information splash, letting you know that the Nginx-PHP combo is running as expected.

Ok so hopefully you were able to get to this point. If basic_auth gives you trouble, skip that step and comment out the auth_basic lines in /etc/nginx/sites-available/default. This is the first sub-part of this setup, the rest is dealing with node.js and testing the whole thing together with Nginx, PHP and Socket.IO.

Installing Node.js,  Socket.IO,  node-serialport

One great feature of socket.io is cross-browser compatibility. If your browser or mobile device does not support web sockets natively, it will fall back to other techniques to still make the socket fast enough for realtime action. The node.js version that I found to be working with socket.io was 0.10.28. I tried other versions and had a ton of trouble, spent many hours trying to get past tons of build errors. So I will stick with 0.10.28 for now and hopefully all these stable versions will become published in apt-get and npm for easy command line install.
Make a directory for Node.js and install it from the arm binaries:

sudo mkdir /opt/node
cd ~/sources
sudo wget http://nodejs.org/dist/v0.10.28/node-v0.10.28-linux-arm-pi.tar.gz
sudo tar xvzf node-v0.10.28-linux-arm-pi.tar.gz
sudo cp -r node-v0.10.28-linux-arm-pi/* /opt/node

Now add node to the environment PATH so you can execute it from anywhere:

sudo nano /etc/profile
...
NODE_JS_HOME="/opt/node"
PATH="$PATH:$NODE_JS_HOME/bin"
export PATH
...

Then log out and log back in to register the node PATH. You can now check the installed version of node and npm:

node -v   (0.10.28)
npm -v    (1.4.9)

You may now delete the ~/sources temporary directory where you downloaded and unpacked the source/binaries, and an apt-get clean, to free up some space:

sudo rm –rf ~/sources
sudo apt-get clean

Next up is socket.io and node-serialport. Typically node packages will install in a subdirectory where you execute the npm install command, ie in the directory where your node app is located. In my case I am running from a home subdirectory dedicated to the Moteino gateway:

sudo mkdir ~/moteino
sudo chown pi moteino
cd ~/moteino

Now install node-serialport, which is needed to interface node.js to your Arduino/Moteino or whatever else you are relaying to your home automation hardware. This module gave me a bit of trouble also, but at the time of writing this it seems that it works fine with node 0.10.12:

npm install serialport

The install command should download the dependencies and source and build the module. Then you can check that it was built correctly by running this command that I found somewhere on StackOverflow during the hurdles I’ve had installing serialport, thanks to whoever came up with it, you should get “Looks OK” if everything went ok:

cd ~/moteino/node_modules/serialport
if [ "x`find . -name "serial*node"`" = "x" ]; then echo "Something sucks"; else echo "Looks OK"; fi

Also, here’s a screenshot of the output during the install and the above check of serialport:
serialportinstall

Finally, install socket.io in the same directory:

cd ~/moteino
sudo npm install socket.io

This installs socket.io version 1.2.1.

Testing it all

If you got to this point and everything went ok, you’re ready to test the whole stack we’ve insatlled so far. Let’s create a simple node.js chat application (adapted from here):

cd ~/moteino
sudo nano chat.js

Now paste this content and save the file:

//this port has to be configured the same number in file /etc/nginx/sites-available/default on line "proxy_pass http://localhost:8080"
var io = require('socket.io').listen(8080);

//to test your GPIO serial/uart connected Moteino/Arduino uncomment the following:
//var serialport = require("serialport");
//var serial = new serialport.SerialPort("/dev/ttyAMA0", { baudrate : 115200, parser: serialport.parsers.readline("\n") });
 
//authorize handshake - make sure the request is coming from nginx, not from the outside world
//if you comment out this section, you will be able to hit this socket directly at the port it's running at, from anywhere!
//this was tested on Socket.IO v1.2.1 and will not work on older versions
io.use(function(socket, next) {
 var handshakeData = socket.request;
 console.log("AUTHORIZING CONNECTION FROM " + handshakeData.connection.remoteAddress + ":" + handshakeData.connection.remotePort);
 if (handshakeData.connection.remoteAddress == "localhost" || handshakeData.connection.remoteAddress == "127.0.0.1")
 next();
 next(new Error('REJECTED IDENTITY, not coming from localhost'));
});
 
// usernames which are currently connected to the chat
var usernames = {};
 
// classic chat demo used for socket.io
io.sockets.on('connection', function (socket) {
 // when the client emits 'sendchat', this listens and executes
 socket.on('sendchat', function (data) {
 // we tell the client to execute 'updatechat' with 2 parameters
 io.sockets.emit('updatechat', socket.username, data);
 });
 
 // when the client emits 'adduser', this listens and executes
 socket.on('adduser', function(username){
 // we store the username in the socket session for this client
 socket.username = username;
 // add the client's username to the global list
 usernames[username] = username;
 // echo to client they've connected
 socket.emit('updatechat', 'SERVER', 'you have connected');
 // echo globally (all clients) that a person has connected
 socket.broadcast.emit('updatechat', 'SERVER', username + ' has connected');
 // update the list of users in chat, client-side
 io.sockets.emit('updateusers', usernames);
 });
 
 // when the user disconnects.. perform this
 socket.on('disconnect', function(){
 // remove the username from global usernames list
 delete usernames[socket.username];
 // update list of users in chat, client-side
 io.sockets.emit('updateusers', usernames);
 // echo globally that this client has left
 socket.broadcast.emit('updatechat', 'SERVER', socket.username + ' has disconnected');
 });
});
 
// to test your GPIO serial/uart connected Moteino/Arduino uncomment the following:
//// listen to data from Moteino and just dump it on a socket pipe
//serial.on("data", function (data) {
// io.sockets.emit('SERIALOUTPUT', data);
//});

Create a HTML file that will interface with the chat app:

cd /var/www/default
sudo nano chat.html

Paste this content and save, note that you will need to get the socket.io.js client script from here and place it in the same /var/www/default directory, also replace YOUR_PI_ADDRESS with the real thing:

<!-- MAKE SURE TO CHANGE THE IP ADDRESS TO YOUR OWN, SEE BELOW -->
<script src="https://cdn.socket.io/socket.io-1.2.1.js"></script>
<script src="https://code.jquery.com/jquery-1.11.1.js"></script>
<script type="text/javascript">
 //be sure to replace the string with the IP address of your PI
 //if you hit it from outside your LAN then put your public IP address and
 //enable port forwarding on your router such that all WWW (port 80) traffic goes to your internal PI address
 var socket = io.connect('192.168.2.107', {'connect timeout': 1000});

 // on connection to server, ask for user's name with an anonymous callback
 socket.on('connect', function(){
 // call the server-side function 'adduser' and send one parameter (value of prompt)
 socket.emit('adduser', prompt("What's your name?"));
 });
 
 // listener, whenever the server emits 'updatechat', this updates the chat body
 socket.on('updatechat', function (username, data) {
 $('#conversation').append('<b>'+username + ':</b> ' + data + '<br>');
 });
 
 // listener, whenever the server emits 'updateusers', this updates the username list
 socket.on('updateusers', function(data) {
 $('#users').empty();
 $.each(data, function(key, value) {
 $('#users').append('<div>' + key + '</div>');
 });
 });
 
 // on load of page
 $(function(){
 // when the client clicks SEND
 $('#datasend').click( function() {
 var message = $('#data').val();
 $('#data').val('');
 // tell server to execute 'sendchat' and send along one parameter
 socket.emit('sendchat', message);
 });
 
 // when the client hits ENTER on their keyboard
 $('#data').keypress(function(e) {
 if(e.which == 13) {
 $(this).blur();
 $('#datasend').focus().click();
 }
 });
 });
</script>
<div style="float:left;width:100px;border-right:1px solid black;height:300px;padding:10px;overflow:scroll-y;">
 <b>USERS</b>
 <div id="users"></div>
</div>
<div style="float:left;width:300px;height:250px;overflow:scroll-y;padding:10px;">
 <div id="conversation"></div>
 <input id="data" style="width:200px;" />
 <input type="button" id="datasend" value="send" />
</div>

Now start the chat.js app:

cd ~/moteino
node chat.js &             #this runs your chat.js app in the background

Open a browser on another computer and load your Pi’s address + chat.html, you may see the SSL warning you’ve seen before in the PHP test step (or depending on your browser, it may actually remember your answer and never ask you again as long as the certificate won’t change). Then you should get the basic authentication dialog where you need to enter the credentials you generated for the .htpasswd file, then finally your chat app should show up, ask for your name and then you can enter messages that will be broadcast to all the users connected to it. Note that you are requesting the web content and the websocket connection at the same address and port, because of the websocket proxy feature of Nginx, and the socket will reject any direct requests:

And here’s the console output:

Congratulations! You are now ready to build a secure gateway to your home automation using this or a similar stack. There are always tradeoffs to any stack that makes something like this possible. As other users suggested, there are other alternatives that run a webserver+websocket+SSL all in one python or node.js process. Those could work just fine, but I prefer a scalable modular approach which makes use of robust parts that have an established footprint in their respective market, and which are best at doing the one thing I need them for.

You can already see how long this post ended up being, and it made sense to keep this one all together. And lots of things can go wrong during such an install. I suppose that’s one of the disadvantages that come with having a modular stack like the one presented here. I actually had to go through this setup twice myself to prepare this post, and took notes at every step to make sure it all works. Hopefully I did not miss anything.

In the next and perhaps final part of this post series, I will show the firmware/hardware that I used to control my garage door using a Moteino, complete with sensors that detect whether the door is either open/closed or somewhere in between, and the ability to trigger a door open/close command from anywhere in the world.