Home automation with Moteino & MQTT

[nicbrink]

I wanted to take the time to report SUCCESS!  I really like the mysensors protocol and the flexibility it provides.  Now that it supports moteinos and RFM69 radios, I think it is the perfect combination.

Using the development branch of mysensors: https://github.com/mysensors/Arduino/tree/development/libraries/MySensors I was able to get a serial gateway and sensor node working by downloading it to my libraries folder.  Here are some notes / tips I discovered along the way. 

1.  Mysenors development branch does not have the latest rfm69 code from Felix, specifically the rx dead lock scenario.
     https://github.com/LowPowerLab/RFM69/commit/ed2fd5b8d55d011ed8164d9d517f364cc7841a0c
However it was very easy to update with the latest code.
2.  I had to edit the mySensors\MyConfig.h file prior to uploaded my sketch.  The file is dead simple, it really only had 2 options.  The first is to define a sensor / serial gateway / or MQTT.  The second option is for the radio.  Obviously RFM69.
3.  Next I had to edit the mySensors\MyDriverRF69.h to define all the cool rfm options such as the encrypt key, the network ID, and IS_RFM69HW option.
4.  It does not look like the gatewayid or nodeid in this file have any effect.  You have to set the nodeID in your main sensor sketch.
5.  I then basically implemented the code from this post on the mysensors forum.  http://forum.mysensors.org/topic/655/serial-gateway-connection-to-openhab/3.  I did not have to make other changes to the serialgateway sketch.
6.  In the sensor sketch from the previous post I had to change MySensor gw(48, 49); to MySensor gw;

Out side of that everything was good to go.  I wired up a dht, uploaded the sketches and I was off any running.

The next step is to try and get an MQTT gateway up and running.  I have a enc28j60 module, and it looks like the only change I would need to make is to the libraries\UIPEthernet\utility\Enc28J60Network.h file so that the CS pin can be redefined since it is already used from the RFM69 radio. 

//#define ENC28J60_CONTROL_CS     SS
#define ENC28J60_CONTROL_CS     7
#define SPI_MOSI        MOSI
#define SPI_MISO        MISO
#define SPI_SCK         SCK
//#define SPI_SS          SS
#define SPI_SS          7


I am not convinced that I need an mqtt gateway.  A moteino is small and pretty easy to attach to a raspberypi or beaglebone gateway.  There might be scenarios where someone might want the flexibility of a lan port provides.  I am not sure what they memory requirements will look like, or if there is a way to secure the MQTT messages.  I know that the protocol allows for an ssl type encryption, but I dont know if the arduino library supports it.

Thanks again to Felix, Hek, and everyone on this forum and the mysensors forum of all the great information. 

[Fwessels]

HI nicbrink any success with the MQTT gateway?

[BrianB]

Hi All,
Though I would jump in here with some thoughts and opinions because, well, that's what forums are for

I've been doing a LOT of research the past weeks into the current state of affairs in the home automation industry and the more recent injection of open source community projects and initiatives, which seem to be, indeed, moving and shaping the industry. Nice to see that!

I worked in the IT industry for more than 20 years and finished my career as a systems architect. Fancy title for the guy who figures out how to make crap work But, doing that, and being an engineer, taught me to try to look at things first from a 20,000 foot level and then drill down to find, and use, best practices and technologies that were most likely to achieve wide spread adoption and support, thus likely to endure the longest. This is my observations and opinions...just opinions guys

I see the likes of OpenHAB and PiDome trying to be all things to all people and trying to create a platform that will congregate the data and control of everything that's deployed out there already and everything that is to come in the future. In doing that, within their architectures there will always be "chasing the dog" scenarios in which for every new sensor/node/controller rolled out, a new definition has to be created to fit into the framework. Sorta like computer hardware guys that have to also build drivers for every stinking version of Windows! It will never end as long as every manufacturer out there is using a different protocol. As well, in the case of people like us on this forum who are experimenting with Felix's very cool Moteino's...many are concerned now too with building, or implementing, tools and/or gateways of some sort to translate your custom made protocol into something that can go back and forth with some kind of management/presentation platform which may turn out to be just a "flavor of the day".

I guess it goes without saying that the industry needs standards to move forward in a meaningful way if we are ever to really achieve interoperability like that envisioned in the Internet Of Things (IoT). Manufactures HATE standards because without their proprietary pieces, then they have to directly compete, and be better than, the competition. BUT..as we have seen in many industries, eventually standards are developed and adopted by large and small developers alike and the world becomes a better place. Think TCP/IP...

From what I can see, MQTT is going to become the defacto standard as a protocol for automation. Given that major industry leaders are quietly getting involved in, or buying other companies, that utilize MQTT, that will start to drive it's wide spread awareness, utilization, and implementation.  In examining MQTT myself, I believe it has all the needed flexibility and expandability to work well for automation. It's easy to imagine it being structured like SNMP but with a whole lot better security and much lower bandwidth requirements!!

It's also my opinion that manufacturers of devices, and that includes people like us here building cool Mote's, that the MQTT message structure should be pushed all the way back to the firmware on the device. When building your payload message, why not use an MQTT convention? In so doing, all your wireless to IP network gateway has to do is take your payload and forward it on to an MQTT broker. From there, some very nice management and control platforms could be built that only have to deal with getting and sending MQTT messages. That, in my opinion, removes a whole lot of the moving parts from the current paradigm that exists out there. Will it work for existing devices with every manufacturers existing protocols and api's...no. But, if every device manufacturer starts having to implement MQTT in their devices in order to be attractive to consumers and fit into some nice management platforms that will evolve, then they will...or die. That's the way this industry works...over and over and over again.

As I'm developing my Mote's, I will be working towards implementing MQTT message structures in some kind of sensible pattern and see how it all works out. Now, speaking of standards, we all need to structure our MQTT messages in the same format...ok, so we will all do it MY way ok?? LOL. I believe that too will come...again..think something like SNMP and it starts to make sense. Well, it does to me anyway...

[TomWS]

@BrianB, I've seen the 'fury' behind MQTT and, being a guy who likes to keep things simple, it was something that I purposefully avoided since I can build my own data structures and packet formats, thankyou...

However, after reading your post, I was intrigued by your argument justifying the likelihood of MQTT's success.  As a result of your post, I've come out of my shell and, at least, scoped out MQTT (on Wikipedia, of course).

Net: Thanks to you, I now have ONE MORE THING to think about and research!   

Thanks,
Tom

[BrianB]

@BrianB, I've seen the 'fury' behind MQTT and, being a guy who likes to keep things simple,

Tom, that is exactly why I think MQTT will be adopted. You will find that after your research (sorry about that) that it truly is a very simple concept and structure that can be easily implemented by manufacturers in firmware. I'm certainly not a sketch expert yet but I can easily envision small, tight, fast code that MQTT formats your payload just as easily, if not easier, then sticking colons between variables

Glad to hear you interested in the discussion at least...

[kiwisincebirth]

As I'm developing my Mote's, I will be working towards implementing MQTT message structures in some kind of sensible pattern and see how it all works out. Now, speaking of standards, we all need to structure our MQTT messages in the same format...ok, so we will all do it MY way ok?? LOL. I believe that too will come...again..think something like SNMP and it starts to make sense. Well, it does to me anyway...

Thanks for your post, and I agree with MQTT as a hub, but think pushing it down to the RFM69 layer is not advisable, especially since one of the main use cases is battery operation, where every byte transmitted costs precious milliamps. I have gone for a very small binary structure, with a Mote with ethernet shield acting as a gateway to MQTT. In that conversion the binary payload gets converted to a Json payload.

Kiwi

[BrianB]

Thanks for your post, and I agree with MQTT as a hub, but think pushing it down to the RFM69 layer is not advisable, especially since one of the main use cases is battery operation, where every byte transmitted costs precious milliamps. I have gone for a very small binary structure, with a Mote with ethernet shield acting as a gateway to MQTT. In that conversion the binary payload gets converted to a Json payload.

Kiwi

Thanks Kiwi. I absolutely agree that a mandate must be to minimize RF data transmission in sending/receiving meaningful messages to devices. Indeed, we can all design our own rf protocols and pack what we want/need into it very tightly, but, again that takes us away from interoperability. If our common objective, as an industry, is to build and deploy devices that can all inter operate together using common protocols and methods, even across rf networks, then I think MQTT out to the device is also necessary.

And, I believe that the MQTT protocol can be deployed in such a way as to also keep data transmission to a minimum as well. For example, in the snmp protocol, devices, objects, and variables are all represented by a byte or two each separated by periods when messages are transmitted. Those numbers are translated to human readable notation by the management/presentation software translating a "mib" file which defines what the device is and what the individual parameters are and mean. In my work so far, I believe its possible to manage and control a device, which has any number of variables which can be read or set, with MQTT messages, in a message payload of of around 40 bytes. or less. That's pretty efficient and low power RF friendly in my opinion.

When a MQTT message is received by the RF to IP gateway device, it appends on a few more sub topics on the front of the MQTT message for identification and management purposes, then sends that on to the MQTT broker. All done.

Now, the presentation/management system, which subscribes to the relevant MQTT topics on the broker, takes those "/ encoded" MQTT messages and translates them into something human readable. This opens up the possibility for an endless number of management station designs using any number of presentation methods. Json may well be a very likely candidate for that. It could be anything but the point is that all the devices out there are all using the same MQTT messaging format so that maybe, someday, we can achieve interoperability!

[ricka]

For battery operated devices and memory constrained you can use mqtt-sn protocol. Works with sleepy devices.

Documentation on both, MQTT & MQTT-SN: http://mqtt.org/documentation

Open source projects:
https://github.com/X13home
https://github.com/ty4tw/MQTT-SN
https://github.com/boriz/MQTT-SN-Arduino
https://github.com/bittailor/BtMqttSn

[TomWS]

For battery operated devices and memory constrained you can use mqtt-sn protocol. Works with sleepy devices.

Documentation on both, MQTT & MQTT-SN: http://mqtt.org/documentation

Open source projects:
https://github.com/X13home
https://github.com/ty4tw/MQTT-SN
https://github.com/boriz/MQTT-SN-Arduino
https://github.com/bittailor/BtMqttSn

Thanks for the tip and links!  I'll look into it (now that my 'service' provider decided to restore service to our house! - been out since Sunday night and it only took 4 calls and two tech visits to fix...).

Tom

[BrianB]

Thanks ricka...

I've been watching the MQTT-SN development as well. The way I look at it is that, yes, it's pushing the MQTT protocol out to the node, but the architecture still requires a gateway to forward the messages on to the broker on the IP network. An RF to IP gateway will be required however you try to do this. Implementing the MQTT-SN library into each node requires additional code overhead, memory, and resources. As well, it requires that each node be pre-programmed (hard coded) with very specific variables to talk with specific gateways and mqtt message topics.

I don't think it's necessary to put an MQTT library on the devices. In my mind and opinion, all the devices have to do is transmit their data formatted into a predefined MQTT topic format. The device doesn't even need to know anything about MQTT or MQTT messages, it simply has to format it's output strings/messages into a format that is MQTT friendly and send it to the gateway. No more overhead then how everyone currently formats their data in one way or another as a string of data separated by colons or whatever. The "RF to IP" gateway takes care of understanding how to talk to an MQTT broker. With this concept, and in my rough design so far, any given device only needs to be hard coded with a manufacturer id, model number, and unique identifier (say a mac address), and a default network id (say 254) to start up. From their, the gateway, through a dhcp type exchange, can assign it a new network id and now knows it exists and what it is and what mqtt topics to get and send to it for management.

In my mind, to achieve interoperability, any given device, designed to work on a "moteino" rf network should be able to just plug in a go...lol..in my mind. Let's see if we can get there

[ricka]

MQTT-SN protocol specification allows you don't implement all features. You can implement only required features:

6.8 PUBLISH with QoS Level -1
This feature is defined for very simple client implementations which do not support any other features except
this one. There is no connection setup nor tear down, no registration nor subscription. The client just sends its
PUBLISH messages to a GW (whose address is known a-priori by the client) and forgets them. It does not care
whether the GW address is correct, whether the GW is alive, or whether the messages arrive at the GW.

MQTT-SN protocol specification is a good roadmap how to implement your protocol with minimal efforts and wheel reinvention. For RF protocol you can use Felix's library, jeelabs or radiohead library. It will be interoperability issues  with devices using different software libraries/RF protocols, RF chips, RF modulations and etc.

[TomWS]

MQTT-SN protocol specification allows you don't implement all features. You can implement only required features:

6.8 PUBLISH with QoS Level -1
This feature is defined for very simple client implementations which do not support any other features except
this one. There is no connection setup nor tear down, no registration nor subscription. The client just sends its
PUBLISH messages to a GW (whose address is known a-priori by the client) and forgets them. It does not care
whether the GW address is correct, whether the GW is alive, or whether the messages arrive at the GW.

<snip...>

Gee, those guys must have been looking over my shoulder when they came up with this one...   

Still have to read the specs.  It should be an interesting learning time, in any case...

Tom