Pi-Gateway alternative, Moteino Gateway (MGW)

[zynzel]

Hello all!

First of all, Felix - thank you for yours hard work on Moteino.
This is really great piece of hardware.

I plan to use Moteino boards as base hardware for my alarm/metric system.

Unfortunately i was not happy with Pi-Gateway because of:
* hard to implement custom actions
* some metrics was not saved properly (no update in DB)
* everything in one place - gateway.js
* nodejs

So i start working on my own gateway. You can find result at https://github.com/bkupidura/mgw

A few words about MGW:
* python
* MQTT (http://mqtt.org/)
* twitter bootstrap
* JSON

Project assumptions:
* Modular, in future you can extend it with new code
* Accept any kind of metrics (not only already know by MGW)
* Allow to easy write any kind of action
* Allow to easy build some action logic (notify when value==X and alarm is armed but only 1 time per Y)
* Rock-solid (handle as many failure scenarios as possible)

What is already done:
* moteino-gateway - save metrics to DB and handle actions
* moteino-srl - read data from serial and pass it via MQTT to gateway
* moteino-dbaq - allow to perform DB queries (ex. find missing sensors)
* moteino-fence - arm/disarm alarm based on geofencing
* moteino-api - dashboard+api which allow to send messaged to other services

There is no need to use all services if You dont need them. Every service is independent from others, all use MQTT to communicate.

Have external sensor boards without MQTT/serial? Use HTTP API to send metrics!

Sensor definition:
  "motion": {
    "action": [{"name": "send_sms_at", "failback": [{"name": "send_sms"}]}, {"name": "log"}],
    "check_if_armed": {"default": 1, "except": ["12"]},
    "action_interval": 360,
    "threshold": "lambda x: int(x)==1",
    "fail_count": 0,
    "fail_interval": 0,
    "message_template": "Motion detected on board {board_desc} ({board_id})",
    "priority": 0
  }

action will perform send_sms_at (use local modem to send SMS) and log message to file. If send_sms_at will fail, execute send_sms (use external API to send SMS).
Action failbacks can be nested (if A fail, execute B, if B fail, execute C...)

check_if_armed, action will be executed only when alarm is armed, unless metric is from board=12.

action_interval - action can't be performend more often than this

threshold - about what values we want to be informed? This part use python eval() to check data reported by boards

fail_count - how many values in threshold should be reported before action

fail_interval - time window for fail_count, older values will be removed

message_template - action can use this template to send 'human' messages instead of 'board with ID 10 reports value 20 for metric type voltage'

priority - priority of metric. Lowest priority will be handled first. If 100 boards reports voltage in the same time, and 1 board report motion - motion actions will be handled first.

What next:
* Switch from sqlite to smth else (some sql+rrd?)
* html part is ugly
* work on documentation
* prepare more actions

Any help/contribution/opinion/ideas are welcome!

PS. sorry for my bad english - im working on it, but always could have been better.

[WhiteHare]

More detail on exactly how the external sensor with rfm69 uses http api to send mqtt message?  Perhaps a concrete example with details would help.

[zynzel]

Standard moteino boards comes only with RFM69 so in my environment they still connect to gateway board which send sensor data to serial (moteino-srl).

But there is no problem to build moteino based boards with ethernet shield which will report directly to MQTT.

Now you have 3 ways to report to MGW:
* via serial (mgw-srl)
* via API (http proxy to MQTT)
* via MQTT

I you use some ethernet/ip/tcp compatible board, you can report via HTTP with equivalent of
curl localhost:8080/api/action/mqtt -H "Content-Type: application/json" --data '{"data":{"board_id": "1", "sensor_type": "voltage", "sensor_data": "1"}, "topic":"mgw/metric"}'

Or directly to MQTT:
mosquitto_pub -t 'mgw/metric' -m '{"board_id": "1", "sensor_type": "voltage", "sensor_data": "1"}'

[WhiteHare]

 

[Sergegsx]

very interestring !

In order to learn python I am also coding my own gateway.py script.

I wanted to ask you, I want to implement Websocket connection in order to transmit data between html and python (in the future also from android app to be done).

I got working a websocket example (https://gist.github.com/jkp/3136208) but only allows single client. Any help on how to allow multiple clients? threading?
I see felix has used socket.io, I want to know your recommendation on how to implement similar thing using python.

Thanks a lot

[zynzel]

Hello,
I never used websocket.

But maybe instead of implementing everything from scratch use some ready code?

As i see autobahn-python is asynchronous.

https://github.com/crossbario/autobahn-python
http://autobahn.ws/python/websocket/programming.html

[Sergegsx]

after many hours looking into it, having a Socket example working, I learned I needed WebSocket. After one more full day I found Autobahn. It seems like the way to go for what I want to do.

I could have saved many hours with your post before this last weekend 
I appreciate your answer !

In your python implementation, you dont use WebSockets? How do you update a web browser, or any future Android app for example?

thanks a lot zynzel

[zynzel]

MGW use twitter bootstrap+ajax for dashboard frontend.
This way dashboard should look (and work) the same on any web-browser. I dont plan to write dedicated iOS/Android App.
User can use phone web-browser (+VPN).

Data from backend is simply downloaded from JSON API via ajax.

You can try to play with API with standard wget/curl:

# curl localhost:8080/api/action/status
{"armed": 0, "msd": 1, "mgw": 1, "srl": 1, "fence": 1}

# curl localhost:8080/api/node/1
[{"data": [["uptime", "98"], ["version", "6"]], "name": "1", "desc": "Gateway"}]

# curl -H "Content-Type: application/json" -X POST -d '{"start": 1449413530}' localhost:8080/api/graph/rssi
[{"data": [[1449413547000, -70.0], [1449413579000, -70.0], [1449413612000, -69.0]], "name": "Salon - balkon p"}]


Underneath HTML/Ajax part are calling API and show response to user in formatted way.
Please take a look at https://github.com/bkupidura/mgw/blob/master/moteino_sensors/api/static/index.html#L85-L126

[WhiteHare]

If you guys are casting a wide net, but want something simpler/easier,  Blynk is reportedly easy to get up and running on.  http://www.blynk.cc/

Node Red has its fans also.  https://learn.adafruit.com/raspberry-pi-hosting-node-red/what-is-node-red

[syrinxtech]

If you guys are casting a wide net, but want something simpler/easier,  Blynk is reportedly easy to get up and running on.  http://www.blynk.cc/

Node Red has its fans also.  https://learn.adafruit.com/raspberry-pi-hosting-node-red/what-is-node-red

I've been using Blynk for a while now.  If you're thinking of trying it, you can still use it for free!  They just posted a couple of days ago that they will be moving to some form of a pay-service in early January.  They haven't quite figured out how to monetize it yet but I'm sure they will.  I'm currently using Blynk to control an ESP8266 and a relay to allow my wife to control a large fan in the room.