Moteino and Emoncms [Solved]

[mattt]

Hi, I've been searching the forums and google extensively on how to integrate a moteino temp node/gateway to send data via a python script to Emoncms. I was wondering if anyone has had success with this? I am using the oem_gateway python script from https://github.com/Jerome-github/oem_gateway. My sketches are from the struct_send, struct_receive examples sketches.

I have managed to get the python script to read the serial data from a modified struct_receive example, but the oem_gateway script does not recognize the serial data and wont post to the Emoncms database. I have tried to comment out all of the data not needed by the oem_gateway script but this caused the script not to receive any data at all while the arduino serial monitor was displaying just the values from the payload.

I have tried to write my own python script but all I can do is read the serial output from the struct_receive sketch. Does anyone tried any alternatives to Emoncms? I am just trying to set up a temp sensor network with a gateway connected to a server running Debian. Then I want to process the data received into web based GUI.

[Spexx]

My Raspberry is running with the oem_gateway since several weeks without problems 

If I understand you correct, in generally you receive the serial data from the Moteino? How do you connect the Moteino to the Raspberry, via the GPIO or via USB?

You need to edit the oemgateway.conf that serial data will be accepted, check this:

Code Select Expand

# OemGateway Configuration file example
# Copy this as oemgateway.conf (or any custom name) and edit settings

# Each listener and each buffer has 
# - a [[name]]: a unique string
# - a type: the name of the class it instantiates
# - a set of init_settings (depends on the type)
# - a set of runtime_settings (depends on the type)
# Both init_settings and runtime_settings sections must be defined, 
# even if empty. Init settings are used at initialization, and runtime
# settings are refreshed on a regular basis.

# All lines beginning with a '#' are comments and can be safely removed.

####################
# Gateway settings #
####################
[gateway]
# loglevel must be one of DEBUG, INFO, WARNING, ERROR, and CRITICAL
# see here : http://docs.python.org/2/library/logging.html
loglevel = DEBUG

#############
# Listeners #
#############
[listeners]

# This listener manages the RFM2Pi module
#[[RFM2Pi]]
#    type = OemGatewayRFM2PiListener
#    [[[init_settings]]]
#        com_port = /dev/ttyUSB0
#    [[[runtime_settings]]]
#        sgroup = 210
#        frequency = 4
#        baseid = 15
#        sendtimeinterval = 0

# This listener gets data from a socket
#[[Socket]]
#    type = OemGatewaySocketListener
#    [[[init_settings]]]
#        port_nb = 50011
#    [[[runtime_settings]]]

# This listener manages the serial
[[Serial]]
    type = OemGatewaySerialListener
    [[[init_settings]]]
        com_port = /dev/ttyUSB0
    [[[runtime_settings]]]


###########
# Buffers #
###########
[buffers]

# The two following buffers instantiate the same class, 
# that formats the data for an emoncms instance.
# One sends the data to a local instance, the other one
# to a distant one.
# If active is set to False, the buffer neither records nor sends any data,
# but it holds unsent data until active becomes True.
[[emoncms_local]]
    type = OemGatewayEmoncmsBuffer
    [[[init_settings]]]
    [[[runtime_settings]]]
        domain = localhost
        apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        protocol = http://
        active = False
        path = /emoncms

[[emoncms_remote]]
    type = OemGatewayEmoncmsBuffer
    [[[init_settings]]]
    [[[runtime_settings]]]
        domain = emoncms.org
        apikey = xxxxxxxxxx change to your own apikey xxxxxxxxxxxx
        protocol = http://
        active = True
        path =

Don't forget to change the apikey. In case of using the GPIO, you have to change the com_port to /dev/ttyAMA0.

With the Moteino you have to send the serial data in the format NodeID Value1 Value2 Value3 and so on. I use a very simple sketch with the RFM69:

Code Select Expand

// Note: Serial data which are beginning with "> " will be ignored from oem_gateway
#include <RFM69.h>
#include <SPI.h>

// ########################################################################################################################
// Setup for RFM69
// ########################################################################################################################
#define NODEID      1
#define NETWORKID   222
#define FREQUENCY   RF69_868MHZ         // Match this with the version of your Moteino! (others: RF69_433MHZ, RF69_868MHZ)
#define KEY         "1234567890123456"  // has to be same 16 characters/bytes on all nodes, not more not less!
#define LED         9
#define SERIAL_BAUD 115200

RFM69 radio;
boolean promiscuousMode = false;        // set to 'true' to sniff all packets on the same network
boolean ACK = true;                    

// ########################################################################################################################
// Data Structure to be sent
// ########################################################################################################################
typedef struct {
  int Vcc;        // Supply voltage
  int Value1;     // Temperatur
  int Value2;
} 
Payload;
Payload theData;

// ########################################################################################################################
// SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP
// ########################################################################################################################
void setup() {
  Serial.begin(SERIAL_BAUD);
  delay(10);

  radio.initialize(FREQUENCY,NODEID,NETWORKID);
  //radio.setHighPower(); //uncomment only for RFM69HW!
  radio.encrypt(KEY);
  radio.promiscuous(promiscuousMode);

  Serial.println("> RFM69-Receiver  ***************");
  Serial.println("> Listening @868 Mhz ************");
  Serial.println("> NodeID Vcc Value1 Value2 RSSI *");
  Serial.println("> *******************************");
}

// ########################################################################################################################
// LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP 
// ########################################################################################################################
void loop() {
  if (radio.receiveDone())
  {
    if (promiscuousMode)
    {
      Serial.print("> Node[");
      Serial.print(radio.SENDERID, DEC);
      Serial.print("] >> Node[");
      Serial.print(radio.TARGETID, DEC);
      Serial.println("]");
    }

    if (radio.DATALEN != sizeof(Payload))
      Serial.println("> Invalid payload received, not matching Payload struct!");
    else
    {
      theData = *(Payload*)radio.DATA; //assume radio.DATA actually contains our struct and not something else

      Serial.print(radio.SENDERID, DEC);
      Serial.print(" ");
      Serial.print(theData.Vcc);  
      Serial.print(" ");
      Serial.print(theData.Value1);
      Serial.print(" ");
      Serial.print(theData.Value2);
      Serial.print(" ");
      Serial.println(radio.readRSSI());
    }

    if (radio.ACK_REQUESTED && ACK)
    {
      byte theNodeID = radio.SENDERID;
      radio.sendACK();
      Serial.println("> ACK sent");
    }

    Blink(LED,3);
  }
}


void Blink(byte PIN, int DELAY_MS)
{
  pinMode(PIN, OUTPUT);
  digitalWrite(PIN,HIGH);
  delay(DELAY_MS);
  digitalWrite(PIN,LOW);
}

I hope this will help you.

[mattt]

This is great thanks for the help! Would you be able to provide me with the node sketch also? I have the Moteino usb plugged into the usb on an Udoo. I didnt know that the oem_gateway script needed a ">" in front of serial data to ignore that specific data. My outputs are nodeid, temp, humidity and RSSI. The temp and humidity are float variables. I know i'm close this is the output I am getting from oem_gateway:

Code Select Expand

2014-07-25 17:53:20,710 INFO Serial RX: > ACK sent
2014-07-25 17:53:25,366 INFO Serial RX: 2 23.81 56.44 -44
2014-07-25 17:53:25,367 WARNING Misformed RX frame: ['2', '23.81', '56.44', '-44']

[Spexx]

Hmm I know this warning regarding the misformed RX frame, but I don't remember the reason for it. But maybe because of using float values? Try to use only integer values. You can easily convert the data directly in emoncms.org to the correct format.

Attached please find a simple sketch for the node. I don't know which sensors you are using so you have to modify it, but for testing it should works.

Code Select Expand

#include <RFM69.h>
#include <SPI.h>

#define NODEID      15
#define NETWORKID   222
#define GATEWAYID   1
#define FREQUENCY   RF69_868MHZ         //Match this with the version of your Moteino! (others: RF69_433MHZ, RF69_868MHZ)
#define KEY         "1234567890123456"  //has to be same 16 characters/bytes on all nodes, not more not less!

#define SERIAL_BAUD 115200

boolean requestACK = true;              
#define ACK_TIME    15                  // Numbers of ms to wait for an ack

RFM69 radio;

// ########################################################################################################################
// Data Structure to be sent
// ########################################################################################################################
typedef struct {
  int Vcc;        // Supply voltage
  int Value1;
  int Value2;
} 
Payload;
Payload theData;

// ########################################################################################################################
// SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP SETUP
// ########################################################################################################################
void setup(){
  Serial.begin(SERIAL_BAUD);  
  Serial.println("*** RF-Sender (868 MHz) ***");
  Serial.print("*** NodeID: "); 
  Serial.print(NODEID);
  Serial.print(", Group: "); 
  Serial.println(NETWORKID);
  Serial.println("***************************");

  radio.initialize(FREQUENCY,NODEID,NETWORKID);
  radio.encrypt(KEY);
  radio.sleep();
}

// ########################################################################################################################
// LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP LOOP
// ########################################################################################################################
void loop() {

  theData.Value1 = 1000;
  theData.Value2 = 2000;

  theData.Vcc = readVcc();       // Get supply voltage

  //Serielle Ausgabe
  Serial.print("Value1: ");
  Serial.println(theData.Value1);
  Serial.print("Value2: ");
  Serial.println(theData.Value2);
  Serial.print("Vcc: ");
  Serial.println(theData.Vcc);

  radio.send(GATEWAYID, (const void*)(&theData), sizeof(theData), requestACK); 
  if (requestACK)
  {
    if (waitForAck(GATEWAYID))
      Serial.println("> ACK ok");
    else 
      Serial.println("> no ACK!");
  }
  Serial.println("*****************");
  radio.sleep(); 

  delay(5000);
}


// ########################################################################################################################
// waitForAck
// ########################################################################################################################
//wait a few milliseconds for proper ACK to me, return true if indeed received
static bool waitForAck(byte theNodeID)
{
  long now = millis();
  while (millis() - now <= ACK_TIME) {
    if (radio.ACKReceived(theNodeID))
      return true;
  }
  return false;
}

// ########################################################################################################################
// Read current supply voltage
// ########################################################################################################################
long readVcc() {
  bitClear(PRR, PRADC); 
  ADCSRA |= bit(ADEN);             // Enable the ADC
  long result;
  // Read 1.1V reference against Vcc
#if defined(__AVR_ATtiny84__)
  ADMUX = _BV(MUX5) | _BV(MUX0);   // For ATtiny84
#else
  ADMUX = _BV(REFS0) | _BV(MUX3) | _BV(MUX2) | _BV(MUX1);   // For ATmega328
#endif
  delay(2);                        // Wait for Vref to settle
  ADCSRA |= _BV(ADSC);             // Convert
  while (bit_is_set(ADCSRA,ADSC));
  result = ADCL;
  result |= ADCH<<8;
  result = 1126400L / result;      // Back-calculate Vcc in mV
  ADCSRA &= ~ bit(ADEN); 
  bitSet(PRR, PRADC);              // Disable the ADC to save power   
  return result;
}

If needed I could provide you a sketch for the DS18B20 temperature sensor, which includes the "Low Power Library" for saving battery power.

[mattt]

Thanks for the sketch it looks almost identical to mine accept I am sending temp/hum values from an adafruit HTU21D-F. I tried changing the values to int instead of float. It still does the same. It does not make sense the string is printed as the oemgateway script is expecting. 3 values separated by one space. I've been looking through the oemgatewaylistener.py script and from what I can see it accepts float values. This is getting frustrating now I seem to have everything oemgateway expects but it still returns the same error.

Code Select Expand

2014-07-27 19:40:17,240 INFO Logging level set to DEBUG
2014-07-27 19:40:17,240 INFO OemGateway v1.0.2
2014-07-27 19:40:17,241 INFO Opening gateway...
2014-07-27 19:40:17,241 INFO Creating buffer emoncms_local
2014-07-27 19:40:17,241 INFO Creating buffer emoncms_remote
2014-07-27 19:40:17,242 INFO Creating listener RFM2Pi
2014-07-27 19:40:17,243 DEBUG Opening serial port: /dev/ttyUSB0
2014-07-27 19:40:17,245 INFO Setting send time interval to 0
2014-07-27 19:40:17,245 INFO Setting RFM2Pi | frequency: 4
2014-07-27 19:40:18,247 INFO Setting RFM2Pi | sgroup: 100
2014-07-27 19:40:19,249 INFO Setting RFM2Pi | baseid: 1
2014-07-27 19:40:20,251 INFO Creating listener Socket
2014-07-27 19:40:20,252 DEBUG Opening socket on port 50011
2014-07-27 19:40:20,259 INFO Serial RX: > RFM69-Receiver  ***************
2014-07-27 19:40:20,461 INFO Serial RX: > Listening @868 Mhz ************
2014-07-27 19:40:20,664 INFO Serial RX: > NodeID Vcc Value1 Value2 RSSI *
2014-07-27 19:40:20,866 INFO Serial RX: > *******************************
2014-07-27 19:40:21,068 INFO Serial RX: 2 25 56 -50
2014-07-27 19:40:21,068 WARNING Misformed RX frame: ['2', '25', '56', '-50']
2014-07-27 19:40:21,274 INFO Serial RX: > ACK sent
2014-07-27 19:40:25,327 INFO Serial RX: 2 25 56 -50
2014-07-27 19:40:25,328 WARNING Misformed RX frame: ['2', '25', '56', '-50']
2014-07-27 19:40:25,530 INFO Serial RX: > ACK sent
2014-07-27 19:40:30,193 INFO Serial RX: 2 25 56 -50
2014-07-27 19:40:30,194 WARNING Misformed RX frame: ['2', '25', '56', '-50']
2014-07-27 19:40:30,404 INFO Serial RX: > ACK sent
2014-07-27 19:40:35,275 INFO Serial RX: 2 25 56 -50
2014-07-27 19:40:35,276 WARNING Misformed RX frame: ['2', '25', '56', '-50']
2014-07-27 19:40:35,491 INFO Serial RX: > ACK sent

Here is the sketch from my node:

Code Select Expand

#include <RFM69.h>
#include <SPI.h>
#include <SPIFlash.h>
#include <Wire.h>
#include "Adafruit_HTU21DF.h"

#define NODEID      2
#define NETWORKID   100
#define GATEWAYID   1
#define FREQUENCY   RF69_433MHZ //Match this with the version of your Moteino! (others: RF69_433MHZ, RF69_868MHZ)
#define KEY         "thisIsEncryptKey" //has to be same 16 characters/bytes on all nodes, not more not less!
#define LED         9
#define SERIAL_BAUD 115200
#define ACK_TIME    5000  // # of ms to wait for an ack
int TRANSMITPERIOD = 5000; //transmit a packet to gateway so often (in ms)
byte sendSize=0;
boolean requestACK = false;
SPIFlash flash(8, 0xEF30); //EF40 for 16mbit windbond chip
Adafruit_HTU21DF htu = Adafruit_HTU21DF();
float celsius;
float humidity;
RFM69 radio;

typedef struct {		
  int           dataone; //store this nodeId
  //unsigned long uptime; //uptime in ms
  float         temp;   //temperature maybe?
  float         hum;
} Payload;
Payload theData;

void setup() {
  Serial.begin(SERIAL_BAUD);
  radio.initialize(FREQUENCY,NODEID,NETWORKID);
  radio.setHighPower(); //uncomment only for RFM69HW!
  radio.encrypt(KEY);
  char buff[50];
  sprintf(buff, "\nTransmitting at %d Mhz...", FREQUENCY==RF69_433MHZ ? 433 : FREQUENCY==RF69_868MHZ ? 868 : 915);
  Serial.println(buff);
  
  if (flash.initialize())
    Serial.println("SPI Flash Init OK!");
  else
    Serial.println("SPI Flash Init FAIL! (is chip present?)");
  if (!htu.begin()) {
    Serial.println("Couldn't find sensor!");
    while (1);
  }

}

long lastPeriod = -1;
void loop() {
	celsius = htu.readTemperature();
	humidity = htu.readHumidity();
  //process any serial input
  if (Serial.available() > 0)
  {
    char input = Serial.read();
    if (input >= 48 && input <= 57) //[0,9]
    {
      TRANSMITPERIOD = 100 * (input-48);
      if (TRANSMITPERIOD == 0) TRANSMITPERIOD = 1000;
      Serial.print("\nChanging delay to ");
      Serial.print(TRANSMITPERIOD);
      Serial.println("ms\n");
    }
    
    if (input == 'r') //d=dump register values
      radio.readAllRegs();
    //if (input == 'E') //E=enable encryption
    //  radio.encrypt(KEY);
    //if (input == 'e') //e=disable encryption
    //  radio.encrypt(null);
    
    if (input == 'd') //d=dump flash area
    {
      Serial.println("Flash content:");
      int counter = 0;

      while(counter<=256){
        Serial.print(flash.readByte(counter++), HEX);
        Serial.print('.');
      }
      while(flash.busy());
      Serial.println();
    }
    if (input == 'e')
    {
      Serial.print("Erasing Flash chip ... ");
      flash.chipErase();
      while(flash.busy());
      Serial.println("DONE");
    }
    if (input == 'i')
    {
      Serial.print("DeviceID: ");
      word jedecid = flash.readDeviceId();
      Serial.println(jedecid, HEX);
    }
  }

  //check for any received packets
  if (radio.receiveDone())
  {
    Serial.print('[');Serial.print(radio.SENDERID, DEC);Serial.print("] ");
    for (byte i = 0; i < radio.DATALEN; i++)
      Serial.print((char)radio.DATA[i]);
    Serial.print("   [RX_RSSI:");Serial.print(radio.readRSSI());Serial.print("]");

    if (radio.ACK_REQUESTED)
    {
      radio.sendACK();
      Serial.print(" - ACK sent");
      delay(10);
    }
    Blink(LED,5);
    Serial.println();
  }
  
  int currPeriod = millis()/TRANSMITPERIOD;
  if (currPeriod != lastPeriod)
  {
    //fill in the struct with new values
    theData.dataone = 20;
    //theData.uptime = millis();
    theData.temp = celsius; //it's hot!
    theData.hum = humidity;
    
    Serial.print("Sending struct (");
    Serial.print(sizeof(theData));
    Serial.print(" bytes) ... ");
    if (radio.sendWithRetry(GATEWAYID, (const void*)(&theData), sizeof(theData)))
      Serial.print(" Ok.");
    else Serial.print(" No ACK received.");
    Serial.println();
    Blink(LED,3);
	//Sensortemp();
    lastPeriod=currPeriod;
  }

 

}

void Blink(byte PIN, int DELAY_MS)
{
  pinMode(PIN, OUTPUT);
  digitalWrite(PIN,HIGH);
  delay(DELAY_MS);
  digitalWrite(PIN,LOW);
}

[mattt]

I finally figured this out by adding a number after each Misformed RX frame error message. I have 4 values instead of 5 and has to be an odd number (strange). Also my float values have to be int which you already suggested. I don't understand why as I am losing accuracy from my sensors when using int values. Maybe this is a shortfall for emoncms?

Code Select Expand

elif ((not (len(received) & 1)) or (len(received) < 3)):
            self._log.warning("Misformed RX frame: 3" + str(received))
        
        # Else, process frame
        else:
            try:
                # Only integers are expected
                received = [int(val) for val in received]
            except Exception:
                self._log.warning("Misformed RX frame: 4" + str(received))
            else:
                # Get node ID
                node = received[0]
                
                # Recombine transmitted chars into signed int
                values = []
                for i in range(1, len(received),2):
                    value = received[i] + 256 * received[i+1]
                    if value >= 32768:
                        value -= 65536
                    values.append(value)
                
                self._log.debug("Node: " + str(node))
                self._log.debug("Values: " + str(values))
    
                # Insert node ID before data
                values.insert(0, node)

                return values

[mattt]

Now my values sent to emoncms are completely wrong:

Code Select Expand

2014-07-27 22:57:55,885 INFO Logging level set to DEBUG
2014-07-27 22:57:55,886 INFO OemGateway v1.0.2
2014-07-27 22:57:55,886 INFO Opening gateway...
2014-07-27 22:57:55,886 INFO Creating buffer emoncms_local
2014-07-27 22:57:55,887 INFO Creating buffer emoncms_remote
2014-07-27 22:57:55,888 INFO Creating listener RFM2Pi
2014-07-27 22:57:55,888 DEBUG Opening serial port: /dev/ttyUSB0
2014-07-27 22:57:55,890 INFO Setting send time interval to 0
2014-07-27 22:57:55,891 INFO Setting RFM2Pi | frequency: 4
2014-07-27 22:57:56,893 INFO Setting RFM2Pi | sgroup: 100
2014-07-27 22:57:57,895 INFO Setting RFM2Pi | baseid: 1
2014-07-27 22:57:58,898 INFO Creating listener Socket
2014-07-27 22:57:58,899 DEBUG Opening socket on port 50011
2014-07-27 22:57:58,914 INFO Serial RX: 2 2 25 57 -53
2014-07-27 22:57:58,915 DEBUG Node: 2
2014-07-27 22:57:58,916 DEBUG Values: [6402, -13511]
2014-07-27 22:57:58,917 DEBUG Server localhost/emoncms -> buffer data: [2, 6402, -13511], timestamp: 1406527078.92
2014-07-27 22:57:58,918 DEBUG Server localhost/emoncms -> send data: [2, 6402, -13511], timestamp: 1406527078.92
2014-07-27 22:57:58,919 DEBUG Data string: &time=1406527078.92&node=2&json={1:6402,2:-13511}
2014-07-27 22:57:58,920 DEBUG URL string: http://localhost/emoncms/input/post.json?apikey=2fc4288bf06555fefdcb0d8877b9a7ea&time=1406527078.92&node=2&json={1:6402,2:-13511}
2014-07-27 22:57:58,921 INFO Sending to localhost/emoncms
2014-07-27 22:57:59,073 DEBUG Send ok

This is becoming increasingly frustrating I dont understand this piece of code from oemgatewaylistener.py and why it does not just send the original values:

Code Select Expand

# Recombine transmitted chars into signed int
                values = []
                for i in range(1, len(received),2):
                    value = received[i] + 256 * received[i+1]
                    if value >= 32768:
                        value -= 65536
                    values.append(value)

                self._log.debug("Node: " + str(node))
                self._log.debug("Values: " + str(values))

                # Insert node ID before data
                values.insert(0, node)

                return values

[Spexx]

Do you have the correct settings in oemgateway.conf? According to your last post, you have changed the 'listener'. For the serial connection you need to add to the oemgateway.conf the "OemGatewaySerialListener", as you will see in my first post. I have comment out the RFM2Pi and the Socket listener with a '#'.

I think in case of using one of the another listener, the Python script is awaiting that the data will be sent as bytes. Negative values (like RSSI) will not be accepted, so you will get the error 'misformed RX frame'.

So please add following lines to the oemgateway.conf:

Code Select Expand

# This listener manages the serial
[[Serial]]
    type = OemGatewaySerialListener
    [[[init_settings]]]
        com_port = /dev/ttyUSB0
    [[[runtime_settings]]]

I'm not sure how the Python scripts which listener you want to use, but I think in case of comment out the other ones, there shouldn't be any problem.

[Spexx]

It is also described here http://openenergymonitor.blogspot.de/2014/01/oem-gateway-serial-port-emoncms-link.html

Shouldn't be a problem to use numbers with a decimal point.

[mattt]

It worked! That piece of code was missing from my .conf file. Thanks you've been a great help!