Gateway and PowerBi

[frix]

Hi all,

I've been introduced to Microsoft's Power Bi and I'm trying to get it up and running with my Gateway. I think it would do a really nice integration for the visualization of my sensor network data.

I followed THIS POST about publishing the data to weather underground, but I don't want to create files and scripts reading files, etc. I think that Felix' suggestion on that same post about having an event that sends the data to the service would work better, because then I just need to turn it on or off depending on what node I want to publish.

So, I'm writing my own event (I've wrote other events in the past), and I'm breaking my head trying to figure out where the error may be in my code. When I run

Code Select Expand

node gateway.js

I get this response:

Code Select Expand

pi@FrivoPi:~/gateway $ sudo node gateway.js
[12-29-17_02:08:12.306] [INFO]  LOADING USER METRICS...
[12-29-17_02:08:12.449] [INFO]  LOADING USER METRICS MODULE [_example.js]
[12-29-17_02:08:12.748] [INFO]  LOADING USER METRICS MODULE [frivoMotes.js]
[12-29-17_02:08:12.793] [ERROR] FAIL LOADING USER METRICS MODULE [frivoMotes.js]: Unexpected token ;
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: listen EADDRINUSE :::8080
    at Object.exports._errnoException (util.js:911:11)
    at exports._exceptionWithHostPort (util.js:934:20)
    at Server._listen2 (net.js:1248:14)
    at listen (net.js:1284:10)
    at Server.listen (net.js:1380:5)
    at Server.listen.Server.attach (/home/pi/gateway/node_modules/socket.io/lib/index.js:216:9)
    at new Server (/home/pi/gateway/node_modules/socket.io/lib/index.js:50:17)
    at Function.Server (/home/pi/gateway/node_modules/socket.io/lib/index.js:38:41)
    at Object.<anonymous> (/home/pi/gateway/gateway.js:40:27)
    at Module._compile (module.js:409:26)

So, obviously, my custom event has a syntax problem somewhere that is preventing gateway.js from loading my custom events file and therefore there's an unhandled error that crashes the app. But I just can't figure out in which line or character is the fault (hey Felix, maybe an item for the to the 2Do list would be to have more descriptive error logs?)

Please take a look at my code below and see if you can spot the problem, I'd greatly appreciate any feedback. What I'm trying to do is just send a POST request to Power Bi with the time stamp in the right format, the temperature and the humidity. That's all. The two functions that I'm bringing in are: dateformat and request. I have both modules installed in my Node:

Code Select Expand

sendToPowerBi : { label:'PowerBi Stream', icon:'comment', descr:'post node data to powerBi for better visualization',
  serverExecute:function(node) {
  var request = require('request');
  var dateFormat = require('dateformat');
  var pushUrl = 'https://verylongstring';
  var theDate = Date.now();
  request({
	url: pushUrl,
	method: 'POST',
	headers: {'Content-Type': 'application/json'},
	json:{
	'timestamp': dateFormat(theDate,'yyyy-mm-ddTHH:MM:SSlZ'),
	'temperature':node.metrics['C'],
	'humidity':node.metrics['H']
	},
	function(err,res,body){
	  if(!err){
	   console.log(body);
	  } else {
	   console.log(err);
	  }
       }
     });
},
};

Thanks a lot!

[Felix]

Code looks fine to me, but just looking at it I cannot "compile" or run it
However the first error suggests a syntax error in your frivoMotes.js so I would look some more in there:

Code Select Expand

[12-29-17_02:08:12.793] [ERROR] FAIL LOADING USER METRICS MODULE [frivoMotes.js]: Unexpected token ;

I suggest trying the node console itself to do requests and that would give you good feedback if the request is OK or not.
Just run node (or nodejs, however it's registered) in your Pi command line, then import your modules, set any vars, and execute the code you're trying to debug, works a treat and makes checking unknown code pretty easy.

[frix]

Thanks Felix. I didn't know I could use Node like that, debugging here I go!

The way I got it working was by splitting the functions. I figured the event was trying to do too much and it was a bit too hard for me to debug it this way. So, I moved the request to its own function. I didn't know if I could add functions like this to my custom metrics file, so I moved it to its own global function in gateway.js (following the method described in THIS post):

So, in gateway.js:

Code Select Expand

// This function sends sensor data to a realtime dataset in PowerBi
// using a POST request. It is called by the sendToPowerBi event
global.postRequest = function(ID,TEMP,HUMIDITY,DATE,PUSHURL){
 // the date needs to be in ISO UTC time, so we use 'dateformat' to apply the mask
 var theNow = new Date();
 var fixedDate = dateformat(theNow, 'isoUtcDateTime');

 var postOptions = {
   url: PUSHURL,
   method: 'POST',
   headers: {'Content-Type': 'application/json'},
   json: {
        timestamp : fixedDate,
        nodeID : ID,
        temperature : TEMP,
        humidity :  HUMIDITY
      }
   //text:require('querystring').stringify(postData)
 };
 request.post(postOptions, function(error,response,body){
   if(error)console.error('POWERBI POST ERROR: ' + error);
   else console.log('POWERBI POST SUCCESS: ' + body);
 });
}

And my event now looks just like this:

Code Select Expand

sendToPowerBi : { label:'PowerBi Stream', icon:'comment', descr:'post node data to powerBi for better visualization',
  serverExecute:function(node) {
    if(node.settings['powerBiUrl']){
      console.log('PUSH URL: ' + node.settings['powerBiUrl']);
    postRequest(node._id,node.metrics['C'].value,node.metrics['H'].value,Date.now(),node.settings['powerBiUrl']);
  } else console.log(' UNIQUE POWER BI STREAM SKIPPED ');
  //postRequest(node._id,node.metrics['C'].value,node.metrics['H'].value,Date.now(),settings.general.powerBiAllUrl.value);
}
},

It works like a charm. I can choose which node to publish to powerBi and which not to, all by turning the event on or off.

As you can see, I added a powerBiUrl setting to my custom node, and also a powerBiAllUrl setting in the general settings file. This allows me to post nodes to unique powerBi datasets, but also allows me to combine all selected nodes into one big dataset (or at least that was the idea. It doesn't really work because of how powerBi handles streaming datasets).

Code Select Expand

FrivoMote: {
    label  : 'Temp Node',
    icon   : 'icon_temperature2.png',
    settings: {
      lowTempValue: '',
      highTempValue: '',
      lowVoltageValue: '', //blank will make it inherit from global settings.json lowVoltageValue, a specific value overrides the general setting, user can always choose his own setting in the UI
      powerBiUrl: ''},
  },

By the way, this is how the dataset is configured in powerBi:


After all this, I get a really nice real time dashboard with visuals of all my feeds. It has a max plot range of 60min, so that's what I ask for. I'm really happy.



But, this ended up not working for me at all. PowerBi does not create a database for streaming datasets. It just puts them in cache and plots them. Once the datapoint is gone from the chart, it's gone. So this service does not work for me. I want to plot all things together and do data analysis with it.

I'm now trying to setup datasets in Adafruit IO, which is way too similar to how powerBi works and looks like. It's kinda suspicious. But I don't care as long as it's actually saving my data (which it is) and allowing me to generate the plots I need (which it does). I'm just having some issues understanding how the communication works with their system... Pretty weird, actually, that it was easier for me to understand how to send data to a Microsoft product, than to Adafruit's.

I really like how the events really enhance the gateway and what the nodes can do. Thanks a log for thinking about this, Felix! Now I just need to find a service that actually makes sense for the kind of data collection and analysis that I want to do, set up the event accordingly and boom.

Cheers,
Oscar

[Felix]

Glad to see you worked this out !
Ah the cloud  ... here it is today, gone or mutated tomorrow.

[frix]

Exactly, so we should use it while we can! ha!