Graph Data history.

[Stark]

Hi there.

I am loving the gateway. I see that the graph data is limited to a weeks worth of data. How can this be changed to display a longer period of data, say a month or even a year?

I have looked through the graphHelper.js and I can see that the default display will be 8 hours (which can be adjusted) and that there is reference to a start time. But before I go off messing around there are a number of questions. Are graph logs limited i.e. cleared at intervals? Assume a longer period affects performance, processing time to load a longer period?

Thanks in advance.

[Stark]

It was staring me in the face in gateway.js.

Code: [Select]

dbLog.remove({_id:{$lte: ((new Date().getTime())-604800000)}}, {multi:true}, function(err, count){ console.log('Removed '  + count + ' records'); }); //604800000ms = 1 week
  

I have increased the 1 week (as ms) so hopefully that's all that is needed.

[Felix]

Yeah I should have done a better job of moving all the major settings that people might change in a separate file.
I am working on it and will release a new version soon I hope.
Thanks for your thoughts, glad you like it

[Stark]

Thank Felix.
Actually the code is well commented by you. With a bit of investigation I (a relative noob) have made some user adjustments based on these.
Your time permitting a separate setting file, as you suggest, would be great though.

Thanks again for all the hard work.

[Felix]

A new image with some updates and bug fixes and also settings in a separate file (settings.js) is released, please see the last tab on the gateway page where the link to this image is kept updated. Let me know of any issues.

[crsherm]

Felix,
I am in the process of finalizing several weather shields and wish to update my gateway but I do not wish to loose my prior settings.  How would you recommend updating the gateway program files?

Thanks again for all of the great stuff!

[Felix]

So you want to update the software part, and keep the gateway.db (node settings) and also the gatewayLog.db (node data log)?
Just backup those 2 files, replace everything else, then put them back and override the defaults.

BTW I am about to release a major upgrade so it may be worth to wait a while. This upgrade will involve migrating the log data from gatewayLog.db to a new storage database via a script, it will make graphs much much faster so it's very well worth it

[videl]

Felix,

I installed the gateway two months ago with few sensors in my house. That’s great!
I just added a few lines of code for autoscaling the graphs because otherwise it was impossible to see the variations of atmospheric pressure.
Would it be possible to add this feature to your future release?

Thanks for all your work!

[Felix]

I just added a few lines of code for autoscaling the graphs because otherwise it was impossible to see the variations of atmospheric pressure.
Would it be possible to add this feature to your future release?

Sure, can you post the code here or do you have it posted somewhere else?
Any comments will help expedite the code review, thanks.

[crsherm]

Thanks Felix for the quick response!

I think I will wait for your future release.  Any other new items beyond the database script and graphs??

[Felix]

Start/restarts will be very fast also, previously neDB could take a long time to hydrate all the gatewayLog data points in memory.
Graphs will show a customizable timestamped value for each point (upon click), previously it was just the log raw value (no date or unit).
A few other UI changes and improvements, and hopefully no new bugs , I will blog about it once it's released.
The only manual thing required if you have old data logged in neDB (gatewayLog.db) is to run the conversion script, that's a 1 time command, i will include instructions.

This new storage engine will allow really nice coming features like multi node/graphs analysis

[videl]

Here are the modifications I've done in index.php for adding the autoscaling.
I'm quite new to javascrip/node.js so that's probably not the best way to do it but it works for me.

index.php:

adding yaxis properties to graphOptions

Code: [Select]

     
graphOptions = {
    lines: {show: true, steps: true, fill:true },
    xaxis: { mode: "time", timezone: "browser", min:graphView.start, max:graphView.end},
    yaxis: { min:null, max:null, autoscaleMargin:0.02},
    grid: {hoverable: true, clickable: true, backgroundColor: {colors:['#000', '#666']}},
    selection: { mode: "x" },
};

and the scaling

Code: [Select]

    
socket.on('GRAPHDATAREADY', function(rawData){
    graphData = [];
    var max = Number.NEGATIVE_INFINITY;
    var min = Number.POSITIVE_INFINITY;
    var minmax = 0;
    LOG('Got ' + rawData.graphData.data.length + ' graph data points...');

    //finding the min and max of the dataset
    for(var key in rawData.graphData.data) {
        graphData.push([rawData.graphData.data[key]._id, rawData.graphData.data[key].v]);
        max = Math.max(max, rawData.graphData.data[key].v);
        min = Math.min(min, rawData.graphData.data[key].v);
    }
	
    //defining the upper and lower margin
    minmax=(max-min) * graphOptions.yaxis.autoscaleMargin;
    if (min==max)  // in case of only one value in the dataset (motion detection)
        minmax=graphOptions.yaxis.autoscaleMargin * min;
    min -= minmax;
    max += minmax;
	
    //set the graph scale properties
    graphOptions.xaxis.min = graphView.start;
    graphOptions.xaxis.max = graphView.end;
    graphOptions.yaxis.min = min;
    graphOptions.yaxis.max = max;
	
    graphOptions = $.extend(true, graphOptions, rawData.options); //http://stackoverflow.com/questions/171251/how-can-i-merge-properties-of-two-javascript-objects-dynamically
    //need to defer plotting until after pageshow is finished rendering, otherwise the wrapper will return an incorrect width of "100"
    if (metricGraphWrapper.width()==100)
        $(document).on("pageshow", "#metricdetails", renderPlot);
    else renderPlot();
    });

Hope this can help.


modified on 18/09/2015 to match the 06/08/2015 release

[Felix]

Thanks, I will check it out.
The next release is out, see the blog and the gateway guide.

[videl]

Hi Felix,

Thanks for this new release.

I updated my gateway but I can't see the graph improvements mentionned in your blog ( http://lowpowerlab.com/blog/2015/08/27/gateway-log-engine-upgrade/ )
For example, I don't have the visibility and graphing buttons but still have the sliders and there is no label in the graph.

May be I missed something during the update process: I downloaded the zip file from GitHub and copied (or replaced) index.php, gateway.js, settings.js, metrics.js, logUtil.js + ClientSideUI subfolder.
I also installed upstart.

Thanks in advance,
Vincent

[Felix]

videl,
I have to revisit the Github repo and make sure the files are in sync. To get the latest image go to https://lowpowerlab.com/gateway/#image