LowPowerLab Forum

Software support => Pi Gateway => Topic started by: Stark on July 11, 2015, 07:51:06 AM

Title: Graph Data history.
Post by: Stark on July 11, 2015, 07:51:06 AM
Hi there.

I am loving the gateway. I see that the graph data is limited to a weeks worth of data. How can this be changed to display a longer period of data, say a month or even a year?

I have looked through the graphHelper.js and I can see that the default display will be 8 hours (which can be adjusted) and that there is reference to a start time. But before I go off messing around there are a number of questions. Are graph logs limited i.e. cleared at intervals? Assume a longer period affects performance, processing time to load a longer period?

Thanks in advance.
Title: Re: Graph Data history.
Post by: Stark on July 12, 2015, 02:54:34 PM
It was staring me in the face in gateway.js.

Code: [Select]
dbLog.remove({_id:{$lte: ((new Date().getTime())-604800000)}}, {multi:true}, function(err, count){ console.log('Removed '  + count + ' records'); }); //604800000ms = 1 week
 

I have increased the 1 week (as ms) so hopefully that's all that is needed.

Title: Re: Graph Data history.
Post by: Felix on July 13, 2015, 08:48:16 AM
Yeah I should have done a better job of moving all the major settings that people might change in a separate file.
I am working on it and will release a new version soon I hope.
Thanks for your thoughts, glad you like it :)
Title: Re: Graph Data history.
Post by: Stark on July 14, 2015, 01:17:38 AM
Thank Felix.
Actually the code is well commented by you. With a bit of investigation I (a relative noob) have made some user adjustments based on these.
Your time permitting a separate setting file, as you suggest, would be great though.

Thanks again for all the hard work.
Title: Re: Graph Data history.
Post by: Felix on July 20, 2015, 10:32:33 PM
A new image with some updates and bug fixes and also settings in a separate file (settings.js) is released, please see the last tab on the gateway page (http://lowpowerlab.com/gateway/#sourcecode) where the link to this image is kept updated. Let me know of any issues.
Title: Re: Graph Data history.
Post by: crsherm on August 25, 2015, 08:01:33 PM
Felix,
I am in the process of finalizing several weather shields and wish to update my gateway but I do not wish to loose my prior settings.  How would you recommend updating the gateway program files?

Thanks again for all of the great stuff!
Title: Re: Graph Data history.
Post by: Felix on August 25, 2015, 09:40:35 PM
So you want to update the software part, and keep the gateway.db (node settings) and also the gatewayLog.db (node data log)?
Just backup those 2 files, replace everything else, then put them back and override the defaults.

BTW I am about to release a major upgrade so it may be worth to wait a while. This upgrade will involve migrating the log data from gatewayLog.db to a new storage database via a script, it will make graphs much much faster so it's very well worth it 8)
Title: Re: Graph Data history.
Post by: videl on August 26, 2015, 09:49:13 AM
Felix,

I installed the gateway two months ago with few sensors in my house. That’s great!
I just added a few lines of code for autoscaling the graphs because otherwise it was impossible to see the variations of atmospheric pressure.
Would it be possible to add this feature to your future release?

Thanks for all your work!
Title: Re: Graph Data history.
Post by: Felix on August 26, 2015, 10:43:31 AM
I just added a few lines of code for autoscaling the graphs because otherwise it was impossible to see the variations of atmospheric pressure.
Would it be possible to add this feature to your future release?

Sure, can you post the code here or do you have it posted somewhere else?
Any comments will help expedite the code review, thanks.
Title: Re: Graph Data history.
Post by: crsherm on August 26, 2015, 11:50:36 AM
Thanks Felix for the quick response!

I think I will wait for your future release.  Any other new items beyond the database script and graphs??
Title: Re: Graph Data history.
Post by: Felix on August 26, 2015, 12:15:39 PM
Start/restarts will be very fast also, previously neDB could take a long time to hydrate all the gatewayLog data points in memory.
Graphs will show a customizable timestamped value for each point (upon click), previously it was just the log raw value (no date or unit).
A few other UI changes and improvements, and hopefully no new bugs :P, I will blog about it once it's released.
The only manual thing required if you have old data logged in neDB (gatewayLog.db) is to run the conversion script, that's a 1 time command, i will include instructions.

This new storage engine will allow really nice coming features like multi node/graphs analysis :)
Title: Re: Graph Data history.
Post by: videl on August 26, 2015, 05:33:28 PM
Here are the modifications I've done in index.php for adding the autoscaling.
I'm quite new to javascrip/node.js so that's probably not the best way to do it but it works for me.

index.php:

adding yaxis properties to graphOptions

Code: [Select]
     
graphOptions = {
    lines: {show: true, steps: true, fill:true },
    xaxis: { mode: "time", timezone: "browser", min:graphView.start, max:graphView.end},
    yaxis: { min:null, max:null, autoscaleMargin:0.02},
    grid: {hoverable: true, clickable: true, backgroundColor: {colors:['#000', '#666']}},
    selection: { mode: "x" },
};


and the scaling

Code: [Select]
    
socket.on('GRAPHDATAREADY', function(rawData){
    graphData = [];
    var max = Number.NEGATIVE_INFINITY;
    var min = Number.POSITIVE_INFINITY;
    var minmax = 0;
    LOG('Got ' + rawData.graphData.data.length + ' graph data points...');

    //finding the min and max of the dataset
    for(var key in rawData.graphData.data) {
        graphData.push([rawData.graphData.data[key]._id, rawData.graphData.data[key].v]);
        max = Math.max(max, rawData.graphData.data[key].v);
        min = Math.min(min, rawData.graphData.data[key].v);
    }

    //defining the upper and lower margin
    minmax=(max-min) * graphOptions.yaxis.autoscaleMargin;
    if (min==max)  // in case of only one value in the dataset (motion detection)
        minmax=graphOptions.yaxis.autoscaleMargin * min;
    min -= minmax;
    max += minmax;

    //set the graph scale properties
    graphOptions.xaxis.min = graphView.start;
    graphOptions.xaxis.max = graphView.end;
    graphOptions.yaxis.min = min;
    graphOptions.yaxis.max = max;

    graphOptions = $.extend(true, graphOptions, rawData.options); //http://stackoverflow.com/questions/171251/how-can-i-merge-properties-of-two-javascript-objects-dynamically
    //need to defer plotting until after pageshow is finished rendering, otherwise the wrapper will return an incorrect width of "100"
    if (metricGraphWrapper.width()==100)
        $(document).on("pageshow", "#metricdetails", renderPlot);
    else renderPlot();
    });

Hope this can help.


modified on 18/09/2015 to match the 06/08/2015 release
Title: Re: Graph Data history.
Post by: Felix on August 27, 2015, 08:07:23 AM
Thanks, I will check it out.
The next release is out, see the blog and the gateway guide (http://lowpowerlab.com/gateway).
Title: Re: Graph Data history.
Post by: videl on September 17, 2015, 05:33:08 PM
Hi Felix,

Thanks for this new release.

I updated my gateway but I can't see the graph improvements mentionned in your blog ( http://lowpowerlab.com/blog/2015/08/27/gateway-log-engine-upgrade/ (http://lowpowerlab.com/blog/2015/08/27/gateway-log-engine-upgrade/) )
For example, I don't have the visibility and graphing buttons but still have the sliders and there is no label in the graph.

May be I missed something during the update process: I downloaded the zip file from GitHub and copied (or replaced) index.php, gateway.js, settings.js, metrics.js, logUtil.js + ClientSideUI subfolder.
I also installed upstart.

Thanks in advance,
Vincent
Title: Re: Graph Data history.
Post by: Felix on September 18, 2015, 08:57:03 AM
videl,
I have to revisit the Github repo and make sure the files are in sync. To get the latest image go to https://lowpowerlab.com/gateway/#image
Title: Re: Graph Data history.
Post by: videl on September 18, 2015, 02:30:39 PM
Thanks Felix,

As the Moteino Gateway is not the only application running on my Pi I wanted to avoid using the image. But I managed to extract the moteino and www directories from the image. Now everythink works well. Congratulation for this improvement.

I also modified my previous post with the graph autoscaling function to match the new index.php.

Vincent
Title: Re: Graph Data history.
Post by: Felix on September 18, 2015, 03:25:53 PM
Excellent, I need to make a note about the graph scaling and take a look at it, thanks!
Title: Re: Graph Data history.
Post by: Felix on September 23, 2015, 09:10:15 AM
Vincent,
I reviewed your code. I guess I'm a bit confused how this really helps in certain situations because the graph already does pretty nice autoscaling.
With your code I have to define yaxis for each graph otherwise it will autoscale to the full height of the plot window, which is not really desirable in my opinion.
Can you give an example how your autoscaling helps (before/after screenshots), referring to this statement:

I just added a few lines of code for autoscaling the graphs because otherwise it was impossible to see the variations of atmospheric pressure.
Title: Re: Graph Data history.
Post by: videl on September 23, 2015, 01:12:48 PM
Hi Felix,

I've put screenshots in the attached pdf file. You'll see pressure and temperature graphs with the standard index.php and with the autoscale function.

The pressure graph is useless without min/max scaling. The max y axis value is set correctly by Flot but the min y value is set to zero. As pressure variations are quite small compared to the absolute value, it's impossible to see anything more than a flat graph. I couldn't find any way to automatically set both ymin and ymax with Flot graph options. It's the reason why I added this piece of code. But may be there is a smarter way to do it.

Thanks,
Vincent
Title: Re: Graph Data history.
Post by: Felix on September 23, 2015, 03:34:03 PM
Ok I really see your point now, it makes a lot of sense.
I was not using it with similar data and that's why I didn't see the effect of the scaling.
Also there should be a default scaling factor when none is specified.
I will work to get this included.
Title: Re: Graph Data history.
Post by: videl on September 23, 2015, 03:41:44 PM
That's perfect. Thank you Felix!
Title: Re: Graph Data history.
Post by: Felix on September 23, 2015, 11:26:35 PM
This has has been implemented and a new release (v6) is now available (http://lowpowerlab.com/blog/2015/09/23/gateway-software-update-v6/) that includes this change.
Thanks Vincent for your contribution!
Title: Re: Graph Data history.
Post by: Lensdigital on September 28, 2015, 03:39:22 PM
I also would like to extend data log retention, but I cannot find "dbLog.remove" option anywhere...
I tried V6, and still not sure where it can be configured?
Title: Re: Graph Data history.
Post by: Felix on September 29, 2015, 08:47:16 AM
data log retention?
This discussion is about something else. And there is no data retention configuration.
Title: Re: Graph Data history.
Post by: Lensdigital on September 29, 2015, 09:15:25 AM
data log retention?
This discussion is about something else. And there is no data retention configuration.

Umm first post of this topic:
Hi there.

I am loving the gateway. I see that the graph data is limited to a weeks worth of data. How can this be changed to display a longer period of data, say a month or even a year?

I have looked through the graphHelper.js and I can see that the default display will be 8 hours (which can be adjusted) and that there is reference to a start time. But before I go off messing around there are a number of questions. Are graph logs limited i.e. cleared at intervals? Assume a longer period affects performance, processing time to load a longer period?

Thanks in advance.
Title: Re: Graph Data history.
Post by: Felix on September 29, 2015, 09:49:00 AM
Right, but Retention != displaying of the data. With the v6 image there is no limit to how much data is stored. Performance should not be affected. Also the default view loaded is 1 day worth of data and you can see up to 1 month from the controls, but also you can zoom out and see more than that. The graph will load up to 600 points (you can change that in the source if you want) but at 1 year with 600 points you are losing a lot of resolution simply because you cant see a million points on a 1920px display.

Also we are talking about how much data we are keeping and displaying and also mainly the topic is about the change of how data is displayed - scaling wise.
There is no purging of the data from the UI. If you want to remove data, you can delete the logs manually from the disk.
Title: Re: Graph Data history.
Post by: Lensdigital on September 29, 2015, 09:52:05 AM
Thanks Felix!  Understood! :)

I couldn't see before past 1 week, now I see 1 month after I updated to V6.
Title: Re: Graph Data history.
Post by: Stark on November 16, 2015, 04:41:06 PM
Hi Felix.

As of this weekend I have only just found the time to update from the original gateway (if only I was as productive as you). I must say the graphing is great and vastly improved! Many thanks for all the hard work.
Title: Re: Graph Data history.
Post by: Felix on November 16, 2015, 04:55:32 PM
Hey Stark, good to hear... now if only I found some time to deal with the v6 bugs  ::)