Graph is displayed wrong because of wrong/missing data

[LukaQ]

Hello

Few days ago, for unknown reason, I could not reach gateway, nor PI with terminal. I didn't even seen it being connected to router over wifi. I had to remove its power and reapply it.
Don't know where to search for what could be the problem, that made PI unresponsive.

And the next problem is, that this happened:



It would be really great to have option to edit data, which I think, there is no way (even complex) of doing it, am I right?
I probably can't do nothing about this? Looking data for past few months is this way useless

[Felix]

Hi Luka,

Missing data should not produce weird graphs, unless perhaps there is something you added in the graph config which causes flot to go haywire.

An SD corruption is always a possibility, in my case it usually renders the OS unusable, but i've also seen instances where part of the data files become unresponsive (cannot copy or backup them anymore). So I do manual backups often, that's something I would like to automate but haven't gotten to that yet.
Editing the data is another feature which I would like.

A few questions:

1) Are you running the latest Gateway release?
2) Are you able to copy the data files from your Pi?
3) There is the export feature as well which you could use to get the data in CSV format for external analysis. What happens when you try this on a time span which shows these visual errors?

If I could get your binary data file, I could try reproduce the problem on my end.

[LukaQ]

Hello Felix,

1) I am not, but I do run 8.8, would this upgrade be possible to do over the internet? I wasn't near PI for few months now.
2) Yes I was, I was able to do that after this already happened.
3) There are two 0 data points:

Code Select Expand

1.50063E+12	26.57	7.21.17 10:55 AM
1.50063E+12	26.58	7.21.17 10:55 AM
          0	0	1.1.70 12:00 AM
          0	0	1.1.70 12:00 AM
1.50074E+12	27.48	7.22.17 4:33 PM
1.50074E+12	27.45	7.22.17 4:34 PM

Below is RAR of this node


From metrics.js for this node:

Code Select Expand

C : { name:'C',
	regexp:/\bC\:([-\d\.]+)\b/i,
	value:'',
	duplicateInterval:3600,
	unit:'°',
	pin:1,
	graph:1,
	graphValSuffix:'C',
	graphOptions:	{legendLbl:'Temperature',
			 lines: { steps: false, lineWidth:1 },
			 colors:['#F9c'],
			 //yaxis: { tickSize: 1},
			 series: { curvedLines: { active: true, apply: true, nrSplinePoints: 1}, 
					 lines: { show: true, fill: true, fillColor: "rgba(155, 20, 255, 0.3)" },
      			   	   	points: { show: false, fill: false },
				  autoMarkings: { enabled: true, showMinMax: false, showAvg: true, minMaxAlpha: 0.1, lineWidth: 1, avgcolor: "rgb(255, 0, 0)"},		
				 }
			}
      },

There was also something in logUtil.js I changed, to have more time span buttons, like 2 months, 6 m, 1 year. And IIRC Y-axis scale, that was more fine... nothing that would (probably) do this. Will see, what you get. Thank you!

[Felix]

Thanks for the details,
I haven't tried loading your data in my Pi yet, but I fixed your data to delete the blanks. You should not really need upgrade, 8.8 is ok. And you can do an upgrade but you need ftp access.
I found 3 data points that were all null, meaning null time and null data. One was in the middle of the file, two were at the very end of the file. Can you backup your existing file and replace it with the one in the following link and see if you still get corrupted graph (app restart needed)?
https://drive.google.com/open?id=0B89HtwH5i2d0TVFTUE9ORnpnMzA

In fact given that the data time is null, that corresponds to the very beginning of time, which in unix epoch is 1 january 1970. So maybe your graph is probably simply trying to graph that point at an invalid position. It's just a theory and we could spend hours doing forensics on this and I am not sure it's worth it. For now check to see if the data is fixed.

Then we need to try to find out how that null data got in the dataset in the first place. I have not seen this behavior before, so I am not worried about a bug. I would like to see this repeat again, preferably from someone else before we really dive in.

[LukaQ]

Hello Felix,

your data is fine until 21th, than the same kinda problem, where data is missing up to now
How do you edit that dataset, I would like to be able to do that on PI itself if possible

How to get null? copy over older file, that seems to do it, or remove power, i'm not sure how else

[Felix]

I used a hex editor to find the bad data.
A sequence of 9 '00' (null byte) is the problem. The ones at the end I just truncated, for the one in the middle I copied the data point before it.
Each data point is 9 bytes (1 reserved byte, 4 bytes unix epoch timestamp, 4 bytes decimal value).
Backup your old file, then copy 'fixed' file over it over FTP or whatever method you have.
Then restart gateway from the menu option.

[LukaQ]

Will you have option to add:
- edit dataset over browser?
- set gateway not to use any null string and move to next?

[Felix]

Will you have option to add:
- edit dataset over browser?
- set gateway not to use any null string and move to next?

Those are good improvements, the second part is probably pretty straight forward, the first will be a lot of hours