Author Topic: (wanted features / bug fixes) Duplicate values not correctly drawn...  (Read 208 times)

Uncle Buzz

  • Full Member
  • ***
  • Posts: 144
  • Country: fr
I use the 9.0 Pi Gateway, maybe this is already done in 9.1...

If I'm not wrong, when a duplicate value is sent by a node, it is skipped until 'duplicateInterval' has passed, only a new value will be saved, drawing a line from the first value that will be duplicated to the first different value, the graphic won't represent the stable value but a constant change between those 2 points. If the value changes less than 2%, it's ok, but if we have a large difference the curve doesn't represent correctly the edge.

Is it possible to save the first duplicate value, and if the value is still duplicated, instead of skipping it, could it be possible to update the time of the duplicated entry previously saved to correspond to the last transmission, keeping the 2 extremes points drawing a horizontal line between the first point and the last duplicate one before the edge created by the new value?

Instead of having (5 is duplicated) :
Code: [Select]
1;2;5;........................;1;7...
having this :
Code: [Select]
1;2;5;......................;5;1;7...
who will correctly represent the stable value (5) and the falling edge between the last 5 value and the new 1 value.

alexelite

  • NewMember
  • *
  • Posts: 9
Re: (wanted features / bug fixes) Duplicate values not correctly drawn...
« Reply #1 on: February 06, 2021, 05:58:23 PM »
Hi Uncle Buzz, what you want is step graph and can be achieved by changing a parameter in www/index.html. Search  /*steps: true,*/ and uncomment. 

Uncle Buzz

  • Full Member
  • ***
  • Posts: 144
  • Country: fr
Re: (wanted features / bug fixes) Duplicate values not correctly drawn...
« Reply #2 on: February 06, 2021, 07:34:43 PM »
Thank you for the answer, but I'm almost sure index.html has nothing to do with saved values in database, it can't display data that are not in the database.  This setting probably change the way data are displayed on the graph, but doesn't change how data are saved.

alexelite

  • NewMember
  • *
  • Posts: 9
Re: (wanted features / bug fixes) Duplicate values not correctly drawn...
« Reply #3 on: February 07, 2021, 03:04:47 AM »
You don't have to mess with the database and increase the number of points to get your request. Flot graph module can display the way you want it, and it is called stepped graph. All you need to do is change one little line as said before. You will find 2 definitions in index.html, one is for multigraph, the second on in for metric details.

check out the attached images.

you can even do this in metrics definitions, per metrc. Just add 'steps:true' in graphOptions : { ..., lines: { lineWidth:1 , steps:true },

L.E.
If I missed your point (sorry) and it's not only the way the graph is displayed that you want to alter, but also the database, you can go into gateway.js and around line 870
Code: [Select]
if (isNumeric(graphValue))
you copy the entire if before the original and modify it to save the dbEntry instead of entry, checking before if dbEntry is not null and if a metric value exists.

because of the asynchronous nature of nodejs, you should somehow wait for the first write to finish and then attempt the new value save, otherwise they might get saved the other way arround.
This is for version 9.1 and up, in 9.0 the process is similar but the function is written differently and you have to figure the names and the location of the code.

alex
« Last Edit: February 07, 2021, 05:10:19 AM by alexelite »

Uncle Buzz

  • Full Member
  • ***
  • Posts: 144
  • Country: fr
Re: (wanted features / bug fixes) Duplicate values not correctly drawn...
« Reply #4 on: February 07, 2021, 05:50:27 AM »
My request is not only about the graph, the option you mention change the way the graph is drawn between 2 points, it presumes that the value stays constant until a new value is sent, then the graph fall or rise at the new value.

The anti-duplicate function doesn't save any value which is the same as the last saved one. When the new value is saved, we don't save the last time the value was constant, this is a loss of information. We should only discard the points between the first and the last ones, keeping the last one.

I don't want a rising or a falling edge each time the value changes, but keeping the smooth transition between each value needs to save the last duplicate entry to correctly represent the constant level, then the smooth transition between the last time the value were not changed and the new value.

To my mind there are 2 ways to achieve this :
  • saving the first duplicate value, then when another duplicate value is sent, updating the time of the duplicate value saved to keep the time of the last time this value is sent
  • or saving the duplicate value, and delete the previously saved duplicate value, keeping only the last one and the first one that was
     not a duplicate one

It will keep the anti-duplicate function discarding the duplicate values, but saving only one more point to correctly represent the constant level.

This is about how values are saved, not about how they are drawn.

alexelite

  • NewMember
  • *
  • Posts: 9
Re: (wanted features / bug fixes) Duplicate values not correctly drawn...
« Reply #5 on: February 07, 2021, 06:14:44 AM »
OK, thanks, now you made it a little clearer for me and it is actually not a bad idea.
When an identical value arrives, by default only the time in db is updated not the metric graph log.
You can do it exactly the way I said in the later edit.
When an identical value arrives you update the time in db as usual, but set a flag about this. Nothing else.
After this, when a different value arrives you check the flag, and if true, you first save the current database point in metric graph log, and then save the freshly received value as usual, clearing the flag we introduced earlier.

I will add this to my setup to test and share the code if you want, but I run a modified version of the latest version. 9.2.0 (almost identical with 9.1.0 in code). You have to upgrade to use directly, otherwise adapt it to old 9.0.0 code. It is quite different because Felix rewrote the function.

Just a heads up, because the data point number is reduced when requesting a long timeframe, this mod won't make a difference until you zoom in.

Later edit:
Hmm, not quite the way I said  :-[
Duplicate entry is checked in logUtil - postData so there needs to be some modifications there also.
« Last Edit: February 07, 2021, 07:20:02 AM by alexelite »

alexelite

  • NewMember
  • *
  • Posts: 9
Re: (wanted features / bug fixes) Duplicate values not correctly drawn...
« Reply #6 on: February 07, 2021, 09:02:23 AM »
It can all be done in one function mod.
For metrics without duplicateInterval noting changes, each value is saved, regardless.
For metrics with duplicateInterval set and before this interval expires, it reads the last 2 inserts and if valuen==valuen-1==valuen-2  (n is now) it updates timestampn-1 to timestampn
If duplicateInterval has expired it adds a new entry regardless of valuen==valuen-1==valuen-2

Here you go, modified postData in logUtil.js 9.2.0 version, check the repository for diffs and test it for bugs.

Later edit: Fixed an error in ifs. It should work on 9.0.0 without any change
Code: [Select]
// filename:  binary file to append new data point to
// timestamp: data point timestamp (seconds since unix epoch)
// value:     data point value (signed integer)
// duplicateInterval: if provided a duplicate value is only posted after this many seconds
exports.postData = function post(filename, timestamp, value, duplicateInterval) {
  if (!metrics.isNumeric(value)) value = 999; //catch all value
  var logsize = exports.fileSize(filename);
  if (logsize % 9 > 0) throw 'File ' + filename +' is not multiple of 9bytes, post aborted';

  var fd;
  var buff = Buffer.alloc(9);
  var secondLastValue =0, lastTime = 0, lastValue = 0, pos = 0;
  value=Math.round(value*10000); //round to make an exactly even integer

  //prepare 9 byte buffer to write
  buff.writeInt8(0,0);             //flag byte
  buff.writeUInt32BE(timestamp,1); //timestamp 4 bytes
  buff.writeInt32BE(value,5);     //value 4 bytes

  // If there is at least one value
  if (logsize>=9) {
    fd = fs.openSync(filename, 'r');
    //If at least 2 values in file, read last two
    if (logsize>=18){
      var buf13 = Buffer.alloc(13);
      fs.readSync(fd, buf13, 0, 13, logsize-13);
      secondLastValue = buf13.readInt32BE(0); //read second last value (bytes 0-3 in buffer)
      lastTime = buf13.readUInt32BE(5); //read last timestamp (bytes 5-8 in buffer)
      lastValue = buf13.readInt32BE(9); //read last value (bytes 9-13 in buffer)
      fs.closeSync(fd);
    }
    else{
      // read the only value in the file
      var buf8 = Buffer.alloc(8);
      fs.readSync(fd, buf8, 0, 8, logsize-8);
      lastTime = buf8.readUInt32BE(0); //read timestamp (bytes 0-3 in buffer)
      lastValue = buf8.readInt32BE(4); //read value (bytes 4-7 in buffer)
      fs.closeSync(fd);
    }
    if (timestamp > lastTime)
    {
      //If metric duplicateInterval not passed yet and new value == last value == second last value => update last timestamp to new timestamp
      if ((secondLastValue == lastValue && lastValue == value) && (duplicateInterval!=null && timestamp-lastTime<duplicateInterval)){
        fd = fs.openSync(filename, 'r+');
        fs.writeSync(fd, buff, 0, 9, logsize-9);
        fs.closeSync(fd);
      }
      //only write new value if different than last value or duplicateInterval seconds has passed  or not set
      else if ((secondLastValue != lastValue && lastValue == value && duplicateInterval!=null ) || (value != lastValue || duplicateInterval==null || timestamp-lastTime>duplicateInterval))
      {
        //timestamp is in the future, append
        fd = fs.openSync(filename, 'a');
        fs.writeSync(fd, buff, 0, 9, logsize);
        fs.closeSync(fd);
      }
    }
    else
    {
      //timestamp is somewhere in the middle of the log, identify exact timestamp to update
      fd = fs.openSync(filename, 'r');
      pos = exports.binarySearchExact(fd,timestamp,logsize);
      fs.closeSync(fd);

      if (pos!=-1)
      {
        fd = fs.openSync(filename, 'r+');
        fs.writeSync(fd, buff, 0, 9, pos);
        fs.closeSync(fd);
      }
    }
  }
  else
  {
    //empty log, just append data point
    fd = fs.openSync(filename, 'a');
    fs.writeSync(fd, buff, 0, 9, 0);
    fs.closeSync(fd);
  }

  return value;
}

« Last Edit: February 07, 2021, 11:37:55 AM by alexelite »

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6531
  • Country: us
    • LowPowerLab
Re: (wanted features / bug fixes) Duplicate values not correctly drawn...
« Reply #7 on: February 08, 2021, 01:34:57 PM »
The anti-duplicate function doesn't save any value which is the same as the last saved one. When the new value is saved, we don't save the last time the value was constant, this is a loss of information. We should only discard the points between the first and the last ones, keeping the last one.

I don't want a rising or a falling edge each time the value changes, but keeping the smooth transition between each value needs to save the last duplicate entry to correctly represent the constant level, then the smooth transition between the last time the value were not changed and the new value.

To my mind there are 2 ways to achieve this :
  • saving the first duplicate value, then when another duplicate value is sent, updating the time of the duplicate value saved to keep the time of the last time this value is sent
  • or saving the duplicate value, and delete the previously saved duplicate value, keeping only the last one and the first one that was
     not a duplicate one

It will keep the anti-duplicate function discarding the duplicate values, but saving only one more point to correctly represent the constant level.

This is about how values are saved, not about how they are drawn.
I think your first suggestion would be the better approach, I will keep this in mind.
The current workaround is to not use a duplicate interval, this results in more data being written as a tradeoff (each point).
In my mind, if you need that level of detail, to ensure your line doesn't jump, then just turn off duplicateinterval. Or keep it very low, like every few readings if space is a concern. But really ... 9 bytes per entry ... it's hardly a space eater. My personal system of 2 dozen nodes or so is < 200MB since INCEPTION many years ago. I turn off duplicateintervall if i really care to see full detail. This setting was only implemented because of space considerations, and the redundant transmission of many data points of equal value for graphs.