So, I have graph for one of the metric H not being shown anymore past some date. Nothing was changed at that time, that whole node is still logged ok
{"_id":40,"updated":1508784155631,"rssi":47,"metrics":{
"C":{"label":"C","value":21.21,"unit":"°","updated":1508784155631,"pin":1,"graph":1},
"H":{"label":"H","value":53.87,"unit":"%","updated":1508784155631,"pin":1,"graph":1},
"P":{"label":"P","value":1007.34,"unit":"mBar","updated":1508784155631,"pin":1,"graph":1},
"START":{"label":"START","value":"Started","updated":1508742977303,"pin":0},"RSSI":{"label":"RSSI","value":-41,"unit":"db","updated":1508784155631,"pin":0,"graph":1}},"type":"WeatherMote","label":"Indoor Sensor","descr":"bme280"}
yet for H, last point was written to db on 4/10, and nothing else past that
What to do?
Can you look at the logs to see exactly what data is coming in and if it's the same?
Maybe the bin file is corrupt, are you using good SD cards?
{"_id":5,"updated":1508817989265,"rssi":30,"metrics":{
"C":{"label":"C","value":9.94,"unit":"°","updated":1508817989265,"pin":1,"graph":1},
"RSSI":{"label":"RSSI","value":-34,"unit":"db","updated":1508817989265,"graph":1,"pin":0}},"descr":"ds18b20","type":"WeatherMote","label":"Outdoor Sensor"}
{"_id":40,"updated":1508817993318,"rssi":47,"metrics":{
"C":{"label":"C","value":20.54,"unit":"°","updated":1508817993318,"pin":1,"graph":1},
"H":{"label":"H","value":53.39,"unit":"%","updated":1508817993318,"pin":1,"graph":1},
"P":{"label":"P","value":1011.5,"unit":"mBar","updated":1508817993318,"pin":1,"graph":1},
"START":{"label":"START","value":"Started","updated":1508742977303,"pin":0},"RSSI":{"label":"RSSI","value":-46,"unit":"db","updated":1508817993318,"pin":0,"graph":1}},"type":"WeatherMote","label":"Indoor Sensor","descr":"bme280"}
{"_id":40,"updated":1508818018359,"rssi":47,"metrics":{
"C":{"label":"C","value":20.57,"unit":"°","updated":1508818018359,"pin":1,"graph":1},
"H":{"label":"H","value":53.42,"unit":"%","updated":1508818018359,"pin":1,"graph":1},
"P":{"label":"P","value":1011.46,"unit":"mBar","updated":1508818018359,"pin":1,"graph":1},
"START":{"label":"START","value":"Started","updated":1508742977303,"pin":0},"RSSI":{"label":"RSSI","value":-46,"unit":"db","updated":1508818018359,"pin":0,"graph":1}},"type":"WeatherMote","label":"Indoor Sensor","descr":"bme280"}
That looks to me to be ok and as it was. How do I test SD card for corruption? SD card is 16G from SanDisk, that should be ok, I don't know if it is yet 1year old.
It's also weird that it wouldn't work past some day, I would think whole file would be corrupted, isn't that so? Everything else that I use on that PI works ok, all other graphs are ok
I meant the gateway.sys.log not the gateway.db which only keeps the latest values.
I don't see why it would stop. You will need to investigate further and see what changed. Code doesnt just stop working because it has a bad day like us humans :)
If you find a cause let me know and I will fix it.
[10-04-17_16:22:55.660] [LOG] >: [5] C:16.50 [RSSI:-35][ACK-sent]
[10-04-17_16:22:55.663] [LOG] post: /home/pi/gateway/data/db/0005_C.bin[1507126975,16.5]
[10-04-17_16:22:55.665] [LOG] post: /home/pi/gateway/data/db/0005_RSSI.bin[1507126975,-35]
[10-04-17_16:22:55.671] [LOG] [5] DB-Updates:1
[10-04-17_16:17:21.864] [INFO] LOADING USER METRICS...
[10-04-17_16:17:21.883] [INFO] LOADING USER METRICS MODULE [_example.js]
[10-04-17_16:17:23.025] [LOG] >: [40] C:21.45 H:51.65 P:1016.58 [RSSI:-46][ACK-sent]
[10-04-17_16:17:23.046] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507126643,21.45]
[10-04-17_16:17:23.065] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507126643,51.65]
[10-04-17_16:17:23.066] [ERROR] POST ERROR: undefined
[10-04-17_16:17:23.069] [LOG] post: /home/pi/gateway/data/db/0040_P.bin[1507126643,1016.58]
[10-04-17_16:17:23.084] [LOG] post: /home/pi/gateway/data/db/0040_RSSI.bin[1507126643,-46]
[10-04-17_16:17:23.115] [LOG] [40] DB-Updates:1
[10-04-17_16:37:03.757] [INFO] AUTHORIZING CONNECTION FROM ::1:41990
[10-04-17_16:37:03.769] [INFO] NEW CONNECTION FROM 192.168.2.1
[10-04-17_16:37:11.730] [LOG] >: [5] C:16.50 [RSSI:-35][ACK-sent]
[10-04-17_16:37:11.735] [LOG] post: /home/pi/gateway/data/db/0005_C.bin[1507127831,16.5]
[10-04-17_16:37:11.742] [LOG] post: /home/pi/gateway/data/db/0005_RSSI.bin[1507127831,-35]
[10-04-17_16:37:11.757] [LOG] [5] DB-Updates:1
[10-04-17_16:37:18.796] [LOG] >: [40] C:21.46 H:51.75 P:1016.48 [RSSI:-43][ACK-sent]
[10-04-17_16:37:18.801] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507127838,21.46]
[10-04-17_16:37:18.803] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507127838,51.75]
[10-04-17_16:37:18.804] [ERROR] POST ERROR: undefined
[10-04-17_16:37:18.805] [LOG] post: /home/pi/gateway/data/db/0040_P.bin[1507127838,1016.48]
[10-04-17_16:37:18.810] [LOG] post: /home/pi/gateway/data/db/0040_RSSI.bin[1507127838,-43]
[10-04-17_16:37:18.819] [LOG] [40] DB-Updates:1
[10-04-17_16:37:42.835] [LOG] >: [40] C:21.49 H:51.67 P:1016.47 [RSSI:-46][ACK-sent]
[10-04-17_16:37:42.838] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507127862,21.49]
[10-04-17_16:37:42.846] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507127862,51.67]
[10-04-17_16:37:42.847] [ERROR] POST ERROR: undefined
[10-04-17_16:37:42.852] [LOG] post: /home/pi/gateway/data/db/0040_P.bin[1507127862,1016.47]
[10-04-17_16:37:42.855] [LOG] post: /home/pi/gateway/data/db/0040_RSSI.bin[1507127862,-46]
[10-04-17_16:37:42.874] [LOG] [40] DB-Updates:1
[10-04-17_16:38:07.884] [LOG] >: [40] C:21.49 H:51.70 P:1016.47 [RSSI:-47][ACK-sent]
[10-04-17_16:38:07.889] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507127887,21.49]
[10-04-17_16:38:07.890] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507127887,51.7]
[10-04-17_16:38:07.890] [ERROR] POST ERROR: undefined
[10-04-17_16:38:07.892] [LOG] post: /home/pi/gateway/data/db/0040_P.bin[1507127887,1016.47]
[10-04-17_16:38:07.894] [LOG] post: /home/pi/gateway/data/db/0040_RSSI.bin[1507127887,-47]
[10-04-17_16:38:07.907] [LOG] [40] DB-Updates:1
[10-04-17_16:38:13.913] [LOG] >: [5] C:16.50 [RSSI:-35][ACK-sent]
[10-04-17_16:38:13.917] [LOG] post: /home/pi/gateway/data/db/0005_C.bin[1507127893,16.5]
[10-04-17_16:38:13.919] [LOG] post: /home/pi/gateway/data/db/0005_RSSI.bin[1507127893,-35]
[10-04-17_16:38:13.927] [LOG] [5] DB-Updates:1
[10-04-17_16:38:31.962] [LOG] >: [40] C:21.46 H:51.70 P:1016.54 [RSSI:-45][ACK-sent]
[10-04-17_16:38:31.966] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507127911,21.46]
[10-04-17_16:38:31.968] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507127911,51.7]
[10-04-17_16:38:31.969] [ERROR] POST ERROR: undefined
There are some errors. before this errors started to be, this has come to light
[10-04-17_16:17:20.690] [INFO] LOADING USER METRICS...
[10-04-17_16:17:20.709] [INFO] LOADING USER METRICS MODULE [_example.js]
So _example.js...
H : { name:'H',
regexp:/\bH\:([\d\.]+)\b/i,
value:'',
duplicateInterval:3600,
unit:'%',
pin:1,
graph:1,
graphOptions: {legendLbl:'Humidity',
lines: { steps: false, lineWidth:1 },
colors:['#F9c'],
//yaxis: { tickSize: 0.1},
series: { curvedLines: { active: false, apply: false, nrSplinePoints: 1},
lines: { show: true, fill: true, fillColor: "rgba(205, 20, 255, 0.3)" },
points: { show: false, fill: false },
autoMarkings: { enabled: true, showMinMax: true, showAvg: true, minMaxAlpha: 0.3, lineWidth: 1, avgcolor: "rgb(255, 0, 0)"},
}
}
},
Do you see anything wrong here?
and what would this be?
POST ERROR: undefined
FYI:If I delete the H database of node 40, it's starts new file and it can log again and no POST ERRORs anymore
I dont see anything wrong but I only looked at the code visually. Most likely if it loads and the metrics are recognized, it works fine.
Undefined error doesn't say much. You need to add more debugging code there perhaps.
BTW I've seen very good brand SD cards fail after just a few months.
Quote from: Felix on October 24, 2017, 01:30:33 PM
Undefined error doesn't say much. You need to add more debugging code there perhaps.
Where to?
It is interesting, that if file is deleted and new on is created, db is ok, but if I copy what I had, same thing continues. Maybe there is something to be seen in db file? would you fancy a look at it?
What SD cards do you use? how long do you have them before you do something?
If I recall correctly, I have seen corrupted bin files from you before. Noone else has reported a problem in that area, and I haven't seen any myself except with corrupted SD cards.
I don't think it's worthwhile for me to investigate further just to find the same problem.
The bad ones are Kingston, the good ones are sandisk.
before this, it wasn't corrupted bin file, just one point in log with date, that was far off, like 1970 and it draw graph that way. This is something totally different
Oh yes, that invalid date corrupted the data graph, now I remember. How that point got in your data was still not determined as far as I remember.
For this one .. we still have not found why the data "stopped" so it's not clear if its corruption or some other problem.
In the log there is an unknown exception. That's where you need to dig further. Add some exception handling where the logging happens, here is the logger being invoked in gateway.js:
https://github.com/LowPowerLab/RaspberryPi-Gateway/blob/master/gateway.js#L556
try {
console.log('post: ' + logfile + '[' + ts + ','+graphValue + ']');
dbLog.postData(logfile, ts, graphValue, matchingMetric.duplicateInterval || null);
} catch (err) { console.error(' POST ERROR: ' + err.message); /*console.log(' POST ERROR STACK TRACE: ' + err.stack); */ }
dbLog.postData is located in logUtil.js, you could add more exception handling there.
I suggested this could be corruption, maybe concurrency, but still I would expect it a lot more often, especially in for users who have lots of nodes and data. In years of use I would expect to see this type of thing myself if there was a bug in the code.
as far as I see in the gateway.sys.log, problem starts here:
[10-04-17_16:28:42.849] [LOG] >: [40] C:21.47 H:51.52 P:1016.69 [RSSI:-42][ACK-sent]
[10-04-17_16:28:42.852] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507127322,21.47]
[10-04-17_16:28:42.853] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507127322,51.52]
[10-04-17_16:28:42.854] [LOG] post: /home/pi/gateway/data/db/0040_P.bin[1507127322,1016.69]
[10-04-17_16:28:42.857] [LOG] post: /home/pi/gateway/data/db/0040_RSSI.bin[1507127322,-42]
[10-04-17_16:28:42.863] [LOG] [40] DB-Updates:1
[10-04-17_16:29:07.920] [LOG] >: [40] C:21.47 H:51.54 P:1016.69 [RSSI:-65][ACK-sent]
[10-04-17_16:29:07.923] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507127347,21.47]
[10-04-17_16:29:07.924] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507127347,51.54]
[10-04-17_16:29:07.925] [LOG] post: /home/pi/gateway/data/db/0040_P.bin[1507127347,1016.69]
[10-04-17_16:29:07.926] [LOG] post: /home/pi/gateway/data/db/0040_RSSI.bin[1507127347,-65]
[10-04-17_16:29:07.933] [LOG] [40] DB-Updates:1
[10-04-17_16:29:31.978] [LOG] >: [40] C:21.46 H:51.53 P:1016.67 [RSSI:-45][ACK-sent]
[10-04-17_16:29:31.983] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507127371,21.46]
[10-04-17_16:29:31.984] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507127371,51.53]
[10-04-17_16:29:31.986] [LOG] post: /home/pi/gateway/data/db/0040_P.bin[1507127371,1016.67]
[10-04-17_16:29:31.987] [LOG] post: /home/pi/gateway/data/db/0040_RSSI.bin[1507127371,-45]
[10-04-17_16:29:31.994] [LOG] [40] DB-Updates:1
[10-04-17_16:29:57.047] [LOG] >: [40] C:21.48 H:51.54 P:1016.68 [RSSI:-55][ACK-sent]
[10-04-17_16:29:57.054] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507127397,21.48]
[10-04-17_16:29:57.056] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507127397,51.54]
[10-04-17_16:29:57.057] [LOG] post: /home/pi/gateway/data/db/0040_P.bin[1507127397,1016.68]
[10-04-17_16:29:57.059] [LOG] post: /home/pi/gateway/data/db/0040_RSSI.bin[1507127397,-55]
[10-04-17_16:29:57.065] [LOG] [40] DB-Updates:1
[10-04-17_16:17:20.690] [INFO] LOADING USER METRICS...
[10-04-17_16:17:20.709] [INFO] LOADING USER METRICS MODULE [_example.js]
[10-04-17_16:17:32.891] [LOG] >: [40] C:21.49 H:51.52 P:1016.61 [RSSI:-53][ACK-sent]
[10-04-17_16:17:32.924] [LOG] post: /home/pi/gateway/data/db/0040_C.bin[1507126652,21.49]
[10-04-17_16:17:32.948] [LOG] post: /home/pi/gateway/data/db/0040_H.bin[1507126652,51.52]
[10-04-17_16:17:32.949] [ERROR] POST ERROR: undefined
[10-04-17_16:17:32.954] [LOG] post: /home/pi/gateway/data/db/0040_P.bin[1507126652,1016.61]
[10-04-17_16:17:32.973] [LOG] post: /home/pi/gateway/data/db/0040_RSSI.bin[1507126652,-53]
[10-04-17_16:17:32.998] [LOG] [40] DB-Updates:1
If you look at the time, it was 10-04-17_16:29:57 the last time node was updated... then I rebooted the system for some reason and after boot it must have gotten new time, which was 10-04-17_16:17:20.690 and there the first error started to show. Last logged point was 4:29pm with H being 51.53%. Last 3 points are as follows: 51.52, 51.54, 51.53, but in the log the is also 51.54, which is not on graph. Could this time jump do something like that?
I would really like to edit this bin file, delete those lines where time was, what was before that
There is one thing I would like to know also, how do you edit neDB bin file?
There are some assumptions about the log data. It has to be entered sequentially. The logUtil (https://github.com/LowPowerLab/RaspberryPi-Gateway/blob/master/logUtil.js) code does some checks when you post new data:
- it checks if the data is later than the last point in the file, then it just appends it to the file
- if it determines it's somewhere in the middle, it will search for that same timestamp for updating, if it finds it
if (timestamp > lastTime)
{
if (value != lastValue || (duplicateInterval==null || timestamp-lastTime>duplicateInterval)) //only write new value if different than last value or duplicateInterval seconds has passed (should be a setting?)
{
//timestamp is in the future, append
fd = fs.openSync(filename, 'a');
fs.writeSync(fd, buff, 0, 9, logsize);
fs.closeSync(fd);
}
}
else
{
//timestamp is somewhere in the middle of the log, identify exact timestamp to update
fd = fs.openSync(filename, 'r');
pos = exports.binarySearchExact(fd,timestamp,logsize);
fs.closeSync(fd);
if (pos!=-1)
{
fd = fs.openSync(filename, 'r+');
fs.writeSync(fd, buff, 0, 9, pos);
fs.closeSync(fd);
}
}
neDB is only used for gateway.db which is JSON format
the log data is a custom binary format which I've explained before: 9bytes per record = [1 byte reserved, 4 bytes timestamp, 4 bytes value]
You can try export the data to excel to look at it and determine where it went wrong and if it corresponds to your sys logs.
Regarding your last log, something is really wrong there. If you rebooted, even if the time is wrong, there should be a lot more log information during startup, not just user metrics loading.
Quote from: Felix on October 25, 2017, 08:54:06 AM
- if it determines it's somewhere in the middle, it will search for that same timestamp for updating, if it finds it
Is this really sensible? If the time has passed, it has passed, this search would be only good if you were to edit data, which there is no option now. It's not great if system can do it by itself (that is if clock is moved or time is off)
What is your thinking here?
<5MB file = stupid amount of data, excel is not happy with me
In the file I do see some fields that are
Do you do this in terminal?
seems a lot of work to go from bin to hex to excel and format data and then perhaps back again
The edit is not implemented, but posting to an existing timestamp will update it.
If the time moves to the past, then the inserts will fail, unless they match a past datapoint timestamp. That is bad but it's not critical. Normally your clock should not change willy nilly, especially set backwards in time. Forwards in time is no problem, logutil will append a new datapoint.
The export feature is for viewing, not for editing. It's one way basically. useful if you want to manipulate the data in a different way than the default graph in the app.
The app is free. I fix bugs, if you can reproduce the bug. I cannot fix bugs that don't exist or cannot be reproduced or happen random and cannot be pointed to a code location.
All you have at this point is a file with some bad data, a log file which is completely inconsistent with what I would expect. That is not helpful for me to work on it, I don't know where to begin.
Don't get me wrong, help as much as you can, I don't think you are obligated to so do.
QuoteI fix bugs, if you can reproduce the bug.
Don't know if that is a bug or how you would call it, but here is something you can work with:
(https://lowpowerlab.com/forum/pi-gateway/graph-for-one-metric-not-being-shown-anymore-past-some-date/?action=dlattach;attach=2587)
This are last 3 lines of .bin. And as you can see, last line was not written with 9 bytes. This is the reason it did not work. I removed last line, replaced it on server and now it is logging from this time forward, and there is no data in between old and new point, so they are just connected together like they should be.
maybe you can implement something to check if last point is 9 bytes long and is not null or something like that, if possible
Quote from: LukaQ on October 25, 2017, 12:33:07 PM
maybe you can implement something to check if last point is 9 bytes long and is not null or something like that, if possible
That's possible. But a greater question is why and how did that get corrupt record get there. Just looking at bad data is not a lead. Do you have a suggestion what to do with that data?
The logutil code checks for the timestamp, it will
not insert a NULL at the end of the file.
I see you look at 9 bytes per line, correct? Then how is your last record only 7 bytes? The file should always have a multiple of 9 bytes length.
Anyway isn't this the exact same problem you had before? I recall combing through your data bin file before, finding nulls.
Is anyone else having this data problem? Certainly not aware of it, but if someone does please speak up.
Quote from: Felix on October 25, 2017, 01:20:41 PM
That's possible. But a greater question is why and how did that get corrupt record get there.
reboot possible, if not that, power outage... something that would interrupt writing
Quote from: Felix on October 25, 2017, 01:20:41 PMDo you have a suggestion what to do with that data?
I guess delete it, partial data has no place to be in database anyway.
Quote from: Felix on October 25, 2017, 01:20:41 PMThe logutil code checks for the timestamp, it will not insert a NULL at the end of the file.
Ok, that's good, one less possible problem
Quote from: Felix on October 25, 2017, 01:20:41 PMI see you look at 9 bytes per line, correct? Then how is your last record only 7 bytes? The file should always have a multiple of 9 bytes length.
Correct. My only guess would be power loss or reboot just at that time, when file was in writing, do you think that is possible?
Quote from: Felix on October 25, 2017, 01:20:41 PMAnyway isn't this the exact same problem you had before? I recall combing through your data bin file before, finding nulls.
I would say no, since value was ok, but date was I think 1/1/1970 or something like that. At that point I didn't know how to check .bin file.
That data point should be still somewhere, I will find it in a day or two, to see what was going on there.
An exception did happen, that much is clear, just no detail on it.
Seems like nulls/partial data was written.
0 for the timestamp is the unix epoch which is Jan/1/1970. But that has no place at the end of a file.
Another thing I could do is a check on extraction. But if the file has wrong number of bytes then everything is broken, the logutil assumes the records are sequential and 9 bytes each, that's crucial.
There is an UPTIME indicator in the UI which should tell you if the system restarted recently, after a power outage for instance.
A reboot would not interrupt writing of a 9 byte record, that I am pretty sure of, it's an atomic operation. A corrupt SD card on the other hand is much more likely to cause that. And having seen this (only) from you before I think I would bet my money on that. It almost smells like it based on all I'm seeing. When the gateway starts there is a lot of startup logging which I don't see in your case, it's basically impossible.
ok, so for time being, let's not do something and I'll keep my eye on things. I did fix just now all other bin files, that had 9 bytes of 0 here and there. Now the graphs are perfect, from end of february to today and I must say, this is one of the best logging tools
Great, thanks :)
Like I said, the logger can be improved by more checking of bad data.
BUT what I don't think will help is if partial data is written. For that scenario only some automatic data check/fixing could help, maybe some algorithm that checks if the file is a multiple of 9bytes, if not look for bad records, but that is going to be complex - how do you decide where bad data is, establish patterns of data.
I should note here that there is already a check in logutil.js if the bin file is multiple of 9 each time before posting a new datapoint:
if (logsize % 9 > 0) throw 'File ' + filename +' is not multiple of 9bytes, post aborted';
So I would expect to see that exception in your sys.log if the bin files were corrupted in any way. But it doesnt show in your logs although the logging stops, very weird.
I just noticed when looking at sys.log that this happened twice around that time
[10-04-17_16:16:02.379] [LOG]
[10-04-17_16:16:15.391] [LOG]
[10-04-17_16:16:15.393] [LOG]
[10-04-17_16:16:15.396] [LOG]
[10-04-17_16:16:15.401] [LOG]
[10-04-17_16:16:27.433] [LOG]
[10-04-17_16:16:27.440] [LOG]
[10-04-17_16:16:27.442] [LOG]
[10-04-17_16:16:27.443] [LOG]
[10-04-17_16:16:27.445] [LOG]
[10-04-17_16:16:27.451] [LOG]
[10-04-17_16:16:51.474] [LOG]
[10-04-17_16:16:51.477] [LOG]
[10-04-17_16:16:51.479] [LOG]
[10-04-17_16:16:51.480] [LOG]
[10-04-17_16:16:51.481] [LOG]
[10-04-17_16:16:51.487] [LOG]
[10-04-17_16:17:16.516] [LOG]
[10-04-17_16:17:16.520] [LOG]
[10-04-17_16:17:16.521] [LOG]
[10-04-17_16:17:16.522] [LOG]
[10-04-17_16:17:16.523] [LOG]
[10-04-17_16:17:16.529] [LOG]
[10-04-17_16:17:40.556] [LOG]
[10-04-17_16:17:40.559] [LOG]
[10-04-17_16:17:40.560] [LOG]
[10-04-17_16:17:40.561] [LOG]
[10-04-17_16:17:40.563] [LOG]
[10-04-17_16:17:40.569] [LOG]
[10-04-17_16:18:05.598] [LOG]
[10-04-17_16:18:05.601] [LOG]
[10-04-17_16:18:05.603] [LOG]
[10-04-17_16:18:05.604] [LOG]
[10-04-17_16:18:05.605] [LOG]
[10-04-17_16:18:05.616] [LOG]
[10-04-17_16:18:20.620] [LOG]
[10-04-17_16:18:20.622] [LOG]
[10-04-17_16:18:20.624] [LOG]
[10-04-17_16:18:20.629] [LOG]
[10-04-17_16:18:29.661] [LOG]
[10-04-17_16:18:29.664] [LOG]
[10-04-17_16:18:29.665] [LOG]
[10-04-17_16:18:29.666] [LOG]
[10-04-17_16:18:29.667] [LOG]
[10-04-17_16:18:29.674] [LOG]
[10-04-17_16:18:54.702] [LOG]
[10-04-17_16:18:54.704] [LOG]
[10-04-17_16:18:54.706] [LOG]
[10-04-17_16:18:54.707] [LOG]
[10-04-17_16:18:54.708] [LOG]
[10-04-17_16:18:54.714] [LOG]
[10-04-17_16:19:18.744] [LOG]
[10-04-17_16:19:18.747] [LOG]
[10-04-17_16:19:18.748] [LOG]
[10-04-17_16:19:18.750] [LOG]
[10-04-17_16:19:18.751] [LOG]
[10-04-17_16:19:18.757] [LOG]
[10-04-17_16:19:23.766] [LOG]
[10-04-17_16:19:23.768] [LOG]
[10-04-17_16:19:23.770] [LOG]
[10-04-17_16:19:23.775] [LOG]
[10-04-17_16:19:43.807] [LOG]
[10-04-17_16:19:43.811] [LOG]
[10-04-17_16:19:43.812] [LOG]
[10-04-17_16:19:43.813] [LOG]
[10-04-17_16:19:43.815] [LOG]
[10-04-17_16:19:43.820] [LOG]
[10-04-17_16:20:07.849] [LOG]
[10-04-17_16:20:07.852] [LOG]
[10-04-17_16:20:07.854] [LOG]
[10-04-17_16:20:07.855] [LOG]
[10-04-17_16:20:07.856] [LOG]
[10-04-17_16:20:07.862] [LOG]
[10-04-17_16:20:26.872] [LOG]
[10-04-17_16:20:26.875] [LOG]
[10-04-17_16:20:26.876] [LOG]
[10-04-17_16:20:26.881] [LOG]
[10-04-17_16:20:32.911] [LOG]
[10-04-17_16:20:32.915] [LOG]
[10-04-17_16:20:32.916] [LOG]
[10-04-17_16:20:32.917] [LOG]
[10-04-17_16:20:32.919] [LOG]
[10-04-17_16:20:32.925] [LOG]
[10-04-17_16:20:56.954] [LOG]
[10-04-17_16:20:56.957] [LOG]
[10-04-17_16:20:56.958] [LOG]
[10-04-17_16:20:56.960] [LOG]
[10-04-17_16:20:56.961] [LOG]
[10-04-17_16:20:56.967] [LOG]
[10-04-17_16:21:21.994] [LOG]
[10-04-17_16:21:22.003] [LOG]
[10-04-17_16:21:22.005] [LOG]
[10-04-17_16:21:22.006] [LOG]
[10-04-17_16:21:22.007] [LOG]
[10-04-17_16:21:22.013] [LOG]
[10-04-17_16:21:29.016] [LOG]
[10-04-17_16:21:29.020] [LOG]
[10-04-17_16:21:29.021] [LOG]
[10-04-17_16:21:29.027] [LOG]
[10-04-17_16:21:46.058] [LOG]
[10-04-17_16:21:46.060] [LOG]
[10-04-17_16:21:46.062] [LOG]
[10-04-17_16:21:46.063] [LOG]
[10-04-17_16:21:46.065] [LOG]
[10-04-17_16:21:46.070] [LOG]
[10-04-17_16:22:11.098] [LOG]
[10-04-17_16:22:11.102] [LOG]
[10-04-17_16:22:11.103] [LOG]
[10-04-17_16:22:11.105] [LOG]
[10-04-17_16:22:11.106] [LOG]
[10-04-17_16:22:11.112] [LOG]
[10-04-17_16:22:32.121] [LOG]
[10-04-17_16:22:32.124] [LOG]
[10-04-17_16:22:32.126] [LOG]
[10-04-17_16:22:32.131] [LOG]
[10-04-17_16:22:35.162] [LOG]
[10-04-17_16:22:35.173] [LOG]
[10-04-17_16:22:35.175] [LOG]
[10-04-17_16:22:35.176] [LOG]
[10-04-17_16:22:35.178] [LOG]
[10-04-17_16:22:35.183] [LOG]
[10-04-17_16:23:00.203] [LOG]
[10-04-17_16:23:00.206] [LOG]
[10-04-17_16:23:00.207] [LOG]
[10-04-17_16:23:00.209] [LOG]
[10-04-17_16:23:00.210] [LOG]
[10-04-17_16:23:00.216] [LOG]
[10-04-17_16:23:24.244] [LOG]
[10-04-17_16:23:24.247] [LOG]
[10-04-17_16:23:24.248] [LOG]
[10-04-17_16:23:24.249] [LOG]
[10-04-17_16:23:24.250] [LOG]
[10-04-17_16:23:24.256] [LOG]
[10-04-17_16:23:35.266] [LOG]
[10-04-17_16:23:35.268] [LOG]
[10-04-17_16:23:35.270] [LOG]
[10-04-17_16:23:35.275] [LOG]
[10-04-17_16:23:48.308] [LOG]
[10-04-17_16:23:48.310] [LOG]
[10-04-17_16:23:48.312] [LOG]
[10-04-17_16:23:48.313] [LOG]
[10-04-17_16:23:48.314] [LOG]
[10-04-17_16:23:48.320] [LOG]
[10-04-17_16:24:13.347] [LOG]
[10-04-17_16:24:13.350] [LOG]
[10-04-17_16:24:13.352] [LOG]
[10-04-17_16:24:13.353] [LOG]
[10-04-17_16:24:13.354] [LOG]
[10-04-17_16:24:13.360] [LOG]
[10-04-17_16:24:37.369] [LOG]
[10-04-17_16:24:37.372] [LOG]
[10-04-17_16:24:37.373] [LOG]
[10-04-17_16:24:37.379] [LOG]
[10-04-17_16:25:02.411] [LOG]
[10-04-17_16:25:02.416] [LOG]
[10-04-17_16:25:02.417] [LOG]
[10-04-17_16:25:02.418] [LOG]
[10-04-17_16:25:02.418] [LOG]
[10-04-17_16:25:02.422] [LOG]
[10-04-17_16:25:27.453] [LOG]
[10-04-17_16:25:27.457] [LOG]
[10-04-17_16:25:27.458] [LOG]
[10-04-17_16:25:27.459] [LOG]
[10-04-17_16:25:27.460] [LOG]
[10-04-17_16:25:27.466] [LOG]
[10-04-17_16:25:40.475] [LOG]
[10-04-17_16:25:40.478] [LOG]
[10-04-17_16:25:40.480] [LOG]
[10-04-17_16:25:40.486] [LOG]
[10-04-17_16:25:51.516] [LOG]
[10-04-17_16:25:51.519] [LOG]
[10-04-17_16:25:51.521] [LOG]
[10-04-17_16:25:51.522] [LOG]
[10-04-17_16:25:51.523] [LOG]
[10-04-17_16:25:51.530] [LOG]
[10-04-17_16:26:15.556] [LOG]
[10-04-17_16:26:15.559] [LOG]
[10-04-17_16:26:15.560] [LOG]
[10-04-17_16:26:15.563] [LOG]
[10-04-17_16:26:15.564] [LOG]
[10-04-17_16:26:15.570] [LOG]
[10-04-17_16:26:40.599] [LOG]
[10-04-17_16:26:40.609] [LOG]
[10-04-17_16:26:40.610] [LOG]
[10-04-17_16:26:40.611] [LOG]
[10-04-17_16:26:40.613] [LOG]
[10-04-17_16:26:40.619] [LOG]
[10-04-17_16:26:43.622] [LOG]
[10-04-17_16:26:43.625] [LOG]
[10-04-17_16:26:43.626] [LOG]
[10-04-17_16:26:43.633] [LOG]
[10-04-17_16:27:04.661] [LOG]
[10-04-17_16:27:04.664] [LOG]
[10-04-17_16:27:04.666] [LOG]
[10-04-17_16:27:04.667] [LOG]
[10-04-17_16:27:04.668] [LOG]
[10-04-17_16:27:04.675] [LOG]
[10-04-17_16:27:29.706] [LOG]
[10-04-17_16:27:29.710] [LOG]
[10-04-17_16:27:29.711] [LOG]
[10-04-17_16:27:29.712] [LOG]
[10-04-17_16:27:29.714] [LOG]
[10-04-17_16:27:29.719] [LOG]
[10-04-17_16:27:45.728] [LOG]
[10-04-17_16:27:45.731] [LOG]
[10-04-17_16:27:45.732] [LOG]
[10-04-17_16:27:45.739] [LOG]
[10-04-17_16:27:53.768] [LOG]
[10-04-17_16:27:53.771] [LOG]
[10-04-17_16:27:53.772] [LOG]
[10-04-17_16:27:53.774] [LOG]
[10-04-17_16:27:53.775] [LOG]
[10-04-17_16:27:53.782] [LOG]
[10-04-17_16:28:18.809] [LOG]
[10-04-17_16:28:18.812] [LOG]
[10-04-17_16:28:18.813] [LOG]
[10-04-17_16:28:18.815] [LOG]
[10-04-17_16:28:18.816] [LOG]
[10-04-17_16:28:18.822] [LOG]
[10-04-17_16:28:42.849] [LOG]
[10-04-17_16:28:42.852] [LOG]
[10-04-17_16:28:42.853] [LOG]
[10-04-17_16:28:42.854] [LOG]
[10-04-17_16:28:42.857] [LOG]
[10-04-17_16:28:42.863] [LOG]
[10-04-17_16:28:48.871] [LOG]
[10-04-17_16:28:48.879] [LOG]
[10-04-17_16:28:48.880] [LOG]
[10-04-17_16:28:48.886] [LOG]
[10-04-17_16:29:07.920] [LOG]
[10-04-17_16:29:07.923] [LOG]
[10-04-17_16:29:07.924] [LOG]
[10-04-17_16:29:07.925] [LOG]
[10-04-17_16:29:07.926] [LOG]
[10-04-17_16:29:07.933] [LOG]
[10-04-17_16:29:31.978] [LOG]
[10-04-17_16:29:31.983] [LOG]
[10-04-17_16:29:31.984] [LOG]
[10-04-17_16:29:31.986] [LOG]
[10-04-17_16:29:31.987] [LOG]
[10-04-17_16:29:31.994] [LOG]
[10-04-17_16:29:51.994] [LOG]
[10-04-17_16:29:51.998] [LOG]
[10-04-17_16:29:51.999] [LOG]
[10-04-17_16:29:52.006] [LOG]
[10-04-17_16:29:57.047] [LOG]
[10-04-17_16:29:57.054] [LOG]
[10-04-17_16:29:57.056] [LOG]
[10-04-17_16:29:57.057] [LOG]
[10-04-17_16:29:57.059] [LOG]
[10-04-17_16:29:57.065] [LOG]
[10-04-17_16:17:20.690] [INFO]
[10-04-17_16:17:20.709] [INFO]
[10-04-17_16:17:32.891] [LOG]
[10-04-17_16:17:32.924] [LOG]
[10-04-17_16:17:32.948] [LOG]
[10-04-17_16:17:32.949] [ERROR]
[10-04-17_16:17:32.954] [LOG]
[10-04-17_16:17:32.973] [LOG]
[10-04-17_16:17:32.998] [LOG]
[10-04-17_16:17:41.914] [LOG]
[10-04-17_16:17:41.922] [LOG]
[10-04-17_16:17:41.936] [LOG]
[10-04-17_16:17:41.948] [LOG]
[10-04-17_16:17:56.956] [LOG]
[10-04-17_16:17:56.960] [LOG]
[10-04-17_16:17:56.962] [LOG]
[10-04-17_16:17:56.963] [ERROR]
[10-04-17_16:17:56.965] [LOG]
[10-04-17_16:17:56.967] [LOG]
[10-04-17_16:17:56.975] [LOG]
[10-04-17_16:18:22.016] [LOG]
[10-04-17_16:18:22.020] [LOG]
[10-04-17_16:18:22.027] [LOG]
[10-04-17_16:18:22.031] [ERROR]
[10-04-17_16:18:22.033] [LOG]
[10-04-17_16:18:22.040] [LOG]
[10-04-17_16:18:22.053] [LOG]
[10-04-17_16:18:44.082] [LOG]
[10-04-17_16:18:44.086] [LOG]
[10-04-17_16:18:44.088] [LOG]
[10-04-17_16:18:44.100] [LOG]
[10-04-17_16:18:46.134] [LOG]
[10-04-17_16:18:46.140] [LOG]
[10-04-17_16:18:46.143] [LOG]
[10-04-17_16:18:46.144] [ERROR]
[10-04-17_16:18:46.145] [LOG]
[10-04-17_16:18:46.147] [LOG]
[10-04-17_16:18:46.156] [LOG]
[10-04-17_16:19:11.183] [LOG]
[10-04-17_16:19:11.186] [LOG]
[10-04-17_16:19:11.195] [LOG]
[10-04-17_16:19:11.199] [ERROR]
[10-04-17_16:19:11.205] [LOG]
[10-04-17_16:19:11.207] [LOG]
[10-04-17_16:19:11.218] [LOG]
[10-04-17_16:19:35.228] [LOG]
[10-04-17_16:19:35.232] [LOG]
[10-04-17_16:19:35.234] [LOG]
[10-04-17_16:19:35.235] [ERROR]
[10-04-17_16:19:35.239] [LOG]
[10-04-17_16:19:35.241] [LOG]
[10-04-17_16:19:35.251] [LOG]
[10-04-17_16:19:47.250] [LOG]
[10-04-17_16:19:47.254] [LOG]
[10-04-17_16:19:47.262] [LOG]
[10-04-17_16:19:47.270] [LOG]
[10-04-17_16:20:00.294] [LOG]
[10-04-17_16:20:00.306] [LOG]
[10-04-17_16:20:00.308] [LOG]
[10-04-17_16:20:00.309] [ERROR]
[10-04-17_16:20:00.311] [LOG]
[10-04-17_16:20:00.312] [LOG]
[10-04-17_16:20:00.320] [LOG]
[10-04-17_16:20:24.335] [LOG]
[10-04-17_16:20:24.339] [LOG]
[10-04-17_16:20:24.341] [LOG]
[10-04-17_16:20:24.345] [ERROR]
[10-04-17_16:20:24.346] [LOG]
[10-04-17_16:20:24.348] [LOG]
[10-04-17_16:20:24.355] [LOG]
[10-04-17_16:20:49.383] [LOG]
[10-04-17_16:20:49.387] [LOG]
[10-04-17_16:20:49.389] [LOG]
[10-04-17_16:20:49.390] [ERROR]
[10-04-17_16:20:49.391] [LOG]
[10-04-17_16:20:49.393] [LOG]
[10-04-17_16:20:49.399] [LOG]
[10-04-17_16:21:13.424] [LOG]
[10-04-17_16:21:13.427] [LOG]
[10-04-17_16:21:13.429] [LOG]
[10-04-17_16:21:13.430] [ERROR]
[10-04-17_16:21:13.431] [LOG]
[10-04-17_16:21:13.433] [LOG]
[10-04-17_16:21:13.439] [LOG]
[10-04-17_16:21:38.477] [LOG]
[10-04-17_16:21:38.480] [LOG]
[10-04-17_16:21:38.482] [LOG]
[10-04-17_16:21:38.483] [ERROR]
[10-04-17_16:21:38.484] [LOG]
[10-04-17_16:21:38.486] [LOG]
[10-04-17_16:21:38.493] [LOG]
[10-04-17_16:21:52.508] [LOG]
[10-04-17_16:21:52.511] [LOG]
[10-04-17_16:21:52.513] [LOG]
[10-04-17_16:21:52.521] [LOG]
[10-04-17_16:22:02.549] [LOG]
[10-04-17_16:22:02.554] [LOG]
[10-04-17_16:22:02.556] [LOG]
[10-04-17_16:22:02.557] [ERROR]
[10-04-17_16:22:02.560] [LOG]
[10-04-17_16:22:02.562] [LOG]
[10-04-17_16:22:02.574] [LOG]
[10-04-17_16:22:27.596] [LOG]
[10-04-17_16:22:27.602] [LOG]
[10-04-17_16:22:27.605] [LOG]
[10-04-17_16:22:27.607] [ERROR]
[10-04-17_16:22:27.608] [LOG]
[10-04-17_16:22:27.609] [LOG]
[10-04-17_16:22:27.616] [LOG]
[10-04-17_16:22:51.638] [LOG]
[10-04-17_16:22:51.641] [LOG]
[10-04-17_16:22:51.644] [LOG]
[10-04-17_16:22:51.645] [ERROR]
[10-04-17_16:22:51.646] [LOG]
[10-04-17_16:22:51.650] [LOG]
[10-04-17_16:22:51.657] [LOG]
[10-04-17_16:22:55.660] [LOG]
[10-04-17_16:22:55.663] [LOG]
[10-04-17_16:22:55.665] [LOG]
[10-04-17_16:22:55.671] [LOG]
[10-04-17_16:17:21.864] [INFO]
[10-04-17_16:17:21.883] [INFO]
[10-04-17_16:17:23.025] [LOG]
[10-04-17_16:17:23.046] [LOG]
[10-04-17_16:17:23.065] [LOG]
[10-04-17_16:17:23.066] [ERROR]
[10-04-17_16:17:23.069] [LOG]
[10-04-17_16:17:23.084] [LOG]
[10-04-17_16:17:23.115] [LOG]
[10-04-17_16:37:03.757] [INFO]
[10-04-17_16:37:03.769] [INFO]
[10-04-17_16:37:11.730] [LOG]
[10-04-17_16:37:11.735] [LOG]
[10-04-17_16:37:11.742] [LOG]
[10-04-17_16:37:11.757] [LOG]
[10-04-17_16:37:18.796] [LOG]
[10-04-17_16:37:18.801] [LOG]
[10-04-17_16:37:18.803] [LOG]
[10-04-17_16:37:18.804] [ERROR]
[10-04-17_16:37:18.805] [LOG]
[10-04-17_16:37:18.810] [LOG]
[10-04-17_16:37:18.819] [LOG]
[10-04-17_16:37:42.835] [LOG]
[10-04-17_16:37:42.838] [LOG]
[10-04-17_16:37:42.846] [LOG]
[10-04-17_16:37:42.847] [ERROR]
[10-04-17_16:37:42.852] [LOG]
[10-04-17_16:37:42.855] [LOG]
[10-04-17_16:37:42.874] [LOG]
[10-04-17_16:38:07.884] [LOG]
[10-04-17_16:38:07.889] [LOG]
[10-04-17_16:38:07.890] [LOG]
[10-04-17_16:38:07.890] [ERROR]
[10-04-17_16:38:07.892] [LOG]
[10-04-17_16:38:07.894] [LOG]
[10-04-17_16:38:07.907] [LOG]
[10-04-17_16:38:13.913] [LOG]
[10-04-17_16:38:13.917] [LOG]
[10-04-17_16:38:13.919] [LOG]
[10-04-17_16:38:13.927] [LOG]
[10-04-17_16:38:31.962] [LOG]
[10-04-17_16:38:31.966] [LOG]
[10-04-17_16:38:31.968] [LOG]
[10-04-17_16:38:31.969] [ERROR]
First it went to 16:29, then jumped to 16:17 (where error started to show up), then went up 16:22, jumped again to 16:17 and the made another jump to 16:37. So on first time jump, H developed that error, while other 3 data sets were totally fine. This really was write error (or corruption, but I don't know how only 2 bytes would be affected on SD card or is it just my luck)
A check in logutil.js is good for BEFORE write, nothing to check if it was actually written. In any case, this partial data in my mind is almost impossible to get, so I don't think there you need to do anything.
Problem like I showed in other thread, which is related to whole line of 9 byte to be 0 could be something to work on, IF there is some simple way (bash script or something).
Alternative to this would be to have option to view data as table ;D, you could see erroneous data this way. Again, I suspect a lot of hard if you go and do this. Maybe put it on todo list.
:-\
This is SD corruption most likely. In which case no amount of checking or seeing data in a table will help. When the hard drive itself is rotten, it's hopeless, it cannot self heal by error checking. This will keep on happening, and we will wonder where the invisible bug is. It's hardware from all indications, that easily explains all this trouble.
I have seen my own bad SD cards continue to write data in a very weird way, some files would be fine, others I could not even open or copy over FTP. It took me a while to realize the SD was indeed bad, I looked elsewhere because the Gateway app sort of kept working and I thought "how can I have a bad SD card if the app is running". But very strange phenomena was happening. Until I changed to a different SD brand and never had issues again.
Sure, that is a possible explanation indeed. Do you have any prefered software to check SD card or you just replace it?
The answer depends how much more of your life you want to spend on this problem ;)
In my case I just replaced the SD card.