Author Topic: Pi Gateway datastore.js issue [& fix]  (Read 902 times)

gigawatts

  • NewMember
  • *
  • Posts: 23
Pi Gateway datastore.js issue [& fix]
« on: December 01, 2020, 08:56:22 PM »
I'm getting this error when trying to restart the gateway.service (running on a Raspberry Pi 2B with Raspbian GNU/Linux 9.11 stretch, Gateway version 9.0.0, I haven't gotten an opportunity to upgrade to 9.1 or 9.2 yet)


Code: [Select]
$ tail /home/pi/gateway/logs/gateway.sys.log

[12-01-20_19:48:17.151] [INFO]   LOADING USER METRICS...
[12-01-20_19:48:17.175] [INFO]   LOADING USER METRICS MODULE [_InternetSpeedTest.js]
[12-01-20_19:48:17.202] [INFO]   LOADING USER METRICS MODULE [_example.js]
[12-01-20_19:48:17.228] [INFO]   LOADING USER METRICS MODULE [_garage-auto-close.js]
/home/pi/gateway-v9.0.0/node_modules/nedb/lib/datastore.js:77
    if (err) { throw err; }
               ^

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (2923528192) is greater than possible Buffer: 1073741823 bytes
    at FSReqCallback.readFileAfterStat [as oncomplete] (fs.js:269:11)

I did try disabling both of my custom user metrics, that did not help, same error. I'm guessing my metrics database got too large (based on the error message). I know when I was messing with the gateway last week, I did run the "compact database" command from the web UI, if that makes any difference.


Update 1: Well, I found the problem, my new fancy custom user metric to reset my monthly water usage (on a waterMote) didn't just fire once, it fired A LOT of times, bringing my ~/gateway/data/db/gateway.db file up to 2789 MB. What format is this database file in and what tools can I use to prune the junk out of it, so I can restore it to normal operation? Is it just a giant JSON file, and I just need to find the bad entries and delete them?

Update 2: It was indeed just a giant JSON file, and I was able to use a combination of grep and jq to filter out all the bad junk. Back down to 3MB and everything looks to be working again!

For anyone looking to NOT repeat what I did here, I proudly posted my new function over here last week. If anyone would like to point out where I went wrong there, so myself and others can learn from the mistake, I welcome the feedback =]

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Pi Gateway datastore.js issue [& fix]
« Reply #1 on: December 03, 2020, 05:07:30 PM »
With power comes responsibility, someone once said :-)
It's very true with the PiGateway project.
A 2.7Gig node database would be quite a problem ;-)
Please do post an update when you have a full fix, both here and in the other thread, thanks.

gigawatts

  • NewMember
  • *
  • Posts: 23
Re: Pi Gateway datastore.js issue [& fix]
« Reply #2 on: December 07, 2020, 02:14:04 PM »
My database fix was multi-stepped and basically involved using jq (I personally ran it from my much more powerful linux server, but it could be run on the pi as well, it would just take much longer) to print out all the "updated" timestamps in the gateway.db JSON file, find all the timestamps that had hundreds/thousands of entries, and then use egrep to strip each of those lines from the output and write out a clean version of the gateway database file.

Something along these lines:

Code: [Select]
## print out a list of timestamps that show up the most amount of times
cat gateway.db | jq -r '.updated' | sort -n | uniq -c | sort -n | head

## take each of the timestamps that shows up lots of times then put them into the below egrep (-v flag means print out everything EXCEPT lines matching the following) to filter each offense out
 egrep -v '1606824689482|1606823435523|1606822995876|1606822201524|1606802398018|1606803891784|1606805405522|1606813458754|1606820208156' gateway.db > gateway.db.fixed

Then I just copied the db file back to my pi, started the service back up, and it was happy again!