Author Topic: Database/nodes lost on power loss/improper shutdown [+solution]  (Read 13739 times)

detroit_johnny

  • NewMember
  • *
  • Posts: 35
Database/nodes lost on power loss/improper shutdown [+solution]
« on: September 25, 2015, 03:49:28 PM »
I noticed in working with the gateway a few weeks now that if I update metric.js and I reboot, all my previously programmed nodes come up just as I left them.  This is no problem.

However if my Raspi loses power suddenly without a reboot command, I lose all the nodes and next time I boot up I have to initialize all the nodes again (with node type, label, etc).

Is there a way to save my node configuration on the pi so that it can be restored when that happens?  I am planning to order a MightyBoost or ATXRaspi with a battery so I can get a proper shutdown every time in the future.  In the meantime what options are there?  Thanks in advance.

John
« Last Edit: November 24, 2015, 08:02:12 PM by Felix »

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Saving node configurations on the gateway
« Reply #1 on: September 27, 2015, 11:03:14 PM »
John,
What you're saying is a bit worrying. In the sense that after any node update, it should be persisted forever, until you change it again on the node page. That's regardless of reboot or shutdown or pulling the plug (assuming your Pi can still boot after that power fail).
So ... at this point I'd at least recommend trying the latest image if you can. But still, all previous versions were not losing node unexplicably (not that I know of).

detroit_johnny

  • NewMember
  • *
  • Posts: 35
Re: Saving node configurations on the gateway
« Reply #2 on: September 30, 2015, 07:24:57 PM »
I tried another experiment today.

Case 1:  sudo reboot now --> works fine, system comes back up and the nodes are just as I left them.

Case 2:  sudo shutdown now --> wait about 30 seconds, then I unplug, and plug power back in.  All the nodes are gone.  I am using the latest image v6, and I'm pretty sure this was also happening in v5.  This is so strange, I'm not sure what I'm doing wrong??

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Saving node configurations on the gateway
« Reply #3 on: October 01, 2015, 08:34:05 AM »
Rebooting or shutting down and unplugging power has nothing to do with how the engine stores data. So what you are reporting is completely strange. I usually use sudo halt and sudo reboot and have never seen anything like this.
After a 'missing nodes' situation, what do you see in the /moteino/db subdirectory?
There should be a gateway.db and a bunch of binary files that store the graph data. There is no process in my gateway that deletes or purges those files or their content.

detroit_johnny

  • NewMember
  • *
  • Posts: 35
Re: Saving node configurations on the gateway
« Reply #4 on: October 01, 2015, 09:49:09 PM »
Hi Felix, I used the "sudo halt" this time.  Guess what?  The nodes are all still in place! 

Still, I was curious why "sudo shutdown now" or a loss of power causes the issue (I guess sudo shutdown now is about the same as an instant power loss, I'm still learning Linux).  So... I took a screen shot of the /moteino/db subdirectory before and after I used that command instead.  I attached the results.  As expected, the nodes all disappeared with the "sudo shutdown now" command, which is the same thing I experience if power is suddenly cut from the gateway Pi before a proper shutdown --> now defined as "sudo halt" or "sudo reboot now".

You will see from the comparison of the screen shots that gateway.db file size has gone to zero, despite all the other files still being in place.






Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Saving node configurations on the gateway
« Reply #5 on: October 01, 2015, 11:09:00 PM »
Again, strange. At any time during the gateway's activity, if you download the gateway.db, you should see the content being updated as nodes send data. Basically neDB opens the file and always appends to it, even if it's an update or delete. Then once per day my script recycles that DB to reclaim all the stale entries in that file. Anyway that file should always have content. Now perhaps there is a handle to that file by neDB when power is cut or sudo shutdown now is used and it doesn't get closed properly and hence the file ends up corrupted or somehow empty. That I have not verified or tried, but that would be my best guess based on your findings.

Will need to look further into if/why sudo shutdown now causes this problem.
In the meanwhile use sudo halt instead, and perhaps a MightyBoost to keep your Pi juicy during a power outage :)

detroit_johnny

  • NewMember
  • *
  • Posts: 35
Re: Saving node configurations on the gateway
« Reply #6 on: October 02, 2015, 10:23:22 PM »
Yes I agree, my lesson learned is to use the sudo halt or sudo reboot now.  I think you helped me answer my own question too about a backup... I believe I need to save a copy of the gateway.db file in a worse-case scenario.  However as a Linux newbie I can't seem to figure out how to stop the whole node server and moteino code on the next startup from the command line so I can restore the gateway.db file.  I tried the command killall node but that only stops the thing for maybe 3 seconds before it starts back up again on its own.  Not enough time for me to delete the new gateway.db file and place the backup file in its place.  Any ideas?  Do I need to write a script or something to do all of that quickly?

BTW I guess the command sudo shutdown now must not properly close all open files in the system which I think is why gateway.db goes blank.  But the other commands do close any open files first before losing system power.  Maybe??

detroit_johnny

  • NewMember
  • *
  • Posts: 35
Re: Saving node configurations on the gateway
« Reply #7 on: October 02, 2015, 11:45:13 PM »
I noticed if I write an invalid event in the metrics.js file I can make the moteino system not start back up again.  At that point I can delete the gateway.db file and replace it with my saved backup gateway.db file.  After that I comment out or delete my crummy line of code inside metrics.js, do a sudo reboot now and my old nodes and descriptions are back in place   :)

detroit_johnny

  • NewMember
  • *
  • Posts: 35
Re: Saving node configurations on the gateway
« Reply #8 on: October 03, 2015, 10:37:14 PM »
As I mentioned I'm a newb so it took me another 24 hours to figure out that from the terminal I can also issue a sudo stop gateway and then I can put my backup gateway.db file back in place.  After that simply restart the home automation server with sudo start gateway.  Learning new things everyday...

By the way Felix I got a MightyBoost up and running today for my home automation system, using it with my Pi2.  Wondering there must be a way to have a Moteino with a radio on it acting as both the controller for the MightyBoost and as the gateway for the home automation Pi?  Has anyone done this yet and have a sketch they could share?

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Saving node configurations on the gateway [solved]
« Reply #9 on: October 05, 2015, 09:22:32 AM »
See this project where [Chris] seems to have combined the MightyBoost control sketch with a gateway receiver sketch.
Code here.

detroit_johnny

  • NewMember
  • *
  • Posts: 35
Re: Saving node configurations on the gateway [solved]
« Reply #10 on: October 05, 2015, 10:20:33 PM »
Thank you Felix, the links you posted look very promising, I should be able to use them!  I'll be unable to study it for a week or so but I will get to it for sure and let you and the forums know if I was able to implement my own version of it and share code.  Thank you much! 

Lensdigital

  • Full Member
  • ***
  • Posts: 155
    • Lensdigital
Re: Saving node configurations on the gateway [solved]
« Reply #11 on: October 06, 2015, 02:23:06 PM »
Just FYI, I had same issue today. Lost all my node names, I think Pi crashed or lost power recently...

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Saving node configurations on the gateway [solved]
« Reply #12 on: October 06, 2015, 04:00:27 PM »
Ok so it seems this might be an issue for gateways that don't have any power backup. I'd never would have noticed since I run mine on MightyBoost so when power was lost the backup battery kept it going. I will have to look further into it, but it will require a new setup since it's too difficult to disassemble my gateway as it is now.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Saving node configurations on the gateway [solved]
« Reply #13 on: October 08, 2015, 04:53:03 PM »
I've tried replicating but I can't seem to. The nodes are still there after a sudo shutdown now.
My gateway is using an older image, I will try to test using the official image when I get some extra time.

syrinxtech

  • Sr. Member
  • ****
  • Posts: 346
  • Country: us
    • Syrinx Technologies
Re: Saving node configurations on the gateway [solved]
« Reply #14 on: October 09, 2015, 03:18:43 PM »
Folks, I can also attest to the fact shutting down the Pi unexpectedly causes a total loss of nodes.  I have duplicated this several times including today when I just started up the gateway.  I had 3 nodes saved but when I shut down the Pi I unplugged it without shutting it down properly to simulate a power loss.  When I booted up today I have a gateway.db at zero bytes.

I've been a Unix user for almost 30 years and I've always used "sudo shutdown (-r|-h) now" without issue.  I can't say for sure if it's the shutdown command or the simple loss of power that corrupts the file but it definitely happens.  If it is as simple as shutting down a new way that will take care of 50% of the potential problems.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Saving node configurations on the gateway [workaround]
« Reply #15 on: October 09, 2015, 04:43:27 PM »
For now the workaround is to use sudo halt
This issue is definitely on my radar.
« Last Edit: November 12, 2015, 07:47:14 AM by Felix »

dirkdeb

  • NewMember
  • *
  • Posts: 9
Re: Nodes are lost upon power loss/improper shutdown [workaround]
« Reply #16 on: November 09, 2015, 05:44:38 AM »
Hi I am getting this same problem with power failure and shutdown -h now  or shutdown -h +2
Today I had a few power failures and now if I trie to start /etc/init.d/_nodejs-moteino.sh the system report a datastore.js:67
Error: ENOSPC , write error
I noticed in /moteino/db a file gateway.db ~~  along with the gateway.db file
I tried to restart the gateway again and now this file is missing but now there are  two files gateway_nonmatches.db~ with user:group 'pi:pi' and gateway_nonmatches.db~~ with user:group 'root:root'
Although my one sensor with 5 metrics(s) files are in the db directory and previously was registered in the UI there were no sensors/metrics to be seen before the power loss this morning.
Hope this helps in tracing/catching the culprit.
I start digging to make a plan with this. Maybe one can save/backup the crucial files on change and recover on startup ?!
Thanx

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Nodes are lost upon power loss/improper shutdown [workaround]
« Reply #17 on: November 09, 2015, 08:32:21 AM »
dirkdeb, thanks for your details.
If any of the metrics are logged, can you check (if not too late) whether the metrics log files are still present? They are separate from the general node info DB (gateway.db).

dirkdeb

  • NewMember
  • *
  • Posts: 9
Re: Nodes are lost upon power loss/improper shutdown [workaround]
« Reply #18 on: November 10, 2015, 06:28:55 AM »
Hi, yes they were still intact in the /home/pi/moteino/db/ directory. ie: 00x_xx.bin pi:pi  attrib. 644
The db directory sits with ownership pi:pi
settings.js root:root
package.json nobody:root
metrics.js pi:pi
gateway.js pi:pi all same read write access 644
I must confess I am running as root ie. start/stop /etc/init.d/_nodejs-moteino.sh
I changed the "start )" to only log to standard output/console to cache the errors.
#

dirkdeb

  • NewMember
  • *
  • Posts: 9
Re: Nodes are lost upon power loss/improper shutdown [workaround]
« Reply #19 on: November 12, 2015, 02:08:53 AM »
Hi I started to investigate a way to "robustisize" the whole data integrity issue.
I got hold of a "power pack" / power bank for offline cellphone charging, plugged that in overnight and it's running fine. With that I prevent any possibility of power losses.
I then plugged in a USB stick mount the device over my /home/pi/moteino/db directory and got the data logging done on the USB device. With that the possibility of SD card failure is out of the way.
I stopped my node gateway by hitting "/etc/init.d/_nodejs-moteino.sh stop" a few times and...
After the process stopped  I use "/etc/init.d/_nodejs-moteino.sh  start" to start the gateway again ..........and gone was my only sensor!
The data base directory still have the metrics files in it and looks sound, ownerships are good read/write access are good. But gateway.db shows 0 size ?
command df recons I still have a lot of disk space left on the stick.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Nodes are lost upon power loss/improper shutdown [workaround]
« Reply #20 on: November 12, 2015, 07:45:18 AM »
This is good to know. So gateway.db is getting wiped out. It's probably a neDB issue. I will need to focus on that. Sorry for the lack of updates on this, I know it's pretty important. I have been tied up in other things (personal and LPL related). But still on my priority list. Anyone else is welcome to contribute of course.

dirkdeb

  • NewMember
  • *
  • Posts: 9
Re: Nodes are lost upon power loss/improper shutdown [workaround]
« Reply #21 on: November 13, 2015, 01:40:03 AM »
Thanx Felix,
Me/ someone with the same problem will find time somewhere/ somehow to get to the root of this. So many things to do so Few time :)
Regards

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Nodes are lost upon power loss/improper shutdown [workaround]
« Reply #22 on: November 19, 2015, 12:12:44 PM »
There appears to be a fix in NeDB for this issue, see the change log for version 1.4.1.
I will try to test this today.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Nodes are lost upon power loss/improper shutdown [workaround]
« Reply #23 on: November 23, 2015, 07:52:21 AM »
Looks like this is a known bug and it is being worked on. The creator of neDB reported that the current version is 1.4.0 and that 1.4.2 is not a stable release and does not include the database purging bug fix. I opened a github issue for this and I will watch this to make sure I update as soon as there is a fix.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6605
  • Country: us
    • LowPowerLab
Re: Nodes are lost upon power loss/improper shutdown [workaround]
« Reply #24 on: November 23, 2015, 02:30:10 PM »
UPDATE: Luis Chatriot has released a version (neDB 1.5) that fixes the power-cycle database purging issue. I tested on my own Pi and can confirm the database is not lost any longer. However this latest commit in the github master has yet to be published to NPM so until then, whoever wants to upgrade will need to do the following process manually:

- cd /home/pi/moteino/node_modules/nedb - change folder to nedb package location
- npm update - updates nedb and dependencies
- issue a pkill node to recycle the gateway node app
- npm outdated - this will show your current installed version is up to date (note this screenshot shows WANTED as 1.4.0 because 1.5 wasn't published to NPM yet):

 
« Last Edit: December 19, 2015, 11:04:23 AM by Felix »