Gateway v8.10.0 released!

[Tony Weil]

I have been experiencing similar problems with the garage polling event deleting other events since I installed the garage mote recently. Eventually I would like to implement a garage polling event that closes the door during certain hours at night if it is open like the one ssmall is using below.

I modified the garagePoll event code to incorporate more logging to demonstrate what I am experiencing.  I also changed gateway.js and inserted a console.log right after the db.update on line 639

I tried updating gateway.js (version 8.10) with the new version released today. I just copy the new gateway.js to the /gateway folder and restart the gateway (sudo systemctl restart gateway) or just reboot. Consistently, the web app does not connect until I put the original gateway.js back. What am I doing wrong? Should I also be updating the other new files on Github? I tried adding the new logUtil.js, metrics.js and package.json, but no luck. I admit I'm in a little over my head.

Unrelated, I have noticed in the past that the metrics.js or files in the userMetrics directory are incredibly touchy and edits frequently cause the web app to not connect. In some cases, I have founds minor errors in the files. In other cases, I make the most minor "correct" edits and the web app won't connect until I put back the original metrics.js.

Can someone please explain how I can view the Gateway CONSOLE on the Pi. I'm assuming this is different from the log file.

[Felix]

To update to the latest non released version (ie last version + all sequential changes which haven't made it into an "official" new release), you need all the .js files, and also the content of the www folder. Usually non released changes are always tested, so if you manually download an replace all the files, you should be good to go, but aware that you are ahead of the last release. If unsure, you can download the last release and use that.

Javascript, like any other computer language, is touchy when it comes to syntax errors and bugs, no way around that.

The client side console LOG in the UI is different than the server side log which is much more comprehensive. However you can log to each individually so they are basically independent of each other.

[sparky]

sparky - the polling event bug should now be fixed, sorry it took so long!
Just get the latest gateway.js, see the commit changeset for the actual changes.

Working great so far! 

I appreciate your support Felix, thanks

[Tony Weil]

Problems upgrading to 8.11.1

1. I tried replacing the updated 8.11.1 github versions of gateway.js, logUtil.js, metrics.js, package.json, and the entire userMetrics and www folders from https://github.com/LowPowerLab/RaspberryPi-Gateway. I get the following error in the Console Log:

/home/pi/gateway/gateway.js:49
var port = new serialport(settings.serial.port.value, {baudRate : settings.serial.baud.value});
           ^

TypeError: serialport is not a function
    at Object.<anonymous> (/home/pi/gateway/gateway.js:49:12)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Function.Module.runMain (module.js:441:10)
    at startup (node.js:140:18)
    at node.js:1043:3

Garage Poll Questions:

2. What does the unmodified "Garage Poll" event in metrics.js do. Is it necessary for the basic Garage Mote "status"? What causes it to auto start in the first place, something in gateway.js?

3. How does ssmall's Garage Poll work? I can see generally that it only has the "close" action if the time is between 9pm and 6am and if the "status" has been "CLOSED" for more than 10 minutes. What is "closecount" used for and what does the "nodeRightNow" command do?

4. Is it possible, but maybe not the most functional idea, to create a simpler event that just looks at the current Garage "status" and if it is OPEN and the time is between 9pm and 6am, just CLOSE the door?

Node.js

I guess I finally need to learn node.js and Javascript if I want to understand this better, right? Thanks.

[Tony Weil]

Problems upgrading to 8.11.1

1. I tried replacing the updated 8.11.1 github versions of gateway.js, logUtil.js, metrics.js, package.json, and the entire userMetrics and www folders from https://github.com/LowPowerLab/RaspberryPi-Gateway. I get the following error in the Console Log:

/home/pi/gateway/gateway.js:49
var port = new serialport(settings.serial.port.value, {baudRate : settings.serial.baud.value});
           ^

TypeError: serialport is not a function
    at Object.<anonymous> (/home/pi/gateway/gateway.js:49:12)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Function.Module.runMain (module.js:441:10)
    at startup (node.js:140:18)
    at node.js:1043:3

Trying to do my homework. It appears that the serialport module has been changed to //https://github.com/node-serialport/node-serialport,   version "serialport": "~6.2.1", in the 8.11.1 gateway.js. It used to be //https://github.com/voodootikigod/node-serialport, version "serialport": "~2.0", in 8.10.

Is the change as simple as running "npm update serialport"??

It seems that socket.io, nodemailer and request have also been updated between 8.10 and 8.11.1. Do I need to upgrade those also or will "npm update" upgrade all of the modules? Will this get the versions in the package.json or possibly newer versions in the npm repository, whatever they happen to be?

package.json 8.11.1
    serialport": "~6.2.1",
    "socket.io": "~2.1.1",
    "nedb": "~1.8",
    "json5": "~0.5.0",
    "nconf": "~0.8.2",
    "nodemailer": "~4.6.7",
    "console-stamp": "~0.2.4",
    "colors": "~1.1.2",
    "request": "~2.87",
    "merge": "~1.2.0",
    "suncalc": "~1.8"

package.json 8.10

    "serialport": "~2.0",
    "socket.io": "~1.5",
    "nedb": "~1.8",
    "json5": "~0.5.0",
    "nconf": "~0.8.2",
    "nodemailer": "~1.10",
    "console-stamp": "~0.2.4",
    "colors": "~1.1.2",
    "request": "~2.78",
    "merge": "~1.2.0",
    "suncalc": "~1.8"

[ssmall]

3. How does ssmall's Garage Poll work? I can see generally that it only has the "close" action if the time is between 9pm and 6am and if the "status" has been "CLOSED" for more than 10 minutes. What is "closecount" used for and what does the "nodeRightNow" command do?

Tony, my version of garage poll tries to handle a few situations.  I did not want to close the door as soon as it was detected to be open.  My wife gets up early in the morning to work out and starts her car to warm it up in the winter, so I wanted to have a period of time that the door stayed open so carbon monoxide would not build up in the garage.  The close count is there in case the vehicle is not pulled all the way into the garage and the door is not continuously closed on the vehicle during the night, the code attempts to close a max of 3 three times.  My code used to work before version 8 of the software.  It seems (I could be wrong on this) that any changes that are made to a node and saved in an event that is firing are overwritten by the runAndReschedule method.  If you are OK with the garage door closing as soon as it is detected to be open then you should be good.

-Steve

[sparky]

Steve,

So is the code you posted still working or did you have to re-write due to the new updates?  I would like to try this "auto close" feature myself.  All you changed was the new Poll event?

Thanks

[ssmall]

Sparky,

The code does not work any more in Version 8.x (it used to work in version 7.x) because I am trying maintain state in the node.  The changes that are made to the node inside of the event are saved but they are then overwritten in the runAndReschedule method with a node that does not contain the information that was saved in the event.  If you don't try to save any state in the node it should work.  You just can't take some of the precautions I was trying to take in the code I posted.


Hope that makes sense,

-Steve

[sparky]

ok, thanks Steve.  bummer 

[Felix]

Tony,
If you have not recently installed the gateway then you also have to update nodejs packages. I made some updates/changes to the packages in the recent months.
Get the latest packages.json, then run npm update:

cd ~/gateway
sudo npm update

Guide here:
https://docs.npmjs.com/getting-started/updating-local-packages

If all else fails, save your db folder and settings.json5, and run a clean setup again, indeally on the latest release of raspbian.

[Tony Weil]

Thanks Felix. Still learning...

Some success! Eventually, after some problems I'll describe further below, I got the latest package.json and only updated serialport to 6.2.1 using "npm update serialport". Before this, as you suggested, I also copied the following new files/folders from the 8.11.1 update on Github: gateway.js, logUtil.js, metrics.js and the whole www folder.

After running npm update serialport, npm outdated looks like this and version 8.11.1 seems to be running fine, even though some packages were not updated.

pi@raspberrypi:~/gateway $ npm outdated
Package        Current  Wanted  Latest  Location
colors           1.1.2   1.1.2   1.3.2  colors
console-stamp    0.2.6   0.2.7   0.2.7  console-stamp
json5            0.5.1   0.5.1   2.1.0  json5
merge            1.2.0   1.2.1   1.2.1  merge
nconf            0.8.5   0.8.5  0.10.0  nconf
nodemailer      1.10.0   4.6.8   4.6.8  nodemailer
request         2.78.0  2.87.0  2.88.0  request
serialport       6.2.2   6.2.2   7.0.2  serialport
socket.io        1.5.1   2.1.1   2.1.1  socket.io

==================

My previous failed attempt was as follows.I would love to understand why this didn't work and how to upgrade the remaining packages, if they actually need it.

With new package.json in place, I ran "npm outdated", which showed a number of packages needed updating, serialport and nodemailer being the most in need of a major update.

pi@raspberrypi:~/gateway $ npm outdated
Package        Current  Wanted  Latest  Location
colors           1.1.2   1.1.2   1.3.2  colors
console-stamp    0.2.6   0.2.7   0.2.7  console-stamp
json5            0.5.1   0.5.1   2.1.0  json5
merge            1.2.0   1.2.1   1.2.1  merge
nconf            0.8.5   0.8.5  0.10.0  nconf
nodemailer      1.10.0   4.6.8   4.6.8  nodemailer
request         2.78.0  2.87.0  2.88.0  request
serialport       2.0.6   6.2.2   7.0.2  serialport
socket.io        1.5.1   2.1.1   2.1.1  socket.io

After running "npm update", the web client could not connectand there were errors in the  server log about nodemailer. Unfortunately, those server logs got lost. Attached is npm_update.txt that shows what happen while "npm update" was running.

[Felix]

I don't recall having so many warnings, definitely some odd interaction between your nodejs version and the packages dependencies. From your logs and what you show, I cannot tell if there's a clean way to updating your existing setup. I now realize the socket.io was also updated, which includes a client side script in the www folder, so that whole folder should also be updated to latest.

Then nodejs itself might need updating. I made some changes to the setup script, which is a better install of nodejs IMO. I suspect you might spend a lot more time trying to dig yourself out of warnings and error logs. Rather, I would recommend backing up your existing db folder and settings, and getting latest raspbian and doing a clean install, should take no more than 10 minutes if you have a recent Pi. I am in no way an expert on node/npm specific package errors, dependencies and how to quickly resolve them. I usually just wipe and clean install myself, knowing that's generally a good idea and will fetch the latest Pi security updates as well. Let me know what you decide and how it goes...

[Tony Weil]

I really liked the idea of a clean install instead of fixing one problem after another. It was relatively easy and worked like a charm. Thanks for the good advice.

To make sure I didn't kill my working Gateway, I set up a new 8.11.1 gateway on a second Pi with a different IP address and commented out the MightyHat shutdown (shutdownirg.py). Then I copied over the db, settings.json5 and the UserMetrics folder. Once it was working, I just moved the SD card to the Gateway Pi hardware with the MightyHat, changed the IP address and uncommented the MightyHat shutdown.

I temporarily lost a custom mote definition (my LeakMote) that I forgot was defined in metrics.js and had a custom icon in www/images. I was able to restore and move the mote definition to userMetrics.js so I don't lose it on the next upgrade, but I was not able to figure out how to move the icon to the userMetrics folder. I tried all kinds of paths to the icon.png inside the mote definition. I also temporarily lost some custom Events and a some custom Metrics that I had originally put in metrics.js and had never moved over to userMetrics. All fixed now.

Now that I have a working 8.11.1 with a working Garage Poll, I can get back to what I really wanted to do and work on getting the updated Garage Poll event to close the garage door between midnight and 6am.

[Felix]

I will soon release a new version which will include an example garagePoll with auto-close functionality, based on ssmall's work. Stay tuned.

[Tony Weil]

That is really great news, Felix. So far trying to understand the garagePoll with auto-close has been beyond my current ability. Someday, I would like to understand it.