LowPowerLab Forum

Software support => Pi Gateway => Topic started by: LukaQ on September 12, 2017, 10:42:15 AM

Title: PI gateway also sending data to wunderground.com
Post by: LukaQ on September 12, 2017, 10:42:15 AM
Hello all,

has anyone done this?
I tried my best with no success. The problem I had is, I don't know of a way to read serial stream with more than one app (first one is always gateway itself). If I could store to file or read with another program, I think I could also send reading to site.

What are ways to solve this?
Title: Re: PI gateway also sending data to wunderground.com
Post by: Felix on September 12, 2017, 10:53:38 AM
I haven't tried it.
But ... maybe you could just code an event that happens when you receive the serial data for your node, then invoke a HTTP request to their API to post the data.
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 16, 2017, 03:19:22 AM
I will first start with this: I don't know how to do that.

That being said, you could use, like your said,  invoke a HTTP request to their API to post the data, like you can do with browser or you could used
https://www.npmjs.com/package/wunderground-pws

But first problem for me is, how to get data from serial and not disturb gateway itself.
-This could be like stream, just like gateway does it with
Code: [Select]
serial.on("data", function(data) { processSerialData(data); });

-Or if that would be written to a file, which would be read with other program, which would open file and read it, then send to WU.

-Or extract data from:
Code: [Select]
 var entry = {_id:id, updated:existingNode.updated, type:existingNode.type||undefined, label:existingNode.label||undefined, descr:existingNode.descr||undefined, hidden:existingNode.hidden||undefined, rssi:existingNode.rssi, metrics:Object.keys(existingNode.metrics).length > 0 ? existingNode.metrics : {}, events: existingNode.events, settings: existingNode.settings };
and put them in each variable, then invoke a HTTP request with variables in the URL

Also don't know if I would like to go with already parsed data, which gateway uses or raw data from serial port itself
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 16, 2017, 05:15:04 AM
Saving serial to file can be done like so:
Code: [Select]
var fs = require('fs'); // reqire fileSystem node module
fs.writeFile("/home/pi/gateway/name_of_your_file", data );

writeFile will overwrite file, you could append with
Code: [Select]
fs.appendFile
, but I don't see it being useful just yet
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 16, 2017, 05:52:15 AM
From file I read:
Code: [Select]
[40] C:23.04 H:50.67 P:1022.08   [RSSI:-54][ACK-sent]
and I can extract C: number only with
Code: [Select]
less -FX /home/pi/gateway/name_of_your_file | awk  '{print $2}' | sed -e 's/C://'
and get only
Code: [Select]
23.04
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 16, 2017, 03:02:09 PM
Success!!!

Now I can send from gateway to wunderground.com

Now how to do it:
1)you will need bash calculator
Code: [Select]
sudo apt-get install bc
2)you will need to add
Code: [Select]
var fs = require('fs'); // reqire fileSystem node module
Code: [Select]
fs.writeFile("/home/pi/gateway/LOG_FILE", data );
to gateway.js, right after
Code: [Select]
var entry = {_id:id, updated:existingNode.updated, type:existingNode.type||undefined, label:existingNode.label||undefined, descr:existingNode.descr||undefined, hidden:existingNode.hidden||undefined, rssi:existingNode.rssi, metrics:Object.keys(existingNode.metrics).length > 0 ? existingNode.metrics : {}, events: existingNode.events, settings: existingNode.settings};
By now you will have ability to calculate float in bash and you will be write "data" (which is what serial gets) to file (append/overwrite).
3)After that, you will need bash script: nano SCRIPT_NAME.sh
Code: [Select]
#!/bin/bash
while :
do
var="$(less -FX /home/pi/gateway/LOG_FILE | awk  '{print $1}' )"
echo $var
if [ "$var" == "[5]" ]
then
temp1="$(less -FX /home/pi/gateway/LOG_FILE | awk  '{print $2}' | sed -e 's/C://')"
temp=$(echo "$temp1*1.8+32" | bc)
fi
###########################################################################################################################################################################################################################################
if [ "$var" == "[40]" ]
then
humidity1="$(less -FX /home/pi/gateway/LOG_FILE | awk  '{print $3}' | sed -e 's/H://')"
pressure1="$(less -FX /home/pi/gateway/LOG_FILE | awk  '{print $4}' | sed -e 's/P://')"
humidity=$(echo $humidity1/1 | bc)
baro=$(echo "scale=2;$pressure1/33.863753" | bc)
fi
###########################################################################################################################################################################################################################################
echo $temp
echo $humidity
echo $baro
curl "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=YOURSTATIONNAME&PASSWORD=YOURPASSWORD&dateutc=now&tempf=${temp%$cr}&baromin=${baro%$cr}&humidity=${humidity%$cr}&weather=&clouds=&softwaretype=Arduino-ESP8266&action=updateraw"
sleep 10
done

run ./SCRIPT_NAME.sh
if you are in the same folder as SCRIPT_NAME.sh, else enter full path
Oh and BTW, I do calculations because I have C, and mBar and not F and inHg, which site expects.

Have fun

Note: Have changed the two files names, so that is more clear which is which, since they are both in the same folder
Title: Re: PI gateway also sending data to wunderground.com
Post by: Kilo95 on October 16, 2017, 03:32:55 PM
wonderful! thanks for your hard work. i do have one additional question...I have 4 nodes on one gateway. how can i send the data from just one particular node to WU by this route and leave the rest as is? it'd be node 5
Title: Re: PI gateway also sending data to wunderground.com
Post by: Kilo95 on October 16, 2017, 04:01:22 PM
so i save the bash script as SOMENAME.sh then run SOMENAME.js? i never made a SOMENAME.js file. i'm a little lost. when i:
$ chmod +x ./SOMENAME.js

it says no such file or directory found
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 16, 2017, 11:58:57 PM
wonderful! thanks for your hard work. i do have one additional question...I have 4 nodes on one gateway. how can i send the data from just one particular node to WU by this route and leave the rest as is? it'd be node 5
I have two nodes, 5 and 40. From one I take temp, from other I take pressure and humidity. If you have more nodes you just add more IF cases, each looking for specific node number you have.
One thing would also be to optimize this script, for it to collect all different data over longer time, then send once, not everytime node is read.

You can comment out IF node == 40, and move what is in IF statement into IF statement of node 5

Code: [Select]
while :
do
var="$(less -FX /home/pi/gateway/mojlog | awk  '{print $1}' )"
echo $var
if [ "$var" == "[5]" ]
then
temp1="$(less -FX /home/pi/gateway/mojlog | awk  '{print $2}' | sed -e 's/C://')"
temp=$(echo "$temp1*1.8+32" | bc)
humidity1="$(less -FX /home/pi/gateway/mojlog | awk  '{print $3}' | sed -e 's/H://')"
pressure1="$(less -FX /home/pi/gateway/mojlog | awk  '{print $4}' | sed -e 's/P://')"
humidity=$(echo $humidity1/1 | bc)
baro=$(echo "scale=2;$pressure1/33.863753" | bc)
fi
echo $temp
echo $humidity
echo $baro
curl "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=XXXXXX&PASSWORD=XXXXX&dateutc=now&tempf=${temp%$cr}&baromin=${baro%$cr}&humidity=${humidity%$cr}&weather=&clouds=&softwaretype=Arduino-ESP8266&action=updateraw"
sleep 10
done
this will read all 3 variables from node 5. As if you had BME280 there
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 17, 2017, 12:01:46 AM
so i save the bash script as SOMENAME.sh then run SOMENAME.js? i never made a SOMENAME.js file. i'm a little lost. when i:
$ chmod +x ./SOMENAME.js

it says no such file or directory found
No, you have SOMENAME.sh and no SOMENAME.js, I made a typo there. You create .sh and you run it from terminal. Next thing would be to make it into autorun script, so that it runs in background all the time
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 17, 2017, 12:07:22 AM
Looks great
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 17, 2017, 01:01:27 AM
Now to make it autorun on startup you need to do:

Code: [Select]
sudo nano /etc/rc.local
then before end of the file, before exit 0 you add
Code: [Select]
/path/to/my/SCRIPT_NAME.sh || exit 1   # Added by me

then you save and can reboot
This will start a process under root, which will run this script.


And this should be all you need to send any data WU accepts.
Note: Changed the script name
Title: Re: PI gateway also sending data to wunderground.com
Post by: Kilo95 on October 17, 2017, 10:42:15 AM
wonderful! many thanks! i will have to try this later. when i tried it yesterday, it froze up Felix's GW dashboard (stuck on "waiting for socket connection"). maybe it won't work with Felix's dashboard or maybe it was my incorrect script. If it doesn't work with Felix's GW, i'll just get another RPi and set it up separate
Title: Re: PI gateway also sending data to wunderground.com
Post by: Kilo95 on October 17, 2017, 11:38:33 AM
is the NAME_YOUR_FILE the same file name and extension as SOMENAME.sh or different?
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 17, 2017, 01:21:01 PM
wonderful! many thanks! i will have to try this later. when i tried it yesterday, it froze up Felix's GW dashboard (stuck on "waiting for socket connection"). maybe it won't work with Felix's dashboard or maybe it was my incorrect script. If it doesn't work with Felix's GW, i'll just get another RPi and set it up separate
It does work with his gateway, it's the only one I have and was the most important thing to keep it still working, there is no waiting for socket connection, since it does not interfere in any way. I'll say it again, it works on gateway itself, would be pointless to have another RPi just for this. Check again everything. Then will try to see, what you have different
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 17, 2017, 01:22:19 PM
is the NAME_YOUR_FILE the same file name and extension as SOMENAME.sh or different?
Indeed it is, I wasn't that consistent when now showing what my file is named.
Title: Re: PI gateway also sending data to wunderground.com
Post by: Kilo95 on October 17, 2017, 01:41:43 PM
that what i thought you intended to do (keep the same gateway). ive been working on this on my smartphone. i'll wait a few days til i can look at it on my computer and will update then
Title: Re: PI gateway also sending data to wunderground.com
Post by: Kilo95 on October 26, 2017, 11:24:25 AM
Luka, just wanted to post what finally worked for me that you helped me work out off forum.

1. Made sure the permissions for the "WHATEVERNAME.sh" was set to 744 or 777.
2. Removed the conversions to metric
3. Changed the C in the script for the temperature to F (otherwise kept giving a syntax error)
4. Changed the print function from 2,3,4 ({print $2}, {print $3}, etc) to 3,4,5 ({print $3}, {print $4}, etc) since my output from the node shows the battery voltage in the #2 slot {print 2}
5. Set var [5] and removed the reference to var [40] since all 3 of my metrics comes from one node (#5).
6. One node i have outputs temp as 4 digits (ex 5895 but on the gateway page shows up as 58.95 deg) so i divided the temp by 100. that's on a Weathershield R1. on my WS R2, it comes out as 58.95 without the correction

many thanks to Luka for his advice and help. all the credit goes to him

Code: [Select]
#!/bin/bash
while :
do
var="$(less -FX /home/pi/gateway/NAME_YOUR_FILE | awk  '{print $1}' )"
echo $var
if [ "$var" == "[5]" ]
temp1="$(less -FX /home/pi/gateway/NAME_YOUR_FILE | awk  '{print $3}' | sed -e 's/F://')"
temp=$(echo $temp1/100 | bc)
humidity1="$(less -FX /home/pi/gateway/NAME_YOUR_FILE | awk  '{print $4}' | sed -e 's/H://')"
pressure1="$(less -FX /home/pi/gateway/NAME_YOUR_FILE | awk  '{print $5}' | sed -e 's/P://')"
humidity=$(echo $humidity)
baro=$(echo $pressure1)
fi
###########################################################################################################################################################################################################################################
echo $temp
echo $humidity
echo $baro
curl "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=YOURSTATIONNAME&PASSWORD=YOURSTATIONKEY&dateutc=now&tempf=${temp%$cr}&baromin=${baro%$cr}&humidity=${humidity%$cr}&weather=&clouds=&softwaretype=Arduino-ESP8266&action=updateraw"
sleep 10
done
Title: Re: PI gateway also sending data to wunderground.com
Post by: hickse on October 05, 2019, 12:26:24 PM
Hello, Can you share your gateway.js file?  When I added the 2 lines of code
var fs = require('fs'); // reqire fileSystem node module
fs.writeFile("/home/pi/gateway/LOG_FILE", data );

to the end of the section of the gateway.js file
var entry = {_id:id, updated:existingNode.updated, type:existingNode.type||undefined, label:existingNode.label||undefined, descr:existingNode.descr||undefined, hidden:existingNode.hidden||undefined, rssi:existingNode.rssi, metrics:Object.keys(existingNode.metrics).length > 0 ? existingNode.metrics : {}, events: existingNode.events, settings: existingNode.settings};

I get the following errors in the gateway.sys.log

fs.js:133
  throw new ERR_INVALID_CALLBACK();
  ^

TypeError [ERR_INVALID_CALLBACK]: Callback must be a function
    at maybeCallback (fs.js:133:9)
    at Object.writeFile (fs.js:1179:14)
    at /home/pi/gateway/gateway.js:731:6
    at newArguments.(anonymous function) (/home/pi/gateway/node_modules/nedb/lib/executor.js:29:17)
    at Cursor.execFn (/home/pi/gateway/node_modules/nedb/lib/datastore.js:484:12)
    at callback (/home/pi/gateway/node_modules/nedb/lib/cursor.js:126:19)
    at /home/pi/gateway/node_modules/nedb/lib/cursor.js:193:12
    at /home/pi/gateway/node_modules/nedb/lib/datastore.js:329:14
    at Object.async.eachSeries (/home/pi/gateway/node_modules/nedb/node_modules/async/lib/async.js:130:20)
    at /home/pi/gateway/node_modules/nedb/lib/datastore.js:323:11


It also breaks my pigateway's functionality (until I remove those lines of code).

My code looks like this:
      var entry = {
        _id:id,
        updated:existingNode.updated,
        type:existingNode.type||undefined,
        label:existingNode.label||undefined,
        descr:existingNode.descr||undefined,
        hidden:existingNode.hidden||undefined,
        rssi:existingNode.rssi,
        metrics:Object.keys(existingNode.metrics).length > 0 ? existingNode.metrics : {},
        events:existingNode.events,
        settings:existingNode.settings,
        multiGraphs:existingNode.multiGraphs,
        icon:existingNode.icon||undefined
      };
   var fs = require('fs'); // reqire fileSystem node module
   fs.writeFile("/home/pi/gateway/LOG_FILE", data );

I assume that the LOG_FILE file must be created prior to use. What type of file should the LOG_FILE be?  What file extension should it have?

Thanks much for your work on this.
Title: Re: PI gateway also sending data to wunderground.com
Post by: LukaQ on October 07, 2019, 03:18:44 PM
Here you go:

my file is "mojlog" and has no extension, doesn't need one (it's text file, any linux program will read it, such as bash or nano and similar)
That file is created in same folder as gateway.js, make one yourself, can be empty