Author Topic: PI gateway also sending data to wunderground.com  (Read 27666 times)

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
PI gateway also sending data to wunderground.com
« on: September 12, 2017, 10:42:15 AM »
Hello all,

has anyone done this?
I tried my best with no success. The problem I had is, I don't know of a way to read serial stream with more than one app (first one is always gateway itself). If I could store to file or read with another program, I think I could also send reading to site.

What are ways to solve this?

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: PI gateway also sending data to wunderground.com
« Reply #1 on: September 12, 2017, 10:53:38 AM »
I haven't tried it.
But ... maybe you could just code an event that happens when you receive the serial data for your node, then invoke a HTTP request to their API to post the data.

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: PI gateway also sending data to wunderground.com
« Reply #2 on: October 16, 2017, 03:19:22 AM »
I will first start with this: I don't know how to do that.

That being said, you could use, like your said,  invoke a HTTP request to their API to post the data, like you can do with browser or you could used
https://www.npmjs.com/package/wunderground-pws

But first problem for me is, how to get data from serial and not disturb gateway itself.
-This could be like stream, just like gateway does it with
Code: [Select]
serial.on("data", function(data) { processSerialData(data); });

-Or if that would be written to a file, which would be read with other program, which would open file and read it, then send to WU.

-Or extract data from:
Code: [Select]
 var entry = {_id:id, updated:existingNode.updated, type:existingNode.type||undefined, label:existingNode.label||undefined, descr:existingNode.descr||undefined, hidden:existingNode.hidden||undefined, rssi:existingNode.rssi, metrics:Object.keys(existingNode.metrics).length > 0 ? existingNode.metrics : {}, events: existingNode.events, settings: existingNode.settings };
and put them in each variable, then invoke a HTTP request with variables in the URL

Also don't know if I would like to go with already parsed data, which gateway uses or raw data from serial port itself
« Last Edit: October 16, 2017, 04:29:54 AM by LukaQ »

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: PI gateway also sending data to wunderground.com
« Reply #3 on: October 16, 2017, 05:15:04 AM »
Saving serial to file can be done like so:
Code: [Select]
var fs = require('fs'); // reqire fileSystem node module
fs.writeFile("/home/pi/gateway/name_of_your_file", data );

writeFile will overwrite file, you could append with
Code: [Select]
fs.appendFile
, but I don't see it being useful just yet
« Last Edit: October 16, 2017, 05:16:50 AM by LukaQ »

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: PI gateway also sending data to wunderground.com
« Reply #4 on: October 16, 2017, 05:52:15 AM »
From file I read:
Code: [Select]
[40] C:23.04 H:50.67 P:1022.08   [RSSI:-54][ACK-sent]
and I can extract C: number only with
Code: [Select]
less -FX /home/pi/gateway/name_of_your_file | awk  '{print $2}' | sed -e 's/C://'
and get only
Code: [Select]
23.04

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: PI gateway also sending data to wunderground.com
« Reply #5 on: October 16, 2017, 03:02:09 PM »
Success!!!

Now I can send from gateway to wunderground.com

Now how to do it:
1)you will need bash calculator
Code: [Select]
sudo apt-get install bc
2)you will need to add
Code: [Select]
var fs = require('fs'); // reqire fileSystem node module
Code: [Select]
fs.writeFile("/home/pi/gateway/LOG_FILE", data );
to gateway.js, right after
Code: [Select]
var entry = {_id:id, updated:existingNode.updated, type:existingNode.type||undefined, label:existingNode.label||undefined, descr:existingNode.descr||undefined, hidden:existingNode.hidden||undefined, rssi:existingNode.rssi, metrics:Object.keys(existingNode.metrics).length > 0 ? existingNode.metrics : {}, events: existingNode.events, settings: existingNode.settings};
By now you will have ability to calculate float in bash and you will be write "data" (which is what serial gets) to file (append/overwrite).
3)After that, you will need bash script: nano SCRIPT_NAME.sh
Code: [Select]
#!/bin/bash
while :
do
var="$(less -FX /home/pi/gateway/LOG_FILE | awk  '{print $1}' )"
echo $var
if [ "$var" == "[5]" ]
then
temp1="$(less -FX /home/pi/gateway/LOG_FILE | awk  '{print $2}' | sed -e 's/C://')"
temp=$(echo "$temp1*1.8+32" | bc)
fi
###########################################################################################################################################################################################################################################
if [ "$var" == "[40]" ]
then
humidity1="$(less -FX /home/pi/gateway/LOG_FILE | awk  '{print $3}' | sed -e 's/H://')"
pressure1="$(less -FX /home/pi/gateway/LOG_FILE | awk  '{print $4}' | sed -e 's/P://')"
humidity=$(echo $humidity1/1 | bc)
baro=$(echo "scale=2;$pressure1/33.863753" | bc)
fi
###########################################################################################################################################################################################################################################
echo $temp
echo $humidity
echo $baro
curl "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=YOURSTATIONNAME&PASSWORD=YOURPASSWORD&dateutc=now&tempf=${temp%$cr}&baromin=${baro%$cr}&humidity=${humidity%$cr}&weather=&clouds=&softwaretype=Arduino-ESP8266&action=updateraw"
sleep 10
done

run ./SCRIPT_NAME.sh
if you are in the same folder as SCRIPT_NAME.sh, else enter full path
Oh and BTW, I do calculations because I have °C, and mBar and not °F and inHg, which site expects.

Have fun

Note: Have changed the two files names, so that is more clear which is which, since they are both in the same folder
« Last Edit: October 18, 2017, 12:26:29 AM by LukaQ »

Kilo95

  • Sr. Member
  • ****
  • Posts: 340
  • Country: us
Re: PI gateway also sending data to wunderground.com
« Reply #6 on: October 16, 2017, 03:32:55 PM »
wonderful! thanks for your hard work. i do have one additional question...I have 4 nodes on one gateway. how can i send the data from just one particular node to WU by this route and leave the rest as is? it'd be node 5

Kilo95

  • Sr. Member
  • ****
  • Posts: 340
  • Country: us
Re: PI gateway also sending data to wunderground.com
« Reply #7 on: October 16, 2017, 04:01:22 PM »
so i save the bash script as SOMENAME.sh then run SOMENAME.js? i never made a SOMENAME.js file. i'm a little lost. when i:
$ chmod +x ./SOMENAME.js

it says no such file or directory found
« Last Edit: October 16, 2017, 04:09:10 PM by Kilo95 »

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: PI gateway also sending data to wunderground.com
« Reply #8 on: October 16, 2017, 11:58:57 PM »
wonderful! thanks for your hard work. i do have one additional question...I have 4 nodes on one gateway. how can i send the data from just one particular node to WU by this route and leave the rest as is? it'd be node 5
I have two nodes, 5 and 40. From one I take temp, from other I take pressure and humidity. If you have more nodes you just add more IF cases, each looking for specific node number you have.
One thing would also be to optimize this script, for it to collect all different data over longer time, then send once, not everytime node is read.

You can comment out IF node == 40, and move what is in IF statement into IF statement of node 5

Code: [Select]
while :
do
var="$(less -FX /home/pi/gateway/mojlog | awk  '{print $1}' )"
echo $var
if [ "$var" == "[5]" ]
then
temp1="$(less -FX /home/pi/gateway/mojlog | awk  '{print $2}' | sed -e 's/C://')"
temp=$(echo "$temp1*1.8+32" | bc)
humidity1="$(less -FX /home/pi/gateway/mojlog | awk  '{print $3}' | sed -e 's/H://')"
pressure1="$(less -FX /home/pi/gateway/mojlog | awk  '{print $4}' | sed -e 's/P://')"
humidity=$(echo $humidity1/1 | bc)
baro=$(echo "scale=2;$pressure1/33.863753" | bc)
fi
echo $temp
echo $humidity
echo $baro
curl "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=XXXXXX&PASSWORD=XXXXX&dateutc=now&tempf=${temp%$cr}&baromin=${baro%$cr}&humidity=${humidity%$cr}&weather=&clouds=&softwaretype=Arduino-ESP8266&action=updateraw"
sleep 10
done
this will read all 3 variables from node 5. As if you had BME280 there
« Last Edit: October 18, 2017, 12:27:22 AM by LukaQ »

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: PI gateway also sending data to wunderground.com
« Reply #9 on: October 17, 2017, 12:01:46 AM »
so i save the bash script as SOMENAME.sh then run SOMENAME.js? i never made a SOMENAME.js file. i'm a little lost. when i:
$ chmod +x ./SOMENAME.js

it says no such file or directory found
No, you have SOMENAME.sh and no SOMENAME.js, I made a typo there. You create .sh and you run it from terminal. Next thing would be to make it into autorun script, so that it runs in background all the time

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: PI gateway also sending data to wunderground.com
« Reply #10 on: October 17, 2017, 12:07:22 AM »
Looks great

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: PI gateway also sending data to wunderground.com
« Reply #11 on: October 17, 2017, 01:01:27 AM »
Now to make it autorun on startup you need to do:

Code: [Select]
sudo nano /etc/rc.local
then before end of the file, before exit 0 you add
Code: [Select]
/path/to/my/SCRIPT_NAME.sh || exit 1   # Added by me

then you save and can reboot
This will start a process under root, which will run this script.


And this should be all you need to send any data WU accepts.
Note: Changed the script name
« Last Edit: October 18, 2017, 12:28:14 AM by LukaQ »

Kilo95

  • Sr. Member
  • ****
  • Posts: 340
  • Country: us
Re: PI gateway also sending data to wunderground.com
« Reply #12 on: October 17, 2017, 10:42:15 AM »
wonderful! many thanks! i will have to try this later. when i tried it yesterday, it froze up Felix's GW dashboard (stuck on "waiting for socket connection"). maybe it won't work with Felix's dashboard or maybe it was my incorrect script. If it doesn't work with Felix's GW, i'll just get another RPi and set it up separate

Kilo95

  • Sr. Member
  • ****
  • Posts: 340
  • Country: us
Re: PI gateway also sending data to wunderground.com
« Reply #13 on: October 17, 2017, 11:38:33 AM »
is the NAME_YOUR_FILE the same file name and extension as SOMENAME.sh or different?

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: PI gateway also sending data to wunderground.com
« Reply #14 on: October 17, 2017, 01:21:01 PM »
wonderful! many thanks! i will have to try this later. when i tried it yesterday, it froze up Felix's GW dashboard (stuck on "waiting for socket connection"). maybe it won't work with Felix's dashboard or maybe it was my incorrect script. If it doesn't work with Felix's GW, i'll just get another RPi and set it up separate
It does work with his gateway, it's the only one I have and was the most important thing to keep it still working, there is no waiting for socket connection, since it does not interfere in any way. I'll say it again, it works on gateway itself, would be pointless to have another RPi just for this. Check again everything. Then will try to see, what you have different