LowPowerLab Forum

Software support => Pi Gateway => Topic started by: ekoelle on April 06, 2016, 10:59:06 PM

Title: Exporting Data to CSV [implemented!]
Post by: ekoelle on April 06, 2016, 10:59:06 PM
This may be a pretty basic question but is there a way to export the historical data from my sensors to a CSV file?
Title: Re: Exporting Data to CSV
Post by: Felix on April 07, 2016, 07:57:07 AM
UPDATE: This CSV EXPORT feature is now implemented, see below for video demo.

It's a really good question actually.
The answer right now is ... not really  :'(
The data is stored in binary format. I would need to write a program that exports it according to given date parameters (start end).

Maybe a utility page that allows entering 2 dates and starts a CSV file download?
It could also display the START and END of the stored data, and # of points.
Title: Re: Exporting Data to CSV
Post by: ekoelle on April 07, 2016, 10:46:33 PM
Maybe a utility page that allows entering 2 dates and starts a CSV file download?
It could also display the START and END of the stored data, and # of points.

That would be ideal. I'm looking to implement a temp/humidity tracking system in our warehouses at work but we would need to be able to export the data periodically for generating reports. I'd like to avoid having to tap into an online service as i'd like to keep the system contained within our network and not reliant upon someone else's servers. Is all of the data contained in the gateway.db file?
Title: Re: Exporting Data to CSV
Post by: Felix on April 07, 2016, 11:09:54 PM
I'd like to avoid having to tap into an online service as i'd like to keep the system contained within our network and not reliant upon someone else's servers. Is all of the data contained in the gateway.db file?
That's exactly why I created this gateway software :)

Is all of the data contained in the gateway.db file?
No, the historical logged data is kept in the dedicated log subdirectory, as binary files for each node-metric.
Title: Re: Exporting Data to CSV
Post by: Felix on August 02, 2016, 12:27:59 PM
Happy to say this CSV export feature is now implemented, demo & other features can be seen in this overview:


Clicking the new Export-raw-data button creates a CSV file that you can download and process as you'd like (timestamps in milliseconds since unix epoch):

(https://cloud.githubusercontent.com/assets/3129069/17331843/a2002e90-589a-11e6-99de-a3465b188c0c.png)

The formula used in the video to re-format the timestamp is this: =A2/(1000*60*60*24)+"1/1/1970"+(-5/24)
Title: Re: Exporting Data to CSV [implemented!]
Post by: Nutz on August 09, 2016, 02:57:08 PM
Thanx Felix, this is great. I tested it on my installation and works like a charm. Was busy implementing something similar but you beat me to it.
Title: Re: Exporting Data to CSV [implemented!]
Post by: Seedser on January 24, 2017, 10:36:00 PM
Felix,
Can I export all saved data? I assume data amount is only limited by RPI SD or connected storage device size?
I'm looking to record temperature over the year.
I would like to save at 15 minute intervals.
Data points per sensor per year = 35,040.

Mike
Title: Re: Exporting Data to CSV [implemented!]
Post by: Felix on January 25, 2017, 08:04:36 AM
This function will export the raw data in the zoomed range. So zoom out to a year, then export to CSV, what you have in the binary DB log is what you get, there is no 15minute interval functionality.
If you want 15 minutes data, then send it in/record it at 15 minute intervals.
Title: Re: Exporting Data to CSV
Post by: EnMon on March 15, 2017, 01:06:43 PM
Happy to say this CSV export feature is now implemented, demo & other features can be seen in this overview:


Clicking the new Export-raw-data button creates a CSV file that you can download and process as you'd like (timestamps in milliseconds since unix epoch):

(https://cloud.githubusercontent.com/assets/3129069/17331843/a2002e90-589a-11e6-99de-a3465b188c0c.png)

The formula used in the video to re-format the timestamp is this: =A2/(1000*60*60*24)+"1/1/1970"+(-5/24)

Hello Felix,

Is there a possibility I could insert this time-stamp formula into my gateway.js code, so I download the excel sheet with the actual date and time as opposed to the current way of manually inserting the formula every time I download the CSV file?

If yes, could you point me in the right direction please?