Author Topic: Exporting Data to CSV [implemented!]  (Read 3396 times)

ekoelle

  • Newbie
  • *
  • Posts: 16
Exporting Data to CSV [implemented!]
« on: April 06, 2016, 10:59:06 PM »
This may be a pretty basic question but is there a way to export the historical data from my sensors to a CSV file?
« Last Edit: August 02, 2016, 12:35:32 PM by Felix »

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6007
  • Country: us
    • LowPowerLab
Re: Exporting Data to CSV
« Reply #1 on: April 07, 2016, 07:57:07 AM »
UPDATE: This CSV EXPORT feature is now implemented, see below for video demo.

It's a really good question actually.
The answer right now is ... not really  :'(
The data is stored in binary format. I would need to write a program that exports it according to given date parameters (start end).

Maybe a utility page that allows entering 2 dates and starts a CSV file download?
It could also display the START and END of the stored data, and # of points.
« Last Edit: August 02, 2016, 12:29:08 PM by Felix »

ekoelle

  • Newbie
  • *
  • Posts: 16
Re: Exporting Data to CSV
« Reply #2 on: April 07, 2016, 10:46:33 PM »
Maybe a utility page that allows entering 2 dates and starts a CSV file download?
It could also display the START and END of the stored data, and # of points.

That would be ideal. I'm looking to implement a temp/humidity tracking system in our warehouses at work but we would need to be able to export the data periodically for generating reports. I'd like to avoid having to tap into an online service as i'd like to keep the system contained within our network and not reliant upon someone else's servers. Is all of the data contained in the gateway.db file?

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6007
  • Country: us
    • LowPowerLab
Re: Exporting Data to CSV
« Reply #3 on: April 07, 2016, 11:09:54 PM »
I'd like to avoid having to tap into an online service as i'd like to keep the system contained within our network and not reliant upon someone else's servers. Is all of the data contained in the gateway.db file?
That's exactly why I created this gateway software :)

Is all of the data contained in the gateway.db file?
No, the historical logged data is kept in the dedicated log subdirectory, as binary files for each node-metric.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6007
  • Country: us
    • LowPowerLab
Re: Exporting Data to CSV
« Reply #4 on: August 02, 2016, 12:27:59 PM »
Happy to say this CSV export feature is now implemented, demo & other features can be seen in this overview:


Clicking the new Export-raw-data button creates a CSV file that you can download and process as you'd like (timestamps in milliseconds since unix epoch):



The formula used in the video to re-format the timestamp is this: =A2/(1000*60*60*24)+"1/1/1970"+(-5/24)
« Last Edit: August 02, 2016, 12:38:43 PM by Felix »

Nutz

  • Newbie
  • *
  • Posts: 2
Re: Exporting Data to CSV [implemented!]
« Reply #5 on: August 09, 2016, 02:57:08 PM »
Thanx Felix, this is great. I tested it on my installation and works like a charm. Was busy implementing something similar but you beat me to it.

Seedser

  • Newbie
  • *
  • Posts: 1
  • Country: us
Re: Exporting Data to CSV [implemented!]
« Reply #6 on: January 24, 2017, 10:36:00 PM »
Felix,
Can I export all saved data? I assume data amount is only limited by RPI SD or connected storage device size?
I'm looking to record temperature over the year.
I would like to save at 15 minute intervals.
Data points per sensor per year = 35,040.

Mike

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6007
  • Country: us
    • LowPowerLab
Re: Exporting Data to CSV [implemented!]
« Reply #7 on: January 25, 2017, 08:04:36 AM »
This function will export the raw data in the zoomed range. So zoom out to a year, then export to CSV, what you have in the binary DB log is what you get, there is no 15minute interval functionality.
If you want 15 minutes data, then send it in/record it at 15 minute intervals.

EnMon

  • Newbie
  • *
  • Posts: 32
  • Country: us
Re: Exporting Data to CSV
« Reply #8 on: March 15, 2017, 01:06:43 PM »
Happy to say this CSV export feature is now implemented, demo & other features can be seen in this overview:


Clicking the new Export-raw-data button creates a CSV file that you can download and process as you'd like (timestamps in milliseconds since unix epoch):



The formula used in the video to re-format the timestamp is this: =A2/(1000*60*60*24)+"1/1/1970"+(-5/24)

Hello Felix,

Is there a possibility I could insert this time-stamp formula into my gateway.js code, so I download the excel sheet with the actual date and time as opposed to the current way of manually inserting the formula every time I download the CSV file?

If yes, could you point me in the right direction please?