Author Topic: Merging Exported data into one CSV file  (Read 47075 times)

EnMon

  • NewMember
  • *
  • Posts: 32
  • Country: us
Merging Exported data into one CSV file
« on: February 10, 2017, 06:33:44 PM »
Hello all,

Kudos to Felix for giving us such a wonderful Gateway app, it really fantastic!

Just curious, Is there a way to merge all the data coming from one node into a single CSV file as opposed to the current way of retrieving the parameters separately?
For instance, I'm using the weather shield to retrieve Temp, Humidity and battery voltage data and I want to be able to export all these information into one CSV file instead of manually sorting them out. Can i achieve this from tweaking the node.js code or is this impossible?

Please see attached image for graphical illustration of my question.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Merging Exported data into one CSV file
« Reply #1 on: February 21, 2017, 12:39:14 PM »
Sorry for the late response here. Thanks for the feedback :)

This is not currently directly supported in the app.
I am sure it's not impossible but I haven't thought out a solution for this, but certainly would be a nice feature.
I plan to support multi graphs (multiple data sets on the same graph) in the future, it's being asked for in another thread. That's where this feature would come in handy.
However on thing that will need to reconcile is when the timestamps are close but not identical. Ie ... you are viewing data sets from 2 different nodes, not necessarily from the same one (ie sketch sending multiple data points per 1 timestamp). Not really sure how that would work but I think it should be separate time indexing. Where the data originates from the same node data could be indexed with the same timestamps, but those details have to be planned and maybe some more export options made available.

EnMon

  • NewMember
  • *
  • Posts: 32
  • Country: us
Re: Merging Exported data into one CSV file
« Reply #2 on: February 21, 2017, 03:01:11 PM »
Thanks for your response Felix, almost convinced myself no one was going to respond.
So I tried intercepting the data coming in from the node, via the gateway.js code (picture attached) to include an "All" variable. I was hoping this would be logged as a ".bin" just like other parameters and then the the GETGRAPHDATA function would pull it up as a metric key for graph plot. Unfortunately, this wouldn't work because it a comma separated value data and only NUMERIC data can be stored in DB as a ".bin" file.

I understand you have a lot on your plate, but do you mind giving me an idea of what you'd likely do if you were to embark on doing this.
I seem to have run out of ideas on how to make this possible and I need it for a project with an approaching deadline.  :-[

Suggestions are highly welcome!

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Merging Exported data into one CSV file
« Reply #3 on: February 21, 2017, 03:23:08 PM »
The log you see there is simply a console output log that shows raw data and events etc. The real graph data is stored in binary files. A special algorithm is required to extract it from there, as seen in logutil.js.

I could hack together something in an alpha state, maybe a button on the node page. This would trigger a CSV download for all the graphed metrics, with a single timestamp index column and a separate column for each data point. Where data points are missing for some metrics would be blank cells. Would that work?

PS: So you're using my gateway app huh? In what capacity?

EnMon

  • NewMember
  • *
  • Posts: 32
  • Country: us
Re: Merging Exported data into one CSV file
« Reply #4 on: February 21, 2017, 04:16:14 PM »
That would be awesome. :)

Yes I am using your gateway app for a school project.

Basically, I'm trying to evaluate Moteino's Power consumption compared to other motes such as IRIS, MICA etc.

This will help with making a decision regarding what mote to use for an environmental monitoring system situated in a forest that would barely have supply of solar energy for recharging the batteries. I need a mote that consumes as little power as possible, and Moteino looks very promising.

The gateway app comes in handy because it will help to retrieve the required data. 

Thanks for making life easier for a lot of folks like me.  :D

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Merging Exported data into one CSV file
« Reply #5 on: February 21, 2017, 09:29:26 PM »
Basically, I'm trying to evaluate Moteino's Power consumption compared to other motes such as IRIS, MICA etc.

This will help with making a decision regarding what mote to use for an environmental monitoring system situated in a forest that would barely have supply of solar energy for recharging the batteries. I need a mote that consumes as little power as possible, and Moteino looks very promising.

If you are willing to do a little effort then you can get down to between 2-6uA. If you do a little more effort then you can do down to 0.1uA ;)

OK I will try to look into implementing something, won't promise anything great, just usable.

EnMon

  • NewMember
  • *
  • Posts: 32
  • Country: us
Re: Merging Exported data into one CSV file
« Reply #6 on: February 21, 2017, 09:47:43 PM »
Thanks Felix, I really appreciate your effort  :)!!!!!!

0.1uA  :o ;D 8)

I'm definitely willing to do the little effort, what do I need to do to achieve this?

 

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Merging Exported data into one CSV file
« Reply #7 on: February 21, 2017, 09:59:12 PM »
0.1uA  :o ;D 8)

I'm definitely willing to do the little effort, what do I need to do to achieve this?

UPDATE: see the Moteino-8mhz for even lower power (eliminates the extra quiescent 2uA used by the LDO).

For starters I would look around in the forum, there are some active threads right now that talk about various ways to power a Moteino or Moteino derivations (all 328p based) from coin cells or supercapacitors, small solar panels, or combinations of those things.

You can easily achieve 6uA with the DeepSleep sample sketches. Well, actually those use the WDT for sleeping and waking every 8sec, which uses about 4uA. If you sleep "forever" and disable the WDT you can go to 2uA which is mostly the quiescent current of the MCP1703 regulator. If you want to remove the regulator from a Moteino and power it straight from a coin cell for instance, then you can achieve 0.1uA by disabling the WDT and sleeping "forever". Also in that scenario you'd have to set the fuses to run from the internal 8mhz oscillator rather than the onboard 16mhz one. And by "forever sleep" we mean you can still wake the atmega via a hardware interrupt rather than by its own clock source (the WatchDogTimer).
« Last Edit: February 06, 2018, 04:52:58 PM by Felix »

EnMon

  • NewMember
  • *
  • Posts: 32
  • Country: us
Re: Merging Exported data into one CSV file
« Reply #8 on: February 22, 2017, 10:48:48 AM »
Thanks, I'll try this out and let you know my results.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Merging Exported data into one CSV file
« Reply #9 on: February 22, 2017, 11:31:57 AM »
I think I can implement a generic function to download all the CSV raw data of a node. There would be a default limit of about 100k data points.
The graph would look something like this, note the missing points and merged time index:


EnMon

  • NewMember
  • *
  • Posts: 32
  • Country: us
Re: Merging Exported data into one CSV file
« Reply #10 on: February 22, 2017, 11:56:09 AM »
This would work.  ;) :)

EnMon

  • NewMember
  • *
  • Posts: 32
  • Country: us
Re: Merging Exported data into one CSV file
« Reply #11 on: February 23, 2017, 05:11:13 PM »
Hello Felix,
I know this belongs to a different thread, but this is a feedback to the low power suggestions you gave.
I am having issues with the DeepSleep_usingLowPowerlibrary, this gives me a whooping 5mA current consumption, while the  DeepSleep library gives as low as 6.9uA. I have nothing attached, just my Laptop+USB+FTDI Adapter+Moteino RFM69W
Is there something I'm doing wrong?

I have attached the two codes, though they are the sample codes from the RFM69 library.

DeepSleep_usingLowPowerlibrary
Code: [Select]
//***********************************************************************************************************
// Sample sketch that achieves the lowest power on a Moteino of ~6.5uA
// Everything is put to sleep including the MCU, the radio (if any) and the FlashMem chip
//**** SETTINGS *********************************************************************************************
#define WITH_RFM69              //comment this line out if you don't have a RFM69 on your Moteino
#define WITH_SPIFLASH           //comment this line out if you don't have the FLASH-MEM chip on your Moteino
//***********************************************************************************************************

#include <LowPower.h> //get library from: https://github.com/lowpowerlab/lowpower
                      //writeup here: http://www.rocketscream.com/blog/2011/07/04/lightweight-low-power-arduino-library/

#ifdef __AVR_ATmega1284P__
  #define LED           15 // Moteino MEGAs have LEDs on D15
  #define FLASH_SS      23 // and FLASH SS on D23
#else
  #define LED           9 // Moteinos have LEDs on D9
  #define FLASH_SS      8 // and FLASH SS on D8
#endif

#if defined(WITH_RFM69) || defined(WITH_SPIFLASH)
  #include <SPI.h>                //comes with Arduino IDE (www.arduino.cc)
  #if defined(WITH_RFM69)
    #include <RFM69.h>            //get it here: https://www.github.com/lowpowerlab/rfm69
    RFM69 radio;
  #endif
  #if defined(WITH_SPIFLASH)
    #include <SPIFlash.h>         //get it here: https://www.github.com/lowpowerlab/spiflash
    SPIFlash flash(FLASH_SS, 0xEF30); //EF30 for 4mbit  Windbond chip (W25X40CL)
  #endif
#endif

void setup () {
#ifdef WITH_RFM69
  radio.sleep();
#endif

#ifdef WITH_SPIFLASH
  if (flash.initialize())
    flash.sleep();
#endif

  for (uint8_t i=0; i<=A5; i++)
  {
#ifdef WITH_RFM69
    if (i == RF69_SPI_CS) continue;
#endif
#ifdef WITH_SPIFLASH
    if (i == FLASH_SS) continue;
#endif
    pinMode(i, OUTPUT);
    digitalWrite(i, LOW);
  }
}

void loop ()
{
  //optional blink to know radio/flash sleeping went OK
  pinMode(LED, OUTPUT);
  digitalWrite(LED, HIGH);
  delay(30);
  digitalWrite(LED, LOW);

  //sleep MCU for 8seconds
  LowPower.powerDown(SLEEP_8S, ADC_OFF, BOD_OFF);
}


DeepSleep
Code: [Select]
//***********************************************************************************************************
// Sample sketch that achieves the lowest power on a Moteino of ~6.5uA
// Everything is put to sleep including the MCU, the radio (if any) and the FlashMem chip
//**** SETTINGS *********************************************************************************************
#define WITH_RFM69              //comment this line out if you don't have a RFM69 on your Moteino
#define WITH_SPIFLASH           //comment this line out if you don't have the FLASH-MEM chip on your Moteino
//***********************************************************************************************************

#include <Arduino.h>            // assumes Arduino IDE v1.0 or greater
#include <avr/sleep.h>
#include <avr/wdt.h>
#include <avr/power.h>

#ifdef __AVR_ATmega1284P__
  #define LED           15 // Moteino MEGAs have LEDs on D15
  #define FLASH_SS      23 // and FLASH SS on D23
#else
  #define LED           9 // Moteinos have LEDs on D9
  #define FLASH_SS      8 // and FLASH SS on D8
#endif

#if defined(WITH_RFM69) || defined(WITH_SPIFLASH)
  #include <SPI.h>                //comes with Arduino IDE (www.arduino.cc)
  #if defined(WITH_RFM69)
    #include <RFM69.h>            //get it here: https://www.github.com/lowpowerlab/rfm69
    RFM69 radio;
    #define NETWORKID 100
    #define NODEID 123
    #define FREQUENCY RF69_915MHZ
  #endif
  #if defined(WITH_SPIFLASH)
    #include <SPIFlash.h>         //get it here: https://www.github.com/lowpowerlab/spiflash
    SPIFlash flash(FLASH_SS, 0xEF30); //EF30 for 4mbit  Windbond chip (W25X40CL)
  #endif
#endif

//watchdog interrupt
ISR (WDT_vect) {
  wdt_disable();
}

void setup () {
#ifdef WITH_RFM69
  radio.initialize(FREQUENCY,NODEID,NETWORKID);
  radio.sleep();
#endif

#ifdef WITH_SPIFLASH
  if (flash.initialize())
    flash.sleep();
#endif

//  //optional blink to know radio/flash sleeping went OK
 pinMode(LED, OUTPUT);
 digitalWrite(LED, HIGH);
 delay(80);
 digitalWrite(LED, LOW);
//  delay(50);
//  digitalWrite(LED, HIGH);
//  delay(50);
//  digitalWrite(LED, LOW);

  for (uint8_t i=0; i<=A5; i++)
  {
#ifdef WITH_RFM69
    if (i == RF69_SPI_CS) continue;
#endif
#ifdef WITH_SPIFLASH
    if (i == FLASH_SS) continue;
#endif
    pinMode(i, OUTPUT);
    digitalWrite(i, LOW);
  }

  power_timer1_disable();
  power_timer2_disable();
  power_twi_disable();
}

void loop ()
{
  // disable ADC
  ADCSRA = 0; 
  // clear various "reset" flags
  MCUSR = 0;
  // allow changes, disable reset
  WDTCSR = bit (WDCE) | bit (WDE);
  //set interrupt mode and an interval
  WDTCSR = bit (WDIE) | bit (WDP3) | bit (WDP0); //set WDIE, and 8 seconds delay
  wdt_reset(); //pat the dog...
 
  set_sleep_mode (SLEEP_MODE_PWR_DOWN);
  noInterrupts(); // timed sequence follows 
  sleep_enable();

  // turn off brown-out enable in software
  // BODS must be set to one and BODSE must be set to zero within four clock cycles
  MCUCR = bit (BODS) | bit (BODSE);
  // The BODS bit is automatically cleared after three clock cycles
  MCUCR = bit (BODS);
  interrupts();
  sleep_cpu();

  //cancel sleep as a precaution
  sleep_disable();
}


Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Merging Exported data into one CSV file
« Reply #12 on: February 23, 2017, 10:42:57 PM »
Should work, try it with nothing attached but the GND and VCC (3.5v+).

LukaQ

  • Sr. Member
  • ****
  • Posts: 302
  • Country: si
Re: Merging Exported data into one CSV file
« Reply #13 on: February 24, 2017, 02:22:17 PM »
5mA is far too high. I had far less with my, same meter...not in 6uA range, but if I remember right, average in 60-70uA. Is that constant 5ma?

EnMon

  • NewMember
  • *
  • Posts: 32
  • Country: us
Re: Merging Exported data into one CSV file
« Reply #14 on: February 25, 2017, 03:17:30 PM »
I was able to  get as low as 6.9uA with the DeepSleep_UsingLowPowerLibrary, but sometimes, the current consumption gets stuck at 0.33mA and doesn't go lower than that. Still trying to figure out what is going on.