Author Topic: Logging to CSV file  (Read 1916 times)

tcsaba

  • NewMember
  • *
  • Posts: 6
  • Country: hu
Logging to CSV file
« on: November 03, 2020, 07:04:37 AM »
I am using CR R1 version for almost 2 years with the OLED. Now I need logging and I have upgraded to R3. It works well with Arduino IDE Serial Monitor but I am not able to use with my favorite COM port terminal programs like Hterm or Terminal v1.93b. Those terminals have built in logger, so the job would be done. I hope someone has a good and simple solution for this problem.
Questions:
1. Can be connected the COMxy USB link to an UART terminal? Or what interface could help?
2. I would need a basic timestamp in the log. The most simple idea a relative usec or msec stamp would still be fine. Can somebody suggest a better solution?

Thanks, Csaba
« Last Edit: November 13, 2020, 11:08:56 AM by Felix »

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Logging to csv file
« Reply #1 on: November 03, 2020, 03:42:57 PM »
1) Terminal programs should work with any serial connection regardless of its type. By the way, these days there is no real good way to connect a UART directly to a PC. It's always going through a FTDI Adapter chip, or its a serial over USB like the CR does it. So yes any terminal program should open the COMx from the CurrentRanger.
2) It would be nice if your logger would do the timestamping. Clogging the serial output with timestamps would be a big performance hit for the CR output.
By the way there is a program released here that does a bit of that along with a GUI: https://github.com/MGX3D/CurrentViewer


tcsaba

  • NewMember
  • *
  • Posts: 6
  • Country: hu
Re: Logging to csv file
« Reply #2 on: November 04, 2020, 07:59:10 AM »
1, I usually work with FTDI-like UART/USB interface. I use those two terminal apps above, but those two doesn't work with CR.  I tried Putty, that works with CR. Still I do not understand the situation.
The BT-UART should will work with UART/USB converter if I do the firmware adjustment. But probably USB is the fastest interface.  Do you have the approximate SPS figures for the average/faster/slower sampling setup? As I see your suggestion of MGX3D viewer seems the best for first step to try.
2, I will try!

Thanks!

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Logging to csv file
« Reply #3 on: November 04, 2020, 07:51:05 PM »
The CR emulates the serial connection a little different, maybe that's why your preferred programs dont work, I haven't used those programs so I don't have any advice there.
Putty works. Not sure what else to suggest, maybe write your own format in the CR firmware and work with that.
A nice GUI that is customizable would be great, to control logging output etc, then the CR would not need to keep time or timestamps, it doesn't even make sense, or it would require an RTC and also it would greatly increase the serial output bytes.

I talked to the creator of CurrentViewer about how we can increase the SPS and improve that GUI. An ideal SPS target would be 1kSPS perhaps achievable with some new techniques of sampling and streaming the data. Unfortunately I think he did not wrote this for Windows. And this is just a side project whenever he has time (and me as well!).

tcsaba

  • NewMember
  • *
  • Posts: 6
  • Country: hu
Re: Logging to csv file
« Reply #4 on: November 06, 2020, 02:41:40 PM »
The CR emulates the serial connection a little different, maybe that's why your preferred programs dont work, I haven't used those programs so I don't have any advice there.
I also have heard it, but I know nothing about the details. I suppose there is an interface what can be used.

Quote
I talked to the creator of CurrentViewer about how we can increase the SPS and improve that GUI. An ideal SPS target would be 1kSPS perhaps achievable with some new techniques of sampling and streaming the data.

I think transfer the raw measuring can solve the higher speed or a different ADC. The result is 12bits, the additonal 4 bits can interpret the actual range. 2 bytes per sample instead of ASCII records. But that needs a dedicated logger on the PC side.

Quote
Unfortunately I think he did not wrote this for Windows. And this is just a side project whenever he has time (and me as well!).
Do you mean he wrote for Linux, because OS still not available? I am still struggling with the Win version. Could I expect a better result on Linux?
« Last Edit: November 06, 2020, 02:54:40 PM by tcsaba »

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Logging to csv file
« Reply #5 on: November 09, 2020, 08:57:21 AM »
A low SPS could be a bad cable. I would expect higher than 100sps even on older firmware.
But also, this could depend how early that firmware really is.
I can say that we can only really offer support for the latest available firmware since incremental improvements and features are added, including optimizations to increase sampling speed.
The CurrentViewer is a 3rd party tool and was written in python that can be run on any OS, but was only tested on linux as far as I understand, I tested it on W10 and some bugs I reported are being worked on by Marius (MGX3D on Github).

tcsaba

  • NewMember
  • *
  • Posts: 6
  • Country: hu
Re: Logging to csv file
« Reply #6 on: November 09, 2020, 01:25:49 PM »
Since I use ONLY good quality USB cables, there was no USB transmit problem for me recently.
But to be sure I have changed the cables and the optoisolator too, no change in SPS.
The CR firmware was flashed 1 week ago.

The SPS of Putty logging was the same when putty baudrate was to set to 115000.
So this looks to be built in the system, maybe my logging PC is too weak: Win10 (32bit) & Intel P8700@2.5GHz, 2GB RAM.
I will try on my i7-8750H 2.2GHz, 16GB RAM, maybe will be faster.

I am in contact with Marius. I use the logging PC for CV and decreasing the buffer size from 10000 to 2500 it is working with Python 3.8.6, but not 100% tested. He is working on the CSV logger feature.

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Logging to csv file
« Reply #7 on: November 09, 2020, 05:34:49 PM »
He just made a major upgrade, that includes CSV logging but I haven't tried it yet.

tcsaba

  • NewMember
  • *
  • Posts: 6
  • Country: hu
Re: Logging to csv file
« Reply #8 on: November 11, 2020, 03:11:56 PM »
I have tried the new CV, it is much stable and flexible than previous version.
Logging to CSV file has been included. ;D

I am struggling with corrupted data from CS. Some can be filtered by CV but some not, so I get a corrupted CSV at the end.
I have ordered from Welectron 2 pcs of CS, when arrived I will conduct in more testing, what can be the origin.

What kind USB driver do you recommend for Win7 to connect CS.
I have downloaded a SAMD21 USB driver from Sparkfun, it has installed fine, but no COM port created!

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Logging to csv file
« Reply #9 on: November 11, 2020, 03:50:06 PM »
I can't really provide continued support Win7, it's an obsolete OS. Even MS stopped support for it. I would recommend you upgrade to W10.
That being said, there is a W7 driver in the Moteino SAMD boards package, for the uf2 and for the old bootloader, just point to that directory and W7 will find it by VID/PID.
In the main firmware the serial port should be automatically generated by W7 since its a generic USB CDC serial port.

tcsaba

  • NewMember
  • *
  • Posts: 6
  • Country: hu
Re: Logging to csv file
« Reply #10 on: November 12, 2020, 12:46:44 PM »
W7 > W10: We are all directed in the same ghetto trellis. But please no politics here.

I have installed your Board Manager in Arduino, I should look for the driver in the Boards directory?

See the first CSV based Excel graph based on  Marius (MGX3D Github) new version of CV.
This Post has been reached the goal!

 :) ;) :D ;D