Data packet size limit - Can it be increased?

[luisr320]

Hi!
I'm trying to pass a full NMEA GPS sentence from a node to a gateway and after long hours figuring how to do it, I stumble now on the limit of 61 byte for the Data packet size. Is there a way around this? The library documentation refers that is to make the Moteino compatible with AES hardware encryption. Can I cancel the encryption and increase the data packet size? Or maybe someone can suggest a way to break the NMEA message into 61 bytes parts and then assemble them back on the other end as a single sentence?

Thank you
Luis

[Spexx]

Sorry I don't know if you can increase the size, but can't you split the data in two or more packages and on the receiving end you merge them together?

[luisr320]

Well, that was part of the question. How to do it (I'm a copy/paste programmer). I'll just google some way of doing it. Thanks

[Felix]

Luis,
According to the datasheet page 51, packet mode is limited to 255 bytes payload length when AES is not used. I have hardcoded the limit to 61 bytes (plus header bytes=65 bytes) in my RFM69 library, because keeping a large buffer of 255 bytes seems too much for a 2k RAM memory of the atmega328.
I guess you could disable AES and conditionally use a 255 byte buffer when AES is disabled, and also make the changes to support 255 byte packets.
So, I think it's possible, but you may find that it could be easier to use structs or other techniques to compress long strings into a small efficient packet, or split it like I do for wireless programming, where I transmit a whole hex file over the air in up to 1000 packets or even more.

[TomWS]

Well, that was part of the question. How to do it (I'm a copy/paste programmer). I'll just google some way of doing it. Thanks

Luis, if you look at the Struct_send and Struct_receive examples in the RFM69 library you'll see how to declare structures that contain data other than simple character strings.  And, even though the NMEA payload is a simple string, once you break it up you'll need to include information on how to piece it back together again.

The technique I use it to always have a header in the structure that contains a single byte length field and a single byte 'type' field.  Over the air, the length field is redundant (since len is part of the returned packet), but it's useful to keep with the data since you'll need to know how long each block is when you piece the blocks back together.  So, in your case, you could define a couple of types, depending on how many blocks you might need:

Code: [Select]

enum NMEA_BLOCK_TYPE
{
  NMEA_FIRST_BLOCK,
  NMEA_INTERMEDIATE_BLOCK,
  NMEA_LAST_BLOCK
};

// and define the transmitted block as:
struct NMEA_BLOCK
{
   uint8_t
     len,
     type,
     data[58];    // note that you don't have to fill or send ALL of these if your block is shorter than 58 bytes, 'len' tells the receiver how many bytes to read.
} block;

When you receive a packet, you start a new record when the type is NMEA_FIRST_BLOCK, append subsequent NMEA_INTERMEDIATE_BLOCKs to the record, and, finally, terminate the record when you receive the NMEA_LAST_BLOCK.

On the send side you're just taking chunks of 61-headersize length blocks and sending them in these structured packets.  Easy peasy!

Tom

[luisr320]

I can see that all this is straight forward for the Gurus. Unfortunately I can't make it work.

Felix, I can find the #define RF69_MAX_DATA_LEN      61 in the RFM69.h
Changing it to 75 makes the radio not sending data to the receiver. What else must I change to make it work?

Tom, do you have a working example I can use?

Thank you

[TomWS]

I can see that all this is straight forward for the Gurus. Unfortunately I can't make it work.

Felix, I can find the #define RF69_MAX_DATA_LEN      61 in the RFM69.h
Changing it to 75 makes the radio not sending data to the receiver. What else must I change to make it work?

I recommend that you don't mess with ANY of this!  It is far more complicated than breaking up the sentence into pieces.

Tom, do you have a working example I can use?

How long is the sentence?  Can you post the code that reads the sentence and give me the length of the packet you want to send?

Tom

[Felix]

I can see that all this is straight forward for the Gurus. Unfortunately I can't make it work.
Felix, I can find the #define RF69_MAX_DATA_LEN      61 in the RFM69.h
Changing it to 75 makes the radio not sending data to the receiver. What else must I change to make it work?

Unfortunately it's not really straight forward, and certainly not as easy as changing 61 to 75, IOW just one or two lines of code.
I haven't done this exercise so I could only mentally estimate of what it will take.
FWIW It may be something I will consider for a future change, to allow 255 byte packets for non encrypted transmissions. That would force me to think through more aspects (pros/cons) of having such a work mode in the library.

[luisr320]

How long is the sentence?  Can you post the code that reads the sentence and give me the length of the packet you want to send?

Ok, no more messing with the library.

So here is a NMEA message generated on the transmitter side by the GPS, the is correctly displayed on the serial monitor:

$GPRMC,224553.000,A,1952.6095,S,04355.7367,W,0.34,199.27,160216,,,A*60

All the messages that I want to send start with "$GPRMC," . So that one could be appended on the receiver side in front of the received message before it is Serial.print trough the serial port. That would make the message be 63 byte. But that seems to still be too much.

On the other side I get this:

$GPRMC,230138.000,A,1952.6095,S,04355.7367,W,0.34,199.27,160??

At the end it is truncated and some weird characters added. There are 10 bytes missing.

So I guess 80 bytes would be the maximum ever. If I could split the message in two, that would be enough.

Also, I'm sending radio data with two different payload to two different nodes numbers, but they are the same receiver. The idea is to allow me to use a full 64 bytes of data packet size to try to send the NMEA message. This is required because I'm already sending a data package with the parsed data like altitude and airspeed on a different data package to the same receiver.

Why is this required? Well, I want to make a log of position waypoints on the transmitter SPIFLASH. I can't do it on the receiver side because it is already maxed out on memory size, due to the LCD, menus, buttons, etc, so adding the extra code to allow datalogging is not possible. And since the transmitter was very light on the code, I decided to make the log there.

I'm adding the code I'm using on both the transmitter and the receiver.

This will be, I hope, all part of my next Moteino project, a do all Telemetry System for R/C airplanes/drones, that will allow a fast recovery in case of a fly away. It will also allow the viewing of the model position on a local laptop in real time on Google Maps and to download the position log from the transmitter if no laptop is available.

NMEA message content:

$GPRMC,220516,A,5133.82,N,00042.24,W,173.8,231.8,130694,004.2,W*70

12 coma separated parts:

      1   220516     Time Stamp
      2   A          validity - A-ok, V-invalid
      3   5133.82    current Latitude
      4   N          North/South
      5   00042.24   current Longitude
      6   W          East/West
      7   173.8      Speed in knots
      8   231.8      True course
      9   130694     Date Stamp
      10  004.2      Variation
      11  W          East/West
      12  *70        checksum


eg4. $GPRMC,hhmmss.ss,A,llll.ll,a,yyyyy.yy,a,x.x,x.x,ddmmyy,x.x,a*hh
1    = UTC of position fix
2    = Data status (V=navigation receiver warning)
3    = Latitude of fix
4    = N or S
5    = Longitude of fix
6    = E or W
7    = Speed over ground in knots
8    = Track made good in degrees True
9    = UT date
10   = Magnetic variation degrees (Easterly var. subtracts from true course)
11   = E or W
12   = Checksum

Sorry for the long post.

Thank you for your interest.

[TomWS]

<snip>
Also, I'm sending radio data with two different payload to two different nodes numbers, but they are the same receiver. The idea is to allow me to use a full 64 bytes of data packet size to try to send the NMEA message. This is required because I'm already sending a data package with the parsed data like altitude and airspeed on a different data package to the same receiver.

Why is this required? Well, I want to make a log of position waypoints on the transmitter SPIFLASH. I can't do it on the receiver side because it is already maxed out on memory size, due to the LCD, menus, buttons, etc, so adding the extra code to allow datalogging is not possible. And since the transmitter was very light on the code, I decided to make the log there.
<snip>

I'll look at your code and send you a snippet that will break into blocks and then restore into a single NMEA sentence, thanks for the info.  However, the part I've quoted above has me TOTALLY confused.   I don't know if you're talking about one GPS receiver node that is sending to two different Motes or two GPS receivers talking to one other Mote, or, uh, I don't know what.  Maybe a picture (not a video thank you  would help...

Tom

[damonb]

Just at a superficial glance, it looks like the gps message could be mapped to a struct that would easily fit into 60 bytes. Sending all those numbers as text is the main inefficiency concern. Getting each gps message into a single radio packet will save a lot of code, and avoid the need for error handling for lost packets.

[luisr320]

I'll look at your code and send you a snippet that will break into blocks and then restore into a single NMEA sentence, thanks for the info.  However, the part I've quoted above has me TOTALLY confused.   I don't know if you're talking about one GPS receiver node that is sending to two different Motes or two GPS receivers talking to one other Mote, or, uh, I don't know what.  Maybe a picture (not a video thank you  would help...

Tom

Since the payload is limited to 64 bytes and I'm already sending a payload filled by a data struct that include some info like altitude, airspeed, position, etc, there is no more space to send the NMEA msg.

So my idea was to send two different messages, each identified with a different node ID, but to the same node, the receiver end. These are the messages sent, one right after the other:

    radio.sendWithRetry(GATEWAYID, (const void*)(&Data), sizeof(Data), 2, ACK_TIME);
    radio.sendWithRetry(NMEAID, (const void*)(&nmea), sizeof(nmea), 2, ACK_TIME);

On the receiver end, after the  if (radio.receiveDone()), I check if (radio.TARGETID == GATEWAYID), that is NODE1, or if (radio.TARGETID == NMEAID), which is NODE3, thus allowing me to use a new full payload for both data.

I'm sure this is not a very efficient way to do it. I guess I could send just the NMEA message and try to parse its contents on the receiver side to display on the LCD or maybe just parse the NMEA message on the transmitter side and just sent the data on a struct, without the comas, to be re-assembled on the receiver side back as an NMEA sentence.

What I don't understand is the suggestion that using a struct will be enough to somehow compress the data. How is that possible?

[TomWS]

Ok, Luis, back at ya!

I've added a new file, myNMEA.h, which is referenced by both the node and gateway programs ensuring that their structure definitions, etc, match - this is an ABSOLUTE requirement!

I've removed the unnecessary second Target ID so you might be able to switch off promiscuous mode.

These SHOULD work but I obviously couldn't test and wasn't willing to install the libraries to make sure it built...

Good luck,
Tom

[TD22057]

luisr320,
Don't use separate node ID's to identify your message - use a message ID.  Set the first byte of the struct/message as an identifier and test that when the message arrives.  Something like (pseudo-code) below.  Just make sure you initialize your message instance with the correct ID when you fill in the data.

Code: [Select]

enum MessageId {
   MSG_ALTITUDE = 1,
   MSG_GPS = 2,
};
struct AltMsg {
   uint8_t id; // == MSG_ALTITUDE
   int16_t altitude;
   int16_t airspeed;
   //...
};
struct GpsMsg {
   uint8_t id; // == MSG_GPS
   char msg[24];
};

if ( radio.receiveDone() ) {
   if ( (MessageId)radio.DATA[0] == MSG_ALTITUDE ) {
      AltMsg* msg = (AltMsg*)radio.DATA;
      // process altitude message struct
   }
   else if ( (MessageId)radio.DATA[0] == MSG_GPS ) {
      GpsMsg* msg = (GpsMsg*)radio.DATA;
      // process gps message struct
   }
 }

The reason a struct "compresses" string data is just that you can store a number like 123456 (6 character bytes) in a 4 byte integer to save 2 bytes.  And you don't need delimiters so every comma is a saved byte.  And floats (+/- 3.4028235E+38) are 4 bytes as well.

[luisr320]

Wow. Epiphany all the way!!! 

Thank you all for showing me the way to do this.
I can see now that trying to send a string of characters trough a struct, with 1 byte for each char or a 1, 2 or 4 bytes number makes all the difference.

Back on track.

I'll post a new project soon with what I'm doing.
Some photos:

The Adafruit GPS module piggyback with a Moteino. The "Transmitter".


The Moteino with a Nokia 5110 LCD. The "Receiver".


Flight testing, after 7280Km. Large number is the maximum achieved in flight. 1007Km/H  . The transmitter recorded a position Fix on departure and is calculating how far it is from that point.


Passing the NMEA message to Google Earth. I was on the 13th floor, in Belo Horizonte, Brazil.