Author Topic: Model Airplane Pylon Judging System  (Read 51451 times)

delateurj

  • NewMember
  • *
  • Posts: 14
Model Airplane Pylon Judging System
« on: December 18, 2014, 08:22:42 PM »
Hi,

I posted awhile back in a project for a system used to run model airplane pylon races.  Here is link to that thread that describes basic project https://lowpowerlab.com/forum/index.php/topic,749.0.html and the attached is a diagram of the physical layout of the nodes  (attachment was too large so posting without until I can reduce size).

I started with a basic test of 5 nodes (2 Pylon Judge Ground, 1 Pylon Judge Repeater, 1 Pylon 1 Light Repeater and 1 Pylon 1 Light Ground).  Using RFM69HW's.  Every 2.5 to 3 seconds (actual time is randomly chosen 10 ms between 2.5 and 3 secs.) each Pylon 1 Judge Ground sends a 1 byte struct using sendWithRetry using default settings.  This is sent from Pylon Judge Repeater to the Pylon 1 Light Repeater to the Pylon 1 Light Ground unit each using sendWithRetry and each node ack'ing the response.  In this set up I get about a .5 percent failure rate just in the hop between the Pylon Judge Ground units and Pylon Judge Repeater.  This is all done in my "lab" so the nodes are all within 3 feet of each other.

So how surprised.  I am not terribly surprised as the basic listen before talking is not going to in theory result in a zero collision rate.  I was hoping it would be lower but since I am half doing this for the hobby aspect I am not discouraged I will need to improve or work around.  I see two paths:

1. Adjust retries times, number of retries, randomness of retries, threshold of listen before talk and bitrates to see if I can substantially increase reliability. (looking for 99.99% of each individual hops being successful.

2. Implement a basic token ring strategy probably with some sub rings.  I've got enough time between messages and the topology is static (i.e. the nodes will not be dynamically joining and leaving the network).  So this is my primary path since it guarantees no collisions.

Thoughts?

Thanks,
Joe.
« Last Edit: December 18, 2014, 08:34:47 PM by delateurj »

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Model Airplane Pylon Judging System
« Reply #1 on: December 18, 2014, 08:48:40 PM »
Since any one node cannot control the behavior of other nodes, the only way to achieve reliability (assuming nodes are within range and the medium (air) is free of interference and other signal degrading factors) is to repeat the transmission. This should take into account that if there is a 1 receiver node that receives/processes all data from all N nodes, and that could keep it busy or in a non-receive state.

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Model Airplane Pylon Judging System
« Reply #2 on: December 18, 2014, 08:58:27 PM »
...  In this set up I get about a .5 percent failure rate just in the hop between the Pylon Judge Ground units and Pylon Judge Repeater.  This is all done in my "lab" so the nodes are all within 3 feet of each other.

So how surprised.  ...
You got this error rate with sendWithRetry (with three retries??)?  What frequency are you using and what other traffic might there be?

Honestly, I haven't checked error rate (in my case if I miss a sample now and then, eh, who cares?) but, I'd think in a relatively 'clean' environment with these radios (are you using encryption?) they're fairly robust and you certainly don't have a challenging environment (other than possibly overpowering - and distorting- your receivers by being so close!).

I'm as surprised as you and curious about the cause.

Tom

delateurj

  • NewMember
  • *
  • Posts: 14
Re: Model Airplane Pylon Judging System
« Reply #3 on: December 18, 2014, 10:49:56 PM »
Hi Tom,

Was with three retries, 900 mhz (or whatever the exact default is for the 900mhz).  Went at least ten thousands successfully when there is only one pylon judge talking to the repeater.  When you add a second one then there is the low failure rate which I assume is collisions happening as the two judges both talk to the repeater at nearly the same time occasionally and of course there are acks flying around as well.  Felix, are you surprised there wouldn't be at least some since contending for same medium.  I am still learning but my initial reading up on clear channel check protocols are still susceptible to some error rate due to collision.  Retries help but best approaches require to randomize the time until retry (again, feel free to call bs, I am still learning). 

Unfortunately a failed transmission is a big deal as the pilot is counting on the light to come on saying they made it to the pylon.  If that message fails to get through and they don't get a light they think they are flying too short.

Not using encryption.   I am concerned I am distorting by being too close.  When it stops raining I'll do a test outside.

Felix with regards to "Since any node cannot control the behavior of the other nodes...."  This is actually what a token ring does.  A node is not allowed to speak until it gets a message with the "speaking token".  It then speaks if it has a message and then passes the token on to next node in ring (various approaches in passing the token).  (again still learning) but assuming all nodes are functioning correctly assures clear channel to communicate downside can be complicated to implement if network complicated (not my case).  Creates latency while nodes wait for token (need to do some analysis but may be okay) and final downside is if a node fails the whole network can go down if the token is "lost" with the failed node and you don't have some recovery method (that can get complicated).  Latter one is okay as every node I have is critical so if one goes down, I kind of want the whole thing to collapse so I know that heat needs to be re-run.   (clarification....even with token ring, I would use retries).

Thanks so much for the responses, sorry if I rambled above or if I am incoherent or if I said something stupid.
« Last Edit: December 18, 2014, 11:08:07 PM by delateurj »

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Model Airplane Pylon Judging System
« Reply #4 on: December 19, 2014, 07:49:14 AM »
Thanks, I didn't realize that you had multiple radios running during that test.  One thing you might try (I've thought about adding this code to the RFM69 library) is to alter the retry time based on a different value for each node.  You might use the node ID or part of the MAC as a multiplier, so that, when two devices do collide, their respective retries probably won't collide as well.

Tom

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Model Airplane Pylon Judging System
« Reply #5 on: December 19, 2014, 07:58:03 AM »
...
Not using encryption.   I am concerned I am distorting by being too close.  When it stops raining I'll do a test outside.

Felix with regards to "Since any node cannot control the behavior of the other nodes...."  This is actually what a token ring does.  A node is not allowed to speak until it gets a message with the "speaking token".  It then speaks if it has a message and then passes the token on to next node in ring (various approaches in passing the token).  (again still learning) but assuming all nodes are functioning correctly assures clear channel to communicate downside can be complicated to implement if network complicated (not my case).  Creates latency while nodes wait for token (need to do some analysis but may be okay) and final downside is if a node fails the whole network can go down if the token is "lost" with the failed node and you don't have some recovery method (that can get complicated).  Latter one is okay as every node I have is critical so if one goes down, I kind of want the whole thing to collapse so I know that heat needs to be re-run.   (clarification....even with token ring, I would use retries).

Thanks so much for the responses, sorry if I rambled above or if I am incoherent or if I said something stupid.
I replied too quickly in my last reply.  I don't think you have to "move outside" to get sufficient separation, even another room will be sufficient to keep the radios within their AGC range.

Yeah, I'm quite familiar with Token Ring having worked for 'that company' in my time...  I agree that it should be effective at eliminating collisions, assuming that the Token is passed with retry (and you're not getting out of sync due to timeouts).  I'd lean toward distortion as probable cause (unless there are any cordless phones or other 900 MHz interference sources nearby) given this implementation.

No rambling detected and it's perfectly allowed from my perspective (probably perfectly implemented too!).

Tom

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Model Airplane Pylon Judging System
« Reply #6 on: December 19, 2014, 03:38:30 PM »
I am still learning but my initial reading up on clear channel check protocols are still susceptible to some error rate due to collision.  Retries help but best approaches require to randomize the time until retry (again, feel free to call bs, I am still learning). 
Sounds like you're doing the right things. Learning is fun when you like what you're learning.

Felix with regards to "Since any node cannot control the behavior of the other nodes...."  This is actually what a token ring does.  A node is not allowed to speak until it gets a message with the "speaking token".  It then speaks if it has a message and then passes the token on to next node in ring (various approaches in passing the token).  (again still learning) but assuming all nodes are functioning correctly assures clear channel to communicate downside can be complicated to implement if network complicated (not my case).  Creates latency while nodes wait for token (need to do some analysis but may be okay) and final downside is if a node fails the whole network can go down if the token is "lost" with the failed node and you don't have some recovery method (that can get complicated).  Latter one is okay as every node I have is critical so if one goes down, I kind of want the whole thing to collapse so I know that heat needs to be re-run.   (clarification....even with token ring, I would use retries).

Thanks so much for the responses, sorry if I rambled above or if I am incoherent or if I said something stupid.

There are no stupid mistakes, just happy accidents :)
I was not familiar with token ring, but it sounds like it would introduce a lot of latency because of the extra messages. Each packet, depending on payload, will take somewhere around 5-30ms to send (transmitter to modulate out and receiver to modulate in). I am not really sure how reliable and fast the RSSI reading is. Currently the "listen-before-speak" approach is implemented by reading the RSSI which is the only indicator of "energy in the channel". I am not a radio guru so I might be doing something wrong, or setting radio parameters incorrectly before doing this reading, and maybe this could be improved. Perhaps sometimes the readings are false or the collision happens right after the reading but before sending the packet. A lot of moving targets here and points of failure. Usually though, a few nodes in the same network are not a problem and with enough retries the packets should get through close to 100%.

delateurj

  • NewMember
  • *
  • Posts: 14
Re: Model Airplane Pylon Judging System
« Reply #7 on: December 25, 2014, 05:07:47 PM »
Update...been focusing on getting some basic carrier boards done and writing code leveraging flash to perform and record results of tests as I progress.

Here is a summary of the boards I have created which are kind of general purpose so may be of use to someone else's project (if so, Merry Christmas).  I am no EE so these are pretty basic just using eagle light and auto router mostly.  I have done basic functional test on each one. The documentation and picture of each of the boards can be found at http://www.rcpylonracing.com/page4/page10/page10.html

First is the board that will be used for the judge ground, lap counter and starter ground station.  Its called MoteinoPylon2and3Judge but will be used for multiple stations that just need a bunch of inputs for buttons and switches.

The board has the following functions:
- Footprint for Moteino R4.
- Footprint for soldering on a single cell lipo charger I like to use from adafruit (http://www.adafruit.com/product/1905)  (don't need to use, you could just solder your power input of choice to the BAT/GND pads).
- Set up for a power switch to cut off power.
- Voltage divider into A0 to allow monitoring of voltage of power source
- Brings out D3-D7 and A1-A3 for digital inputs using internal pullups for 8 potential inputs via switches and buttons.
- A six pin header to match the pins of the adafruit i2c OLED 128x32 display (http://www.adafruit.com/products/931).  I find this a versatile display that can display a lot of information in a small space and is readable in daylight.  This could be used for some other i2c if you wire into appropriate pads from your i2c device.  I am going to use it for an RTC instead of display on some.

.... gotta go check on the Christmas Turkey, so I'll complete this post later...Merry Christmas.

Turkey is doing fine....

Pylon 1 Light Ground Unit:
- Also has Moteino R4 footprint, power switch, voltage divider, lipo charger and i2c OLED foot print as above.
- Has 8 channel Darlington Driver ULN2803 https://www.sparkfun.com/products/312  I use this to turn on any one of the four lights at pylon 1 by completing the circuit to ground.  I am ganging up two channels for each light because they each draw about 400ma which is at the limit of individual channel capacity and since I only needed 4 and this allow for ganging up, I went that route.
- Voltage divider into A1 to monitor voltage of source that is powering the lights.  The lights in my case our these which are visible in broad daylight from 500 feet.  They are bus light turn signals.https://www.superbrightleds.com/moreinfo/truck-round-lamps/round-led-bus-tail-lights-7in-led-stop-turn-tail-light-with-61-leds/566/
- Footprint for tmp36 temperature sensor https://www.sparkfun.com/products/10988.  My lights should only have a duty cycle of about 15% but thought it would be good to monitor temperature of Darlington to make sure I wasn't overheating it.  So I threw this on there.  Foot print is actually incorrect so you can't push it all the way to board which is okay for my purposes because I plan to bend the leds so its lays on top of the Darlington.
- I beefed up the tracks for the source of the ground line for the lights and the outputs of the Darlington since they'll be carrying around 400 ma each.

Finally the Starter Display Arduino Shield:
- I am using the following DMD from Freetronics (http://www.freetronics.com.au/products/dot-matrix-display-32x16-red#.VJyS0MAAA).  In the basic Arduino/Xbee system I made last year, it worked really well.  Very bright and versatile.  It is all set to work with Arduino Uno. including a special connector that plugs into Arduino Uno.  I messed around a little with trying to get it to work with Moteino but not set up to be 3.3v friendly and only officially support Uno (have a beta for others).  So I decided to create a Arduino Moteino shield.  The Moteino and Arduino talk to each other via I2C using a level converter (https://www.sparkfun.com/products/11955).  This offloads the updating and refreshing of the display to Arduino. 
- Has the voltage divider and I2C OLED header as above.
- Note I started with an Arduino shield eagle part that is the old style that doesn't have the additional IOREF, SCL, SDA pins brought out that the new or the ISCP 6 pin header that the new footprint does, but easily modified to add if you want those.

- Joe.


« Last Edit: December 25, 2014, 06:12:22 PM by delateurj »

delateurj

  • NewMember
  • *
  • Posts: 14
Re: Model Airplane Pylon Judging System
« Reply #8 on: December 29, 2014, 08:43:03 PM »
Update:  Primarily on message reliability....

Following up on my first tests I went back over my test code and found a bug.  I was sending test messages 10 times faster than than I thought and that made the rate of messages almost constant given how long each message took to transmit and ack.....

After fixing that, I set up test with 3 pylon judge nodes talking to 1 judge repeater that repeated message to 1 pylon 1 repeater that repeated the message to 1 pylon 1 ground unit.  Each pylon judge node sends a message every 2.5 to 3.5 seconds destined for pylon 1 light ground.

With default settings got 99.7% success rate on the first hop and 99.98 on second hop and 100% on third hop (ran for 20,000 tests).

I then increased the number of retries to 6 and randomized the retry time to be between 25 and 30 ms  (my successful messages take about 10 ms from time of starting to send to receiving an ack back).

With these default settings achieved 99.95% success on first hop (second and third hops remained similar).  Getting very close to my goal of 99.99%

These tests were done with nodes within inches of each other so time for an open air test at full expected distance test. 

Outside I setup with 3 judge ground units near ground, judge repeater on 10 foot pole, 300 ft away is pylon 1 light repeater on 10 foot pole with pylon 1 light ground unit near ground below it.  Was able to send a total of 1000 messages with no failures (about 20 retries in total (modified the rfm69 library to keep track of retries)).  Got cold so discontinued test but very encouraging.  Need to run for 20,000 to be able compare close quarter stats.

Next steps: 
Order some more Moteinos  :D
Do open air test for 20,000 messages.
Make some minor mods to the boards above.  Some have reset pin of i2c for OLED on D0  (that was a bad idea since used for serial comm....oops) and order some more to finish rest of nodes.
Make a board based on Moteino Mega as I think the starter command and control node will need more program space and perhaps more I/0.  Already using up 21K of program space on judge nodes just mainly withe libraries I am using (rfm69, wire, eeprom, spi, spi_flash and OLED libraries).

- Joe.

« Last Edit: December 29, 2014, 08:45:56 PM by delateurj »

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Model Airplane Pylon Judging System
« Reply #9 on: December 29, 2014, 09:19:57 PM »
Update:  Primarily on message reliability....

Following up on my first tests I went back over my test code and found a bug.  I was sending test messages 10 times faster than than I thought and that made the rate of messages almost constant given how long each message took to transmit and ack.....

After fixing that, I set up test with 3 pylon judge nodes talking to 1 judge repeater that repeated message to 1 pylon 1 repeater that repeated the message to 1 pylon 1 ground unit.  Each pylon judge node sends a message every 2.5 to 3.5 seconds destined for pylon 1 light ground.

With default settings got 99.7% success rate on the first hop and 99.98 on second hop and 100% on third hop (ran for 20,000 tests).

I then increased the number of retries to 6 and randomized the retry time to be between 25 and 30 ms  (my successful messages take about 10 ms from time of starting to send to receiving an ack back).

With these default settings achieved 99.95% success on first hop (second and third hops remained similar).  Getting very close to my goal of 99.99%

These tests were done with nodes within inches of each other so time for an open air test at full expected distance test. 
I'd like to know if randomizing the retry (or more specifically, making it deterministic based on ID or something unique) really helps.  I would think it would.  If you think about it, if a collision takes place and ALL nodes retry in the SAME interval, wouldn't a collision be probable?  I think something that distinctly altering the retry value would have the most, uh, 'value'...

UPDATE: the other thing I think you should try is to separate your devices a bit.  I do think that getting the RSSI value into the high 40s/low 50s is ideal for these radios.  RF errors occur for a variety of reasons - overpowering is one of them...

<...snip>
Next steps: 
Order some more Moteinos  :D
Oh yes, do that!  It will make Felix SOOOOOO happy!   :D

BTW, learn to use F() macro for your debug message strings...  You'll get back a LOT of dynamic storage!

T
« Last Edit: December 29, 2014, 09:24:48 PM by TomWS »

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6866
  • Country: us
    • LowPowerLab
Re: Model Airplane Pylon Judging System
« Reply #10 on: December 29, 2014, 10:52:20 PM »
delateurj, your results are very impressive! They should make the skeptics into believers :)
Actually RSSI of around -50dBm is still very strong signal. Even -80 is strong and below -90 is where it gets weaker.

delateurj

  • NewMember
  • *
  • Posts: 14
Re: Model Airplane Pylon Judging System
« Reply #11 on: December 30, 2014, 12:12:11 AM »
Hi Tom,
Unfortunately I altered two variables at once so I don't know how much was randomizing retries and how much was simply upping number of retries.  It takes several hours to get in 20,000 messages so I went ahead and changed two variables as didn't want to wait to run two different experiments.  I think I can make an improvement on the retry and I like the idea (I think from you earlier) about making it unique to each node as opposed to random.  Right now the randomization is on the call to send with retry so the retry time is different for each call but within the call is using the same retry.  Better would be randomize within the call.

When I get in a 20,000 message long range test that will give data if close proximity is hurting.  If I am reading the data sheet right there is something about automatic gain adjustment, so that might help when too close...but data will be the best indication.
« Last Edit: December 30, 2014, 12:15:00 AM by delateurj »

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Model Airplane Pylon Judging System
« Reply #12 on: January 01, 2015, 10:34:06 PM »
<...snip>

When I get in a 20,000 message long range test that will give data if close proximity is hurting.  If I am reading the data sheet right there is something about automatic gain adjustment, so that might help when too close...but data will be the best indication.
The receive AGC seems to work well if the incoming signal RSSI is no stronger than ~-35.  Any stronger than this and the AGC is already at its limit and you'll begin to get non-linearities in the receive amp.  Not sure how much this affects PER, but it can't help...

You'll be able to get into AGC operating range if the radios are in separate rooms.  And, as Felix pointed out, data integrity is very good even when it would seem weak.  I set my auto-transmit level to target -59 RSSI at the far endpoint.  This is a good balance between power consumption and signal integrity.

Tom
PS: I personally don't like 'randomizing' as, in the words of Donald Knuth, most random generators aren't...
PPS: I bet it's been a LONG time since you've seen a DK quote!

delateurj

  • NewMember
  • *
  • Posts: 14
Re: Model Airplane Pylon Judging System
« Reply #13 on: January 03, 2015, 12:47:56 PM »
Hi Tom,

Thanks for clarification on AGC, that makes sense.  Love DK quotes.

Progress Update:  No new data.  Since encouraged by initial proof of concepts decided to refactor code a little bit since kind of threw it together for proof of concept.  That took some time especially since I was reminded again to appreciate the difference between declaration and definition and only declare in headers, don't define.  Got more moteino's and boards on the way.

Did spend a little time looking at what xbee's do and at the radio head library.  Xbee's (at least the 900 mhz which are annoyingly slightly different from 2.4 in firmware), randomize the retry time on the CCA step.  On this series you cannot adjust the retry time on failed to receive ack, just the number of retries.  Another thing I don't like about them is that this makes how long an attempt may take less than transparent and not controllable.

The radio head RF69 library interestingly enough has "if'd" out the call to getting a fresh RSSI and just returns last rx RSSI saying that it would hang forever.  I guess for this reason, they don't do CCA.  Almost exact same code as Felix uses so I think its just an old artifact the author's never changed.

Radiohead:
Code: [Select]
int8_t RH_RF69::rssiRead()
{
    // Force a new value to be measured
    // Hmmm, this hangs forever!
#if 0
    spiWrite(RH_RF69_REG_23_RSSICONFIG, RH_RF69_RSSICONFIG_RSSISTART);
    while (!(spiRead(RH_RF69_REG_23_RSSICONFIG) & RH_RF69_RSSICONFIG_RSSIDONE))
;
#endif
    return -((int8_t)(spiRead(RH_RF69_REG_24_RSSIVALUE) >> 1));
}

Felix's version:
Code: [Select]
int RFM69::readRSSI(bool forceTrigger) {
  int rssi = 0;
  if (forceTrigger)
  {
    //RSSI trigger not needed if DAGC is in continuous mode
    writeReg(REG_RSSICONFIG, RF_RSSI_START);
    while ((readReg(REG_RSSICONFIG) & RF_RSSI_DONE) == 0x00); // Wait for RSSI_Ready
  }
  rssi = -readReg(REG_RSSIVALUE);
  rssi >>= 1;
  return rssi;
}

Don't see how this works and radio head version would not.  Maybe I missing something.

 I will post to their discussion group.  Saw no other big differences and will stick with Felix's library with a few mods such as being able to know number of retries.

Also spent some time to finally learn Git.  Kept putting it off, I think I have my head wrapped about 80% around it.

Happy new year everyone!

- JOe.
« Last Edit: January 03, 2015, 12:59:48 PM by delateurj »

TomWS

  • Hero Member
  • *****
  • Posts: 1930
Re: Model Airplane Pylon Judging System
« Reply #14 on: January 03, 2015, 05:15:50 PM »
Hi Tom,

Thanks for clarification on AGC, that makes sense.  Love DK quotes.
:)
Progress Update:  No new data.  Since encouraged by initial proof of concepts decided to refactor code a little bit since kind of threw it together for proof of concept.  That took some time especially since I was reminded again to appreciate the difference between declaration and definition and only declare in headers, don't define.  <...snip>
I hear ya...  I've spent the last two days refactoring code on all the parts of my Mote->MoteinoGateway->WiFiGateway->HomeServer code.  It's like that when your initial prototype gets far enough along that it's sort of working and at the level that you understand everything implemented thus far.  It's best to step back at that point, refactor to clean it up, re-test and then move on to the next set of 'features'/unknowns.  Your final code is much more stable in the end, but a bit frustrating while you're doing it.

Also spent some time to finally learn Git.  Kept putting it off, I think I have my head wrapped about 80% around it.
Yeah, this is something I've got to do.  I've had some updates to the RFM69 library, but only sent them into this forum rather than using Git.  In my previous experience, 'git' was something you said to a coyote who wandered onto your property...
I've taken a stab at it with some updates to an adafruit library, but I get a sense that they really don't pay that close attention to pull requests...  (At last mine anyway  ;)


Happy new year everyone!

- JOe.
Thanks, same from me!
Tom