(Semi)Automatic Wireless programming AND long sleep

[msjfb]

Hi,
I saw an old post on this subject and I am at the point where I want to do something similar.
Has anyone worked on this situation, i.e. how to wirelessly reprogram a node that wakes up every 15 minutes ?

I have done all the tests with the samples and got everything to work fine. So no problem with the basics.

I have a couple of ideas along the lines presented by Felix in that older post: queuing a condition in the gateway so that the on the next transmission from the remote node a message will be sent back to the node  advising it to stay awake for x seconds and expect a wireless upload (sit in a tight loop) .

I am thinking of using a dedicated 'secondary' gateway (different id that the main one) that would listen in 'promiscuous mode' at the network, and would handle the 'stay awake' info to send back to the node.

I am a beginner with Python, so how would you go about automatically starting the script from code in the gateway ?
I would think that you would have to start the script and have it wait in a loop until the gateway sent it the OK to start uploading. Just a thought...

Is it possible to write a sketch that will store the file in the gateway memory when running the script manually ? I am assuming yes since what we would have to do is the same as is happening on the node (temp storage).

So if I can manually upload the file onto the (secondary) gateway, can the python script that handles the communications protocol be written as a sketch ?

If this is possible, then we could manually upload a file in the gateway for a given node, and the gateway would automatically handle the transfer when the node wakes up.

Francois

[TomWS]

Francois,  the way I handle this is exactly as you suggested, ie the gateway queues up a packet for the target node and, on the next transmission from the node, the gateway responds with an Ack (as always) but this Ack has content that tells the node that the gateway 'has something' for the node.  At that point the node can decide to accept the 'invitation' immediately or simply wait until the next 'wake' time.

There is one difference, however, from what you're thinking about.  In my case, I've designed my gateways with microSD card slots where the images to be distributed to node can be queued as well.  In operation, the home server sends the image to the gateway and, once the image is successfully downloaded to the gateway and stored on SD, then the Home server tells the gateway which nodes should get the image.

With this approach the image can be packetized so that nodes don't need to load the whole image at once but can download pieces from the gateway over a period appropriate to the battery limitations of the node.  Eventually, when the whole image has been transferred, the node can reboot with the new image.

I don't use the python code or WirelessProgram library at all.

I'm not sure I see the benefit of using a separate gateway for this.  Maybe if it's a 'roving' gateway that you can manually copy the image to SD, insert the SD into the second gateway, and then place the gateway in the vicinity of the target node.  You can eliminate a lot of gateway/server interaction in this case as well as simplifying your primary gateway as it wouldn't have to deal with the program update problem.

Tom

[msjfb]

Tom,
The gateway I am using at present is a NANODE which has ethernet and microSD card. Not sure if I am going to keep this, but you offered interesting comments.
If I understand correctly:
-  you push the file to the gateway SD card (IP or serial, just curious)
- the gateway transfers wirelessly (in packets) the file to the node which stores it in temp memory
- the node reboots and uses new image.

I am glad to see we can 'hack'  the ACK message for this purpose (using the RFM69 library?). I was about to look into this.

thanks. Francois

[TomWS]

Tom,
The gateway I am using at present is a NANODE which has ethernet and microSD card. Not sure if I am going to keep this, but you offered interesting comments.
If I understand correctly:
-  you push the file to the gateway SD card (IP or serial, just curious)

I use IP.  My Gateway has a WiFi module.

- the gateway transfers wirelessly (in packets) the file to the node which stores it in temp memory
- the node reboots and uses new image.

Yes, the node stores the image data in flash memory just like is done with the WirelessProgramming library.  Exactly the same way except the header is not programmed until the entire image is successfully stored.  Then the image header is written so that, on the next reboot, the DualOptiloader will copy the new image into program memory.

Note that the packets sent to the node using a 'pull' methodology - the node controls how many packets it wants per sample cycle by requesting each packet from the gateway.  As an example, I have some Temperature Humidity modules that send samples every 10 minutes.  Typically they are only awake for 2 seconds.  When they start downloading an image, they request packets for 10 seconds more and then go back to sleep.  Each packet contains about 32 bytes of actual image data.  I don't recall how many packets are fetched during each sample time, but it takes about 2 hours to completely transfer about 25K image.

I am glad to see we can 'hack'  the ACK message for this purpose (using the RFM69 library?). I was about to look into this.

Actually this is not a 'hack'.  It is a feature of the RFM69 library - it is very useful.  I actually use this feature 'under the covers' to return the sender's RSSI back to the sender so that it can tune it's transmit power to an optimum level.

thanks. Francois

pas de quoi...

Tom

[msjfb]

After trying out different calls and looking at the library code, I finally was able to get at the (possible) data passed back with the ACK. But I had to use the properties (ACK_REQUESTED and ACK_RECEIVED) to do it (in both nodes). The methods did not seem to work.

This is a code sample:
    if (radio.sendWithRetry(GATEWAYID, (const void*)(&theData), sizeof(theData)))
    {
      Serial.print(" Sent!");
    }
    else Serial.print(" Error...");
    Serial.println();
//    if (radio.ACKReceived(GATEWAYID))  <---  DOES NOT WORK, ALWAYS RETURNS FALSE
    if (radio.ACK_RECEIVED)   
    {
      Serial.print(" - ACK  received from ");Serial.print (GATEWAYID);
      if (radio.DATALEN != 0) {
        Serial.print(" with Message: ");
        for (byte i = 0; i < radio.DATALEN; i++)
        Serial.print((char)radio.DATA);
      }
      Serial.println();
    }     
    else
    {
      Serial.print(" - ACK  NOT received from ");Serial.print (GATEWAYID);Serial.println();
    }   


I also tried the .SEND method (with requestACK set  to TRUE), but the other end never sees the ACK request (using either the method or property to look for it).

I may have been doing something wrong, but I got it to work using the properties.
Maybe this will help others struggling with this.

Now on to the next step: transfer the image...
Any comments are welcome.
Thanks.

[TomWS]

Francois,
here is what I use in my gateway and nodes to pass info back during an ack.

First, from the gateway side, since the gateway had to queue up something the server wanted to send to a sleeping device, it placed the packet in a 'gotSomething' queue, as in, I've got something for this device.  The queue is indexed by node ids.  When a node checks in, just prior to sending the Ack, the gateway performs the following code:

Code Select Expand

    
    if (ackReq && target == mwgwMote.MY_DEVICE_ID)  // only Ack if specifically addressed to me
    {
      // check to see if we have a request pending for this device
      TwsMoteResponse *pR = (TwsMoteResponse*)rcvBuf;
      int i = -1;
      if (pR->h.type!=TM_REQ_ACK)           // this might be a TM_REQ_ACK, don't send gotSomething if it is!
        i=checkGotSomething(sender);
      if (i>=0) 
      {
        if (bVerboseOn) {Serial.print(F( "Processing GotSomething..." ));}
        reqSomething.request = TM_GOT_SOMETHING;
        reqSomething.data[0] = gotSomethingList[i].something;
        mwgwMote.radio[b].sendACK(&reqSomething,reqSomething.h.len);[/b]
        if (bVerboseOn) {Serial.print(F( " - ACK with gotSomething=" )); Serial.println(reqSomething.data[0],HEX);}
      } else 
      {
        mwgwMote.radio.sendACK();
        if (bVerboseOn) {Serial.print(F( " - ACK sent" )); }
      }
    }

Note that the 'gotSomething' packet only sends a single byte.  This byte is a 'key' to the 'something'.  Once the node decides that it's actually ready to receive the 'something', it will send a request back to the gateway and stay awake for the 'something'.

Then, on the node side, who, after waking up, sent a packet to the gateway using my sendPacket method. 

Code Select Expand

    
    TwsMoteHeader tmDataIn=TM_H_INIT(TM_DATA_IN);
    memcpy(tx_buf, &tmDataIn, tmDataIn.len);   // encapsulate our data packet into a DataIn wrapper
    thMote.buildPacket(tx_buf, &th_data.h);
    thMote.sendPacket((TwsMoteHeader*)tx_buf);
    byte *pAck = thMote.getAckData();     // See if we got something in return.
    if (pAck)
    {
      // got something, parse the packet
      parseRecvPacket(pAck, thMote.GATEWAYID, thMote.MY_DEVICE_ID, pAck[0]);  
    }
   

Finally, the code 'under the covers', (sendPacket, getAckData, etc) is here:

Code Select Expand

/******************************************************************************
*
*   sendPacket() - common task, saves typing...
*
******************************************************************************/
bool  TwsMote::sendPacket(TwsMoteHeader*pkt, uint8_t to) {
  bool rc;
  if (to==NULL) 
    to = devDescriptor.gateway;  // default is to send to gateway
    
  rc = radio.sendWithRetry(to, pkt, pkt->len);
  return AckCheck(rc);  // see if we got any data with the Ack (THIS NEEDS TO BE DONE IMMEDIATELY AFTER THE SEND!!!!)

}

/******************************************************************************
*
*   AckCheck() - local method to check to see if there is any AckData and save it locally if there is.
*
******************************************************************************/
bool TwsMote::AckCheck(bool rc)
{
  if (rc) 
  {
    noInterrupts();
    // tranmission ok, see if any data was Ack'd
    if (radio.DATALEN && radio.DATALEN==radio.DATA[0]) // got a valid packet?
    {
      // yup, save it
      for (int i=0; i<radio.DATALEN; i++) 
        sendAckData[i] = radio.DATA[i];
    }
      else
        sendAckData[0]=0;
    interrupts();
  }
  return rc;
}

/******************************************************************************
*
*   getAckData() - return ptr to locally saved ack data
*
******************************************************************************/
byte *TwsMote::getAckData(void)
{
  if (sendAckData[0]) return &sendAckData[0];
  return NULL;
}

One additional comment: in the code you provided the fatal flaw was using print statements between the sendWithRetry and the check for data.  These need to be done immediately together.  Additionally, the check for ACK_RECEIVED is totally redundant - you wouldn't have gotten a positive response from sendWithRetry if you hadn't received an Ack...

Tom

[msjfb]

Tom,
Thanks for your detailed reply. It is very much appreciated.
I have one question: it appears that you still have to check the validity of the packet(s) found  in the .DATA[] buffer when looking for any data sent back with the ACK:
  // tranmission ok, see if any data was Ack'd
    if (radio.DATALEN && radio.DATALEN==radio.DATA[0]) // got a valid packet?

Does this mean that it is possible  that the buffer might contain data that is not from the ACK, or even NOT from the node we sent the 'SendWithRetry' to ? for instance in a situation when there is a high volume of messages sent by many nodes ? In other words can collisions be a problem in terms of data integrity ? I understand that a sendMessage might fail due to collisions (I read the <Model Airplane Pylon Judging System> post, I just want to make sure that if I do get something (out of the radio), it is what I was expecting.

Finally, you are of course right about the code logic. In  my defense  this debug code was initially using the .SEND method to try and understand how the ACK was handled, then I  changed to .SendWithRetry without changing the code structure before posting. Will watch out for this next time...


Cheers,

Francois

[TomWS]

Francois, my working assumption is that receive packets could come in at any time, including interrupting an Ack.  Frankly this isn't likely in my network except in the case of the gateway which could get valid packets at any time.  Hence the code tries to accommodate this.  What is likely is that there will be packets that are 0 length, simply because they are Acks. 

In the example you cite, my packets all lead with a length byte, so if its a valid packet the DATALEN will ALWAYS equals the first byte in the packet.

In general, wireless networks are amazing that they work at all.  Anyone who assumes they are perfect hasn't spent any time watching un-demodulated waveforms.  The best we can hope for is some logic that gets MOST of the stuff and doesn't break when it gets what it didn't expect (or vice versa).

Good luck with your project.  What are you building?

Tom
PS: You don't need a 'defense' - how do you think I know that you shouldn't put print statements there??? 

[msjfb]

Tom,
Ok, I'm back...
The way I understand it, if the SendWithRetry method  returns TRUE it guarantees that a valid ACK was sent back to me from the destination.
However, to get at the possible data sent back with the ACK it looks like all we have is the .DATA property (to be checked right after the .Send) that theoretically could contain anything and we are relying on 'timing' to assume that it is what we are looking for unless we do more validation.
So if I got it right, I want to have a few bytes to ensure that the message is intended for me and that it is valid.

I am thinking that the ACK message could have the following structure (3 control bytes, message, Checksum):
<HEX06><DestinationNodeId><Number of Bytes>...Message...<Checksum>
Unless I am missing something that does not require this...

My project is Home Automation in general, but with particular focus on: Remote monitoring of temperature (spa, outdoor shed, etc.), energy monitoring (both smart meter Infrared pulse counting for whole house and current monitoring of certain circuits) I also have an interest for 'trigger systems' that will do 'something' based on certain conditions.
One project in particular I am working on (I have not seen anything similar around):
Automatic closure of my house water supply :
-  5volt (60 ma) ball valve (found in China)
- Switch mote based ( I like the faceplate!)
- Mighty  boost for standby power (I want the valve to operate even during a power outage)
- Moteino RFM69 (of course)
- local water leak detector
- Loud alarm buzzer
Obviously the idea is to have the valve close if the local detector senses water, but also the mote should be able to react to a message from other water detector motes spread out through the house, or from any other condition (ex: when the alarm system is armed). All this action is monitored by the Gateway and eventually pushed out (PC, cloud, etc) for further processing.
I am in the final process of assembling all the hardware, and should begin on the software soon.

François

[Felix]

Francois,
Check out how the garageMote is implemented.
Generally speaking, just setting the target and sending an ACK is enough and the receiver (if in RX/listen mode) will receive it, and sendWithRetry will resend up to N times (3 by default) to ensure a packet was received and ACK was returned.

[TomWS]

Tom,
Ok, I'm back...
The way I understand it, if the SendWithRetry method  returns TRUE it guarantees that a valid ACK was sent back to me from the destination.
However, to get at the possible data sent back with the ACK it looks like all we have is the .DATA property (to be checked right after the .Send) that theoretically could contain anything and we are relying on 'timing' to assume that it is what we are looking for unless we do more validation.
So if I got it right, I want to have a few bytes to ensure that the message is intended for me and that it is valid.

<snip...>

The SENDERID field in the Ack will tell you whether it's a response from your destination or a packet from someone else.  It won't tell you directly if you received a packet from that sender JUST as you sent your packet (with some slight timing difference since it is half-duplex), however, coupled with ACK_RECEIVED, you have a pretty good confidence level that it's the Ack to your send.

Also, if you take a look at Felix's blog about the RFM69 library (http://lowpowerlab.com/blog/2013/06/20/rfm69-library/) you'll see the format of the entire packet that is sent over the air.  You'll notice that the packet already has CRC built in so your checksum is, to my mind, unnecessary.  I also use encryption, not to keep my temperature packets secret     but to add another level of error detection to the content.

I do packetize all content in a very specific way, but it's primarily to make it easy to parse the content on the receiving end.

Tom

[msjfb]

Tom and Felix,
Thanks for the input.
I really wanted to understand how the 'data returned with ACK' worked because I find it is a really efficient way to respond to certain messages. So I should be good to go.

François