Moteinos + DS3234 RTC sharing SPI - require hard reset each time?

[OSBSS]

Also, a follow up on previous hanging up issue on SPI:

Ensuring the Moteino doesn't go into slave mode doesn't always work with Arduino IDE 1.6.7 apparently. I realized that even after adding that extra line of code in the beginning, one of the nodes still got hanged and was consuming 16mA continuously for a week till the battery died and only realized it later when the gateway wasn't receiving any data from it. When I switched back to IDE v1.0.5 and re-uploaded the code, it appears to works fine, going into sleep mode without any issue. Not sure why or what the new IDE does or if that's an issue at all, but just thought I'd add it here.

[Felix]

Yup we are getting repeated reports of 1.6.x not performing.
I am still on IDE 1.0.6 which works great for everything.

[TomWS]

Following up on this:

I've been working with several Moteino nodes that rely on the DS3234 RTC to wake them up from sleep mode and I chose this over the WatchDog Timer because my application requires precise time-keeping over a long period of time. I realize that WDT method can be scalable to a reasonably large number of nodes in the same network, since they don't necessarily send data all at the same time (i.e., depends on when they were powered on and when the timer was initiated). However, in my network, since all the clocks are maintaining relatively accurate time, I expect all of them to transmit data at around the same time (+/- 0.5 seconds or so).

This starts becoming an issue when I have more than 5 or 6 nodes in the network, as it appears that the network may get busy and I start to lose some data packets from a few nodes that were further away from the gateway. While I do use sendWithRetry(), specify a wait time of 40ms (even tried with 100ms) and 5 retries, I still end up with one or two nodes dropping some packets occasionally. Is there a specific way to ensure a more robust data transfer or perhaps a retry method so this doesn't happen when I have to add more Moteinos in my network? I do realize that while the node ID can theoretically go up to 255, it doesn't mean that I expect it to work with over 200 nodes with my current setup, but I expected at least 20-30 nodes to be able to work reliably with one gateway.

I use the node ID to shift a node's transmit time by a small amount based on its node id multiplied by (how long a transmission takes and how much variability there is in the timing - ie, some RTC's can only be set to within 1 second which gives you 2 second variability).

The other problem I observe with my setup, which can also be a side-effect of using a precise clock to maintain time on all nodes, is the method of uploading data through my gateway. I have two approaches:

1. Wait till the gateway receives data from all nodes, and then upload all the data online at the same time. The obvious problem with this is if data packets from one of the nodes fails to transmit due to network congestion (or whatever the issue is, I'm assuming it's a congested network), the gateway is now stuck waiting for that node and no data is uploaded online till the packet actually goes through from that one node.

2. Upload as soon as the gateway receives data from ANY node. This is fine, but again, due to the precise time-keeping, all data is expected to be received at about the same time. If the gateway is busy uploading instead of listening for data from the next node, then a few packets tend to get lost yet again.

A third approach is a hybrid of the first two.  The gateway knows when all the reports are to come in.  It provides a window large enough for all the reports, accumulating the data that does come in, and then sends the bulk.  Anything that comes in after this, but before the next window, is going to need some time correction anyway.

While switching to WDT may solve some of these issues by letting the nodes send the data at different times, I would prefer to maintain accurate timing with my setup.

The WDT is horribly inaccurate and shouldn't be used for tight timing applications.

Tom

[OSBSS]

@Tom - thanks for the tip. I implemented a similar approach, and it works well, as long as correct time is maintained. Working on syncing time on each node with server time (dwelling into wireless programming - keep getting impressed by Felix's work)

I'm wondering though, like you said, the minimum variability in timing with the RTC I use is 1 second. This method works well for 5-7 nodes, but my logging interval can be as small as 10 seconds, depending on the application (some are used in research - decay of gases, quick environmental changes in a controlled chamber, some are used around an academic building to monitor building activity, etc.) I can see this method to be a problem in scalability when I add more nodes in the network, and if number of nodes exceeds 10, then the setup reaches its limit. If I increase the logging interval to 30 or 60 seconds, I am able to add more nodes, but anything beyond that, I would be compromising on the quality of data collected. I currently have 8 nodes, so the method works fine for now, but will be increasing the number soon, so just thinking ahead.

Another approach would be to use more than one gateway and allocate a different network ID to it and the few nodes it serves. I haven't tried this yet as most of my nodes are currently in use, but since they would be operating in the same frequency but just have a different network ID, I'm wondering how that would affect congestion and data loss. If I have 30 nodes and 3 gateways inside a building (10 nodes per gateway) for example, how would they behave when working in tandem? I would assume there would be some type of interaction by a node to all three gateways till it figures out which one has the correct network ID, and during this interaction, other nodes that are supposed to be sending data to their gateways might not be able to do so.

Is this a proper way to look at it? Maybe I'm confusing the principles on how the modules work.

Edit: Just reading up on network IDs and hardware filtering. Nevermind that.

[joelucid]

If you measure the offset of your rtc against the correct second using the moteino's resonator, calibrate the wdt by measuring it against the rtc and then sleep when the rtc wakes you using the calibrated wdt to pass the time until the real second hits you should be able to get down to around 15ms resolution.

Some Rtcs also can be restarted so that they start their second counter when you tell them to. That would make it even easier.

[TomWS]

I concur with Joe on this one, especially since your RTC is accurate.  Use it to determine to calibrate both millis() and WDT loops it takes between 1.000 second interrupts (do the measurements at two different times so millis() isn't affected by WDT) and then use that as a scale factor for LowPower Sleep timings and finish up using millis() to zero in on the correct post time.

I use the AB18X5 RTC, which has resolution to 10 milliseconds (including its alarm) function, but the crystal needs to be calibrated (ie, each board using it) and that can be a PITA if you don't have the equipment.

Tom
EDIT: You'll have to do calibration cycles periodically as temperature alone will cause the WDT and probably processor clock to drift.

[OSBSS]

Joe and Tom - that's an interesting idea. Yeah, my first thought was that wide swings in temperature would create more drifts.

Just to clarify: currently the RTC is programmed to generate interrupts exactly when its time reaches 00 seconds for example, so I offset this by node ID like you suggested, and it will wake each node at 01 second, 02, 03, etc respectively. So if my logging interval is 30 seconds, it will automatically wake up at 31, 32, 33rd second of the minute (RTC's time) every time. So instead of this, do I wake all nodes at exactly 00, but offset that by immediately going to sleep and then relying on WDT to wake up after a very short duration which varies based on node ID?

Or should I actually measure the difference in the actual "second" measured by RTC (interrupt every second) and what WDT or millis() report as their value at that instant? I recall doing a simple test with millis() couple of years ago and found it to drift significantly - by a minute or two over the course of a day (granted it was the internal oscillator, but even Uno doesn't fair too well). Never really measured the amount, as it always varied. Still not sure how using this difference as a scale factor in the sleep timings helps. Perhaps I didn't fully understand the implementation.

[joelucid]

I just looked at the data sheet in the rtc does have a way to synchronize the time:

The countdown chain is reset whenever the seconds register is written.

So what you do is periodically your node sends a ping to the server and the server responds with the exact time in ms. It can do that since it can record how many millis() have passed since the last second increment from the RTC occurred. Then the client knows how long to wait until the next second increment happens. Wait and then set the seconds at the RTC which causes the counter chain to restart that instance.

The RTC also supports square wave output. So if you then want to report a measurement at 1:30:44:50 you let the RTC wake you at 1:30:44 and then count 50 * 1000 / 1024 interrupts from the square wave output. You can powerDown between the interrupts. You could also use this sleep method when you wait for the next second increment during synchronization.

[perky]

You can do better than that using the Time-sync Protocol for Sensor Networks (TPSN). The trick involves the node sending a timestamp (copy of it's counter) to the server, and the server sending back its timestamp (a copy of it's counter). By making an assumption that the propagation delays in both directions are equal you can get a more accurate synchronzation.
Mark.

[WhiteHare]

You can do better than that using the Time-sync Protocol for Sensor Networks (TPSN). The trick involves the node sending a timestamp (copy of it's counter) to the server, and the server sending back its timestamp (a copy of it's counter). By making an assumption that the propagation delays in both directions are equal you can get a more accurate synchronzation.
Mark.

Is the idea of having the node send its counter to the server just to have the same size packet as the one it's receiving?  i.e. Could the node just as easily fill the packet with a payload of the same length with but with other data  (such as data it might have been sending anyway) and/or padding?  Provided the node records its counter in the same way it would have anyway, then (if I get the gist of the idea), all it has to do is wait for the server's return packet, note the time of it's arrival, cut the roundtrip time in half to calculate what its counter should have read at the time the server received its packet, and then change its counter with a correction offset so that they would have been the same?  Sounds promising.  Alternatively, I suppose it could do much the same thing just by receiving two time-stamped packets in a row from the server.

Cool.  That sounds simple enough that I think I'll actually try it!  I never did like the idea of just "spraying and praying" packets for lengthy periods just to have a chance that one of them might hit the Listen window.  Having the server send a relatively few synchronized packets would be far more preferable, even if the tradeoff requires a bit of extra work by the node.  If one can accept the RFM69's counter in place of an RTC, then it would be great if one could maybe (?) even do it using non-modified Moteino's that lack an RTC....  Offhand I don't recall whether the RFM69's counter can be directly changed to a particular value for an instantaneous time correction like perhaps an RTC could be, but  I do remember  JoeLucid has demonstrated on another thread that it can be gradually modified using recalibration to satisfy a similar objective.

[perky]

I looked at this some time ago for my project, it was OTT for what I needed but I wanted to understand the principle. I was wrong in the above post about the node having to send its counter value, it just needs to sample it.

Reference the attached diagram. The top horizontal line represents the server's RTC counter (maybe a 16 bit 32.768kHz timer counter). The bottom horizontal line is the node's counter. The goal is to adjust the node's counter at time T5 to be exactly the same as the server's counter at time T6. I've seen this diagram and the final equations stated several times in documents, but no real explanation of how the equations were derived, so here it is:

The node samples it's counter at T1 and saves it. It then sends a packet to the server. The server samples it's counter immediately it receives the packet at T2 and saves it. When it is able to send a reply, the server samples it's counter at T3, and sends its T3 and previously stored T2 values back to the node. The node samples its counter at T4. The node now has 4 counter values, it's own counts of T1 and T4, and the server's counts T2 and T3.

A few assumptions:
1) Transfers to/from the radio are done with SPI in the MHz frequency range, so can be ignored for time.
2) The time for the node to sample its counter at time T1, send the packet, and the server to sample its counter at time T2 is d.
3) The time for the server to sample its counter at time T3, send the packet and the receiver to sample its counter at T4 is also d.
4) The packet sizes sent by the node and by the server are the same size.

Now the logic. Let's assume the node's counter is behind the server's counter by a value Tdelta (in counter units). We need to calculate that and add it to the node's counter so that the node's counter at time T5 is the same as the server's counter at time T6.

We can write some equations.
A:  T2 = T1 + Tdelta + d
B:  T4 = T3 + Tdelta + d
C:  T4 - T1 = T3 - T2 + 2d

We can now find out what Tdelta is in terms of the other known values.

Re-arrange C:
D:  d = (T4 - T1 - T3 + T2)/2

Re-arrange A:
E:  Tdelta = T2 - T1 - d

Substitute D:
Tdelta = T2 - T1 - (T4 - T1 - T3 + T2)/2

Therefore:
F: Tdelta = ( T2 - T1 -T4 + T3)/2

Re-writing these we get:

d = ((T2 - T1) + (T4 - T3))/2, and

Tdelta = ((T2 - T1) - (T4 -T3))/2.

All the node needs to do is add Tdelta to it's current counter value to be in sync with the server. The beauty of this is it is independent of how long it takes the server to get round to sending the reply packet, it may for example be using listen before talk. Also the calculations are independent of bit rate, as long as the packet is the same size.

Mark.

[SadE54]

Also , Time flooding syncronization protocol (TFSP)could be used :
http://www.eecs.harvard.edu/~mdw/course/cs263/papers/ftsp-sensys04.pdf

The timestamp on the first payload byte from the gateway is regularly broadcasted and the frame is timestamped too on the node at the reception.
You can compare the 2 because at the reception of the propagation time , all other timing can be determined.
A linear regression on several couple of timestamps is then used to set the node clock.
It seems it could be very precise (1us)

I would implement that someday , but some specs seem blur to me :-/ concerning the computation of some times
And I guess a 32 bit timer could be welcome 

[snorp]

The other problem I observe with my setup, which can also be a side-effect of using a precise clock to maintain time on all nodes, is the method of uploading data through my gateway. I have two approaches:

1. Wait till the gateway receives data from all nodes, and then upload all the data online at the same time. The obvious problem with this is if data packets from one of the nodes fails to transmit due to network congestion (or whatever the issue is, I'm assuming it's a congested network), the gateway is now stuck waiting for that node and no data is uploaded online till the packet actually goes through from that one node.

2. Upload as soon as the gateway receives data from ANY node. This is fine, but again, due to the precise time-keeping, all data is expected to be received at about the same time. If the gateway is busy uploading instead of listening for data from the next node, then a few packets tend to get lost yet again.

While switching to WDT may solve some of these issues by letting the nodes send the data at different times, I would prefer to maintain accurate timing with my setup.

Any ideas, comments, criticisms or suggestions regarding this setup are appreciated.

It seems to me that you really don't want the nodes themselves deciding when to send data. Instead, you should poll each one of them from the gateway. That way you can fully orchestrate the collection of data and ensure that you don't have collisions. If the timing of the collection on each node is critical, you can still use your RTC on the nodes to determine when to do that -- just don't send it until you are asked by the gateway.

[OSBSS]

I've been experimenting with various setups as mentioned here by others (thanks a lot for the feedback guys)

I agree with you snorp. Been reading up on listen mode on the radio and trying to understand how it works. I'm not overly concerned about the node gaining a few uAs in sleep mode, so it seems like this approach would work well. I think WhiteHare's suggestion of using a GPS module to maintain exact time on the gateway is ideal for maintaining exact time. If not, I can use my RTC and just sync it to server time either on every upload or at the end of the day, so it always stays accurate. This way I can also eliminate the RTC on the nodes, save a few bucks, and avoid all these SPI issues.

I'm still not entirely familiar with listen mode though and wondering about time delays. For my application, if I have a logging interval of, say 5 mins, the node remains in listen mode indefinitely until the clock on the gateway strikes the exact 00 second of the interval, or whatever the interval is, and sends a "wake" command. The node can then wake up, take whatever measurements, send it, wait for ACK. Then the gateway receives data, sends the ACK, and the node goes back to sleep in listen mode. What would the duration of this entire cycle be? If it's short enough, there's a good chance I can place over 20-30 nodes and still get data on exact time intervals from each one. And since the gateway is only requesting from one node at a time, congestion and collisions are less of an issue. If I'm mistaken on the basics of listen mode, do let me know. I've occasionally seen nodes to have some timeout issues where it gets stuck in RX mode indefinitely on basic setup - still unsure what causes this, probably SPI issues with the clock. Will debug more if it happens in listen mode as well.

[perky]

Depending on bit rate and packet length I'm sure you can get something like 20ms total for the request packet and the ACK back. There's a bit of uncertainty at the beginning, so maybe allow 10ms or so for that.

I can see a system that uses 32.768kHz 10ppm watch crystals on both the gateway and the node. The gateway would transmit request packets regularly with an incrementing node address in one of the fields. The node would initally listen for a packet, when it get's one it then knows from the address field where in the sequence it is, and then go to sleep so that it wakes up when the gateway sends the request packet that is addressed specifically to that node. It'll then go back to sleep and wait for it's next 'time slot'. If we say 32 nodes, with 5 mins between uploads, the gateway would be transmitting a request packet once every 9.4 seconds. The 32.768kHz crystal would drive the timer/counter of the micro and permanently run even when asleep (for about 1uA current), so it can be used to generate a wakeup interrupt. The drift difference would be in total 20ppm over 5 mins (about 6ms) , that would set the initial uncertainty window size, but you'd want a node to re-calculate it's next slot if a packet isn't received when it expected one so the uncertainty window can grow until after n packets are lost it tries to re-aquire lock again (which is a battery-thirsty operation, you'll want to do that rarely).

You could get more adventurous with this and use a pseudo-random sequence for the time slots, that would allow multiple systems to co-exist in the same rf neighbourhood.
Mark.