RFM69 TX power testing & library fix

The RFM69 library has been very robust and stable over the years. It is widely used in many hobby and home automation projects as well as research and commercial products. But that doesn’t mean we can’t find occasional things to fix and improve!

Is there a problem?

Recently, after renewed attention to a long time issue relating to power levels being incorrectly set, I took measurements and found the setPowerLevel() function was behaving somewhat unexpected. Instead of a smooth  Namely it as making available the lowest possible transmit power levels on the HW/HCW (aka high power) variants of RFM69. Since a commit in 2015 the 32 power levels were actually made into 16 steps and only using PA1+PA2 in high power mode. Not so much a loss except for those hoping to maximize battery life by running at the lowest power levels on the HCW – in which case they’d be better off using the CW instead. But this is enough to find a fix and take full benefit anyway of what the HCW has to offer.

Basic TX power testing

To produce power profile I put together a set of Transmitter and Receiver examples. To measure power output you can just use the transmitter sketch – it toggles TX mode and you can navigate the power levels from min to max. Below is a simple hardware test setup that can be used to take some power measurements: a CurrentRanger reads the current going into a Moteino+RFM69HCW that radiates an unmodulated carrier signal into a dipole. Idle MCU current (TX off) is read and then subtracted from total current with TX enabled:

To also measure RSSI (poor man’s power meter) we can just use another Moteino with the receiver sketch mentioned above, which will simply sample RSSI continuously (in RX mode), at a max AGC setting of -48dB to simulate a long “distance” since we’re doing all this in close proximity (the same room!). Open the transmitter sketch in a serial terminal and toggle TX mode with t and increase power with + & - in steps or < & > in dB – there is a new function setPowerDBm() in RFM69 library v1.5.0, in addition to setPowerLevel(). The CurrentRanger will show the current drained by the Moteino when transmitting at different levels and the RSSI on the receiver will also reflect different output power levels.

This setup produced the following power profile on the W/CW radio, nothing unexpected since there’s only PA0 to be used with no other registers to set, power levels being set in RegPaLevel.OutputPower. The only thing to notice is the ‘step’ in TX current and RSSI which indicates something’s definitely being switched inside the transceiver itself as _powerLevel crosses the middle of the RegPaLevel.OutputPower register, even if PA settings remain unchanged (only PA0 is used).

On the HW/HCW, in v1.4.3 (prior to the fix this blog is about) the power profile looked like this:

As it turns out the startup power is correctly set to a valid level 31, but calling setPowerLevel() even once results in values being set from 0-16 (all “forbidden” – see tables below) and never back to the original 31. This results in a ‘step ladder’ of transmit power levels vs. current. I know this sounds confusing, because it is.

The theory – datasheets and registers

To better understand what’s really going on, first we need to clear some confusion and understand that RFM69 radios have several possible transmitter PA (power amplifier) combinations for different output power profiles as outlined in the datasheet:

For W/CW radios things are very simple – only PA0 is wired to RFIO output pin and it can produce from -18 to 13dBm, corresponding to 32 possible power levels that can be set in RegPaLevel, as clearly reflected in the W/CW graph above.

For RFM69 HW/HCW radios things get more complicated, there are two possible PAs  and a power boost mode to be enabled to reach the highest output of 18 to 20dBm, according to section 3.3.7:

I know of only one other good article that outlines all this in detail, a blog by Andre Hessling, I found it accurate and so my focus is not to replicate that here, but rather use it as reference and complement for those wanting to get a full picture, so check it out if you want to get more in depth.

So!  The RFM69 library v1.4.3 and prior, has been only using PA1+PA1 in boost mode, at power levels 0-16, except at startup where _powerLevel was initialized and set to 31 and never divided by 2 like in setPowerLevel() – bug!.  In addition, the datasheet says levels below 16 are off limits for the PA1+PA2 combination, but because this works it went unnoticed for a long time, at least in my library (semi bug?). If you’ve read Andre’s article, you will notice PA1 alone can be used and he proposes using the following PA combinations for HCW:

  • PA1 (and PA1 only!) from -2 to 13 dBm output
  • PA1+PA2 from 14 to 17dBm output
  • PA1+PA1+HighPower for 18 to 20dBm

If we revisit the datasheet, we realize there is actually a power profile overlap between these 3 combinations:

Notice for instance we can achieve -13dBm in 3 different ways on the HW/HCW:

  • using PA1 only with RegPaLevel.OutputPower=31
  • using PA1+PA2 with RegPaLevel.OutputPower=27
  • using PA1+PA2 with RegPaLevel.OutputPower=24

Enough already, what’s the conclusion?

Since theoretically the output is the same, it would be interesting to see the current profile against each of these as well. To cut this (really) short, I will link the spreadsheet with my tests and graphs and just mention the final conclusions. It matters which PA combination you pick, as a general rule, using more than one PA will result in higher current for the same effective transmitted output, at first this seemed counter intuitive but that’s what the meter and RSSI shows. I determined the best combination for HW/HCW to be the following:

If you notice, there are now only 24 _powerLevel‘s with slight overlaps between steps 14,15 and 16,17. The light orange band PA1+PA2 combination is sandwiched between PA1 and PA1+PA2+HighPower and helps optimize current used at those theoretical dBm outputs. This is similar but a little different than Andre’s proposed bands – I suspect he may have looked at RSSI but not at current used. Overall this produced the smoothest output power vs. RSSI vs. current curves that I could determine – with minimal ‘step’ jumps on the current and RSSI, and this is what we’d expect to properly use the power levels, especially in the RFM69_ATC extension which automatically regulates the power level based on a desired RSSI. And here are the measurements in a graph:

For the W/CW nothing really changes since there was no real issue, the graph will be the same as the first one included in this article.

Update the library and test on your own!

I made a new release v.1.5.0 which should address all these issues and includes the TX power testing examples which can be used by anyone with perhaps different/better test setups to verify these findings. Please update  and try this out. I may update this blog with further info if need be in the future since it’s more of a reference for RFM69 PA modes.

Moteino SAMD 1.5.0 release

There is a major new release for the Moteino SAMD Boards Package 1.5.0. It will popup as an update reminder the next time you restart Arduino IDE, or you can go to the Boards Manager and update from there:

Here are the most significant changes in the SAMD package:

  • All MoteinoM0 and CurrentRanger boards will start shipping with the UF2 bootloader (it’s well worth a read if you’re not familiar with it). The TLDR; is: it supports sam-ba serial protocol uploads as before (via CDC serial, from bossac or via the Arduino IDE) and it also supports drag-drop updates of the firmware as well as the bootloader itself (via a MSC flash drive that appears when the M0 is running the bootloader). Extremely useful if you want to allow an end user to update the firmware and/or bootloader with a newFirmware.uf2 file drag-drop to the “flash-drive” simulated by the bootloader, without the need for the IDE. You could enter the UF2 with a RST double-tap as before, and you’d see a new “flash-drive” on your system (the CURRENT.UF2 is the actual firmware loaded in the MCU – useful to back up before an update):
  • To top off the UF2 awesomeness, MoteinoM0’s will continue to support updates of the firmware from the external FLASH-MEM chip, after an OTA upload via RFM69. The latest RFM69 library release 1.4 has been updated to support this.
  • SerialUSB is now completely removed from the MoteinoM0 and CurrentRanger variant definitions:
    • On MoteinoM0 Serial is now the USB serial, Serial0 is the UART on pins 30/31, and Serial1 is the UART on pins 0/1.
    • On CurrentRanger Serial is now the USB serial.
    • SERIAL_PORT_USBVIRTUAL is now Serial by default
  • You might notice in the MoteinoM0/CurrentRanger boards menu, there are now some options like choosing the USB stack (Arduino, TinyUSB) and more notably the Crystal selection. You can compile for the external crystal (default for MoteinoM0 R1) and “crystal-less” ie. the internal ultra low power 32.768kHz clock). When running without the external crystal, the internal clock is tuned using the USB bus clock which is very precise.

Note that the Moteino AVR boards package is now at v1.6.1. You are encouraged to update both of these packages. Older boards running the sam-ba bootloader may be flashed with the new bootloader included in the 1.5.0 package via SWD programmer. I may even offer to do this for free if you’re willing to return the board and pay for shipping back to you. Please report any bugs or issues in the MoteinoM0 or CurrentRanger forums.

RFM69 10bit node addresses

I made some changes to the RFM69 library to support 10bit node addresses, and you should see version 1.3.0 available in your Arduino IDE Library Manager. Previously the standard was a 1byte (8bits) address for sender and receiver, a total of 2 bytes used in each packet for addressing. I wanted to avoid using another 2 bytes to increase addressing to 16bit (65k addresses) because having that many nodes is impractical, and 2 more bytes is an “expense” most users won’t need. So I used the bottom 4 bits of the CTL byte, to add 2 bits to each of the existing to/from address bytes. The network addresses are kept at 1byte (8bit).

This yields an address space of 1024 total addresses. Address 0 is now reserved for the BROADCAST address, and you are left with 1023 possible addresses. A gateway is suggested to be kept at address 1 and the rest of the nodes should occupy incremental addresses.

There should be no impact on anyone not using more than 255 nodes as before. One required change for those wanting to use addresses larger than 1 byte (255) is they need to use uint16_t variables in their sketch. I’m looking forward for constructive feedback and any bug reports at this time. The ListenMode part of the code (which is experimental) was left untouched and it only supports 8bit addressing as before.

Here is the latest RFM69 packet header structure including a breakdown of the CTL byte:

Other important updates:

  • The BROADCAST address was changed to 0 (from 255 previously).
  • The previous RFM69 v1.2.0 included SPI Transactions, another important change to the library.
  • After it was brought up in the forum that Moteino AVR sketches compile to larger sketch size than Arduino, I recently released a new  AVR Moteino Boards definition (based on Arduino 1.6.23 AVR boards) which results in significant compiled size reductions for all Moteino sketches, please give this a try. In Arduino IDE’s Boards Manager you should be able to easily upgrade to the latest version.

RFM69 and SPIFlash libraries released via Arduino LibraryManager

The LowPowerLab RFM69 and SPIFlash libraries are now published to the Arduino Library directory, and available to install in the IDE’s Library Manager (under Tools>Manage Libraries). Note that if you already have these library installed manually, you may need to remove it and re-install using the manager:

As new versions are released, you can update to the latest or switch back to an older version if you have a need to do so. If you’re new to using the Library Manager, it’s worth to read the official Arduino Library guide, and also check the Library Manager FAQ, to understand how Arduino libraries work, and how manage them (adding, switching, updating, manual library installation, deleting local libraries, etc).

As usual, please report any issues or bugs in the forum or open a Github issue where appropriate.

Custom ArduinoIDE: Boards & Ports toolbar add-on

Have you ever done testing of lots of different boards with the Arduino IDE?
Were you frustrated with having to search your target board and port in the Tools>Boards submenu every time? Do this a few dozen times and you realize this is a major pain and time waste. Especially true if:

  • your boards menu is getting ever longer with every version and new package
  • with every new board there is a new serial port to select
  • you are in a hurry, make a wrong selection, get compile/upload errors, start selecting your board/port from scratch, face-palm!

I think if I add all the time I spent doing that in the past decade I could go on a short vacation in the carribean, wouldn’t that be nice!

I spent some time dusting off my Java skills and understanding the Arduino IDE source code and made this little add-on toolbar:

Here’s a video overview of how this works:

Continue reading

Gateway app updates

There are some notable changes and some new features for the Gateway app – these are published in this Github release. Below is a video overview and a summary of changes:

New settings:

  • serial baud is now 19200 (old: 115200), this is to eliminate any serial corruption when lots of serial data is going in/out to Moteino/MightyHat
  • keepFilesOnDelete (default: false) ensures the binary log files are removed when node/metrics are removed
  • graphMaxPoints – graph data point resolution (total points graphed from raw data)

image

New features

  • List reordering (desktop only) – drag & drop nodes to reorder list & broadcast to all clients:
    image
  • Graph legend div reformat + count of total raw data points that produced a graph are now displayed on top left.
  • Also there is a new Export-Raw-Data button that generates a CSV file from the selected time window. This was requested here.
    image

Wireless Programming just got 50% faster!

The Wireless Programming GUI app that I introduced in an earlier post is now at version 1.2 (get exe here). It features the ability to bundle up to 3 HEX file lines per RF packet (instead of the default 1), which yields a significant transfer speed increase. Let’s just do the TL;DR first and gaze at the results of transferring a compiled sketch of 13,858 bytes:

  • running with 1 line per packet (the same as before) yields 866 packets and ~43s OTA time:
  • running with 3 lines per packet yields 290 packets and ~18s OTA time:

That’s an average 58% speed gain, not too bad!
By comparison, the same sketch uploaded directly via FTDI/serial takes about 13s, so much closer to that figure than before. I think a few other techniques could be used to squeeze yet more speed (increasing RF bitrate!) but that’s for another day.

Note that the same exact data is transferred, except in fewer longer packets. I had to make some changes to the WirelessHEX69 library to support these longer packets. Also I fixed some bugs that emerged during this development, see the github commit for all the changes.

To support the new GUI v1.2 speed increase you only need to update your WP programmer Moteino with a new version of the WirelessProgramming_gateway sketch (to refresh the WirelessHEX69 code) to make it support the faster WP speed (ie also update the WirelessHEX69 lib!). Everything else is completely backwards compatible, and you can use the new GUI v1.2 with an older WP programmer Moteino if need be (1 line per packet only), and also all old target nodes that are OTA/WP programmable are not affected at all and can be updated at the new speed. FWIW you can actually use 1 or 2 (not sure why you would) or 3 lines per packet (the new default going forward). Experiment and see what your own speed gains are! Please report any bugs or issues in this forum!

RFM69_ATC: Automatic Transmission Control

There is now a new extension to the RFM69 library. It’s called RFM69_ATC aka Automatic Transmission Control. Many thanks to Tom Studwell who implemented this and shared it in the forum. The basic idea behind this extension is to allow your nodes to dial down transmission power based on the received signal strength indicator (RSSI). For instance a sleepy battery node like a MotionMote sits still inside the home and usually has a very strong received signal, somewhere in the range of -30 to -40dBm when transmitting at full power. You could manually tune that down using radio.setPowerLevel(..) in code but its tedious and is a static change, if you move the node or conditions change it will not be smart to adjust the power level to the new environment. However with RFM69_ATC this is done automatically for you, you just need to indicate a target RSSI. On each packet sent and ACK received (using sendWithRetry(…) is required), the node analyzes the actual RSSI and continuously adjusts its own transmission power level of the RFM69 transceiver to attempt to match the target RSSI (+ or -). This way that end node is only “loud” enough to be heard by the gateway, not much louder. Close by nodes can “whisper” while nodes farther away or with more obstacles will “speak up” as necessary but avoid that default fixed maximum “screaming” level. This is “polite” in terms of “RF pollution” and efficient in terms of power consumption. Even non-battery nodes where power is no problem should implement this for the sake of the “polite” factor.

I have updated the Node and Gateway examples to have ATC enabled, and also the MotionMote example is now ATC enabled, in this case with a target RSSI of -90dBm. The noise floor is somewhere around -100dBm with the default RFM69 lib settings, so for static nodes that won’t experience a lot of movement or temperature drifts, a -90dBm target is pretty safe. That will keep the transmitter power to a minimum and save power. We know that the greatest toll on a battery powered node are the spikes of current used when the transmitter is active (up to 130mA for RFM69HW). Reducing the transmit power level to the minimum required will exponentially reduce that spike and result in longer battery life and a more “quiet” sensor network that won’t reach across your whole neighborhood. This is really awesome!

Here is a sample transmission using the Gateway and Node examples linked above. Note how the node starts transmitting at full power, then dials down power to match a test RSSI target of -68dBm. This is output from the Gateway end:

ATC

There are a few required changes to a sketch where you want to use ATC:

  • you must #include <RFM69_ATC.h> in addition to #include <RFM69.h>
  • you must use RFM69_ATC radio; instead of RFM69 radio;
  • for the gateway/receiver end the above two changes are sufficient, for the end node that does the power level adjustment you must also do the following:
  • in your setup() function (after all RFM69 initialization is complete) call this function to set your target RSSI: radio.enableAutoPower(targetRSSI);
    • targetRSSI is a negative integer, should be from very strong (-30) to very weak (-95). Usually you would want to be closer to the noise floor end (-100) since you want to reduce transmit power to the bare minimum
    • this is a one time call that enables the dynamic adjustment of the output power on that node
  • you must use radio.sendWithRetry() instead of just radio.send() or a combination of radio.send() and radio.receiveDone() for the radio to be able to receive the important headers from the gateway that tells it how to handle its power, this was done in the MotionMote example linked above

Once these changes are implemented, the node will start to progressively dial down power (assuming it starts transmitting at full power) with each packet sent, until the RSSI meets the target. When the RSSI is below the target power is dialed up again and so on, in an attempt to stay as close to the target as possible.

In the examples that I mentioned there is a pattern that I followed by implementing a define directive (#define ENABLE_ATC) which when left uncommented will enable ATC at compile time, if removed/commented the sketch will run normally without ATC. As always, bug reports, suggestions and contributions are welcome, the forum is the best place for that purpose.

I must mention here that at this time I have only partially implemented his variant with some adjustments to Tom’s implementation (I left the differences commented out in my version). For now I chose to leave my power control resolution as is. He actually went into more detail allowing a finer control of the power control – this is because of the differences between RFM69W and HW which handle output power control differently. If you’d like to try his variant with that extra control check this forum post or his Github repo. Thanks Tom for your great contributions and inspiring this piece of work and also the awesome forum projects you’ve posted!

Gateway Source Code Released

A few weeks went by since I blogged about the new Gateway interface. I managed to get the code ready and I posted the links on the dedicated gateway page where I prepared a guide to install the stack needed to support it (this will be improved). The code itself and sources are published at Github. There you will find the old interface sample I had posted a long time ago (in the OLD subdirectory, kept there for reference); the files are replaced with the new interface files.

There are 2 directories where you will need to copy files from here after you’ve completed the setup of your Pi from this guide (I may release a Pi image that has everything setup and ready to go). Please note: when you download stuff from Github you must use the Download Zip button in the repository rather than downloading individual files.
First there is the /var/www /default directory, you should copy these files there:

Second there is the /home/pi/moteino directory, you should copy these files there:

Once the gateway starts running it will create the databases for you and start logging data as it comes in from your remote nodes through the gateway Moteino attached to your Pi’s serial port configured in the gateway.js script.

This post is not meant to replace the guide found at the lowpowerlab.com/gateway page. I will post further details there. I created a new forum for discussing this and other gateway related topics so please post your questions there.

Moteino now on CodeBender.cc!


IMPORTANT NOTE: unfortunately codebender announced they are shutting down on or after Oct. 20, 2016. Hence forth this page is only kept for reference and it will eventually be removed. The codebender programming instances below might become inactive, display an error or show a blank space once codebender shuts down these services. You are encouraged to start programming or move your code in the Arduino IDE.


This is really awesome news!
The guys at codebender.cc have done a great job creating an online place to create, edit, store, share your Arduino sketches. It’s very easy to install and use right in your browser!

UPDATE: Check the new Moteino Programming Guide.

And now Moteino has been added to their fleet of supported devices, along with the RFM69, SPIFlash libraries. Once your Moteino is connected to your computer, you can edit your sketch, or load the example(s) of the many integrated libraries (like the Node example in the RFM69 library) edit the settings to match your hardware and upload it to your board. Another awesome feature of codebender is embeddable sketches, that’s right, any sketch can be embedded in another web page, like the Node example below, you can upload it right from this blog post (see below for browser plugin requirements, or check this getting started guide):

To be able to use CodeBender you will need to create an account, then install a browser plugin that allows access to your computer’s serial ports, which in turn make it possible to reset and upload sketches to your Moteinos (or whatever other board you may have and they support). Also you need to choose your target board, protocol (USBtinyISP for Moteino) and serial port it’s attached to, then click “Run on Arduino” and that will compile/upload the sketch. That’s about it. Please check their getting started guide as well.

I warmly recommend using CodeBender with Moteinos since it’s easy and much nicer than the clunky Arduino IDE, and it’s probably going to get even better as they add more features. Plus I will work with them to ensure the libraries are in sync with the latest releases, and new examples will also be posted on CodeBender. Of course, you can always add your own sketches to your CodeBender repository, just like you did in Arduino, or you can clone an example and load your own settings and modify the code as you like. I will probably start using embedded sketches whenever I want to share code since it allows CodeBender users to upload it right from this blog rather than going to GitHub, copy pasting, making sure they got all the latest updated libraries, etc.

To celebrate this great milestone I am reducing the pricing on all Moteinos by $2, a saving of 10-15% depending on your options, to last through the end of December!

Happy CodeBending!