Upload Error on a Moteino USB R4 and loosing Digital Pins control

[luisr320]

Hi.
After uploading a sketch into a USB Moteino via USB (my Gateway), I get this error:

Code Select Expand

avrdude: reading on-chip flash data:
Reading | ################################################## | 100% 3.68s
avrdude: verifying ...
avrdude: verification error, first mismatch at byte 0x007b
         0x80 != 0x00
avrdude: verification error; content mismatch
avrdude: safemode: lfuse reads as 0
The uploader process failed
avrdude: safemode: hfuse reads as 0
avrdude: safemode: efuse reads as 0
avrdude: safemode: Fuses OK (H:00, E:00, L:00)
avrdude done.  Thank you.

If I use any other Moteino, all goes well. The script load and all seems to be working fine. However, I've been dealing with a major problem for the last month or so, where all my nodes stop responding after a while (couple of hours). I would still get the ACK back but the digital pins weren't activated.
I have replaced it by another one and all works fine now.

Not really sure what is the problem but I nearly pulled every single one of my hairs finding out what was wrong.

I have the latest libraries and cores for the Moteino installed.

Is this Moteino broken? Any suggestion to make it behave?

Luis

[Felix]

What revision is this, and approx when was it purchased?

If I use any other Moteino, all goes well.

Is that another MoteinoUSB?

There have been changes to RFM69 lib lately, no reports of misbehavior, no issues that I can detect on my own either.

[luisr320]

I think I bought that USB moteino in 2015. It's an R4 USB. It still works well after giving the 0x007b byte error. I was just wondering if I could do something to it to clear that error.

Anyway, that was a dead end. I though it was the reason for ALL my moteinos acting weird and not responding to the Gateway but that's not it. I replaced it with a different moteino and the problem is still there. I just can't figure out why I'm having this specific problem.

The symptom is this: During some hours after I re-upload a sketch, or if I reset a node by powering off then on, it works well, and all the digital outputs work properly. But after some time, 2, maybe 3 hours after that, some nodes stop responding to the Gateway commands and eventually, all of them stop responding. I can ping them and receive an ACK back, so I know they can receive and transmit. Its only the digital pins that don't work anymore as outputs.

But the really funny thing is that the node that was in my car, that I use to open the garage door and the outer gate by flashing the lights, that has a direct link to the nodes, still worked flawlessly even when the gateway stopped to be able to control the nodes. I had an old version of the RFM69 library in the car node, with the #include <wirelesslHEX69.h> library instead of the OTA.h. BUT... after I uploaded the same exact sketch that it had before but with an updated RFM69 library with the OTA.h, it now starts to loose control of the gates nodes too.

I'm recovering the original old version of the RFM69 that I used to program the car node and will upload a new sketch in there to see if it regains control of the gates and come back with the results. Any ideas?

[Felix]

Yes we need to try to eliminate hardware as a fault.
We need to come to a point where we can compare a working setup and non-working, and determine what the real problem is.
I would go back with an older RFM69 version, all versions and releases are in github history: https://github.com/LowPowerLab/RFM69/
Not sure what else to suggest.

RE The MoteinoUSB - Is it the version that includes two 27 ohm resistors under the FTDI chip (FT231XS)? Or without?

[luisr320]

Oh boy. Now I really messed everything up. I uploaded a new sketch to each moteino with an older version of the RFM69 with the WirelessHEX69.h instead of the RFM69_OTA.h and now, even though it gives no error on uploading wirelessly, the old sketch is not replaced if I try to upload something else in them. So I guess I have to dig them out and manually load a working sketch with the RFM69_OTA.h
This is driving me nuts.

[luisr320]

Regarding the USBMoteino that gives the error, I'm not sure of what resistors you are talking about. Here's a photo:

[HeneryH]

R4 USB Motes have a design problem with missing resistors and a pull-up that make it so that I have never been able to get any of my computers to successfully upload to them.

[luisr320]

I can upload both via the USB cable and via OTA. It's just that I never have noticed that error before and it's probably there since day one. But other then that it works fine.

[Felix]

R4 USB Motes have a design problem with missing resistors and a pull-up that make it so that I have never been able to get any of my computers to successfully upload to them.

That's sometimes, not always.
At least that's a known quirk.
Luis mentioned he has other new nodes that can upload without any messages, but behave in a way he doesn't expect.
I would be more concerned if many people would report the same problem.
This sort of hints at the possibility (small though) the gateway (or the nodes) are drifted out of frequency, they sort of work but not repeatedly.
At the very least try to make all the nodes run the same library version. Even though... every time I make changes I test against nodes running older lib versions, and I have nodes that are very very "old" (ie old/aged xtals).

[luisr320]

So I set up a spank new moteino on my desk with the same firmware I have on my Gate control node, with a couple of LEDs to monitor the digital output pins behavior in parallel with my gate controller.

I can see in my Homeseer Log (my home automation system) that the protocol to control the gate was sent, received and acknowledged.
Initially I get both moteinos to work well, the doors open and close and the leds on the one on my desk blink accordingly. But after some hours, they both stop responding.



I replaced the Gateway with a new moteino and I still can't control the gate or get the LEDs to blink.

I then turned of the one on my table just to make sure it wasn't creating a problem and the result is the same.

Turning the moteino in my table back on makes it now responsive to the gateway but not the one on my gate.

By turning the gate moteino OFF then ON, I can now control it for a couple of hours.

[Felix]

Wait, I thought you said:

I can ping them and receive an ACK back, so I know they can receive and transmit. Its only the digital pins that don't work anymore as outputs.

So it's the digital inputs having some problem. Right?
The radio link is still good, correct?

If so, can you add some blinking on the receiving nodes, that loose control, and/or check with a DMM if the pins actually flip.

[luisr320]

The radios work fine. I have added some Serial.print statements to the moteino on my desk to monitor if when it stops working I still geral the correct control protocol message trough. Now all I have to do is wait for it to fail again and I'll measure the voltage at those output pins. And I'll add some more debug message to see exactly what is received:

Code Select Expand

// **********************************************************************************************************
// Outer Gate Node 
// Ver. 3.3
// 2014-05-22 (C) [email protected]
// Adapted from [email protected], http://www.LowPowerLab.com mote.ino
// **********************************************************************************************************
// This sketch parses a command received from the Gateway to make pin 16 or 18 pulse high/low, thus pulsing the Outer Gate relays to open or close the gate.
// **********************************************************************************************************
// Creative Commons Attrib Share-Alike License
// You are free to use/extend this code/library but please abide with the CCSA license:
// http://creativecommons.org/licenses/by-sa/3.0/
// **********************************************************************************************************

#include <RFM69.h>         //get it here: https://github.com/lowpowerlab/RFM69
#include <RFM69_OTA.h>     //get it here: https://github.com/lowpowerlab/RFM69
#include <SPIFlash.h>      //get it here: https://github.com/lowpowerlab/spiflash
#include <SPI.h>           //included with Arduino IDE install (www.arduino.cc)

#define NODEID 5 //unique for each node on same network
#define GATEWAYID 1 //Gateway number on this network
#define NETWORKID 100 //the same on all nodes that talk to each other
#define FREQUENCY RF69_433MHZ
#define ENCRYPTKEY "encryptkey" //exactly the same 16 characters/bytes on all nodes!
#define ACK_TIME 3000 // max # of ms to wait for an ack
#define LED 9 // Moteinos have LEDs on D9
#define RELAY1 5 // Output pin connected to the relay
#define RELAY2 7 // Output pin connected to the relay
#define SERIAL_BAUD 115200

// Hardware Init
SPIFlash flash(8, 0xEF30); //EF40 for 16mbit windbond chip
RFM69 radio; // Call the radio

			 // Outbound Data Struct
typedef struct
{
	char data_Type[6];
	long int out_data1;
	float out_data2;
	float out_data3;
} Payload;
Payload outData;

//Inbound Data Struct
typedef struct
{
	long int in_data;  // Command received from Gateway
} in_Payload;
in_Payload inData;

//Initialize some variables
bool promiscuousMode = false; //set to 'true' to sniff all packets on the same network
long lastPeriod = 0;
int transmitPeriod = 15000; //transmit a ping to gateway so often (in ms)

void setup()
{
	//Setup the pins
	pinMode(LED, OUTPUT);
	pinMode(RELAY1, OUTPUT);
	pinMode(RELAY2, OUTPUT);

	//Reset the in_data variable to 0
	inData.in_data = 0;

	//Start the Serial port
	Serial.begin(SERIAL_BAUD);

	//Initialize the radio
	radio.initialize(FREQUENCY, NODEID, NETWORKID);
	radio.setHighPower(); //only for RFM69HW!
	radio.encrypt(ENCRYPTKEY);//Turn encryption ON
	radio.promiscuous(promiscuousMode);
}

void loop()
{
	/////////////////////////////////////////////////////////////////////////////
	//**********CHECK IF ANY INSTRUCTIONS WERE RECEIVED FROM HOMESEER**********//
	/////////////////////////////////////////////////////////////////////////////
	if (radio.receiveDone())//If some packet was received by the radio, wait for all its contents to come trough
	{
		//////////////////////////////////////////////////////////////////////////
		//               Check if data is wirelees programing                   //
		//////////////////////////////////////////////////////////////////////////
		CheckForWirelessHEX(radio, flash, true); // set to true for debug. This will check if there is a new wireless program to load

		if (radio.TARGETID == NODEID)//Check if the packet destination is this radio (NODE 5)
		{
			//Extract data from the incoming payload
			inData = *(in_Payload*)radio.DATA;


			Serial.println(inData.in_data);

			//////////////////////////////////////////////////////////////////////////
			//            Process the received data from the Gateway                //
			//////////////////////////////////////////////////////////////////////////
			if (inData.in_data == 1)// If a command was sent from the Gateway to toggle RELAY1, a "1" was received
			{
				digitalWrite(RELAY1, HIGH);
				Serial.println("RELAY1-ON");
				delay(500);
				digitalWrite(RELAY1, LOW);
				Serial.println("RELAY1_OFF");
			}
			else if (inData.in_data == 2) // If a command was sent from the Gateway to toggle RELAY2, a "2" was received
			{
				digitalWrite(RELAY2, HIGH);
				Serial.println("RELAY2-ON");
				delay(500);
				digitalWrite(RELAY2, LOW);
				Serial.println("RELAY2_OFF");

			}
			/*
			else if (in_theData.in_data == 3)//Alarm received
			{
				strcpy(theData.dataType, "PANE!");
				outData.out_data1 = 1;
				outData.out_data2 = 0;
				outData.out_data3 = 0;
				radio.sendWithRetry(GATEWAYID, (const void*)(&theData), sizeof(theData), 5, ACK_TIME);
			}
			*/
			inData.in_data = 0; //Reset the in_data Variable content
			Blink(LED, 25);//Blink the transmitter led to show the data was successfully received
			if (radio.ACKRequested())
			{
				radio.sendACK();
			}
		}
	}

	//////////////////////////////////////////////////////////////////////////
	//                          Ping the Gateway                            //
	//////////////////////////////////////////////////////////////////////////

	int currPeriod = millis()/transmitPeriod;
	if (currPeriod != lastPeriod)
	{
		lastPeriod=currPeriod;

		//fill in the struct with new values
		strcpy(outData.data_Type,"PING!");
		outData.out_data1 = 99999999;
		outData.out_data2 = 0;
		outData.out_data3 = 0;

		if (radio.sendWithRetry(GATEWAYID, (const void*)(&outData), sizeof(outData), 2, ACK_TIME))
		{
			Blink(LED,25);//Blink the node led to show a successfully ping
		}
	}
}


void Blink(byte PIN, int DELAY_MS)//Local led blinking function
{
	pinMode(PIN, OUTPUT);
	digitalWrite(PIN, HIGH);
	delay(DELAY_MS);
	digitalWrite(PIN, LOW);
}

[Felix]

maybe worth doing a before/after comparison, as well as with the pin as connected in your project, then with it disconnected (to see if the load perhaps has any influence on the pin).
BTW - what is your estimated load on those pins that stop working?

[luisr320]

So this morning I checked the COM Monitor connected to the moteino I have on my desk (running in parallel with my troublesome gate node) and I could see all the times that the loop() resets the control variable to 0 displayed every 15 seconds, as a result of the pings sent to the gateway every 15 seconds. This happens because when a ping is sent, an ACK is sent back to the node and satisfies the if (radio.receiveDone() and the if (radio.TARGETID == NODEID)//Check if the packet destination is this radio (NODE 5) and at the end hitting the variable reset instruction, inData.in_data = 0; //Reset the in_data Variable content and resetting it to 0. So there are a bunch of zero there showing that all this worked properly.

But then, somewhere during the night, the 0 becomes a -1338048512. And now, if I send a command to open a door, a 1, it becomes a -1338048511 and a command to close the door, a 2 becomes a -133804850.

But these numbers vary a bit. Here is a log for a 15 second period:

Code Select Expand

2019-01-26 05:07:06	Message	-1338002432
2019-01-26 05:07:15	Message	-1338048510
2019-01-26 05:07:15	Message	-1338048510
2019-01-26 05:07:21	Message	-1338048512
2019-01-26 05:07:21	Message	-1338002412
2019-01-26 05:07:36	Message	-1338002432
2019-01-26 05:07:36	Message	-1338002412

And here is when the transition happened:


Looking at the COM Port Logs I can see that it took 10 hours to to fail.
And it looks like it is an overflow problem.

The variable used to hold the control data is a long int. I don't really need a long int. A uint8_t would hold numbers from 0 to 255 and since the control data is a 1 or a 2 (or a 0 when nothing is going on), that should be enough. And looking at all the other nodes that control relays and that are giving me this same problem, they all have that in common: the control variable is a long int.

So I will change that variable to uint8_t and wait to see if it still overflows.

But even if this works, the mystery is not solved. Why is it overflowing to a weird number like that?
Here is a reference list of the maximum values that a variable in C++ can have:

boolean (8 bit) - simple logical true/false byte (8 bit) - unsigned number from 0-255 char (8 bit) - signed number from -128 to 127. unsigned char (8 bit) - same as 'byte'; if this is what you're after, you should use 'byte' instead, for reasons of clarity word (16 bit) - unsigned number from 0-65535 unsigned int (16 bit)- the same as 'word'. Use 'word' instead for clarity and brevity int (16 bit) - signed number from -32768 to 32767. unsigned long (32 bit) - unsigned number from 0 to -4,294,967,295. The most common usage of this is to store the result of the millis() function, which returns the number of milliseconds the current code has been running long (32 bit) - signed number from -2,147,483,648 to 2,147,483,647 float (32 bit) - signed number from -3.4028235E38 to 3.4028235E38.

In my code I have assigned millis() to an int variable. So that will clearly be a problem too. Could it be the root problem? Let's see...

[Felix]

And it looks like it is an overflow problem.
In my code I have assigned millis() to an int variable. So that will clearly be a problem too. Could it be the root problem? Let's see...

It probably is. A number is just a bunch of bits. How it is interpreted (both on the node, and your gateway) depends on how the variables are defined, signed or unsigned etc, and if any casting is involved. An unsigned int of any length is never negative. So this behavior (after 10 hours) suggests some kind of memory overflow/corruption or logical bug. It's pretty common that a variable counting time (otherwise defined properly) to be assigned to another and through casting to become negative.