Pi Gateway becomes unresponsive - reboot fixes it

[Tony Weil]

On two occasions in the last few weeks my Pi Gateway has become unresponsive. Reboot by pushing Mighty Hat shutdown button works. Is there anywhere I can look at logs or something to try and figure out what happened?

Can anyone suggest a good way to monitor the Pi Gateway for failure? I only discover the problem when I try to load the web page or notice I'm not getting SMS when garage opens. I could set up a computer to ping it and send alert if it fails, but I don't really want to dedicate a computer to it. I suppose I could set up another Pi to do that.

When the Pi Gateway becomes unresponsive, I notice the following status. Everything on the Pi and MightyHat hardware (LEDs and LCD) look normal:

Pi 3 B+:
Red power light is on
Green Activity light flashes occasionally
Hooked up a monitor and keyboard, but nothing on screen.
Can't ping the local IP address.
Can't SSH to the Pi

MightyHat:
Mighty Hat is on and receiving data on LCD
5Von green LED (Pi power) is on
Square Pi on/off button is green
Boot OK LED is on
Boost LED is on
Backup Lipo battery is fully charged

[HeneryH]

On the monitoring topic.  I use https://anturis.com/ free plan which offers something like 5 minute intervals for up to about 5 probes.  You can use either client probe apps or you can to a simple http probe.

[Tony Weil]

I have tried some of the free online monitoring services with no luck.  I have port forwarding set up on my router so the Pi Gateway is accessible from the Internet at WAN IP address of my router and am able tologin from anywhere on the Internet.  I assume this is a required step or is there another way to do this. I am using the self signed cert that is the default  Pi Gateway setup. The cert or something's else is making the web site appear "down" to the monitoring services. Whenever I try to login to the gateway manually from Chrome or Safari I always get warning about the cert that I have to click through. How can I make this work? Thanks

[HeneryH]

I have tried some of the free online monitoring services with no luck.  Anturis (not sure about the others) just simply emulates a web connection either with or without login credentials.   Without the login credentials, it just checks to see if the web server responds with anything, with the credentials it checks to see if it can log in with the credentials you provide the monitor service.  There isn't much to go wrong there.  It can either reach and login to the server or not.

[Tony Weil]

I was able to get the monitoring working. I don't know what my problem was, probably rushing. Of the free services, I set up 3 different ones to test them. All were pretty good for a free service.

On all of these services the default check is between every 1 and 5 minutes. I found that Comcast has a very unreliable connection, especially in the middle of the night, not that I really care if my Pi Gateway is available from the Internet 24x7. I changed the checks to 30 minutes to minimize alert noise.

Anturis works well. The setup is a little cryptic and I needed to uncheck the default HTTP status since this is an HTTPS server. Maybe I goofed somewhere. It has the option for authentication and checking page content like "IoT Gateway Dashboard". It has basic mail notification, but no IOS app.

My.Pingdom only provides 1 free monitor and a second free one if you send a tweet advertising their product. The interface is nice, although most menu items require an upgrade which makes the interface busy and annoying. It has the option for authentication and checking page content like "IoT Gateway Dashboard". It has basic mail notification and a really nice IOS app with alerts on the phone.

Uptime Robot provides 50 free monitors and is similar to my.Pingdom including authentication I, but no content checking I could find. It has basic mail notification and a really nice IOS app with alerts on the phone. It also includes a "TV Mode" for displaying stats on a large monitor such as in a help desk environment.

[Felix]

How about a hardware watchdog?

[Tony Weil]

How about a hardware watchdog?

Can you please elaborate on this approach?

[Felix]

We could implement the atmega's reset watchdog. It's a step "below" an actual reset button press.
Nevermind, I somehow was thinking this as MightyHat locking up.

If the software itself is locking up then that's a potential bigger problem. Either some bug I havent found, or most often its something the user tries to implement by copy-pasting-adjusting and they introduce a bug that way, which may only trigger when an event fires or some time later. Not the easiest to debug but the gateway.sys.log is the way to check for such errors.

[HeneryH]

FYI - I've built two recent systems based on the stretch-lite method of building the pi and the MightyHat.  Both have been rock solid.   One is a newer 3b+ flavor while the other is an older A+.

This isn't a proof that it is perfect but I offer two recent data points.