[SOLVED] gateway service fails when logfile gets large

[kni]

Using this version of moteino gateway:
$ git rev-parse --short HEAD
39114ac

A few weeks ago my gateway service failed. Restarting would cause the service to start but would fail again after ~5 seconds.  After some troubleshooting, I deleted the log file and then noticed the service remained running. I didn't think much of it at the time.

This just happened again. I noticed the service had failed. Trying to start it, gives the same symptoms as a few weeks ago. The service would start but then fail again:

Code: [Select]

$ sudo systemctl start moteino-gateway
$ sudo systemctl status moteino-gateway
● moteino-gateway.service - Moteino Gateway Service
   Loaded: loaded (/etc/systemd/system/moteino-gateway.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-07-03 11:26:15 CDT; 216ms ago
  Process: 5162 ExecStop=/bin/kill -s SIGTERM $MAINPID (code=exited, status=1/FAILURE)
 Main PID: 5172 (sh)
    Tasks: 7 (limit: 4915)
   Memory: 5.4M
      CPU: 212ms
   CGroup: /system.slice/moteino-gateway.service
           ├─5172 /bin/sh -c /usr/bin/node /home/abauer/git/RaspberryPi-Gateway/gateway.js >> /var/log/moteino-gateway.log 2>&1
           └─5176 /usr/bin/node /home/abauer/git/RaspberryPi-Gateway/gateway.js

Jul 03 11:26:15 build-droid systemd[1]: Started Moteino Gateway Service.


abauer@build-droid:~$ sudo systemctl status moteino-gateway
● moteino-gateway.service - Moteino Gateway Service
   Loaded: loaded (/etc/systemd/system/moteino-gateway.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2018-07-03 11:26:20 CDT; 2s ago
  Process: 5270 ExecStop=/bin/kill -s SIGTERM $MAINPID (code=exited, status=1/FAILURE)
  Process: 5241 ExecStart=/bin/sh -c /usr/bin/node /home/abauer/git/RaspberryPi-Gateway/gateway.js >> /var/log/moteino-gateway.log 2>&1 (
 Main PID: 5241 (code=exited, status=1/FAILURE)
      CPU: 1.584s

Jul 03 11:26:20 build-droid systemd[1]: moteino-gateway.service: Failed with result 'exit-code'.
Jul 03 11:26:20 build-droid systemd[1]: moteino-gateway.service: Service hold-off time over, scheduling restart.
Jul 03 11:26:20 build-droid systemd[1]: Stopped Moteino Gateway Service.
Jul 03 11:26:20 build-droid systemd[1]: moteino-gateway.service: Start request repeated too quickly.
Jul 03 11:26:20 build-droid systemd[1]: Failed to start Moteino Gateway Service.
Jul 03 11:26:20 build-droid systemd[1]: moteino-gateway.service: Unit entered failed state.
Jul 03 11:26:20 build-droid systemd[1]: moteino-gateway.service: Failed with result 'exit-code'.

Starting the service directly works:

Code: [Select]

abauer@build-droid:~$ sudo /usr/bin/node /home/abauer/git/RaspberryPi-Gateway/gateway.js

I've got plenty of space on the disk:

Code: [Select]

abauer@build-droid:~$ df -lh
Filesystem      Size  Used Avail Use% Mounted on
udev            930M     0  930M   0% /dev
tmpfs           200M  6.6M  193M   4% /run
/dev/sda1       115G   16G   94G  14% /
tmpfs           997M     0  997M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           997M     0  997M   0% /sys/fs/cgroup
tmpfs           997M     0  997M   0% /tmp
/dev/mmcblk1p1   15G  1.4G   13G  10% /media/mmcboot
log2ram          50M   15M   36M  30% /var/log
tmpfs           200M     0  200M   0% /run/user/1001

I noted my log file was 36MB. After I deleted it, the service would start and stay started.

I'm curious if anyone else has seen this behavior. The next time it happens, I need to take a closer look at the syslog and moteino log.

[kni]

And... I think I just answered my own question.
Look at how much free space is in the log2ram filesystem shown above.... 36MB.

[kni]

In my case, I migrated my Odroid over to an SSD, rather than use the sd card, so I do not have a need for the log2ram service.

I chose to disable it:

sudo systemctl stop log2ram
sudo systemctl disable log2ram

Problem solved.

[Felix]

Thanks for sharing this, might come useful for others!