Reached heap limit Allocation failed - JavaScript heap out of memory

[Lukapple]

I have been running my Moteino Gateway without any problems for several years, but in the last few days, it has started to crash with the following error message:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory.

Any idea how to debug this?

Log:
[code]
[04-08-23_10:30:27.908] [LOG]    MQTT client successfully connected to broker
[04-08-23_10:30:27.912] [LOG]    MQTT client successfully connected to broker
[04-08-23_10:30:28.041] [LOG]    **** DB COMPACTION DONE
[04-08-23_10:30:28.043] [LOG]    **** DB COMPACTION DONE
[04-08-23_10:30:28.044] [LOG]    **** DB COMPACTION DONE
[04-08-23_10:30:28.045] [LOG]    **** DB COMPACTION DONE
[04-08-23_10:30:28.058] [INFO]   **** SCHEDULING EVENT - nodeId:11 event:sendTimeSyncDMD to run in ~16h29m
[04-08-23_10:30:28.059] [INFO]   **** SCHEDULING EVENT - nodeId:201 event:weatherStationPoll to run in ~10m
[04-08-23_10:30:28.060] [INFO]   **** SCHEDULING EVENT - nodeId:202 event:seaTemperaturePoll to run in ~10m10s
[04-08-23_10:30:28.110] [INFO]   [11] DB-Updated: schedule(sendTimeSyncDMD):dbNode={"_id":11,"updated":1617973191476,"label":"DMD display","descr":"Bazen tabla","metrics":{"START":{"label":"START","value":"Started","updated":1617973191476},"RSSI":{"label":"RSSI","value":-74,"unit":"db","updated":1617973191476,"graph":1},"TIMESYNC":{"label":"TIMESYNC","value":"SNC","updated":1617804977648,"pin":1,"graph":1}},"events":{"sendTimeSyncDMD5":{"enabled":0},"sendTimeSyncDMD":{"enabled":1,"executeDateTime":"2023-04-09T01:00:00.001Z"},"timeSyncListener":{"enabled":1}}}
[04-08-23_10:30:28.113] [INFO]   [201] DB-Updated: schedule(weatherStationPoll):dbNode={"_id":201,"updated":1680456050495,"label":"Temperatura/de

[Felix]

Something in the app is eating memory and hitting the allowed limit. If you have any custom code I would look into that first for any potential memory leaks.
I have also been running it for years without problems, have not seen this type of error.

[Lukapple]

The issue was actually caused by the gateway.db database file. During the compacting process, the CPU usage spiked to 100% and eventually ran out of memory. However, once the gateway.db file was deleted, everything returned to normal.

[Felix]

That's strange, how large was that database at time of compaction?
Compacting it more often would probably be the best approach.

[Lukapple]

Apologies for the delayed response.

The gateway.db file was only a few KB in size. The largest file, gateway_nonmatches.db = 52M. I believe the issue was connected to JSON parsing. While debugging the process, I encountered some unexpected nil errors, although I can't recall the specifics. I have a backup of that database, and if you're interested, I can send it to you for analysis.

Code Select Expand

199K - 0002_RSSI.bin
240K - 0002_Status.bin
223K - 0003_RSSI.bin
362K - 0003_Status.bin
354K - 0004_M.bin
3.0M - 0004_RSSI.bin
489K - 0004_V.bin
1.1M - 0006_C.bin
1.2M - 0006_RSSI.bin
368K - 0006_V.bin
207B - 0007_CM.bin
1.5M - 0007_Količina.bin
846K - 0007_RSSI.bin
332K - 0008_M.bin
2.9M - 0008_RSSI.bin
4.4M - 0008_V.bin
7.3K - 0009_RING.bin
 23K - 0009_RSSI.bin
 14K - 0010_Alarm.bin
1.6M - 0010_RSSI.bin
3.6K - 0011_RSSI.bin
189B - 0011_TIMESYNC.bin
8.5K - 0012_Alarm.bin
 11K - 0012_RSSI.bin
1.1K - 0013_HB.bin
 21K - 0013_Mailbox.bin
192K - 0013_RSSI.bin
 20K - 0013_Stanje vrat.bin
169K - 0013_Status.bin
3.4K - 0014_Alarm.bin
6.6K - 0014_HB.bin
 48K - 0014_M.bin
 50K - 0014_RSSI.bin
8.4K - 0015_HB.bin
9.1K - 0015_RSSI.bin
5.8K - 0016_HB.bin
543K - 0016_RSSI.bin
620K - 0016_Vlaga.bin
 18B - 0091_RSSI.bin
1.1M - 0201_C.bin
858K - 0201_Rain.bin
321K - 0202_C.bin
7.8K - gateway.db
 52M - gateway_nonmatches.db

[Felix]

The gateway.db being small is a good thing and I see no problem with parsing that.
I would delete the non_matches db because that's not really useful at this point, you're not going to try to harvest any of that data.
If this keeps happening then it might be worth more investigation, although, I have never ran into this type of error, the software is very stable.

[Lukapple]

Thanks for the suggestions. If the error occurs again, I will dedicate more time to analyzing and identifying the cause of the error.