InfluxDB and Raspberry Pi - frenemies?
A quick look at why I moved my InfluxDB data away from one of my Raspberry Pi.
I have been running InfluxDB as one of my time-series database, to collect various information, such as home automation sensors (temperature, humidity, ...) as well as pushing OPNsense metrics to Telegraf.
When I initally set it up, I didn't really spend a lot of time on deciding on which system I would install it: since I had a Raspberry Pi 3 with Raspbian, running my Domoticz and Grafana, I just installed InfluxDB on it.
The only precaution I took was to use a dedicated USB key for InfluxDB, as I didn't want it to kill the Pi's micro-SD card (InfluxDB obviously writes a lot of data to disk).
This setup ran perfectly for more than a year, and then all of sudden, it failed.
By failed I mean the Pi was not reachable anymore, either by HTTP or SSH.
Pluging a screen/keyboard into the Pi, I saw that it was not happy with the USB key (it couldn't find it and failed booting), so I assumed the USB key was dead and replaced it.
I then copied a backup of my InfluxDB data (yes, it's always a good idea to have backups ;) and restarted InfluxDB, thinking "problem fixed", and not even looking at the logs, as it was obvious to me that the problem was the USB key which I replaced.
Trying to look at my Grafana dashboard, I got the infamous Bad Gateway 502 error message.
Humf, "problem's not fixed" and I did some more troubleshooting:
- stopping and starting InfluxDB was OK
- but a
netstat -an |grep LISTENwould not list InfluxDB listening on port 8086 - running a
top, I found out that the load average was increasing a lot, before dropping - and so did the memory - the Pi 3 has only 1GB of RAM, so it can be eaten very fast! Which was the case here
Attempt #1
Looking for similar issues, I found this one on GitHub; quite long and with many people having the same issue as me.
This comment was interesting, and recommended to:
- change InfluxDB format to
tsi(see this page for more details on tsi - add swap space to the Pi (by setting CONF_SWAPSIZE=2048` in /etc/dphys-swapfile) - and then reboot
- set
index-version="tsi1"in the influxdb.conf file
I tried that and had some hopes, but I still have the same issue when looking at InfluxDB journal (`journaltcl -fu influxdb); after reading the data and trying to compress them, InfluxDB was crashing and restarting, with the below message:
Jun 20 18:47:36 domoticz influxd[1173]: runtime: out of memory: cannot allocate 8192-byte block (988643328 in use)
This other comment mentioned moving away from Raspbian and installing an unofficial Ubuntu 18.04 64-bit build, which seems to work much better.
Since I'm using this Pi not only for InfluxDB but for other applications, I didn't want to do that.
Attempt #2 - Fixing it for real!
I had a "sleeping" server running Debian 10, with 32GB of RAM, meaning more than enough for InfluxDB needs.
So I ended up:
- installing InfluxDB on it,
- restoring my backup,
- pointing Grafana and OPNsense to this new server (instead of the Pi).
And voilà, problem fixed for good this time :-)
Probably with a Pi4 and Ubuntu 64-bit, this would have worked fine, but that would have meant also reinstalling the other applications, which I didn't want to do.
Tags: Tips, IT, Opensource