Search | Sailfish OS | Running | PineTime | All Posts

InfluxDB and Raspberry Pi - frenemies?

June 21, 2020 — Nico Cartron

A quick look at why I moved my InfluxDB data away from one of my Raspberry Pi.

I have been running InfluxDB as one of my time-series database, to collect various information, such as home automation sensors (temperature, humidity, ...) as well as pushing OPNsense metrics to Telegraf.

When I initally set it up, I didn't really spend a lot of time on deciding on which system I would install it: since I had a Raspberry Pi 3 with Raspbian, running my Domoticz and Grafana, I just installed InfluxDB on it.
The only precaution I took was to use a dedicated USB key for InfluxDB, as I didn't want it to kill the Pi's micro-SD card (InfluxDB obviously writes a lot of data to disk).

This setup ran perfectly for more than a year, and then all of sudden, it failed.
By failed I mean the Pi was not reachable anymore, either by HTTP or SSH.
Pluging a screen/keyboard into the Pi, I saw that it was not happy with the USB key (it couldn't find it and failed booting), so I assumed the USB key was dead and replaced it.
I then copied a backup of my InfluxDB data (yes, it's always a good idea to have backups ;) and restarted InfluxDB, thinking "problem fixed", and not even looking at the logs, as it was obvious to me that the problem was the USB key which I replaced.

Trying to look at my Grafana dashboard, I got the infamous Bad Gateway 502 error message.
Humf, "problem's not fixed" and I did some more troubleshooting:

  • stopping and starting InfluxDB was OK
  • but a netstat -an |grep LISTEN would not list InfluxDB listening on port 8086
  • running a top, I found out that the load average was increasing a lot, before dropping
  • and so did the memory - the Pi 3 has only 1GB of RAM, so it can be eaten very fast! Which was the case here

Attempt #1

Looking for similar issues, I found this one on GitHub; quite long and with many people having the same issue as me.

This comment was interesting, and recommended to:

  • change InfluxDB format to tsi (see this page for more details on tsi
  • add swap space to the Pi (by setting CONF_SWAPSIZE=2048` in /etc/dphys-swapfile) - and then reboot
  • set index-version="tsi1" in the influxdb.conf file

I tried that and had some hopes, but I still have the same issue when looking at InfluxDB journal (`journaltcl -fu influxdb); after reading the data and trying to compress them, InfluxDB was crashing and restarting, with the below message:

Jun 20 18:47:36 domoticz influxd[1173]: runtime: out of memory: cannot allocate 8192-byte block (988643328 in use)

This other comment mentioned moving away from Raspbian and installing an unofficial Ubuntu 18.04 64-bit build, which seems to work much better.
Since I'm using this Pi not only for InfluxDB but for other applications, I didn't want to do that.

Attempt #2 - Fixing it for real!

I had a "sleeping" server running Debian 10, with 32GB of RAM, meaning more than enough for InfluxDB needs.

So I ended up:

  • installing InfluxDB on it,
  • restoring my backup,
  • pointing Grafana and OPNsense to this new server (instead of the Pi).

And voilĂ , problem fixed for good this time :-)

Probably with a Pi4 and Ubuntu 64-bit, this would have worked fine, but that would have meant also reinstalling the other applications, which I didn't want to do.

Tags: Tips, IT, Opensource

I don't have any commenting system, but email me (nicolas at ncartron dot org) your comments!
If you like my work, you can buy me a coffee!