Nico's blog

PowerDNS LMDB Backend

In this article, I want to compare the performances of a PowerDNS Authoritative server using a SQL backend, versus one using a LMDB backend.

---

LMDB Backend

What is LMDB?

LMDB stands for Lightning Memory-Mapped Database. As usual, I'll defer to the excellent Wikipedia article.

In a nutshell, it provides an embedded transactional database in the form of a key-value store, and provides:

• greatly-increased performance,
• guaranteed data integrity and reliability.

LMDB in PowerDNS

One of the (many) nice features of the PowerDNS Authoritative server is its ability to use a variety of backends from which it's getting its data. This includes "classical" SQL backends (MySQL, PostgreSQL, ...), "flat-files" backends, but also the "Pipe" backend, allowing to, well, pipe the query to this backend (or more precisely, question is read on standard input and answer on standard output).

The LMDB backend was added in 2019 with the version 4.2.0 of the Authoritative server, but it was still experimental.
In December 2020, it was made production ready with version 4.4.0.

Let's go!

Environment

I'll be using PowerDNS 4.5.2 (latest version available when this article is being written), running on a FreeBSD 13.0-RELEASE-p4.

Install and configure PowerDNS

I'll be using FreeBSD's packages and not port, as we don't need fancy options, and this will also be faster.

• run pkg install powerdns
  • the LMDB backend is installed as part of this package
• also, add pdns_enable="YES" to your /etc/rc.conf
• For the PowerDNS configuration file (/usr/local/etc/pdns/pdns.conf), please see further down.

PostgreSQL

You need to be careful when installing PGSQL: if you look for the latest version, you'll see:

root@freebsd:/usr/local/etc/pdns # pkg search postgresql14
pgtcl-postgresql14-2.1.1_2 TCL extension for accessing a PostgreSQL server (PGTCL-NG)
postgresql14-client-14.1 PostgreSQL database (client)
postgresql14-contrib-14.1 The contrib utilities from the PostgreSQL distribution
postgresql14-docs-14.1 The PostgreSQL documentation set
postgresql14-plperl-14.1 Write SQL functions for PostgreSQL using Perl5
postgresql14-plpython-14.1 Module for using Python to write SQL functions
postgresql14-pltcl-14.1 Module for using Tcl to write SQL functions
postgresql14-server-14.1 PostgreSQL is the most advanced open-source database available anywhere

then doing a

pkg install postgresql14-server-14.1

will output the following message:

The following 5 package(s) will be affected (of 0 checked):
Installed packages to be REMOVED:
postgresql13-client: 13.5
powerdns: 4.5.2

New packages to be INSTALLED:
llvm11: 11.0.1_3
postgresql14-client: 14.1
postgresql14-server: 14.1

Number of packages to be removed: 2
Number of packages to be installed: 3

The process will require 740 MiB more space.

Proceed with this action? [y/N]:

Wait! I don't want PowerDNS to be removed!

Let's see the PowerDNS package dependencies with:

pkg info -dx powerdns

which gives me:

powerdns-4.5.2:
lua52-5.2.4
curl-7.80.0
boost-libs-1.72.0_6
sqlite3-3.35.5_4,1
postgresql13-client-13.5
mysql57-client-5.7.36
lmdb-0.9.29,1

Right: I'll have to install PGSQL version 13 - which is fine with me, since I don't really want some fancy new features, I just need a basic PGSQL.

Once PGSQL installed with a pkg install postgresql13-server-13.5, you can carry on.
As usual with FreeBSD ports/packages, the post-install instructions are clear:

To initialize the database, run
/usr/local/etc/rc.d/postgresql initdb

You can then start PostgreSQL by running:
/usr/local/etc/rc.d/postgresql start

[...]
To run PostgreSQL at startup, add
'postgresql_enable="YES"' to /etc/rc.conf

PowerDNS setup for PostgreSQL

Cool, now that PostgreSQL is installed and running, we need to set it up for PowerDNS:

• login with psql -U postgres
• then create database powerdns; to create the PDNS database
  • you can check with \l that the powerdns database has been created
• then \c powerdns to switch to this database
• then paste the PowerDNS PGSQL Schema
  • you can check with \d that the PowerDNS tables have been created
• now we want a specific PGSQL user for PowerDNS, not the default one
  create user powerdns with password 'your_password';
• and we give him all privileges to the PowerDNS DB:
  grant all privileges on database powerdns to powerdns;

Feeding PowerDNS

Finally, we want to have some zones to test against.
Since we just want to compare performances, I'm not creating a super complex setup, so a loop who creates 10k zones with the same content will do:

$ORIGIN .
test101.org 3600 IN NS ns1.test1.org.
test101.org 3600 IN SOA a.invalid hostmaster.test101.org 0 10800 3600 604800 3600
www.test101.org 3600 IN A 1.1.1.1
www.test101.org 3600 IN A 42.42.42.42

Testing

Right, so now it's time to test!
I'll test from the same host where PowerDNS is running.

The pdns.conf file for PostgreSQL looks like this:

local-address=192.168.x.x
local-port=53
launch=gpgsql
gpgsql-host=127.0.0.1
gpgsql-port=5432
gpgsql-dbname=powerdns
gpgsql-user=powerdns
gpgsql-password="<mypassword>"

and the one for LMDB:

local-address=192.168.x.x
local-port=53
launch=lmdb
lmdb-filename=/var/spool/powerdns/pdns.lmdb

As noted in the LMDB settings page, if you're running PowerDNS on Linux with systemd, pay attention to the location of your LMDB file, as ProtectSystem is set to full.
I'm running FreeBSD, so I'm not impacted by this :-)

With dnsperf

I first tested with dnsperf, a well known DNS benchmark tool, initially developed by Nominum (now part of Akamai), and whose development has now been handed to DNS-OARC.

dnsperf help (available via dnsperf -h or via its manpage) is pretty clear, so I can do some testing with e.g.:

dnsperf -d input.txt -s 192.168.x.x

to use input.txt as entry file, and 192.168.x.x as the target DNS server.
dnsperf will stop once it has gone through the file, which in my case is composed of 10.000 lines.

Let's tweak this a bit more:

• I want to cycle multiple times through the file, I'll use -n,
• I can also simulate several clients with -c,
• if my system is multi-threaded, I can also use -T to specify the number of threads
• Finally, I can have dnsperf to output the results every X seconds with -S

dnsperf -d input.txt -s 192.168.x.x -n 250 -c 10 -S 1

• Results with the PGSQL backend:

  Response codes: NOERROR 2500000 (100.00%)
Average packet size: request 33, response 49
Run time (s): 18.021062
Queries per second: 138726.563396

  Average Latency (s): 0.000702 (min 0.000019, max 0.034880)
Latency StdDev (s): 0.000777

• And with the LMDB backend:

  Response codes: NOERROR 2500000 (100.00%)
Average packet size: request 33, response 49
Run time (s): 17.368094
Queries per second: 143942.104413

  Average Latency (s): 0.000677 (min 0.000018, max 0.010012)
Latency StdDev (s): 0.000173

So 138k QPS with the PGSQL backend, versus 143k QPS with the LMDB backend.
Not a huge difference.

Let's try with another, more stressful tool: Flamethrower.

With Flamethrower

The fine folks at NS1 have created Flamethrower and made it available with an open source license.
The previous article gives a lot of details, but in a nutshell, Flamethrower is a lot more flexible, and will stress the DNS server a bit more :)

Even better, Flamethrower is available as a FreeBSD package! <3
And it's even the latest version, i.e. 0.11.0

Using it is super easy (in doubt, use flame -h):

flame 192.168.x.x -f input.txt -R

• PGSQL Backend:

  0.868047s: send: 80100, avg send: 80100, recv: 465, avg recv: 465, min/avg/max resp: 2.54784/34.849/67.6803ms, in flight: 79625, timeouts: 0
1.86871s: send: 92000, avg send: 86050, recv: 0, avg recv: 465, min/avg/max resp: 0/nan/0ms, in flight: 171625, timeouts: 0
2.87327s: send: 86200, avg send: 86100, recv: 296, avg recv: 380, min/avg/max resp: 1.38499/200.513/299.907ms, in flight: 257529, timeouts: 0
3.8808s: send: 93700, avg send: 88000, recv: 0, avg recv: 380, min/avg/max resp: 0/nan/0ms, in flight: 271064, timeouts: 80165
4.88136s: send: 91400, avg send: 88680, recv: 575, avg recv: 445, min/avg/max resp: 0.666835/153.186/250.928ms, in flight: 269934, timeouts: 91955
5.88143s: send: 91100, avg send: 89083, recv: 0, avg recv: 445, min/avg/max resp: 0/nan/0ms, in flight: 275465, timeouts: 85569

  total sent : 779000
total rcvd : 2430
avg r qps : 485
avg s qps : 86554
timeouts : 776570 (99.6881%)

  Hum, looks like something is not quite right, let's look at /var/log/messages:

  18:49:48 freebsd pdns[77479]: Done launching threads, ready to distribute questions
18:49:48 freebsd pdns[77479]: 5001 questions waiting for database/backend attention. Limit is 5000, respawning
18:49:49 freebsd kernel: Limiting icmp unreach response from 65055 to 200 packets/sec
18:49:49 freebsd pdns[77330]: Our pdns instance exited with code 1, respawning
18:49:50 freebsd kernel: Limiting icmp unreach response from 85745 to 200 packets/sec

  and indeed in the Auth settings page, we can see the max-queue-length setting:

  Integer
Default: 5000

  If this many packets are waiting for database attention, consider the situation hopeless and respawn.

  (The FreeBSD kernel message about limiting ICMP unreach response, is because PowerDNS' Guardian respawned the Authoritative process, therefore for a few moments, there was nothing listening on UDP/53, while Flamethrower was still sending traffic, therefore FreeBSD's blackhole kicked in.)

  So the above PowerDNS logs show that the PGSQL got overwhelmed right away, as there were a lot of "in flight" requests for the 3 first seconds, before having many timeouts starting at 4th second.

  We can ratelimit Flamethrower using the -Q switch, which I did to find out how many QPS I could reach until the DB started being overwhelmed:

  flame 192.168.x.x -f input.txt -R -Q 10000

  I then gradually increased that number.
  But something weird was happening: it seemed that I could not get the same number every time, the results were super inconsistent. I did some more tests, until I remember that PowerDNS >= 4.5 comes with "Zone cache" (see my article here).
  Basically, Zone Cache "adds a cache of all zones, avoiding backend lookups for zones that do not exist, and for non-existing subzones", thus improving performances.

  To validate that the inconsistencies I was seeing were thanks to/because of Zone Cache, I disabled it by adding zone-cache-refresh-interval=0 to my pdns.conf, restarted PDNS, and tested again.
  This time, I could barely get 20k QPS out of PDNS: so indeed, as my input file consisted of ~10k records/domains, the first test I ran with Flamethrower at 10k QPS caused those lookups to be cached, and when I increased the ratio, I wouldn't hit the DB but rather PowerDNS' Zone Cache.

  I will leave it enabled, as it's a default, and we want a fair comparison with LMDB.

  So the results are: ~18k average for a 60-seconds test

  Now, if I warm up the cache by gradually increasing the QPS, I can get a lot more QPS:

  e.g. by using --qps-flow "10000,10000;50000,10000;100000,10000;", I'm telling Flamethrower to send 10k QPS for 10 seconds, then 50k QPS for 10 seconds, and then 100k QPS for 10 seconds (and carry on at 100k QPS until I stop it).
  By doing so, I managed to get ~85k QPS, but that's a bit cheating ^^

• LMDB Backend:

  I got ~95k average, still for a 60-second test.
  I ran Flamethrower without any ratelimit, i.e. no need to warm up the cache.
  I was a bit disappointed by those results, until I saw this line at the very beginning of Flamethrower's output:

  flaming target(s) [192.168.x.x] on port 53 with 10 concurrent generators, each sending 10 queries every 1ms on protocol udp

  How about we try to increase the number of concurrent generators, with -c?
  This time I've let the test run for 2 minutes (120 seconds):

  • with 15 concurrent traffic generators: ~115k QPS (with ~2.6% timeout)
  • with 20 concurrent traffic generators: ~150k QPS (with ~3.6% timeout)
  • with 25 concurrent traffic generators: ~175k QPS (with ~15% timeout)

  I didn't test with more than 25, as 15% timeout is quite a lot.

  (Note that I also tested the PGSQL backend with more concurrent traffic generators, but that overwhelmed the DB straight away).

Wrap Up

Right, so looking at "pure" numbers, we have:

• 18k QPS for PGSQL
• versus 95k QPS for LMDB (and even more if we accept some timeout, but let's stick to 95k).

So it looks like using the LMDB backend does indeed increase the performances drastically.

Bear in mind that the tests were run on a local machine, with both the client and the server running there.

In an upcoming article, I want to:

• test against a "real" Authoritative DNS server in the wild,
• and also look at how to deal with spreading the configuration across multiple PowerDNS Authoritative servers using LMDB as backend (since that's one of the main reasons people are using the SQL backend: to benefit from PDNS' Native replication).

---

Tags: DNS, FreeBSD