Search | Sailfish OS | Running | PineTime | All Posts

Testing PowerDNS' Lightning Stream to sync LMDB backends

May 19, 2023 — Nico Cartron

What is it?

The PowerDNS folks recently announced in a blog post support for an ultra fast LMDB synchronisation for the Authoritative DNS server.

It is called Lightning Stream and will be released with PowerDNS Authoritative 4.8.0 - and can now be tested with the beta-1!

Installing PowerDNS Auth 4.8.0-beta1 on FreeBSD

When I started writing that blog post, the -alpha1 version was the first release to include Lightning Stream support, but -beta1 got released a few days ago, so I upgraded my setup. You may notice some references to -alpha1, just ignore them :)

After you've grabbed the Tarball from the PowerDNS Downloads website, you can install with:

 export CXXFLAGS=-std=c++17
 ./configure --with-modules="lmdb" --prefix=/root/pdns
 gmake install

Once done, let's use a very basic pdns.conf:


Then let's launch PowerDNS:

 freebsd# /root/pdns/sbin/pdns_server
 Apr 26 22:33:05 This is a standalone pdns
 Apr 26 22:33:05 Listening on controlsocket in '/var/run/pdns/pdns.controlsocket'
 Apr 26 22:33:05 UDP server bound to
 Apr 26 22:33:05 TCP server bound to
 Apr 26 22:33:05 PowerDNS Authoritative Server 4.8.0-alpha1 (C) 2001-2022 PowerDNS.COM BV
 Apr 26 22:33:05 Using 64-bits mode. Built using clang 13.0.0 ( llvmorg-13.0.0-0-gd7b669b3a303) on Apr 26 2023 22:26:27 by root@freebsd.
 Apr 26 22:33:05 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
 Apr 26 22:33:05 Polled security status of version 4.8.0-alpha1 at startup, no known issues reported: Unsupported pre-release
 Apr 26 22:33:06 Creating backend connection for TCP
 Apr 26 22:33:06 About to create 3 backend threads for UDP
 Apr 26 22:33:06 Done launching threads, ready to distribute questions

Lightning Stream

Install it

I have followed those instructions:

git clone
cd lightningstream

Update pdns.conf

The above page recommends default server settings, so my updated pdns.conf looks like:


Configure it

I am using Backblaze B2 as S3 target.

Here's my YAML configuration file:

 instance: dns-lenovo
     # Auth 'lmdb-filename'
     path: /var/spool/pdns-4.8/pdns.lmdb
     schema_tracks_changes: true
       no_subdir: true
       create: true      # optional for 'main', as auth will create it on startup, if needed
       map_size: 1000MB  # for create=true, make sure to match auth's lmdb-map-size
     # Auth 'lmdb-filename' plus '-0' for the first shard
     path: /var/spool/pdns-4.8/pdns.lmdb-0
     schema_tracks_changes: true
       no_subdir: true
       create: true      # strongly recommended for shards
       map_size: 1000MB  # for create=true, make sure to match auth's lmdb-map-size

   type: s3
     access_key: XXX
     secret_key: YYY
     bucket: pdns-lightning
     create_bucket: false

   address: ":8500"  # for status and metrics

Launch it

 freebsd# ./bin/lightningstream --config=/root/lightningstream/bin/config.yaml sync
 INFO[0000] Storage backend initialised                   storage_type=s3
 INFO[0000] [main          ] Opening LMDB                 db=main lmdbpath=/var/spool/pdns-4.8/pdns.lmdb
 INFO[0001] startup phase completed successfully          starttracker=main
 INFO[0001] startup phase completed successfully          starttracker=shard

Second PowerDNS instance

I used a second FreeBSD instance to test the sync, so I basically did the same (installing PowerDNS 4.8.0-beta1, then Lightning Stream) - the only things I changed in the configuration files are:

  • the listener IP address for PowerDNS,
  • the name of the instance for Lightning Stream.

Testing the setup

Checking the S3 bucket

Looking at the Backblaze bucket, I could confirm that the sync was happening:

Checking the Lightning Stream logs

The Lightning Stream logs on both servers also looked ok - here's an extract after adding a DNS zone (with pdnsutil create-zone) on the first FreeBSD server:

Server 1:

 INFO[0027] [main          ] Stored snapshot              compression_ratio="1:1.74" db=main instance=dns-lenovo snapshot_name=main__dns-lenovo__20230518-183511-748512453__G-0000000000000000.pb.gz snapshot_size="404 B" time_acquire=0s time_compress=2ms time_copy_shadow=0s time_dump=1ms time_gc=3ms time_store=1.211s time_total=1.215s txnID=14 uncompressed_size="702 B"
 INFO[0027] [shard         ] Stored snapshot              compression_ratio="1:1.73" db=shard instance=dns-lenovo snapshot_name=shard__dns-lenovo__20230518-183511-748553077__G-0000000000000000.pb.gz snapshot_size="309 B" time_acquire=0s time_compress=2ms time_copy_shadow=0s time_dump=0s time_gc=2ms time_store=1.296s time_total=1.298s txnID=6 uncompressed_size="536 B"

Server 2:

 INFO[0023] [shard         ] Snapshot downloaded          component=downloader db=shard instance=dns-nuc shorthash=58a9e07 snapshot_instance=dns-lenovo time_load_storage=206ms time_load_total=206ms timestamp=20230518-183511-748553077
 INFO[0023] [main          ] Snapshot downloaded          component=downloader db=main instance=dns-nuc shorthash=c3362ac snapshot_instance=dns-lenovo time_load_storage=258ms time_load_total=258ms timestamp=20230518-183511-748512453
 INFO[0024] [shard         ] Loaded remote snapshot       db=shard instance=dns-nuc shorthash=58a9e07 snapshot_instance=dns-lenovo time_total=2ms time_write_lock=2ms timestamp=20230518-183511-748553077 txnID=5                                
 INFO[0024] [main          ] Loaded remote snapshot       db=main instance=dns-nuc shorthash=c3362ac snapshot_instance=dns-lenovo time_total=2ms time_write_lock=2ms timestamp=20230518-183511-748512453 txnID=13

Checking the DNS content

I had created one zone from each server:

  • freebsd1.local from the Server 1,
  • nuc.local from the Server 2.

A quick check on both servers also confirmed that the data were identical:

 # pdnsutil list-all-zones

Updating the zone with pdnsutil edit-zone also got the update synchronised to the other server, and vice-versa.

Checking the metrics

Lightning Stream comes with a handy metrics web server (which also allows pulling metrics from Prometheus), so you can easily check the status of each instance.

In my case, I could once again confirm that the content was the same on both instances:

The power of the Opensource community!

When I initially tested Lightning Stream, for some reason I could not make it work properly:

  • the configuration looked fine,
  • I could see the LMDB snapshots uploaded to my S3 bucket whenever I made a change,
  • but changes were not visible on the other server (a zone created on server 1 would not appear on server 2),
  • although when I tried creating the same zone on the second server, pdnsutil would tell me that the zone already existed... even though a pdnsutil list-all-zones didn't show it!

I then sent an email to the PowerDNS mailing-list asking for help, and I got an answer pointing out that I forgot the line:


... and without that line, the sync wouldn't work!

Adding it fixed my issue straight away of course :-)

Wrap Up

Lightning Stream is a very convenient and efficient way to synchronise a LMDB database between multiple PowerDNS instances - the LMDB backend is extremely fast and up to now, there was no way to synchronise its content to other servers.

Tags: DNS, FreeBSD

I don't have any commenting system, but email me (nicolas at ncartron dot org) your comments!