Testing PowerDNS' Lightning Stream to sync LMDB backends
What is it?
The PowerDNS folks recently announced in a blog post support for an ultra fast LMDB synchronisation for the Authoritative DNS server.
It is called Lightning Stream and will be released with PowerDNS Authoritative 4.8.0 - and can now be tested with the beta-1!
Installing PowerDNS Auth 4.8.0-beta1 on FreeBSD
When I started writing that blog post, the -alpha1 version was the first release to include Lightning Stream support, but -beta1 got released a few days ago, so I upgraded my setup. You may notice some references to -alpha1, just ignore them :)
After you've grabbed the Tarball from the PowerDNS Downloads website, you can install with:
export CXXFLAGS=-std=c++17
./configure --with-modules="lmdb" --prefix=/root/pdns
gmake
gmake install
Once done, let's use a very basic pdns.conf:
local-address=192.168.42.124
local-port=53
launch=lmdb
lmdb-filename=/var/spool/pdns-4.8/pdns.lmdb
Then let's launch PowerDNS:
freebsd# /root/pdns/sbin/pdns_server
Apr 26 22:33:05 This is a standalone pdns
Apr 26 22:33:05 Listening on controlsocket in '/var/run/pdns/pdns.controlsocket'
Apr 26 22:33:05 UDP server bound to 192.168.42.124:53
Apr 26 22:33:05 TCP server bound to 192.168.42.124:53
Apr 26 22:33:05 PowerDNS Authoritative Server 4.8.0-alpha1 (C) 2001-2022 PowerDNS.COM BV
Apr 26 22:33:05 Using 64-bits mode. Built using clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) on Apr 26 2023 22:26:27 by root@freebsd.
Apr 26 22:33:05 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Apr 26 22:33:05 Polled security status of version 4.8.0-alpha1 at startup, no known issues reported: Unsupported pre-release
Apr 26 22:33:06 Creating backend connection for TCP
Apr 26 22:33:06 About to create 3 backend threads for UDP
Apr 26 22:33:06 Done launching threads, ready to distribute questions
Lightning Stream
Install it
I have followed those instructions:
git clone https://github.com/PowerDNS/lightningstream.git
cd lightningstream
./build.sh
Update pdns.conf
The above page recommends default server settings, so my updated pdns.conf
looks like:
local-address=192.168.42.124
local-port=53
launch=lmdb
lmdb-filename=/var/spool/pdns-4.8/pdns.lmdb
lmdb-shards=1
lmdb-random-ids=yes
lmdb-flag-deleted=yes
lmdb-map-size=1000
zone-cache-refresh-interval=0
zone-metadata-cache-ttl=0
Configure it
I am using Backblaze B2 as S3 target.
Here's my YAML configuration file:
instance: dns-lenovo
lmdbs:
main:
# Auth 'lmdb-filename'
path: /var/spool/pdns-4.8/pdns.lmdb
schema_tracks_changes: true
options:
no_subdir: true
create: true # optional for 'main', as auth will create it on startup, if needed
map_size: 1000MB # for create=true, make sure to match auth's lmdb-map-size
shard:
# Auth 'lmdb-filename' plus '-0' for the first shard
path: /var/spool/pdns-4.8/pdns.lmdb-0
schema_tracks_changes: true
options:
no_subdir: true
create: true # strongly recommended for shards
map_size: 1000MB # for create=true, make sure to match auth's lmdb-map-size
storage:
type: s3
options:
access_key: XXX
secret_key: YYY
bucket: pdns-lightning
create_bucket: false
endpoint_url: https://s3.us-west-000.backblazeb2.com
http:
address: ":8500" # for status and metrics
Launch it
freebsd# ./bin/lightningstream --config=/root/lightningstream/bin/config.yaml sync
INFO[0000] Storage backend initialised storage_type=s3
INFO[0000] [main ] Opening LMDB db=main lmdbpath=/var/spool/pdns-4.8/pdns.lmdb
[...]
INFO[0001] startup phase completed successfully starttracker=main
INFO[0001] startup phase completed successfully starttracker=shard
Second PowerDNS instance
I used a second FreeBSD instance to test the sync, so I basically did the same (installing PowerDNS 4.8.0-beta1, then Lightning Stream) - the only things I changed in the configuration files are:
- the listener IP address for PowerDNS,
- the name of the instance for Lightning Stream.
Testing the setup
Checking the S3 bucket
Looking at the Backblaze bucket, I could confirm that the sync was happening:
Checking the Lightning Stream logs
The Lightning Stream logs on both servers also looked ok - here's an extract after adding a DNS zone (with pdnsutil create-zone) on the first FreeBSD server:
Server 1:
INFO[0027] [main ] Stored snapshot compression_ratio="1:1.74" db=main instance=dns-lenovo snapshot_name=main__dns-lenovo__20230518-183511-748512453__G-0000000000000000.pb.gz snapshot_size="404 B" time_acquire=0s time_compress=2ms time_copy_shadow=0s time_dump=1ms time_gc=3ms time_store=1.211s time_total=1.215s txnID=14 uncompressed_size="702 B"
INFO[0027] [shard ] Stored snapshot compression_ratio="1:1.73" db=shard instance=dns-lenovo snapshot_name=shard__dns-lenovo__20230518-183511-748553077__G-0000000000000000.pb.gz snapshot_size="309 B" time_acquire=0s time_compress=2ms time_copy_shadow=0s time_dump=0s time_gc=2ms time_store=1.296s time_total=1.298s txnID=6 uncompressed_size="536 B"
Server 2:
INFO[0023] [shard ] Snapshot downloaded component=downloader db=shard instance=dns-nuc shorthash=58a9e07 snapshot_instance=dns-lenovo time_load_storage=206ms time_load_total=206ms timestamp=20230518-183511-748553077
INFO[0023] [main ] Snapshot downloaded component=downloader db=main instance=dns-nuc shorthash=c3362ac snapshot_instance=dns-lenovo time_load_storage=258ms time_load_total=258ms timestamp=20230518-183511-748512453
INFO[0024] [shard ] Loaded remote snapshot db=shard instance=dns-nuc shorthash=58a9e07 snapshot_instance=dns-lenovo time_total=2ms time_write_lock=2ms timestamp=20230518-183511-748553077 txnID=5
INFO[0024] [main ] Loaded remote snapshot db=main instance=dns-nuc shorthash=c3362ac snapshot_instance=dns-lenovo time_total=2ms time_write_lock=2ms timestamp=20230518-183511-748512453 txnID=13
Checking the DNS content
I had created one zone from each server:
- freebsd1.local from the Server 1,
- nuc.local from the Server 2.
A quick check on both servers also confirmed that the data were identical:
# pdnsutil list-all-zones
freebsd1.local
nuc.local
Updating the zone with pdnsutil edit-zone also got the update synchronised to the other server, and vice-versa.
Checking the metrics
Lightning Stream comes with a handy metrics web server (which also allows pulling metrics from Prometheus), so you can easily check the status of each instance.
In my case, I could once again confirm that the content was the same on both instances:
The power of the Opensource community!
When I initially tested Lightning Stream, for some reason I could not make it work properly:
- the configuration looked fine,
- I could see the LMDB snapshots uploaded to my S3 bucket whenever I made a change,
- but changes were not visible on the other server (a zone created on server 1 would not appear on server 2),
- although when I tried creating the same zone on the second server, pdnsutil would tell me that the zone already existed... even though a pdnsutil list-all-zones didn't show it!
I then sent an email to the PowerDNS mailing-list asking for help, and I got an answer pointing out that I forgot the line:
lmdb-shards=1
... and without that line, the sync wouldn't work!
Adding it fixed my issue straight away of course :-)
Wrap Up
Lightning Stream is a very convenient and efficient way to synchronise a LMDB database between multiple PowerDNS instances - the LMDB backend is extremely fast and up to now, there was no way to synchronise its content to other servers.