as42184.net

TKRZ Stadtwerke GmbH

DNS Anycast

In this post we want to introduce you to our new project ‘DNS-Anycast with ExaBGP’ and create a tutorial for you. This setup has been active for the past few weeks and has proven to be functional.



What is Anycast

Anycast is a routing method that uses fewer IP addresses, is more secure against DDOS and distributes traffic based on source location, resulting in better latency and redundany. Cloudflare has a good article explaining the functionality and other aspects of Anycast.


What are the requirements

For this tutorial you will need at least 2 servers and 2 routers. In our case we used 4 Debian 12 VMs with unbound as resolver and 4 Juniper routers. I assume that you know how to set up a secure resolver and create an active BGP session between your routers.


Resolver configuration

First you need to assign the Anycast IP address to the server. You can edit the addresses here:

nano /etc/network/interfacesCode language: Shell Session (shell)

You will need to configure a new loopback interface with your address. The IPs must be the same on all resolvers.

# The anycast IP
auto lo:1
iface lo:1 inet static
        address 185.XXX.XXX.XXX/32

iface lo:1 inet6 static
        address 2a02:XXX:XXX::X/128Code language: plaintext (plaintext)

And restart the service.

systemctl restart networkingCode language: Shell Session (shell)

You will also need to add the Anycast addresses to your resolver and restart its service. If you are using unbound, you can add them here:

nano /etc/unbound/unbound.conf.d/interfaces-access.confCode language: Shell Session (shell)

ExaBGP

ExaBGP is a software to perform dynamic network changes. We use it to announce the Anycast-Address when DNS is working. If the resolver doesn’t work, the route is no longer announced. First of all you need to install the packages.

apt install python3-exabgp python3-dnspythonCode language: Shell Session (shell)

You need DNS-Python to check if the resolver works with a Python script.
Then you have to create the configuration file of ExaBGP.

nano /etc/exabgp/exabgp.confCode language: Shell Session (shell)

Now you need to configure your BGP sessions and processes to check that the resolver is working as expected. Here is an example:

process announce-routes {
 run /etc/exabgp/dns-check.sh;
 encoder text;
}

process announce-v6-routes {
 run /etc/exabgp/dns-v6-check.sh;
 encoder text;
}

neighbor 185.XXX.XXX.XXX {
    local-address 185.XXX.XXX.XXX;
    local-as 42184;
    peer-as 42184;

    api {
        processes [ announce-routes ];
    }

}

neighbor 2a02:XXX:XXX::X {
    local-address 2a02:XXX:XXX::X;
    local-as 42184;
    peer-as 42184;
    router-id 185.XXX.XXX.XXX;

    api {
        processes [ announce-v6-routes ];
    }

}Code language: JSON / JSON with Comments (json)

You can also use private ASNs instead if you don’t have a public one.
To finish the ExaBGP configuration, you have to create one more file.

exabgp --fi > /etc/exabgp/exabgp.envCode language: Shell Session (shell)

To make the DNS check work, you also need to create the Python scripts shown below.

IPv4:

nano /etc/exabgp/dns-check.shCode language: Shell Session (shell)
#!/usr/bin/env python3
import socket
from sys import stdout
from time import sleep

import dns.resolver

def is_alive(host, resolverIP):
    resolver = dns.resolver.Resolver()
    resolver.nameservers = [resolverIP]
    try:
        answers = resolver.resolve(host, 'A')
        for rdata in answers:
            if rdata.address:
               return True
    except dns.resolver.NoAnswer:
        return False
    except dns.exception.Timeout:  # Handle timeout exception
        return False
    except dns.resolver.NXDOMAIN:  # Handle NXDOMAIN exception
        return False
    except dns.resolver.NoNameservers:  # Handle NoNameservers exception
        return False
    except Exception as e:  # Handle other exceptions
        print("An error occurred:", e)
        return False

resolver_ip = '127.0.0.1'
while True:
    if is_alive('tkrz.de', resolver_ip):
        stdout.write('announce route 185.XXX.XXX.XXX/32 next-hop self' + '\n')
        stdout.flush()
    else:
        stdout.write('withdraw route 185.XXX.XXX.XXX/32 next-hop self' + '\n')
        stdout.flush()
    sleep(5)Code language: Python (python)

IPv6:

nano /etc/exabgp/dns-v6-check.shCode language: Shell Session (shell)
#!/usr/bin/env python3
import socket
from sys import stdout
from time import sleep

import dns.resolver

def is_alive(host, resolverIP):
    resolver = dns.resolver.Resolver()
    resolver.nameservers = [resolverIP]
    try:
        answersv6 = resolver.resolve(host, 'AAAA')
        for rdata in answersv6:
            if rdata.address:
               return True

    except dns.resolver.NoAnswer:
        return False
    except dns.exception.Timeout:  # Handle timeout exception
        return False
    except dns.resolver.NXDOMAIN:  # Handle NXDOMAIN exception
        return False
    except dns.resolver.NoNameservers:  # Handle NoNameservers exception
        return False
    except Exception as e:  # Handle other exceptions
        print("An error occurred:", e)
        return False

resolver_ip = '::1'
while True:
    if is_alive('tkrz.de', resolver_ip):
        stdout.write('announce route 2a02:XXX:XXX::X/128 next-hop self' + '\n')
        stdout.flush()
    else:
        stdout.write('withdraw route 2a02:XXX:XXX::X/128 next-hop self' + '\n')
        stdout.flush()
    sleep(5)Code language: Python (python)

However, it is possible that your BGP sessions may be interrupted by hold timer exceptions. To avoid this problem, you can edit the ExaBGP environment file and change the following part.

nano /etc/exabgp/exabgp.env

Old:

[exabgp.api]
ack = true
Code language: JavaScript (javascript)

New:

[exabgp.api]
ack = false
Code language: JavaScript (javascript)

Finally, you can create the ExaBGP service file and start the process.

nano /etc/systemd/system/exabgp.service
Description=ExaBGP
Documentation=man:exabgp(1)
Documentation=man:exabgp.conf(5)
Documentation=https://github.com/Exa-Networks/exabgp/wiki
After=network.target
ConditionPathExists=/etc/exabgp/exabgp.conf

[Service]
User=exabgp
Group=exabgp
Environment=exabgp_daemon_daemonize=false
RuntimeDirectory=exabgp
RuntimeDirectoryMode=0750
ExecStartPre=-/usr/bin/mkfifo /run/exabgp/exabgp.in
ExecStartPre=-/usr/bin/mkfifo /run/exabgp/exabgp.out
ExecStart=/usr/sbin/exabgp /etc/exabgp/exabgp.conf
ExecReload=/bin/kill -USR1 $MAINPID
Restart=always
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.targetCode language: plaintext (plaintext)
systemctl restart exabgp.serviceCode language: Shell Session (shell)

Anycast Router configuration

In this section we will show you a possible configuration for Juniper routers.
We assume that you already have active BGP sessions and know how to configure and debug them. However, you will need to start with a prefix list:

set policy-options prefix-list v4-dns-anycast-list 185.XXX.XXX.XXX/32
set policy-options prefix-list v6-dns-anycast-list 2a02:XXX:XXX::X/128Code language: plaintext (plaintext)

Then configure the policy statements.

set policy-options policy-statement v4-dns-anycast term prefixes from prefix-list v4-dns-anycast-list
set policy-options policy-statement v4-dns-anycast term prefixes then accept
set policy-options policy-statement v4-dns-anycast term reject then reject
set policy-options policy-statement v6-dns-anycast term prefixes from prefix-list v6-dns-anycast-list
set policy-options policy-statement v6-dns-anycast term prefixes then accept
set policy-options policy-statement v6-dns-anycast term reject then rejectCode language: plaintext (plaintext)

And finally, bring up the BGP sessions. If you’re not using the same ASN on the router and server, you’ll need to change the ‘group type’ to external.

set protocols bgp group v4-dns type internal
set protocols bgp group v4-dns description "Peering to DNS"
set protocols bgp group v4-dns mtu-discovery
set protocols bgp group v4-dns import then-reject
set protocols bgp group v4-dns family inet unicast
set protocols bgp group v4-dns export then-reject
set protocols bgp group v4-dns local-as 42184
set protocols bgp group v4-dns graceful-restart
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX local-address 185.XXX.XXX.XXX
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX import v4-dns-anycast
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX import then-reject
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX export then-reject
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX peer-as 42184
set protocols bgp group v6-dns type internal
set protocols bgp group v6-dns description "v6: Peering to DNS"
set protocols bgp group v6-dns mtu-discovery
set protocols bgp group v6-dns import then-reject
set protocols bgp group v6-dns family inet6 unicast
set protocols bgp group v6-dns export then-reject
set protocols bgp group v6-dns local-as 42184
set protocols bgp group v6-dns graceful-restart
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X local-address 2a02:XXX:XXX::X
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X import v6-dns-anycast
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X import then-reject
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X export then-reject
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X peer-as 42184Code language: plaintext (plaintext)

You are now at the end of this tutorial. Please check if the BGP sessions are established and if the route gets announced. If everything is working as expected, you can shut down a server and check if the DNS is still available.
Now you have successfully set up your DNS servers with an Anycast IP and implemented lifechecks for more redundancy.

Update 2024/07/08: Updated systemd service definition, which now creates the fifo files with minimal rights in a more suitable location (/run/exabgp/).

Hello World

A while back, some network engineers at AS42184 decided to set up a (then) Twitter account to share glimpses of their day-to-day work. And tech porn. Mostly tech porn. Now that the platform is crumbling, we thought about moving to Mastodon (or even Bluesky?), but hosting a Mastodon instance sounds like a lot of work, and we could only reach a subset of our followers. So we decided to take a step back and start this blog. With the added benefit of keeping posted stories more visible (and searchable) on the web.

Read more