In this post we want to introduce you to our new project ‘DNS-Anycast with ExaBGP’ and create a tutorial for you. This setup has been active for the past few weeks and has proven to be functional.
What is Anycast
Anycast is a routing method that uses fewer IP addresses, is more secure against DDOS and distributes traffic based on source location, resulting in better latency and redundany. Cloudflare has a good article explaining the functionality and other aspects of Anycast.
What are the requirements
For this tutorial you will need at least 2 servers and 2 routers. In our case we used 4 Debian 12 VMs with unbound as resolver and 4 Juniper routers. I assume that you know how to set up a secure resolver and create an active BGP session between your routers.
Resolver configuration
First you need to assign the Anycast IP address to the server. You can edit the addresses here:
nano /etc/network/interfacesCode language: Shell Session (shell)You will need to configure a new loopback interface with your address. The IPs must be the same on all resolvers.
# The anycast IP
auto lo:1
iface lo:1 inet static
address 185.XXX.XXX.XXX/32
iface lo:1 inet6 static
address 2a02:XXX:XXX::X/128Code language: plaintext (plaintext)And restart the service.
systemctl restart networkingCode language: Shell Session (shell)You will also need to add the Anycast addresses to your resolver and restart its service. If you are using unbound, you can add them here:
nano /etc/unbound/unbound.conf.d/interfaces-access.confCode language: Shell Session (shell)ExaBGP
ExaBGP is a software to perform dynamic network changes. We use it to announce the Anycast-Address when DNS is working. If the resolver doesn’t work, the route is no longer announced. First of all you need to install the packages.
apt install python3-exabgp python3-dnspythonCode language: Shell Session (shell) You need DNS-Python to check if the resolver works with a Python script.
Then you have to create the configuration file of ExaBGP.
nano /etc/exabgp/exabgp.confCode language: Shell Session (shell)Now you need to configure your BGP sessions and processes to check that the resolver is working as expected. Here is an example:
process announce-routes {
run /etc/exabgp/dns-check.sh;
encoder text;
}
process announce-v6-routes {
run /etc/exabgp/dns-v6-check.sh;
encoder text;
}
neighbor 185.XXX.XXX.XXX {
local-address 185.XXX.XXX.XXX;
local-as 42184;
peer-as 42184;
api {
processes [ announce-routes ];
}
}
neighbor 2a02:XXX:XXX::X {
local-address 2a02:XXX:XXX::X;
local-as 42184;
peer-as 42184;
router-id 185.XXX.XXX.XXX;
api {
processes [ announce-v6-routes ];
}
}Code language: JSON / JSON with Comments (json) You can also use private ASNs instead if you don’t have a public one.
To finish the ExaBGP configuration, you have to create one more file.
exabgp --fi > /etc/exabgp/exabgp.envCode language: Shell Session (shell)To make the DNS check work, you also need to create the Python scripts shown below.
IPv4:
nano /etc/exabgp/dns-check.shCode language: Shell Session (shell)#!/usr/bin/env python3
import socket
from sys import stdout
from time import sleep
import dns.resolver
def is_alive(host, resolverIP):
resolver = dns.resolver.Resolver()
resolver.nameservers = [resolverIP]
try:
answers = resolver.resolve(host, 'A')
for rdata in answers:
if rdata.address:
return True
except dns.resolver.NoAnswer:
return False
except dns.exception.Timeout: # Handle timeout exception
return False
except dns.resolver.NXDOMAIN: # Handle NXDOMAIN exception
return False
except dns.resolver.NoNameservers: # Handle NoNameservers exception
return False
except Exception as e: # Handle other exceptions
print("An error occurred:", e)
return False
resolver_ip = '127.0.0.1'
while True:
if is_alive('tkrz.de', resolver_ip):
stdout.write('announce route 185.XXX.XXX.XXX/32 next-hop self' + '\n')
stdout.flush()
else:
stdout.write('withdraw route 185.XXX.XXX.XXX/32 next-hop self' + '\n')
stdout.flush()
sleep(5)Code language: Python (python)IPv6:
nano /etc/exabgp/dns-v6-check.shCode language: Shell Session (shell)#!/usr/bin/env python3
import socket
from sys import stdout
from time import sleep
import dns.resolver
def is_alive(host, resolverIP):
resolver = dns.resolver.Resolver()
resolver.nameservers = [resolverIP]
try:
answersv6 = resolver.resolve(host, 'AAAA')
for rdata in answersv6:
if rdata.address:
return True
except dns.resolver.NoAnswer:
return False
except dns.exception.Timeout: # Handle timeout exception
return False
except dns.resolver.NXDOMAIN: # Handle NXDOMAIN exception
return False
except dns.resolver.NoNameservers: # Handle NoNameservers exception
return False
except Exception as e: # Handle other exceptions
print("An error occurred:", e)
return False
resolver_ip = '::1'
while True:
if is_alive('tkrz.de', resolver_ip):
stdout.write('announce route 2a02:XXX:XXX::X/128 next-hop self' + '\n')
stdout.flush()
else:
stdout.write('withdraw route 2a02:XXX:XXX::X/128 next-hop self' + '\n')
stdout.flush()
sleep(5)Code language: Python (python)However, it is possible that your BGP sessions may be interrupted by hold timer exceptions. To avoid this problem, you can edit the ExaBGP environment file and change the following part.
nano /etc/exabgp/exabgp.envOld:
[exabgp.api]
ack = true
Code language: JavaScript (javascript)New:
[exabgp.api]
ack = false
Code language: JavaScript (javascript)Finally, you can create the ExaBGP service file and start the process.
nano /etc/systemd/system/exabgp.serviceDescription=ExaBGP
Documentation=man:exabgp(1)
Documentation=man:exabgp.conf(5)
Documentation=https://github.com/Exa-Networks/exabgp/wiki
After=network.target
ConditionPathExists=/etc/exabgp/exabgp.conf
[Service]
User=exabgp
Group=exabgp
Environment=exabgp_daemon_daemonize=false
RuntimeDirectory=exabgp
RuntimeDirectoryMode=0750
ExecStartPre=-/usr/bin/mkfifo /run/exabgp/exabgp.in
ExecStartPre=-/usr/bin/mkfifo /run/exabgp/exabgp.out
ExecStart=/usr/sbin/exabgp /etc/exabgp/exabgp.conf
ExecReload=/bin/kill -USR1 $MAINPID
Restart=always
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.targetCode language: plaintext (plaintext)systemctl restart exabgp.serviceCode language: Shell Session (shell)Anycast Router configuration
In this section we will show you a possible configuration for Juniper routers.
We assume that you already have active BGP sessions and know how to configure and debug them. However, you will need to start with a prefix list:
set policy-options prefix-list v4-dns-anycast-list 185.XXX.XXX.XXX/32
set policy-options prefix-list v6-dns-anycast-list 2a02:XXX:XXX::X/128Code language: plaintext (plaintext)Then configure the policy statements.
set policy-options policy-statement v4-dns-anycast term prefixes from prefix-list v4-dns-anycast-list
set policy-options policy-statement v4-dns-anycast term prefixes then accept
set policy-options policy-statement v4-dns-anycast term reject then reject
set policy-options policy-statement v6-dns-anycast term prefixes from prefix-list v6-dns-anycast-list
set policy-options policy-statement v6-dns-anycast term prefixes then accept
set policy-options policy-statement v6-dns-anycast term reject then rejectCode language: plaintext (plaintext)And finally, bring up the BGP sessions. If you’re not using the same ASN on the router and server, you’ll need to change the ‘group type’ to external.
set protocols bgp group v4-dns type internal
set protocols bgp group v4-dns description "Peering to DNS"
set protocols bgp group v4-dns mtu-discovery
set protocols bgp group v4-dns import then-reject
set protocols bgp group v4-dns family inet unicast
set protocols bgp group v4-dns export then-reject
set protocols bgp group v4-dns local-as 42184
set protocols bgp group v4-dns graceful-restart
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX local-address 185.XXX.XXX.XXX
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX import v4-dns-anycast
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX import then-reject
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX export then-reject
set protocols bgp group v4-dns neighbor 185.XXX.XXX.XXX peer-as 42184
set protocols bgp group v6-dns type internal
set protocols bgp group v6-dns description "v6: Peering to DNS"
set protocols bgp group v6-dns mtu-discovery
set protocols bgp group v6-dns import then-reject
set protocols bgp group v6-dns family inet6 unicast
set protocols bgp group v6-dns export then-reject
set protocols bgp group v6-dns local-as 42184
set protocols bgp group v6-dns graceful-restart
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X local-address 2a02:XXX:XXX::X
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X import v6-dns-anycast
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X import then-reject
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X export then-reject
set protocols bgp group v6-dns neighbor 2a02:XXX:XXX::X peer-as 42184Code language: plaintext (plaintext) You are now at the end of this tutorial. Please check if the BGP sessions are established and if the route gets announced. If everything is working as expected, you can shut down a server and check if the DNS is still available.
Now you have successfully set up your DNS servers with an Anycast IP and implemented lifechecks for more redundancy.
Update 2024/07/08: Updated systemd service definition, which now creates the fifo files with minimal rights in a more suitable location (/run/exabgp/).
