Docker container DNS resolution issue in air-gapped network

Problem

The scenario is where docker containers are running on a host system with Ubuntu, which is connected in an air gapped network, with the router configured with a list of local DNS servers. The host doesn’t have any manual DNS entries, though it is able to make DNS queries via the LAN’s upstream DNS servers. Containers running on Bridge mode, on the other hand, are trying to resolve from 8.8.8.8 instead of using LAN’s DNS. The expected behaviour is that containers should resolve from the LAN’s upstream servers.

Analysis

Here is a bit of related information to understand how Ubuntu DNS resolution works & how docker container’s DNS configuration is being handled:

Host DNS

Ubuntu has changed the DNS resolution flow since 18.04 with the introduction of systemd-resolved as the default resolver.

systemd-resolved is a systemd service that provides network name resolution to local applications via three interfaces:

A D-Bus API

GNU C Library (glibc) getaddrinfo and its related resolver functions, including gethostbyname via Name Service Switch (NSS) with plugin nss-resolve module.
(nss-resolve via systemd-resolved get nameserver list from /etc/resolv.conf, /etc/hosts and/or /etc/hostname files based on the /etc/nsswitch.conf configuration)

A local DNS stub listener on 127.0.0.53

In short, systemd-resolved daemon also acts as a DNS server by listening on IP address 127.0.0.53 on the local loopback interface. lo is hardcoded, it cannot listen on any other network interfaces such as docker0, eth0,en0, etc.

Any non-local DNS queries, in our case, will be forwarded to the upstream DNS server of the LAN. System gets this DNS server details during the DHCP connection, which will be updated in /etc/resolv.conf file dynamically.

/etc/resolv.conf file shouldn’t be edited manually, rather it should be symlinked to one of the following files as per your use-case, where you can specify hardcoded nameservers if there any:

/run/systemd/resolve/stub-resolv.conf
/usr/lib/systemd/resolv.conf
/run/systemd/resolve/resolv.conf

Container DNS

When docker containers are started in Bridge mode, docker daemon copies any non-localhost entries of /etc/resolv.conf, /etc/hosts and /etc/hostname files from the host to the container; if no non-localhost entries are found, container’s /etc/resolv.conf will be initialized with hardcoded nameserver 8.8.8.8.

fig 1: Docker DNS issue in air-gapped network

Whenever the host machine receives new DNS configuration over DHCP, it will be updated to the docker container once it gets restarted. When upstream DNS configuration changes, there is no way the running containers to know about the changes.

Solution

Hardcoding nameservers in instance doesn’t solve the issue, because whenever a nameserver change is required, it needs to be updated in every system of the network. All running docker container instances need to be restarted as well in order to get the updated DNS list.

One solution for this problem would be to enable host system to listen on docker interface apart from the local loopback interface. i.e., forwarding all DNS queries from the docker containers to the host’s 127.0.0.53.

Since systemd-resolved cannot listen on docker0 interface, we can use dnsmasq to bind to docker interface. dnsmasq will accept queries from the containers and relay to the systemd-resolvd.

$ sudo apt-get install dnsmasq

/etc/dnsmasq.conf

interface=docker0
bind-interfaces

We also need to configure docker daemon to point all container DNS queries to forward to the docker host system.

/etc/docker/daemon.json

{
    "dns": ["172.17.0.1"]
}

Then restart both docker & dnsmasq services:

$ sudo service docker restart
$ sudo service dnsmasq restart

fig 2: dnsmasq relaying DNS queries from docker interface to loopback interface

Now you would be able to get LAN’s updated upstream DNS servers inside docker containers whenever host’s DHCP connection changes, and will be reflected within the containers without restart.

Networking Devops