How to configure a Linux host to resolve and access Kubernetes services by name

In this post, I will explain how to configure a Ubuntu host to have access to services inside a Kubernetes cluster installed with microk8s.

Photo by Evan Dennis on Unsplash

How Kubernetes internal DNS works?

All Pods and Services are connected to a private virtual network with IPs in the range 10.x.x.x and the host where Kubernetes is installed is not connected to this network and that is why it does not has access to services or pods.

To exemplify, let’s say you have created a database and exposed it by a service simply named my-db as in this simple YAML file:

apiVersion: v1
kind: Service
metadata:
name: my-db

As described in the official documentation, Kubernetes will create a full name to assign to this service in the format:

[service-name].[namespace].svc.cluster.local

And in our example it will be: my-db.default.svc.cluster.local

Internally, the Pods will find this service because the Kubernetes’ name system (DNS) will convert this name into an IP in the range 10.x.x.x.

It is not required to type the suffix .default.svc.cluster.local because the DNS server is also configured to auto-complete it for us.

How to connect to the Kubernetes private network

There is an addon for microk8s named host-access that will connect the host to the Kubernetes’ private network and it will grant access to the internal IPs in the cluster.

The addon official description is:

The host-access add-on enables access to services running on the host machine via a fixed IP.

To enable the host-access addon, just run the command:

microk8s enable host-access

A new network interface will be created in the host with the name lo:microk8s and with the IP 10.0.1.1 connecting the host to the Kubernetes’ private network.

Now that the host is connected, the access by IPs is enabled, but the name resolution still needs to be configured.

How to find the Kubernetes’ internal DNS IP

We need to discover which is the IP of the service converting names into IPs for Kubernetes so we can use the same in our host.

To find the Kubernetes’ internal DNS server IP we just need to list the running services in the kube-system namespace:

$ microk8s.kubectl get service --namespace kube-system
NAME TYPE CLUSTER-IP
kube-dns ClusterIP 10.152.183.10

Many other services might be running, but the important one is just the kube-dns one and its IP, and in this case, it is 10.152.183.10.

PS: This IP may be different for you depending on your installation and configuration, so it is extremely necessary to run the above command to find the right IP.

Configure the DNS resolution for Kubernetes’ services

Ubuntu, since 16.04, uses the service systemd-resolved to manage the DNS servers being used by the system, and it is in this service folder we are going to make changes.

Two files are auto-generated by the service systemd-resolved:

/run/systemd/resolve/stub-resolv.conf
/run/systemd/resolve/resolv.conf

The first one is just an empty file without any effect in the system, while the second one is the file containing the desired settings.

Unix systems read the file /etc/resolv.conf to discover which DNS servers are available, and if we analyze this file on Ubuntu with the following command:

$ ls -l /etc | grep resolv
> resolv.conf -> /run/systemd/resolve/stub-resolv.conf

It is possible to see that it is just a symbolic link pointing to the stub file mentioned above, that has no effect, leaving the responsibility of resolving names into IPs to the DHCP server of your network, and it doesn’t know anything about services running inside the Kubernetes cluster.

Let’s remove the symbolic link and create another one pointing to the right file:

$ sudo rm /etc/resolv.conf
$ sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

By default on Ubuntu, systemd-resolved comes with only one configuration file: /etc/systemd/resolve.conf, but we cannot change this file because everything will be lost in the next system update, therefore it is NOT recommended to change this file.

The solution to avoid losing the configuration is to create a new folder and put our file inside that new folder.

Our new configuration file might have any name as long as it ends with the extension .confg:

$ sudo mkdir -p /etc/systemd/resolved.conf.d
$ sudo vim /etc/systemd/resolved.conf.d/00-k8s-dns-resolver.conf

Content of: /etc/systemd/resolved.conf.d/00-k8s-dns-resolver.conf

[Resolve]
Cache=yes
DNS=10.152.183.10
Domains=default.svc.cluster.local svc.cluster.local cluster.local

Breakdown the config file:

  • Cache=yes: Keep in memory the already discovered names to make the next request faster.
  • DNS=10.152.183.10: This is the IP of the Kubernetes internal DNS, we discovered it in the previous step.
  • Domains=default.svc.cluster.local svc.cluster.local cluster.local: Instruct the DNS server to retry, in case of failure, appending these domains as suffices to the requested name.

And now to finish it and load the configuration, restart the systemd-resolved service:

$ sudo service systemd-resolved restart

Testing the configuration:

Now it is time to test if it is possible to find the service from the beginning of the post, the my-db service.

Simply run the ping command to find out if the name is resolved to the actual IP:

$ ping my-db
PING my-db.default.svc.cluster.local (10.152.183.5)...
64 bytes from my-db.default.svc.cluster.local (10.152.183.5)...
64 bytes from my-db.default.svc.cluster.local (10.152.183.5)...
64 bytes from my-db.default.svc.cluster.local (10.152.183.5)...

As we can see by the ping result, the name my-db was converted to its full-name, also know as Full Qualified Domain Name (FQDN), to the IP 10.152.183.5.

The configuration was written to include multiple variations of the service name, so all of these commands are valid:

ping my-db
ping my-db.default
ping my-db.default.svc

In case you have multiple namespaces, not only the default, it is just a matter of including another domain in our config file: /etc/systemd/resolved.conf.d/00-k8s-dns-resolver.conf.

I hope this post can help you configure a Kubernetes cluster as a working environment or as your Personal cloud.

Frontend Engineer for more than 15 years now, in love with tech in general and not tied do a single stack. I like to make experiments with Node and DevOps