How do I get individual pod hostnames in a Deployment registered and looked up in Kubernetes?

Question

I need to know all the hostnames for all the pods in a Deployment in Kubernetes.

Based on https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/, I tried:

apiVersion: v1
kind: Service
metadata:
  name: default-subdomain
spec:
  selector:
    name: busybox
  clusterIP: None
  ports:
  - name: foo
    port: 1234
    targetPort: 1234
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox1
  labels:
    name: busybox
spec:
  replicas: 2
  selector:
    matchLabels:
      name: busybox
  template:
    metadata:
      labels:
        name: busybox
    spec:
      hostname: dummy <---- effect of this line 
      subdomain: default-subdomain
      containers:
      - image: busybox
        command:
          - sleep
          - "99999"
        name: busybox
        stdin: true
        tty: true

1. If I don't add the hostname, no pods are registered with DNS
2. If I do add the hostname value, there is only one entry in the DNS

How can I get every pod in a deployment to be registered, preferably using the pod name, and looked up by fqdn of the pod - e.g. pod_name.subdomin.namespace.svc.cluster.local?

Answer 1

CoreDNS creates A and SRV records only for Services. It doesn't generate pods' A records as you may expect after reading the documentation:

The pods insecure option is provided for backward compatibility with kube-dns. You can use the pods verified option, which returns an A record only if there exists a pod in same namespace with matching IP. The pods disabled option can be used if you don’t use pod records.

with the one exception: if you create a Headless service (when you specify ClusterIP: None in the Service spec)

So, here is my example of Headless Service based on your YAML:

NAME                TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
default-subdomain   ClusterIP      None            <none>        1234/TCP       50s

Here is the list of pods created by the above deployment on my cluster:

NAMESPACE       NAME                                        READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
default         busybox1-76745fcdbf-4ppsf                   1/1     Running   0          18s   10.244.1.22   kube-node2-1   <none>           <none>
default         busybox1-76745fcdbf-d76q5                   1/1     Running   0          18s   10.244.1.23   kube-node2-1   <none>           <none>

In this case, instead of one A and one SRV record for Service's ClusterIP, we have two A and two SRV records with the same name, and IP addresses of Pods which are Endpoints for the Headless Service:

default-subdomain.default.svc.cluster.local. 5 IN A 10.244.1.22
_foo._tcp.default-subdomain.default.svc.cluster.local. 5 IN SRV 0 50 1234 10-244-1-22.default-subdomain.default.svc.cluster.local.
default-subdomain.default.svc.cluster.local. 5 IN A 10.244.1.23
_foo._tcp.default-subdomain.default.svc.cluster.local. 5 IN SRV 0 50 1234 10-244-1-23.default-subdomain.default.svc.cluster.local.

To resolve SRV records, A records also has been created for both Headless Service endpoints.

If you don't specify specify hostname and subdomain for pods, A records will be created with IP addresses as a hostnames:

10-244-1-22.default-subdomain.default.svc.cluster.local. 5 IN A 10.244.1.22
10-244-1-23.default-subdomain.default.svc.cluster.local. 5 IN A 10.244.1.23

But if you are specify both of them you will get these record as follows:

dummy.default-subdomain.default.svc.cluster.local. 5 IN A 10.244.1.22
dummy.default-subdomain.default.svc.cluster.local. 5 IN A 10.244.1.23

SRV records will look as follows in this case (yes, there are still two of them and they are the same):

_foo._tcp.default-subdomain.default.svc.cluster.local. 5 IN SRV 0 50 1234 dummy.default-subdomain.default.svc.cluster.local.
_foo._tcp.default-subdomain.default.svc.cluster.local. 5 IN SRV 0 50 1234 dummy.default-subdomain.default.svc.cluster.local.

CoreDNS server resolve such records in "random" way (IP addresses is changing):

root@ubuntu:/# ping dummy.default-subdomain.default.svc.cluster.local -c 1 | grep PING
PING dummy.default-subdomain.default.svc.cluster.local (10.244.1.27) 56(84) bytes of data.
root@ubuntu:/# ping dummy.default-subdomain.default.svc.cluster.local -c 1 | grep PING
PING dummy.default-subdomain.default.svc.cluster.local (10.244.1.27) 56(84) bytes of data.
root@ubuntu:/# ping dummy.default-subdomain.default.svc.cluster.local -c 1 | grep PING
PING dummy.default-subdomain.default.svc.cluster.local (10.244.1.26) 56(84) bytes of data.
root@ubuntu:/# ping dummy.default-subdomain.default.svc.cluster.local -c 1 | grep PING
PING dummy.default-subdomain.default.svc.cluster.local (10.244.1.27) 56(84) bytes of data.
root@ubuntu:/# ping dummy.default-subdomain.default.svc.cluster.local -c 1 | grep PING
PING dummy.default-subdomain.default.svc.cluster.local (10.244.1.26) 56(84) bytes of data.
root@ubuntu:/# ping dummy.default-subdomain.default.svc.cluster.local -c 1 | grep PING
PING dummy.default-subdomain.default.svc.cluster.local (10.244.1.26) 56(84) bytes of data.
root@ubuntu:/# ping dummy.default-subdomain.default.svc.cluster.local -c 1 | grep PING
PING dummy.default-subdomain.default.svc.cluster.local (10.244.1.27) 56(84) bytes of data.

To debug it, I've used zone transfer feature that CoreDNS supports. To enable it you should add transfer to * line to coredns ConfigMap. You can replace * with specific IP for security. Example:

$ kubectl get cm coredns -n kube-system -o yaml
apiVersion: v1
data:
 Corefile: |
   .:53 {
       errors
       health
       kubernetes cluster.local in-addr.arpa ip6.arpa {
          transfer to *   <---- enable zone transfer to anyone(don't use in production) 
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf
       cache 30
       loop
       reload
       loadbalance
   }
kind: ConfigMap
metadata:
 creationTimestamp: "2019-05-07T15:44:02Z"
 name: coredns
 namespace: kube-system
 resourceVersion: "9166"
 selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
 uid: f0646569-70de-11e9-9af0-42010a9c0015 

After that you'll be able to list all DNS records from cluster.local zone using the following command:

dig -t AXFR cluster.local any

More information can be found here:

• support for zone transfer for kubernetes #1259
• Feature request: support zone transfers in Kubernetes middleware #660