Demystifying Endpoints in Kubernetes

1. Overview

Endpoints are typically an overlooked Kubernetes resource that many users don’t know much about. This is because we don’t need to create or interact with them directly. Instead, they’re automatically created and configured by Kubernetes in the background.

In this tutorial, we’re going to explain what endpoints are and what their role is in Kubernetes.

2. Ephemeral Pod IPs in Kubernetes

Pods are ephemeral by nature in Kubernetes, meaning that they can be terminated and recreated at any time. To maintain the availability and performance requirements for our services, we use a higher-level controller like a deployment to monitor our pods and keep their desired number as expected.

This ephemeral nature introduces a challenge with IP addressing. Whenever a pod gets deleted and recreated, a new IP address is assigned to it. This change in IP address prevents clients who want to connect to the pod from having a stable destination IP to send the traffic to. So, if a client was connecting to a pod with a specific IP, and this pod was terminated and recreated with a new IP, the client will disconnect, as the previous IP no longer exists.

Now, let’s see this in action by creating a deployment and terminating one of its pods:

$ kubectl create deploy test --image=nginx --replicas=1
deployment.apps/test created

$ kubectl get pods -owide
NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
test-764c85dd84-c6v5t   1/1     Running   0          7s    10.244.1.3   node01   <none>           <none>

In the above snippet, we created a deployment with the name test that includes one replica. When listing the pods, we can see that the pod is running and has an IP address of 10.244.1.3.

Let’s delete this pod and see what happens:

$ kubectl delete pod test-764c85dd84-c6v5t
pod "test-764c85dd84-c6v5t" deleted

$ kubectl get pods -owide
NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
test-764c85dd84-6nrzp   1/1     Running   0          21s   10.244.1.4   node01   <none>           <none>

As we can see, a new pod was created automatically, but with a new IP address of 10.244.1.4. Now, this new IP address assignment will disconnect any client that was previously connecting to the old pod.

3. Kubernetes Services

To overcome this problem of changing the IP address, Kubernetes introduced networking objects called services. A service is an abstraction that provides a stable IP address for clients to connect to, and then it proxies the traffic it receives to one or more backend pods.

We can think of a service as a load balancer, where there’s a stable frontend IP that is exposed to clients, and it forwards the traffic to other backend IPs that run the actual application. The clients are now not concerned with any change in the backend IPs, since they only need to know about the stable frontend IP.

Services automatically handle any change in the backend IP addresses. So, when a pod is terminated and recreated, the service will detect this, add the new IP address for the new pod, and remove the old one. In order for a service to track specific pods in the cluster, we use labels and selectors.

Now, let’s create a service object:

$ cat service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

The above output is our service YAML manifest. As we can see in the selector field, it will forward traffic to backend pods with label app: my-app. 

Let’s create a deployment that matches this label:

$ cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Here, we create our deployment YAML manifest using the nginx image and the same label of app: my-app.

Finally, we need to create our objects by deploying the YAML manifests to the cluster:

$ kubectl apply -f deployment.yaml 
deployment.apps/nginx-deployment created

$ kubectl apply -f service.yaml 
service/my-service created

Now, if we try to connect to our service, we’ll find that the response comes from the nginx pod:

$ curl my-service
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
---------------- OUTPUT TRIMMED ------------

So, this ensures that our service is working fine and forwarding the traffic to the correct pods.

4. Service Endpoints

As our service gets linked to its backend pods, Kubernetes automatically creates some objects called endpoints. An endpoint object represents the IP address and port number for one of the backend pods. In other words, the endpoints define the target that the service should forward the traffic to. Whenever a pod is created or removed, its corresponding endpoint is updated to keep the service in sync.

Let’s check the endpoints for our previous service:

$ kubectl describe svc my-service
Name:              my-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=my-app
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.103.145.6
IPs:               10.103.145.6
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.1.2:80,10.244.2.2:80,10.244.2.3:80

Above, we use the kubectl describe command to get more details about our service. We can see that the Endpoints are showing in the output with a list of IP addresses. Let’s verify these are the actual IP addresses of our pods:

$ kubectl get pods -owide
NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
nginx-deployment-79557f68d5-9lnlt   1/1     Running   0          95m   10.244.1.2   node01   <none>           <none>
nginx-deployment-79557f68d5-f6rqt   1/1     Running   0          95m   10.244.2.2   node02   <none>           <none>
nginx-deployment-79557f68d5-g4s4f   1/1     Running   0          95m   10.244.2.3   node02   <none>           <none>

And we can see here that our pods have the same IP addresses mapped to the service by the endpoints. Finally, let’s delete one of our pods and see what happens:

$ kubectl delete pod nginx-deployment-79557f68d5-9lnlt
pod "nginx-deployment-79557f68d5-9lnlt" deleted

$ kubectl get pods -owide
NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
nginx-deployment-79557f68d5-76kdl   1/1     Running   0          26s   10.244.1.8   node01   <none>           <none>
nginx-deployment-79557f68d5-f6rqt   1/1     Running   0          99m   10.244.2.2   node02   <none>           <none>
nginx-deployment-79557f68d5-g4s4f   1/1     Running   0          99m   10.244.2.3   node02   <none>           <none>

$ kubectl describe svc my-service
Name:              my-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=my-app
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.103.145.6
IPs:               10.103.145.6
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.1.8:80,10.244.2.2:80,10.244.2.3:80

With the first command above, we deleted the pod with the IP address 10.244.1.2. We can see that our pod was automatically replaced, thanks to our deployment controller, and the new pod was assigned an IP address of 10.244.1.8. If we look at our service endpoints, we’ll find that the new IP automatically replaced the old one. So now, any traffic coming to our service can be forwarded successfully to the new active pod.

5. Conclusion

In this article, we’ve covered the concept of Kubernetes endpoints. We described the problem of ephemeral IP addresses for pods, and then we explained how services provide a stable IP to overcome this challenge. Finally, we checked how endpoints represent the IP address for each of our backend pods, and how it automatically gets updated whenever a pod is replaced.