Troubleshooting Empty ADDRESS in Kubernetes Ingress

1. Introduction

Kubernetes ingress provides HTTP and HTTPS routing to services within the cluster. However, configuring ingress can sometimes lead to issues, such as the ingress resource showing an empty ADDRESS field.

In this tutorial, we’ll troubleshoot this common problem and explore setting up, configuring, and debugging ingress in Kubernetes. First, we’ll understand Kubernetes ingress and its components. Then, we’ll look into creating and configuring ingress resources.

Finally, we’ll examine common issues regarding Kubernetes ingress and their troubleshooting steps. Let’s get started!

2. Understanding Kubernetes Ingress

Ingress in Kubernetes acts as a bridge, allowing external traffic to reach services within the cluster. It consists of two main components: the ingress controller, which manages the routing rules, and the ingress resource, which defines the rules for routing traffic.

Let’s examine these two in more detail, as we need to understand them carefully before we begin our troubleshooting process.

2.1. The Ingress Controller

The ingress controller is a specialized load balancer for Kubernetes environments. It manages the traffic routing rules specified in the ingress resource.

Typically, we deploy the controller as a pod within the Kubernetes cluster, with several popular options, such as NGINX, Traefik, and HAProxy. Its primary function is to watch the Kubernetes API for changes to ingress resources and then update its configuration to reflect these changes. By dynamically adjusting its configuration, the ingress controller can effectively distribute traffic across multiple backend services, ensuring high availability and scalability.

Additionally, many ingress controllers can handle SSL/TLS termination, which offloads the encryption workload from the backend services, providing an extra layer of security. Furthermore, the controller supports custom routing rules, allowing us to direct traffic based on hostnames, paths, or even request headers.

In essence, the ingress controller is the brain behind traffic management in a Kubernetes cluster, adapting in real time to the specified ingress resources.

2.2. The Ingress Resource

The ingress resource is a Kubernetes object that defines the rules for routing external HTTP and HTTPS traffic to services within the cluster.

Typically, we define the ingress resource using a YAML file, which includes specifications for hostnames, paths, backend services, and, optionally, TLS settings. This resource allows us to set up rules to direct traffic to different services based on the request’s URL path or hostname.

For example, we can route traffic to example.com/api to one service while traffic to example.com/web goes to another. Also, if we require TLS, the ingress resource can specify which secret contains the SSL certificate and private key, ensuring secure communication. Moreover, we can use annotations to provide additional configuration options for the ingress controller, such as setting custom timeouts or enabling specific features.

Essentially, the ingress resource is the configuration file the ingress controller reads to understand how to route incoming traffic. It provides a flexible way to manage access to services within a Kubernetes cluster. By understanding both the ingress controller and the ingress resource, we can effectively manage and troubleshoot external traffic flow into our Kubernetes environment.

3. Setting Up the Ingress Controller

To use ingress, first, we need to install the ingress-nginx controller:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
deployment.apps/ingress-nginx-controller created
service/ingress-nginx-controller created

This command deploys the necessary resources for the ingress-nginx controller, including the namespace, serviceaccount, configmap, roles, role bindings, deployment, and service.

Then, after installation, we need to verify that the ingress controller is running.

To do this, let’s check the status of the pods in the ingress-nginx namespace:

$ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-controller-6d8bc4c7d5-9l6r8   1/1     Running     0          2m

Our output indicates that the ingress-nginx controller pod is running. The READY column shows that the pod is fully ready, and the STATUS column confirms it’s in the Running state. Ideally, the RESTARTS column should be 0, indicating no restarts have occurred.

However, common pitfalls during this setup include not waiting for the controller to be fully deployed or omitting necessary permissions. These issues can cause the controller to fail to start or function properly. We need to ensure that all resources are created and running as expected.

Notably, if we’re using Minikube, we might need to enable the ingress addon to facilitate the use of the ingress controller:

$ minikube addons enable ingress
The 'ingress' addon is enabled

Enabling the ingress addon in Minikube is an additional step that ensures seamless integration and functionality.

4. Creating and Configuring Ingress Resources

After setting up our ingress controller, we can now create an ingress resource to define our routing rules.

An ingress resource specifies how external traffic should be routed to services within our cluster.

Let’s see an example manifest for creating an ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: example.k8s.io
    http:
      paths:
      - path: /service
        pathType: Prefix
        backend:
          service:
            name: example-service
            port:
              number: 80

In this manifest, we define an ingress resource example-ingress. The annotations section includes an nginx-specific annotation that rewrites the target of incoming requests to the root path. This is useful for ensuring that requests are correctly routed to the backend service.

Under the spec section, we define a rule for the host example.k8s.io. This rule specifies that any HTTP requests to example.k8s.io/service should be routed to the backend service example-service on port 80. We set the pathType to Prefix, meaning that any request path that starts with /service will match this rule.

Then, we need to save the manifest to a file (such as example-ingress.yaml) and apply this resource to the cluster:

$ kubectl apply -f example-ingress.yaml
ingress.networking.k8s.io/example-ingress created

Our output confirms that the ingress resource has been successfully created in the cluster. The ingress controller will now pick up this resource, configure itself accordingly, and start routing traffic based on the defined rules.

5. Common Issues and Troubleshooting Tips

After setting up and configuring our ingress controller and resources, one common issue we might encounter is seeing an empty ADDRESS in the ingress resource:

$ kubectl get ingress
NAME              CLASS    HOSTS             ADDRESS   PORTS   AGE
example-ingress   <none>   example.k8s.io              80      10m

As we can see, the ADDRESS column is empty, which indicates that the ingress resource doesn’t have an assigned IP address or hostname. This is often the root of connectivity issues and generally indicates a problem with the ingress controller setup or configuration.

Let’s see some steps to troubleshoot it effectively.

5.1. Checking Ingress Controller Status

First and foremost, we need to ensure that the ingress controller is running correctly by checking the status of the pods:

$ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-controller-6d8bc4c7d5-9l6r8   1/1     Running     0          10m

Our output here confirms that the ingress controller pod is Running. If the STATUS column shows any state other than Running, such as CrashLoopBackOff or Error, there’s likely a problem that we need to address.

5.2. Verifying Ingress Annotations and Labels

Next, we need to ensure that the ingress resource has the necessary annotations.

For example, the annotation nginx.ingress.kubernetes.io/rewrite-target is often required for proper request routing:

annotations:
  nginx.ingress.kubernetes.io/rewrite-target: /

Annotations provide the ingress controller with additional instructions for handling incoming traffic.

Therefore, missing or incorrect annotations can prevent the ingress from functioning correctly.

5.3. Checking Ingress Controller Logs

To diagnose issues, we can also inspect the logs of the ingress controller pod:

$ kubectl logs -n ingress-nginx <controller-pod>

For instance, let’s see a sample inspection:

$ kubectl logs -n ingress-nginx ingress-nginx-controller-6d8bc4c7d5-9l6r8
[error] 20#20: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: example.k8s.io, request: "GET /service HTTP/1.1", upstream: "http://10.0.0.2:80/service"

Here, the log indicates a connection issue between the ingress controller and the backend service. We need to ensure the backend service is running and accessible.

5.4. Describing the Ingress Resource

Another thing we can do is describe the ingress resource to check for warnings or errors.

To do this, we can use the kubectl describe command to get more detailed information about the ingress resource:

$ kubectl describe ingress example-ingress
Name:             example-ingress
Namespace:        default
Address:          <none>
Default backend:  <default>
Rules:
  Host              Path  Backends
  ----              ----  --------
  example.k8s.io
                    /service   example-service:80 (10.0.0.2:80)
Annotations:        nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Warning  Sync    10m (x5 over 10m)  nginx-ingress-controller  Error during sync: error connecting to backend

Our output here shows an event warning about an error during synchronization, which indicates network issues or a possible misconfiguration in the backend service.

5.5. Addressing the Errors

To address this error, or if we see errors or warnings in the logs or descriptions, we need to verify our service endpoints. First, we need to ensure the backend services have the correct endpoints and are accessible.

Next, we should check our network policies to confirm that no policies are blocking traffic between the ingress controller and the backend services. Also, we should validate ingress rules by double-checking them for correct paths and hosts to ensure proper routing.

Lastly, we need to ensure we’ve correctly configured the services the ingress resource references. This is especially important for services using NodePort or LoadBalancer types. For instance, a misconfigured service might not be accessible to the ingress controller.

6. Testing and Verifying Ingress Setup

To ensure that our ingress setup works correctly, we can verify the configuration and functionality using several kubectl commands and other tools.

6.1. Testing Ingress With curl

To verify that the ingress is routing traffic correctly, we can use curl to make a request to the specified host and path:

$ curl http://example.k8s.io/service
<!DOCTYPE html>
<html>
<head>
<title>Welcome to example-service!</title>
</head>
<body>
<h1>Success! The example-service is running.</h1>
</body>
</html>

Our output indicates that the request was successfully routed through the ingress to the backend service example-service, and the expected response was received.

6.2. Monitoring Ingress Logs

For real-time debugging and to monitor the ingress controller’s activity, we can check the logs of the ingress controller pod:

$ kubectl logs -n ingress-nginx <controller-pod>
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.47.0
  Build:         git-6ea333c45
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.6

-------------------------------------------------------------------------------

W0515 10:15:30.123456       6 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
I0515 10:15:30.123456       6 main.go:110] "Creating API client" host="https://10.96.0.1:443"
I0515 10:15:30.123456       6 main.go:154] "Running in Kubernetes cluster" major="1" minor="20" git="git-6ea333c45" state="clean" commit="6ea333c45" platform="linux/amd64"
I0515 10:15:30.123456       6 main.go:96] "Validated ingress-nginx configuration"
I0515 10:15:30.123456       6 main.go:99] "Starting Ingress controller"
I0515 10:15:30.123456       6 nginx.go:250] "Starting NGINX Ingress controller"

Our output here shows the initialization and running status of the nginx-ingress controller.

By keeping a tab on these logs, we can identify any issues or errors that occur during the operation of the ingress controller.

7. Conclusion

In this article, we explored troubleshooting steps for an empty ADDRESS in Kubernetes ingress. By setting up the ingress controller correctly, verifying configurations, and using advanced features, we can ensure our ingress resources function correctly.

Finally, we need to remember that regular monitoring and adherence to best practices will help us maintain a robust and reliable Kubernetes environment.

Full Archive

About Baeldung