If you have a few years of experience in the DevOps ecosystem, and you're interested in sharing that experience with the community, have a look at our Contribution Guidelines.
1. Introduction
In a Kubernetes cluster, certificates are critical in securing communication between different components. However, these certificates aren’t everlasting and can expire, leading to various operational issues.
In this tutorial, we’ll explore how to handle expired certificates in a Kubernetes cluster. We’ll cover checking certificate expiry, renewing certificates, and best practices to ensure a smooth and secure operation. Let’s get started!
2. Understanding Kubernetes Certificates
Kubernetes uses various certificates to ensure secure communication within the cluster:
- API server certificates: secure the Kubernetes API server communication
- kubelet certificates: secure the communication between the API server and the kubelet
- etcd certificates: secure the communication between the API server and the etcd database
- client certificates: used by different components to authenticate to the API server
Each of these certificates has an expiration date, and if they expire, components can fail to communicate securely, resulting in errors and potential downtime.
3. Identifying Expired Certificates
Our first step in dealing with expired certificates is identifying which certificates have expired or are about to expire.
To check the expiration dates of these certificates, we can use openssl (a powerful tool for working with SSL/TLS certificates) and kubeadm.
3.1. Checking a Specific Certificate
Let’s check for the expiration date of a specific certificate, such as the API server certificate:
$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep 'Not After'
Not After : Aug 13 14:32:00 2024 GMT
In this command, we used the following arguments:
- openssl x509 – invokes openssl to process X.509 certificates (a standard for public key certificates)
- -in /etc/kubernetes/pki/apiserver.crt – specifies the certificate file to read
- -noout – suppresses the output of the encoded version of the certificate
- -text – prints the certificate in text form
- grep ‘Not After’ – filters the output to show only the expiration date
As we can see, in this example, the certificate will expire on August 13, 2024, at 14:32:00 GMT.
3.2. Checking All Certificates in a Directory
Instead of checking certificates for their expiration dates individually, we can also use a loop to check the expiration dates of all certificates in the /etc/kubernetes/pki/ directory. This method excludes the ca.crt (Certificate Authority certificate), which is typically not checked for expiration in the same context.
Precisely, we’ll combine the find command with grep to execute a directory search:
$ find /etc/kubernetes/pki/ -type f -name "*.crt" -print | grep -v 'ca.crt$' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {} | grep "Not After"'
/etc/kubernetes/pki/apiserver.crt
Not After : Apr 13 14:32:00 2024 GMT
/etc/kubernetes/pki/front-proxy-client.crt
Not After : Apr 13 14:32:00 2024 GMT
/etc/kubernetes/pki/etcd/server.crt
Not After : Apr 13 14:32:00 2024 GMT
...
Let’s better understand our interaction here:
- find /etc/kubernetes/pki/ -type f -name “*.crt” – finds all files with a .crt extension in the specified directory
- -print – prints the path of each found file
- grep -v ‘ca.crt$’ – excludes the ca.crt file from the list
- xargs -L 1 – executes the command for each input line
- -t – prints the command before executing it
- -i – replaces {} with the file name from the input
- bash -c ‘openssl x509 -noout -text -in {} | grep “Not After”‘ – runs the command to read the certificate and filters for the expiration date
In short, this loop effectively checks the expiration dates of all certificates in the specified directory, allowing us to quickly identify any that have expired or are close to expiring.
3.3. Using kubeadm to Check All Certificates
kubeadm simplifies the process of managing Kubernetes clusters. It includes commands to check and renew certificates.
Let’s use kubeadm to check for certificate expiration:
$ kubeadm certs check-expiration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Oct 23, 2023 07:15 UTC 364d no
apiserver Oct 23, 2023 07:15 UTC 364d ca no
apiserver-kubelet-client Oct 23, 2023 07:15 UTC 364d ca no
controller-manager.conf Oct 23, 2023 07:15 UTC 364d no
front-proxy-client Oct 23, 2023 07:15 UTC 364d front-proxy-ca no
scheduler.conf Oct 23, 2023 07:15 UTC 364d no
As we can see, our output displays the expiration dates of all important certificates. It provides a comprehensive overview of certificate statuses, making it easier to identify which ones need renewal.
4. Renewing Expired Kubernetes Certificates
Once we’ve identified which certificates have expired or are about to expire, the next step is to renew them. Tools like kubeadm still come in handy and simplify this process.
Let’s look into methods to renew expired Kubernetes certificates, both automatically and manually.
4.1. Backup and Restore
Before starting the certificate renewal process, it’s a good practice to back up the current certificates. This ensures we can restore them if something goes wrong during renewal.
To do this, we can copy the contents of the certificates directory (/etc/kubernetes/pki) with the cp command:
$ cp -R /etc/kubernetes/pki /etc/kubernetes/pki.backupHere, we copy the entire pki directory to a backup location.
Now, we can proceed with our certificate renewal process.
4.2. Renewing All Certificates With Kubernetes Upgrade
Kubernetes 1.15+ introduced automatic certificate renewal during upgrades. This means that when we upgrade our Kubernetes cluster using kubeadm, it automatically renews the certificates as part of the upgrade process. This feature helps reduce manual intervention and ensures our cluster remains secure without the need for frequent manual renewals.
For instance, let’s see an example of upgrading our Kubernetes cluster to 1.20.1 with kubeadm upgrade.
To do this, first, we need to check the current version and available upgrades of our Kubernetes cluster with kubeadm upgrade plan:
$ kubeadm upgrade plan
Upgrade to the latest version in the v1.20 series:
COMPONENT CURRENT AVAILABLE
API Server v1.20.0 v1.20.1
Controller Manager v1.20.0 v1.20.1
Scheduler v1.20.0 v1.20.1
Kube Proxy v1.20.0 v1.20.1
CoreDNS 1.7.0 1.7.0
Etcd 3.4.13 3.4.14
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.20.1After reviewing the plan, we can now apply the upgrade with kubeadm upgrade apply:
$ kubeadm upgrade apply v1.20.1
[upgrade] Running pre-flight checks
[upgrade] Making sure the cluster is healthy:
[upgrade] Freeing etcd...
[upgrade] Upgrading to version v1.20.1
[upgrade] DaemonSet "kube-system/kube-proxy" updated
...
We can replace v1.20.1 with our preferred upgrade version.
With these upgrades, Kubernetes handles the certificate renewals seamlessly.
4.3. Renewing All Certificates With kubeadm
If automatic renewal isn’t an option, we can also manually renew the certificates with kubeadm. To do this, we have two options.
First, we can decide to renew all certificates at once:
$ kubeadm certs renew all
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to the kubelet renewed
certificate for the controller manager to talk to the apiserver renewed
certificate for the scheduler manager to talk to the apiserver renewed
certificate for the front proxy client renewed
...
This renews all certificates used by the Kubernetes control plane. The all argument specifies that all certificates should be renewed.
After renewing the certificates, we need to restart the control plane components, including the API server, controller manager, and scheduler, to use the new certificates.
For this, restarting the kubelet service with systemctl usually suffices:
$ sudo systemctl restart kubelet
This restarts the kubelet service.
4.4. Renewing Specific Certificates
If we only need to renew specific certificates, we can specify which ones to renew.
For example, we can decide to renew just the API server certificate:
$ kubeadm certs renew apiserver
certificate for serving the Kubernetes API renewed
Here, apiserver specifies the certificate to renew.
After renewing the certificates, we can re-check the new expiration dates using openssl, as discussed earlier, to verify that the renewal was successful.
4.5. Updating kubeconfig Files
Sometimes, when we renew certificates, we also need to update the kubeconfig files used by various components to authenticate to the API server. kubeadm handles this automatically when we use kubeadm certs renew all, but if we manually renew certificates, we might need to update the kubeconfig files ourselves.
First, we regenerate the files for the admin user:
$ kubeadm init phase kubeconfig admin
[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "admin.conf" kubeconfig fileNext, we regenerate the files for the controller manager:
$ kubeadm init phase kubeconfig controller-manager
...
[certs] Generating "controller-manager.conf" kubeconfig file
Lastly, we regenerate the files for the scheduler:
$ kubeadm init phase kubeconfig scheduler
...
[certs] Generating "scheduler.conf" kubeconfig file
These commands regenerate the kubeconfig files for the admin user, controller manager, and scheduler.
5. Regenerating Kubernetes Certificates
In some cases, we might need to regenerate specific certificates.
For instance, let’s see the detailed steps for regenerating certificates like the API server certificate.
First, we should move the old certificate (if it exists) to a backup location by renaming the existing API server certificate:
$ mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.old
$ mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.oldNext, we can regenerate a new certificate for the API server using kubeadm:
$ kubeadm alpha phase certs apiserver
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.1]
Afterward, we regenerate the new certificates for the API server kubelet client:
$ kubeadm alpha phase certs apiserver-kubelet-client
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver-kubelet-client" certificate and keyAfter regenerating the certificates, we need to update the kubeconfig files to ensure the new certificates reflect:
$ kubeadm alpha phase kubeconfig all
[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "admin.conf" kubeconfig file
[certs] Generating "controller-manager.conf" kubeconfig file
[certs] Generating "scheduler.conf" kubeconfig file
This regenerates all kubeconfig files.
6. Troubleshooting Kubernetes Certificate Issues
In case we encounter any issues in our certificate renewal or regeneration process, let’s briefly examine ways to identify and troubleshoot certificate issues in our Kubernetes cluster.
First, we can check the log files /var/log/kube-apiserver.log and /var/log/kubelet.log for certificate issues. We should look for entries mentioning certificate errors or expiration issues.
Alternatively, we can use journalctl -u kubelet to check the kubelet logs for certificate-related issues.
Lastly, we can view and validate the kubeconfig files with kubectl config view, which displays the current configuration and certificates. This ensures that our configurations are correct and up-to-date.
7. Conclusion
In this article, we explored methods for renewing expired Kubernetes certificates and covered troubleshooting steps for common certificate issues to avoid certificate-related disruptions. By following these guidelines, we can maintain the security and reliability of our Kubernetes clusters and prevent future issues.