The Zombie Apocalypse in Kubernetes: Strategies for Dealing with Zombie Pod Processes in Kubernetes

Zombie Processes in Kubernetes

In Kubernetes, a zombie process refers to a container that is no longer responding to signals, including termination signals, but is still running and consuming resources.

The container is considered a zombie because it is not responding to any commands and is essentially dead, but it continues to consume resources, such as CPU, memory, and network bandwidth.

Zombie processes can occur in any containerized environment, but they are more common in Kubernetes due to its complex architecture and the way it manages container lifecycles.

Unlocking the Full Potential of GPU-Heavy Apps in Kubernetes with Node Templates

Common Causes

• Container crashes: When a container crashes, it may not always clean up after itself
• Network issues: Network problems can prevent a container from receiving termination signals
• Resource constraints: Insufficient resources, such as CPU, memory, or disk space
• Misconfigured containers: Containers that are not configured properly or have incorrect settings
• Kubernetes cluster issues: Problems with the Kubernetes cluster, such as etcd failures or network issues
• Improper termination of a process: If a process is not terminated properly, it can leave a zombie process behind
• Orphaned processes: If a parent process dies without properly reaping its child processes, the child processes can become orphaned
• Software bugs: Some software bugs can cause a process to enter a zombie state, especially if the software does not handle errors or terminations properly

How Zombie Pod Processes Arise

Zombie Pod Processes

In Kubernetes, pods are the basic units of deployment. A pod consists of one or more containers that share the same network namespace and storage volumes. Each container runs a single process (or a group of processes) that performs a specific task.

When a pod is deleted or evicted, Kubernetes sends a SIGTERM signal to each container in the pod. This signal tells the container to gracefully terminate its main process and any child processes it may have spawned. The container has a grace period (default 30 seconds) to perform any cleanup actions before it is forcibly killed by a SIGKILL signal.

However, sometimes the main process or its child processes do not handle the SIGTERM signal properly and ignore it.

This can happen for various reasons:

• Process is blocked on an I/O operation or a system call
• Process has installed a custom signal handler that overrides the default behavior
• Process has spawned a detached child process that does not inherit the signal disposition of the parent
• Process has created a subshell that does not propagate the signal to its commands

In these cases, the container will not exit until it receives the SIGKILL signal, which cannot be caught or ignored. However, this signal only kills the main process of the container, not its child processes. If the main process has left behind any orphaned child processes, they will become zombies and remain in the process table until they are reaped by another process.

Impact

• Resource consumption: Zombie processes consume resources, such as CPU, memory, and network bandwidth, which can lead to performance issues and slow down the cluster
• Container crashes: Zombie processes can cause container crashes, which can lead to data loss and affect the overall availability of the application
• Cluster instability: Zombie processes can cause the Kubernetes cluster to become unstable, leading to issues such as node failures, pod failures, and service disruptions
• Security risks: Zombie processes can pose security risks, as they can be used to launch attacks on other containers or the host system

Detection Strategies

• Check the process table: You can use the ps command to check the process table and look for processes in the “Z” state. However, this method may not be accurate in all cases, as some systems may not update the process table immediately after a process has terminated
• Use the Kubernetes API: You can use the Kubernetes API to list pods and their running containers. If a container has exited but is still running, it may be a zombie process. You can use the kubectl get pods command to list pods and their status
• Use the Docker API: If you are using Docker containers in your Kubernetes cluster, you can use the Docker API to inspect containers and their running processes. If a process has exited but is still running, it may be a zombie process. You can use the docker inspect command to inspect containers and their processes

Prevention Strategies

• Monitor container health: Regularly monitor container health
• Use liveness probes: Liveness probes can help detect when a container is no longer responding and terminate it
• Use readiness probes: Readiness probes can help ensure that a container is ready to receive traffic before it is exposed to the network
• Use a proper signal handler in the container entrypoint: When a pod is terminated, Kubernetes sends a SIGTERM signal to the main process in each container. The process should catch this signal and perform a graceful shutdown, such as closing open files, releasing resources, or notifying other services. If the process does not exit within a grace period (default 30 seconds), Kubernetes sends a SIGKILL signal to force-kill it
• Configure containers correctly: Ensure that containers are configured correctly and have appropriate resource requests and limits to prevent resource constraints
• Use resource constraints: Use resource constraints to limit the resources that a container can consume
• Use pod placement constraints: Use pod placement constraints to prevent pods from being scheduled on nodes that are experiencing issues
• Regularly update Kubernetes components: Regularly update Kubernetes components, such as the kernel, to address known issues
• Use Kubernetes network policies: Use Kubernetes network policies to restrict network access to and from containers
• Use a pod-level PID namespace: A PID namespace isolates the process ID space for a group of processes. By default, each container in a pod has its own PID namespace, which means that it cannot see or interact with processes in other containers or on the host. However, Kubernetes also supports sharing a PID namespace across all containers in a pod. To enable pod-level PID namespace, set shareProcessNamespace: true in the pod spec
• Monitor Kubernetes logs: Monitor Kubernetes logs to detect issues that may lead to zombie processes
• Use third-party tools: Use third-party tools, such as container performance monitoring and security tools

Tools for Detecting and Preventing Pod Zombie Processes

• kubectl: Command-line tool can be used to list and delete pods and containers in the cluster. For example, to list all pods that are not in Running state on a node, run kubectl get pods — field-selector=status.phase!=Running — all-namespaces -o wide — node <node-name>. To delete a pod forcefully, run kubectl delete pod <pod-name> — force — grace-period=0
• tini, dumb-init, s6-overlay: Use an init container to run a process reaper that periodically scans and kills zombie processes. This can be done by using a tool such as dumb-init or tini, or by writing a custom script that uses the ps and kill commands
• node-problem-detector: Is an agent that runs on each node and detects various node problems, such as disk pressure, kernel deadlock, or docker daemon failure
• kube-zombie: Controller that watches for pods that are stuck in Terminating state and deletes them automatically after a configurable timeout

Conclusion

Zombie processes in Kubernetes can cause performance issues, stability problems, and security risks. Understanding the causes of zombie processes and implementing best practices to mitigate the risks can help prevent issues and ensure the health and stability of the Kubernetes cluster. Regularly monitor container health, use liveness and readiness probes, configure containers correctly, and use resource constraints to prevent zombie processes.

Additionally, regularly update Kubernetes components, use Kubernetes network policies, monitor Kubernetes logs, and use third-party tools to detect and mitigate zombie processes.

Chaos Engineering in Kubernetes: A Guide to Building Scalable and Fault-Tolerant Microservices