Interactive Troubleshooter

Published: Oct 18, 2026

Kubernetes Pod Stuck at CrashLoopBackOff - Log Analysis Tree

Time: 15 mins Level: Beginner

By StepDiag Engineering Team

Reviewed by Ibraheem Taofeeq Opeyemi

Exhaustive diagnostic decision tree to isolate and resolve Kubernetes Pod CrashLoopBackOff states caused by application runtime exceptions, container exit codes (OOMKilled, command errors), failing liveness probes, and permission blocks.

Full Diagnostic Tree & Step-by-Step Overview

What container exit code or reason is reported in the 'Last State' field when running 'kubectl describe pod <pod_name>'?

OOMKilled or Exit Code 137 (SIGKILL due to cgroup memory exhaustion)
Exit Code 1 or 255 (Uncaught application runtime exception or fatal error)
Exit Code 127 or 126 (Command not found or script execution permission denied)
Exit Code 0 (Container completed execution and exited cleanly, but restartPolicy forced a reboot)

What memory allocation metric or application layer caused the container to breach its memory envelope?

Container RSS memory exceeded the explicit 'resources.limits.memory' defined in the Pod spec
Java JVM heap allocation failed to respect container limits due to legacy JDK ergonomics
Node host system ran out of swap/RAM, triggering the Linux kernel node-level OOM killer
Application memory leak steadily accumulated uncollected references until container termination

What error pattern or stack trace appears in 'kubectl logs <pod_name> --previous'?

Missing environment variable, secrets, or ConfigMap key lookup failure during startup
Database connection timeout, DNS lookup failure, or upstream service unreachable
Liveness probe failures repeatedly killing a slow-starting or heavily loaded application
Filesystem permission denied (EACCES) when writing to mounted persistent volumes

How is the container entrypoint or command defined within the Pod specification or Dockerfile?

The 'command' or 'args' array references a binary path or shell script that does not exist in the image
An entrypoint wrapper shell script lacks executable (+x) permissions or has Windows CRLF line endings
Dynamic library dependency (.so / DLL) missing or glibc / musl compatibility mismatch

Why did the short-lived container task trigger an unexpected CrashLoopBackOff cycle?

Pod runs a batch job or migration script configured with default 'restartPolicy: Always'
Container process spawns in background and main thread exits immediately (lacking foreground process)

Increasing Pod Memory Limits and Tuning Cgroup Resource Allocation

Solution:

Root Cause: Container Cgroup Memory Limit Exceeded

When a container process exceeds the memory ceiling specified in its Pod manifest (resources.limits.memory), the Linux kernel Cgroup OOM driver terminates the process with SIGKILL (Exit Code 137). Kubernetes captures this event and marks the termination reason as OOMKilled.

Diagnostic Verification:

1.Inspect the Pod status using kubectl describe pod <pod_name> -n <namespace>.

2.Look under the Last State section for Exit Code: 137 and Reason: OOMKilled.

3.Check resource metrics if Metrics Server is running:

bash
   kubectl top pod <pod_name> --containers -n <namespace>

Step-by-Step Fix:

1.Edit the Deployment or Pod Manifest:

Increase the memory limit in the spec file:

yaml
     resources:
       requests:
         memory: "512Mi"
       limits:
         memory: "1Gi"

2.Apply the Manifest Update:

Execute:

bash
     kubectl apply -f deployment.yaml -n <namespace>

3.Verify Pod Stability:

Monitor Pod status to ensure it transitions to Running:

bash
     kubectl get pods -n <namespace> -w

Prevention & Long-Term Monitoring:

Implement Horizontal Pod Autoscalers (HPA) based on memory utilization and set up Prometheus alerts for containers approaching 85% of their memory limits.

Configuring Container-Aware JVM Ergonomics (-XX:+UseContainerSupport)

Solution:

Root Cause: JVM Max RAM Mismatch and Off-Heap Overflows

Legacy Java applications (or modern JVMs without container support flags) read total physical host node memory instead of container cgroup memory limits. The JVM calculates default heap sizes based on host RAM, resulting in physical heap allocations that quickly breach container limits and trigger OOMKilled (Exit Code 137).

Diagnostic Verification:

1.Retrieve stdout/stderr logs from the crashed container:

bash
   kubectl logs <pod_name> -c <container_name> --previous -n <namespace>

2.Check for java.lang.OutOfMemoryError: Java heap space or sudden silent SIGKILL exits.

Step-by-Step Fix:

1.Pass Container-Aware Flags to JVM:

Update environment variables in your Pod manifest to enable container support and set explicit RAM percentages:

yaml
     env:
       - name: JAVA_TOOL_OPTIONS
         value: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0"

2.Balance Heap vs Off-Heap Memory:

Ensure resources.limits.memory accounts for off-heap allocations, Metaspace, and thread stacks (e.g., set container limit to 1.5x of the max heap target).

3.Redeploy the Workload:

Apply the updated manifest via kubectl apply.

Prevention & Long-Term Monitoring:

Always standardize on modern JDK releases (Java 11/17/21) and monitor JVM Native Memory Tracking (NMT) in staging environments.

Diagnosing Host-Level Kernel Memory Pressure and Node Evictions

Solution:

Root Cause: Worker Node RAM Exhaustion and System OOM Kills

When a Kubernetes worker node experiences global memory pressure because node-level resource reservations (system-reserved or kube-reserved) are absent or insufficient, the host Linux kernel OOM killer terminates processes arbitrarily to keep the kernel alive.

Diagnostic Verification:

1.Inspect worker node system logs via kubectl:

bash
   kubectl describe node <node_name>

2.Look for SystemOOM events or conditions showing MemoryPressure = True.

Step-by-Step Fix:

1.Configure Kubelet Resource Reservations:

Add explicit system reservations in the Kubelet configuration file on the worker node:

yaml
     systemReserved:
       memory: "1Gi"
     kubeReserved:
       memory: "1Gi"
     evictionHard:
       memory.available: "500Mi"

2.Restart Kubelet Daemon:

Run systemctl restart kubelet on the affected node.

3.Balance Workload Distribution:

Cordon and drain heavily loaded nodes if needed:

bash
     kubectl drain <node_name> --ignore-daemonsets --delete-emptydir-data

Prevention & Long-Term Monitoring:

Define explicit ResourceQuotas per namespace and enforce mandatory Pod request/limit values across all clusters.

Tracing Application-Level Memory Leaks and Unbounded Heaps

Solution:

Root Cause: Unbounded Application Memory Leak

Software bugs (such as unclosed database cursors, static collection accumulation, or global caching without eviction policies) cause the application process to steadily consume memory over time until it breaches its cgroup memory limit.

Diagnostic Verification:

1.Plot memory consumption over time using kubectl top or Grafana dashboard metrics.

2.Observe a steady linear upward slope in memory usage prior to the CrashLoopBackOff event.

Step-by-Step Fix:

1.Capture Memory Dump Before Crash:

Temporarily increase memory limit to give the container headroom, then execute a heap dump:

bash
     kubectl exec -it <pod_name> -n <namespace> -- jcmd 1 GC.heap_dump /tmp/dump.hprof

2.Extract and Analyze Heap Dump:

Copy the dump file to your local machine for analysis:

bash
     kubectl cp <namespace>/<pod_name>:/tmp/dump.hprof ./dump.hprof

3.Patch Application Code:

Fix identified object memory leaks in the application code and release a new container image tag.

Prevention & Long-Term Monitoring:

Integrate automated load testing into CI/CD pipelines to catch memory leaks before promoting code to production.

Resolving Missing Secrets, ConfigMaps, and Environment Variables

Solution:

Root Cause: Startup Failure Due to Missing Configuration Objects

Applications that fail to find required environment variables, mounted ConfigMaps, or Kubernetes Secrets during initialization throw fatal startup exceptions, causing the container process to crash immediately with Exit Code 1.

Diagnostic Verification:

1.Inspect container logs for the failed execution:

bash
   kubectl logs <pod_name> --previous -n <namespace>

2.Look for stack traces such as KeyError, NullPointerException, or ConfigMap "app-config" not found.

3.Check Pod events:

bash
   kubectl get events -n <namespace> --field-selector involvedObject.name=<pod_name>

Step-by-Step Fix:

1.Verify ConfigMap and Secret Existence:

Confirm that all referenced resources exist in the target namespace:

bash
     kubectl get configmap <config_name> -n <namespace>
     kubectl get secret <secret_name> -n <namespace>

2.Correct Name and Key References:

Match the manifest valueFrom references to actual secret keys:

yaml
     env:
       - name: DATABASE_URL
         valueFrom:
           secretKeyRef:
             name: db-secrets
             key: url

3.Re-apply Configurations:

Create missing resources and re-apply the deployment.

Prevention & Long-Term Monitoring:

Use GitOps tools (like ArgoCD or Flux) to ensure Secrets and ConfigMaps are synchronized before deploying application Pods.

Fixing Downstream Database Timeouts and CoreDNS Name Resolution Drops

Solution:

Root Cause: Network Unreachability and DNS Resolution Failures

If an application requires active database connectivity or downstream service handshakes during startup, network policy blocks, missing DNS records, or database outages will cause the application to crash on startup.

Diagnostic Verification:

1.Check previous logs for connection errors:

bash
   kubectl logs <pod_name> --previous -n <namespace> | grep -iE 'timeout|refused|dns|unknown host'

2.Test DNS resolution and network connectivity using an ephemeral debug pod:

bash
   kubectl run nettest --rm -i --tty --image=busybox -- nslookup <service_name>.<namespace>.svc.cluster.local

Step-by-Step Fix:

1.Verify Downstream Service and Endpoints:

Ensure target database or microservice endpoints are active:

bash
     kubectl get endpoints <service_name> -n <namespace>

2.Audit NetworkPolicies:

Confirm that NetworkPolicies permit outbound traffic on the required ports (e.g., 5432 for PostgreSQL, 6379 for Redis).

3.Add Retry Backoff Logic in Application:

Update application startup code to retry database connections with exponential backoff rather than terminating instantly.

Prevention & Long-Term Monitoring:

Implement initContainers to wait for downstream dependencies to become reachable before launching the main container.

Tuning Liveness Probe Initial Delays and Timeout Thresholds

Solution:

Root Cause: Premature Liveness Probe Failures During Cold Boot

If an application takes longer to initialize than allowed by the livenessProbe configuration (initialDelaySeconds + failureThreshold), Kubernetes assumes the application is dead and sends a SIGKILL, trapping the Pod in a CrashLoopBackOff state.

Diagnostic Verification:

1.Inspect Pod events for liveness probe failure entries:

bash
   kubectl describe pod <pod_name> -n <namespace>

2.Look for event text stating: Liveness probe failed: HTTP probe failed with statuscode 500 or Readiness probe failed: Connection refused.

Step-by-Step Fix:

1.Adjust Probe Delays and Thresholds:

Increase initialDelaySeconds and failureThreshold in the Pod manifest:

yaml
     livenessProbe:
       httpGet:
         path: /healthz
         port: 8080
       initialDelaySeconds: 45
       periodSeconds: 10
       failureThreshold: 6

2.Utilize Startup Probes for Slow Initialization:

Add a startupProbe to shield slow-starting containers during boot:

yaml
     startupProbe:
       httpGet:
         path: /healthz
         port: 8080
       failureThreshold: 30
       periodSeconds: 10

3.Apply Updated Spec:

Run kubectl apply -f deployment.yaml -n <namespace>.

Prevention & Long-Term Monitoring:

Always separate startupProbe logic from livenessProbe to accommodate cold boots without sacrificing fast fault detection.

Resolving Persistent Volume Claim (PVC) Permission Blocks (fsGroup / runAsUser)

Solution:

Root Cause: Persistent Volume Mount Permission Denied (EACCES)

When a container runs as a non-root user (configured via securityContext.runAsUser), but mounted Persistent Volumes (PVCs) belong to root:root, the application crashes upon attempting to write log files or database locks to the volume.

Diagnostic Verification:

1.Check previous container logs for permission errors:

bash
   kubectl logs <pod_name> --previous -n <namespace> | grep -i 'permission denied'

2.Inspect security context in the Pod manifest:

bash
   kubectl get pod <pod_name> -n <namespace> -o yaml | grep -A 10 securityContext

Step-by-Step Fix:

1.Configure Pod-Level fsGroup:

Set fsGroup in the Pod's securityContext to automatically recursively change volume group ownership on mount:

yaml
     spec:
       securityContext:
         runAsUser: 10001
         runAsGroup: 10001
         fsGroup: 10001

2.Use InitContainers to Chown Volume (Alternative):

If fsGroup is unsupported by the storage provider, use an initContainer running as root:

yaml
     initContainers:
       - name: fix-permissions
         image: busybox:latest
         command: ["sh", "-c", "chown -R 10001:10001 /data"]
         volumeMounts:
           - name: storage-volume
             mountPath: /data

3.Apply Updated Manifest:

Run kubectl apply -f deployment.yaml -n <namespace>.

Prevention & Long-Term Monitoring:

Test non-root container deployment security policies in lower environments prior to cluster migration.

Fixing Invalid Container Entrypoint Paths and Missing Binaries (Exit Code 127)

Solution:

Root Cause: Missing Executable Binary or Invalid Command Path

An Exit Code 127 indicates that the Linux shell or container runtime could not locate the executable binary specified in the container command or args array (or Dockerfile ENTRYPOINT / CMD).

Diagnostic Verification:

1.Check kubectl describe pod <pod_name> for Exit Code: 127.

2.Review configured container command:

bash
   kubectl get pod <pod_name> -n <namespace> -o jsonpath='{.spec.containers[*].command}'

Step-by-Step Fix:

1.Verify Executable Path in Container Image:

Run an interactive temporary container using the same image to verify binary paths:

bash
     kubectl run debug-test --rm -i --tty --image=<your_image> -- /bin/sh

2.Update Manifest Command Definition:

Fix typos or provide absolute paths in the manifest spec:

yaml
     command: ["/usr/local/bin/app"]
     args: ["--config", "/etc/app/config.yaml"]

3.Re-apply Manifest:

Execute kubectl apply -f deployment.yaml -n <namespace>.

Prevention & Long-Term Monitoring:

Use multi-stage Docker builds to ensure final runtime images contain all necessary binaries and correct $PATH exports.

Correcting Script Execution Permissions (+x) and Windows CRLF Line Endings

Solution:

Root Cause: Script Execution Permission Denied or Shebang Parsing Failure

An Exit Code 126 or 127 occurs when an entrypoint shell script (entrypoint.sh) lacks executable file mode permissions (chmod +x), or contains Windows CRLF (\r\n) line endings that prevent the Linux kernel shebang interpreter (#!/bin/bash\r) from running.

Diagnostic Verification:

1.Inspect container termination status in kubectl describe pod for Exit Code 126 or 127.

2.Check logs for messages like /bin/bash^M: bad interpreter: No such file or directory.

Step-by-Step Fix:

1.Fix Line Endings in Repository:

Convert CRLF to LF format on your source code file:

bash
     dos2unix entrypoint.sh

2.Grant Executable Permissions:

Update Git file permissions:

bash
     chmod +x entrypoint.sh
     git update-index --chmod=+x entrypoint.sh

3.Rebuild and Push Container Image:

Rebuild the Docker image and update the deployment image tag in Kubernetes.

Prevention & Long-Term Monitoring:

Configure .gitattributes to enforce * text eol=lf across all project repositories.

Resolving Shared Library (.so) and C-Runtime (glibc/musl) Mismatches

Solution:

Root Cause: Missing Dynamic C-Libraries or Musl/Glibc Incompatibility

When compiling binaries on a glibc-based OS (like Ubuntu/Debian) and copying them into a musl-based container image (like Alpine Linux), executing the binary fails with No such file or directory (Exit Code 127) because the dynamic linker (/lib/ld-linux-x86-64.so.2) is missing.

Diagnostic Verification:

1.Check logs for missing dynamic library errors:

bash
   kubectl logs <pod_name> --previous -n <namespace>

2.Test binary dependencies using ldd inside a debug container.

Step-by-Step Fix:

1.Switch Base Image to Match C-Runtime:

Match compiler and runtime base images (e.g., use debian:bookworm-slim for glibc binaries or alpine for musl binaries).

2.Compile Statically Linked Binaries:

For Go/C++ applications, build fully static binaries to eliminate dynamic library dependencies:

bash
     CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .

3.Rebuild Image and Deploy:

Push updated container image and reload the Kubernetes deployment.

Prevention & Long-Term Monitoring:

Enforce consistent base operating systems across build stages in multi-stage Dockerfiles.

Configuring 'restartPolicy: Never' / 'OnFailure' for Batch Jobs

Solution:

Root Cause: Incorrect Restart Policy on Short-Lived / Batch Workloads

When a container runs a short-lived script (such as a database migration or batch processing task) that completes successfully with Exit Code 0, Kubernetes default restartPolicy: Always causes the kubelet to restart the container perpetually, eventually trapping the Pod in CrashLoopBackOff.

Diagnostic Verification:

1.Run kubectl describe pod <pod_name> and check:

State: Terminated, Exit Code: 0.

Restart Policy: Always.

Step-by-Step Fix:

1.Convert Workload to Kubernetes Job:

For one-off batch tasks, use a Kubernetes Job or CronJob spec instead of a Deployment:

yaml
     apiVersion: batch/v1
     kind: Job
     metadata:
       name: db-migration
     spec:
       template:
         spec:
           restartPolicy: OnFailure
           containers:
             - name: migration
               image: my-migration-tool:v1

2.Update Standalone Pod Spec:

If using standalone Pods, change restartPolicy to OnFailure or Never:

yaml
     spec:
       restartPolicy: OnFailure

3.Deploy Updated Object:

Delete old deployment and apply the Job manifest.

Prevention & Long-Term Monitoring:

Never deploy short-lived processing tasks using standard Kubernetes Deployment manifests.

Keeping Container Processes Running in Foreground (PID 1)

Solution:

Root Cause: Main Process Detaching or Running in Background

Docker containers require a foreground process associated with PID 1 to remain active. If an entrypoint script starts a daemon in the background (e.g., nginx, apache2ctl start, or systemctl) and exits, PID 1 terminates with Exit Code 0, causing Kubernetes to continually restart the container.

Diagnostic Verification:

1.Check kubectl logs <pod_name> --previous.

2.Observe that the log ends normally after executing a daemon start command without keeping STDOUT open.

Step-by-Step Fix:

1.Force Foreground Execution in Container Command:

Update the container command or Dockerfile to run daemons in the foreground:

For Nginx: nginx -g 'daemon off;'

For Apache: httpd -DFOREGROUND

2.Use Tail or Wait in Entrypoint Scripts:

If executing complex background initialization scripts, keep PID 1 alive at the end of the script:

bash
     #!/bin/sh
     /usr/sbin/service my-service start
     exec tail -f /var/log/my-service.log

3.Apply Updated Spec:

Rebuild image, update deployment manifest, and execute kubectl apply.

Prevention & Long-Term Monitoring:

Avoid using traditional Linux init systems (systemd/sysvinit) inside container images.

Related Diagnostic Guides

High Server Load But Low CPU: iowait & Zombie Process Tree

MySQL Too Many Connections Error - Connection Leak Diagnostic

Docker Container Exits With Code 137: OOM Diagnostic

Nginx 502 Bad Gateway After Deployment - Upstream Diagnostic

Ubuntu Black Screen After Nvidia Driver Install - Secure Boot Diagnostic

Subscribe to StepDiag

Embed Guide on Your Website