What error pattern appears in 'kubectl logs --previous'?

Diagnostic options for this step: Application throws unhandled runtime exceptions, missing environment variables, or database connection failures (Exit Code 1)., Kubelet reports 'exec format error', 'no such file or directory', or command entrypoint failure (Exit Code 127 / 126).

Interactive Troubleshooter

Published: Sep 4, 2026

Kubernetes Pod CrashLoopBackOff: Root Cause Fix

Time: 15 mins Level: Beginner

By StepDiag Engineering Team

Reviewed by Ibraheem Taofeeq Opeyemi

An exhaustive interactive diagnostic guide to fix Kubernetes Pod CrashLoopBackOff errors, OOMKilled states, and container startup failures. Follow structured troubleshooting steps for k8s memory limits, probe timeouts, and volume mount issues.

Full Diagnostic Tree & Step-by-Step Overview

What specific container exit status, exit code, or event signature is displayed when running 'kubectl describe pod <pod-name>'?

Container terminates with Exit Code 137 or Last State shows 'OOMKilled: true'.
Container exits immediately with Exit Code 1 or Exit Code 127 during application startup.
Pod runs temporarily but fails Liveness or Readiness probes repeatedly until restarted.
Container crashes due to missing Secrets, ConfigMaps, or persistent volume mount failures.

What resource metric or node-level event triggers the memory termination?

Container memory usage hits the hard limit set in resources.limits.memory.
Worker node experiences system-wide memory pressure causing pod eviction (NodeMemoryPressure).

Fix OOMKilled (Exit Code 137) Memory Excursions

Solution:

Root Cause: Container Exceeding Defined Memory Limits

When a container attempts to allocate more memory than defined in its resources.limits.memory spec, the Linux kernel Out-Of-Memory (OOM) killer sends a SIGKILL signal (exit code 137) to the process. Kubernetes then marks the container as failed and initiates a back-off restart cycle.

Step-by-Step Fix:

1Verify OOM Termination in Pod Events:

Run kubectl describe pod <pod-name> -n <namespace>.

Check the Last State block for Exit Code: 137 and Reason: OOMKilled.

2Analyze Container Memory Metrics:

Check real-time usage using kubectl top pod <pod-name> --containers.

Review historical memory growth in Prometheus/Grafana to identify memory leaks versus baseline resource undersizing.

3Increase Spec Memory Limits and Requests:

Edit your deployment manifest (kubectl edit deployment <deployment-name>).

Update the container spec limits:

yaml
     resources:
       requests:
         memory: "512Mi"
         cpu: "250m"
       limits:
         memory: "1Gi"
         cpu: "500m"

4Optimize Application Garbage Collection:

For Java (JVM) runtimes, set explicit heap parameters via environment variables (e.g., -Xmx768m) to ensure JVM memory remains below the container limit.

Resolve CPU Throttling & Node Memory Pressure Evictions

Solution:

Root Cause: Worker Node Resource Exhaustion and Kubelet Eviction Thresholds

When a worker node memory utilization breaches the memory.available eviction threshold set on the kubelet, the control plane marks the node under MemoryPressure. Kubelet begins evicting BestEffort and Burstable pods, forcing continuous container restarts.

Step-by-Step Fix:

1Inspect Worker Node Conditions:

Check node status: kubectl get nodes and kubectl describe node <node-name>.

Look under Conditions for MemoryPressure: True or DiskPressure: True.

2Identify Resource-Heavy Pods:

Run kubectl top nodes and kubectl top pods --all-namespaces --sort-by=memory.

3Apply Guaranteed Quality of Service (QoS) Class:

Ensure critical pods have identical requests and limits for both CPU and memory to assign Guaranteed QoS class, protecting them from eviction during node pressure:

yaml
     resources:
       requests:
         memory: "1Gi"
         cpu: "1000m"
       limits:
         memory: "1Gi"
         cpu: "1000m"

4Drain and Rescale Node Capacity:

Cordon and drain the overloaded node: kubectl cordon <node-name> then kubectl drain <node-name> --ignore-daemonsets.

Scale your cluster autoscaler or add additional worker nodes to redistribute workloads.

What error pattern appears in 'kubectl logs <pod-name> --previous'?

Application throws unhandled runtime exceptions, missing environment variables, or database connection failures (Exit Code 1).
Kubelet reports 'exec format error', 'no such file or directory', or command entrypoint failure (Exit Code 127 / 126).

Fix Application Runtime Errors, Missing Secrets, and Env Vars

Solution:

Root Cause: Runtime Unhandled Exceptions & Missing Application Dependencies

Exit Code 1 indicates that the container application started, encountered a fatal runtime exception (such as a missing environment variable, database connection timeout, or invalid configuration syntax), and exited abruptly. Follow the Kubernetes Official Debugging Guide for standard pod state inspection routines.

Step-by-Step Fix:

1Examine Terminated Container Logs:

Fetch logs from the crashed container instance: kubectl logs <pod-name> -c <container-name> --previous.

Look for stack traces, missing key errors, or failed endpoint connections.

2Verify Environment Variables, ConfigMaps, and Secrets:

Check that all envFrom or valueFrom referenced ConfigMaps and Secrets exist in the namespace:

bash
     kubectl get configmap -n <namespace>
     kubectl get secret -n <namespace>

Confirm key names in the manifest match the exact keys defined inside the Secret/ConfigMap.

3Test Database and Dependent Service Connectivity:

Launch an ephemeral debug container inside the pod network namespace:

bash
     kubectl debug -it <pod-name> --image=busybox -- sh

Test DNS resolution and port reachability: nslookup <service-name> and telnet <service-name> <port>.

Fix Missing Entrypoint Executables, Shell Paths, and Exit Code 127

Solution:

Root Cause: Invalid Container Entrypoint, Command Syntax, or Shell Script Line Endings

Exit Code 127 occurs when the container runtime cannot locate the command specified in the image ENTRYPOINT, CMD, or manifest command field. Exit Code 126 occurs when the binary exists but lacks execution permissions (chmod +x). Windows-style CRLF line endings in shell scripts (entrypoint.sh) also cause interpreter errors like /bin/sh^M: bad interpreter.

Step-by-Step Fix:

1Verify Container Command and Args Manifest Spec:

Inspect your deployment manifest for incorrect command overrides:

yaml
     spec:
       containers:
       - name: app-container
         image: myapp:v1.0
         command: ["/bin/sh", "-c"]
         args: ["/app/start.sh"]

2Validate Script Line Endings (CRLF vs LF):

Convert Windows CRLF line endings to Linux LF before building Docker images:

bash
     dos2unix entrypoint.sh

3Inspect Image Entrypoint Path:

Run a local container interactively to verify executable paths:

bash
     docker run --rm -it --entrypoint /bin/sh myapp:v1.0
     ls -la /app/start.sh

Ensure proper execution permissions exist (chmod +x /app/start.sh) inside the Dockerfile.

Which specific health probe failure is reported in the pod event log?

Liveness probe failed: HTTP probe failed with statuscode 500 or connection refused.
Slow application startup causes Liveness probe to trip before initial boot completes.

Adjust Liveness and Readiness Probe Failure Thresholds

Solution:

Root Cause: Aggressive Health Probe Timing or Invalid HTTP Health Endpoints

If a livenessProbe fails continuously for failureThreshold attempts, the kubelet kills the container and triggers a restart. If the health check endpoint returns an unexpected HTTP status code (non 200-399 range) or exceeds timeoutSeconds, the pod enters CrashLoopBackOff.

Step-by-Step Fix:

1Check Probe Events in Describe Logs:

Run kubectl describe pod <pod-name> and check the Events section for messages like Liveness probe failed: HTTP probe failed with statuscode 500.

2Validate Health Endpoint Manually:

Forward local port to pod to test the health endpoint directly:

bash
     kubectl port-forward pod/<pod-name> 8080:8080
     curl -i http://localhost:8080/healthz

3Adjust Probe Timing Parameters in Manifest:

Increase initialDelaySeconds, timeoutSeconds, and failureThreshold:

yaml
     livenessProbe:
       httpGet:
         path: /healthz
         port: 8080
       initialDelaySeconds: 30
       periodSeconds: 10
       timeoutSeconds: 5
       failureThreshold: 3

Configure Startup Probes for Slow Booting Applications

Solution:

Root Cause: Slow Container Initialization Tripping Liveness Checks

Legacy microservices, Java applications, or database migration containers often require 60+ seconds to initialize caches or complete schema migrations. If only a livenessProbe is configured, it will repeatedly kill the container mid-boot, creating a perpetual CrashLoopBackOff cycle.

Step-by-Step Fix:

1Implement a Dedicated Startup Probe:

Add a startupProbe to disable liveness and readiness checks until the container finishes initial bootstrapping:

yaml
     startupProbe:
       httpGet:
         path: /healthz
         port: 8080
       failureThreshold: 30
       periodSeconds: 10
     livenessProbe:
       httpGet:
         path: /healthz
         port: 8080
       periodSeconds: 10

In this setup, the application has up to 300 seconds (30 * 10s) to complete boot tasks before liveness probing begins.

2Apply Manifest and Verify Boot Completion:

Apply deployment configuration: kubectl apply -f deployment.yaml.

Stream logs to confirm boot sequence completes: kubectl logs -f <pod-name>.

What type of storage, configuration, or security restriction is failing?

Volume mount permission denied (EACCES) or non-root user securityContext restrictions.
Port conflict or socket binding error (EADDRINUSE) on container network interface.

Fix Volume Mount Failures and SecurityContext Permission Denied Errors

Solution:

Root Cause: File System Permission Mismatch & Security Context User Lockouts

When securityContext.runAsUser specifies a non-root User ID (UID), but mounted volumes (PersistentVolumeClaims or hostPaths) are owned by root:root (UID 0), the containerized application fails to open configuration or data files (Permission denied / EACCES) and crashes instantly.

Step-by-Step Fix:

1Identify Permission Error in Logs:

Check application failure logs: kubectl logs <pod-name> --previous | grep -i permission.

2Configure fsGroup in Pod SecurityContext:

Add fsGroup to the pod security context so Kubernetes automatically changes group ownership of mounted volumes to match the container user group:

yaml
     spec:
       securityContext:
         runAsUser: 10001
         runAsGroup: 10001
         fsGroup: 10001
       containers:
       - name: app
         image: myapp:v1.0

3Remediate PersistentVolume Ownership via InitContainer:

If using hostPath or rigid volume plugins, add an initContainers block running as root to chown the volume mount path prior to main application execution:

yaml
     initContainers:
     - name: volume-permissions
       image: busybox:latest
       command: ["sh", "-c", "chown -R 10001:10001 /var/data"]
       volumeMounts:
       - name: data-volume
         mountPath: /var/data

Resolve Container Port Binding Conflicts and Socket Errors

Solution:

Root Cause: TCP/UDP Port Collisions and HostPort Allocation Locks

If a container attempts to bind to a network port (EADDRINUSE) that is already bound by another process in the same pod container network namespace, or if multiple pods using hostPort land on the same worker node, socket creation fails and the container terminates.

Step-by-Step Fix:

1Review Port Allocation in Container Spec:

Inspect container deployment for duplicate port definitions or hostPort usage:

bash
     kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].ports}'

2Remove HostPort Bindings and Utilize K8s Services:

Remove hostPort assignments from pod specs to avoid node-level socket conflicts. Rely on standard Kubernetes Service (ClusterIP, NodePort, LoadBalancer) or Ingress routing.

3Update Application Listening Port:

If running multiple containers inside a single Pod manifest, ensure each container binds to distinct listening ports (e.g., App on 8080, Metrics Exporter on 9090).

Related Diagnostic Guides

Windows Update Error 0x80070002 Missing Files Diagnostic Guide

System Restore Failed 0x80070005 Antivirus Conflict Tree

Windows Photo Viewer Not Showing Thumbnails: Codec Diagnostic

Safe Mode Password Not Working With Microsoft Account Fix

Clipboard History Not Working Windows 11: Sync Service Tree

Subscribe to StepDiag

Embed Guide on Your Website