Full Diagnostic Tree & Step-by-Step Overview
What exit code or primary state is reported when running 'docker inspect <container_id>' or 'docker ps -a'?
- Exit Code 137 (Fatal error: Out of memory / Killed signal)
- Exit Code 1, 126, 127, or application startup error/crash
- Exit Code 0 (Exited cleanly immediately after starting)
- Container reports (unhealthy) status under Docker Healthcheck
What specific OOM indicator is reported in container inspection or system kernel logs?
- 'OOMKilled: true' is set in 'docker inspect --format="{{.State.OOMKilled}}"'
- 'OOMKilled: false', but system host kernel dmesg shows Out of memory kills
Fix Container Memory Limit Exceeded (OOMKilled: true)
Solution:
Root Cause: Container Exceeded Set Memory Limit (OOMKilled: true)
The Docker daemon forcibly terminated PID 1 inside the container because its RAM consumption exceeded the hard memory limit defined via
--memory or
deploy.resources.limits.memory in Docker Compose.
# Step-by-Step Fix:
Verify OOM State via Docker Inspect:Execute: docker inspect CONTAINER_ID --format 'OOMKilled: {{.State.OOMKilled}}, ExitCode: {{.State.ExitCode}}'.Confirm OOMKilled: true and ExitCode: 137.Increase Memory Allocation in Docker Compose / Command:In docker-compose.yml, update memory limits: yaml
deploy:
resources:
limits:
memory: 2GB
reservations:
memory: 512MB
For CLI invocation: docker run -d --memory="2g" --memory-swap="3g" my-image.Identify & Fix Application Memory Leaks:Monitor real-time consumption: docker stats CONTAINER_ID.Inspect heap dumps or profile application garbage collection parameters (e.g., set -Xmx for Java applications to stay within container limits).
Fix Host System Memory Exhaustion & Linux Kernel OOM Killer
Solution:
Root Cause: Host System Memory Exhaustion Invoking Linux Kernel OOM Killer
The host operating system ran completely out of physical RAM and swap space. The Linux kernel OOM killer selected the Docker container process for termination to prevent a total host kernel panic.
# Step-by-Step Fix:
Inspect Host Kernel Logs:Run command on host: dmesg -T | grep -i -E 'oom|killed process'.Confirm if containerd-shim or your application process was killed by Out of memory: Kill process.Check Host RAM & Swap Consumption:Check host resource usage: free -h and vmstat 1 5.Configure Memory Overcommit and Add Swap:Create host swap space if unavailable: sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile.
Set OOM Score Adjustments or Host Enforced Limits:Ensure all running containers have hard memory limits set so no single container can deplete total host RAM.
What specific output or error pattern is displayed at the bottom of 'docker logs <container_id>'?
- exec user process caused: no such file or directory OR Exit code 126/127
- Unhandled exception, missing environment variable, or database connection error
Fix Entrypoint Shebang, Windows CRLF Line Endings, or Binary Path Error
Solution:
Root Cause: Windows CRLF Line Endings or Missing Binary in Entrypoint Script
When entrypoint shell scripts are edited on Windows host machines, CRLF (\r\n) line endings are injected into the file. When executed inside a Linux container, the shebang line fails (e.g., looking for /bin/sh\r), resulting in Exit Code 127 or no such file or directory.
# Step-by-Step Fix:
Convert CRLF Line Endings to LF (Unix Format):Convert the entrypoint script on your host machine: dos2unix entrypoint.sh or use VS Code status bar to change CRLF -> LF.
Verify Shell Shebang and Execution Permissions:Ensure the top line of entrypoint.sh is exactly #!/bin/sh or #!/bin/bash without extra characters.Ensure executable permissions: chmod +x entrypoint.sh.Validate Command Path in Dockerfile:Verify ENTRYPOINT or CMD uses valid JSON array syntax: ENTRYPOINT ["/bin/sh", "/app/entrypoint.sh"] instead of string syntax if path resolution fails.
Rebuild Image Without Cache:Run: docker build --no-cache -t my-image:latest ..
Fix Application Runtime Exceptions & Missing Environment Variables
Solution:
Root Cause: Unhandled Application Runtime Exception or Unset Environment Variable
The container process starts successfully, but the inner application encounters an unhandled runtime error (e.g., missing database connection string, invalid API key, missing directory) and exits with Code 1.
# Step-by-Step Fix:
Capture Exhaustive Error Logs:Retrieve the last 200 lines of standard error: docker logs --tail 200 CONTAINER_ID.
Verify Injected Environment Variables:Check injected env vars: docker inspect CONTAINER_ID --format '{{json .Config.Env}}'.Ensure required secrets or .env file values are passed correctly in Docker Compose.Run Container Interactively to Debug:Override entrypoint to launch an interactive shell: docker run -it --entrypoint /bin/sh my-image:latest.
Manually execute your app startup command inside the shell to observe error output in real time.
What type of process is configured as the main CMD/ENTRYPOINT inside the container?
- A background service/daemon (e.g., Nginx, Apache, sshd) that detaches upon startup
- A short-lived batch script, database migration, or CLI utility with 'restart: always'
Fix Background Daemon Process Exit (Ensure PID 1 Remains in Foreground)
Solution:
Root Cause: PID 1 Process Exited cleanly (Daemon Process Forked to Background)
Docker containers only stay running as long as the primary process (PID 1) defined in CMD or ENTRYPOINT remains active in the foreground. If a service forks to the background (e.g., running nginx without daemon off; or service apache2 start), PID 1 exits immediately with status code 0.
# Step-by-Step Fix:
Force Daemon Services to Run in Foreground:Nginx: Change CMD ["nginx"] to CMD ["nginx", "-g", "daemon off;"].Apache: Use CMD ["apache2ctl", "-D", "FOREGROUND"].PHP-FPM: Use CMD ["php-fpm", "-F"].Keep Generic Script Running via Foreground Tail:If running custom scripts, append a blocking command at the end of your script: exec tail -f /dev/null or wait.
Avoid Using Systemd/Service Commands:Do not use service ssh start inside Docker containers. Execute the binary directly in foreground: CMD ["/usr/sbin/sshd", "-D"].
Fix Misconfigured Restart Policy on Batch / One-off Task Containers
Solution:
Root Cause: One-off Utility Container Configured with 'restart: always'
The container ran a batch job, database migration script, or backup task that finished successfully and exited with Code 0. However, because
restart: always or
restart: unless-stopped was specified in Docker Compose, Docker automatically restarts it endlessly.
# Step-by-Step Fix:
Update Restart Policy in Docker Compose:Change restart policy to no or on-failure: yaml
restart: on-failure
Use Docker Compose Profiles or Kubernetes Jobs for Ephemeral Tasks:For database migrations or setup jobs, isolate them using Compose profiles so they do not continuously run alongside persistent web services: docker compose run --rm migration-service.
Remove Automatic Restart from Running Container:Update container policy live: docker update --restart=no CONTAINER_ID.
Why is the Docker engine marking the container status as (unhealthy)?
- Healthcheck command binary missing or returning non-zero exit code
- Application takes longer to boot than healthcheck start_period and timeout allow
Fix Missing Healthcheck Utilities and Failing Check Commands
Solution:
Root Cause: Healthcheck Command Missing Binary or Hitting Incorrect Target Port
The HEALTHCHECK directive in the Dockerfile or Docker Compose relies on a tool (like curl or wget) that does not exist in lightweight base images (e.g., Alpine or Distroless), causing the check command itself to fail.
# Step-by-Step Fix:
Inspect Detailed Healthcheck Failure Output:Run command: docker inspect --format='{{json .State.Health}}' CONTAINER_ID | jq.Review FailingStreak and the specific exit code / stderr of the test command.Replace Missing Tools in Healthcheck Command:If curl is missing in Alpine, install it in Dockerfile (RUN apk add --no-cache curl) or switch to native shell/python checks: HEALTHCHECK CMD wget --quiet --tries=1 --spider http://localhost:8080/health || exit 1.
Verify Internal Port and Localhost Binding:Ensure healthcheck tests localhost on the *container's internal port*, not the external host mapped port.
Adjust Healthcheck Timings and Add Startup Grace Period (start_period)
Solution:
Root Cause: Insufficient Startup Grace Period (start_period) for Heavy Applications
The application requires 30-60 seconds to complete database connections or warm up caches, but the healthcheck begins probing immediately and hits the maximum retry threshold before the app is ready.
# Step-by-Step Fix:
Add start_period to Dockerfile or Compose Configuration:Configure a grace period during which failed health checks do not count toward the failure threshold: yaml
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
interval: 10s
timeout: 5s
retries: 3
start_period: 40s
Increase Timeout and Retry Intervals:If app startup is slow under heavy CPU load, increase retries from 3 to 5 and timeout from 5s to 10s.Test Healthcheck Locally:Manually run test command inside container to verify response time: docker exec -it CONTAINER_ID curl -f http://localhost:8080/health.