Interactive Troubleshooter

Published: Oct 18, 2026

Docker Container Exits With Code 137: OOM Diagnostic

Time: 15 mins Level: Beginner

By StepDiag Engineering Team

Reviewed by Ibraheem Taofeeq Opeyemi

Exhaustive diagnostic decision tree to isolate and resolve Docker container exit code 137 (SIGKILL / OOMKilled) events caused by cgroup memory hard limits, host kernel memory pressure, JVM heap misconfigurations, and IPC shared memory exhaustion.

Full Diagnostic Tree & Step-by-Step Overview

How is the container exit code 137 reported, and what is the output of 'docker inspect <container_id>'?

'OOMKilled: true' with ExitCode 137 in docker inspect output
'OOMKilled: false' with ExitCode 137, but host kernel dmesg shows out-of-memory events
ExitCode 137 occurs during build time ('docker build' or CI/CD pipeline step)
ExitCode 137 occurs only on specific high-throughput applications (e.g., PyTorch, Chromium, PostgreSQL)

What specific cgroup memory parameter or runtime limit is configured on the container?

A hard memory limit (--memory / mem_limit) is defined and reached by application working set memory
A Java / JVM application process exceeds container cgroup limits due to off-heap / Metaspace growth
A Python / Node.js worker leaks memory or buffers large payloads in RAM until hitting cgroup caps
Kernel memory or swap memory limits (--memory-swap) are severely misconfigured or disabled

What host-level resource constraint triggered the unmanaged SIGKILL (Signal 9)?

The host node ran out of physical RAM and swap, triggering the host OS kernel OOM killer
External orchestration (Kubernetes, Docker Swarm, Systemd) sent an ungraceful SIGKILL timeout
Docker Desktop VM (WSL2 or Hyper-V / macOS Lima) hit its allocated virtual machine memory ceiling

Where does the memory allocation failure occur during the image build pipeline?

Heavy compilation steps (e.g., C++/Rust compilation, webpack / Vite bundling, npm build)
BuildKit daemon container exceeding host or runner cgroup resource limits

What specialized Linux subsystem resource is being exhausted inside the container container?

Shared memory (/dev/shm) default 64MB capacity exhausted by multi-process workloads
Database engine (PostgreSQL / MySQL) buffer pool exceeding container limits under concurrent queries

Adjusting Docker Cgroup Hard Memory Limits and Swap Allocations

Solution:

Root Cause: Container Cgroup Memory Hard Limit Breach

When a container's memory consumption reaches its configured cgroup limit (--memory or mem_limit), the Linux kernel cgroup memory controller terminates the process inside the cgroup using a SIGKILL signal (128 + 9 = 137). Docker records this state as OOMKilled: true.

Diagnostic Verification:

1.Run docker inspect <container_name_or_id> and verify:

bash
   docker inspect --format='{{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' <container_id>

2.Check cgroup memory limits vs usage:

bash
   docker stats <container_id> --no-stream

Step-by-Step Fix:

1.Increase Container Memory Allocation:

For CLI executions, raise the memory ceiling:

bash
     docker run -d --memory="2g" --memory-swap="4g" my-app:latest

For Docker Compose setups, update docker-compose.yml:

yaml
     version: '3.8'
     services:
       app:
         image: my-app:latest
         deploy:
           resources:
             limits:
               memory: 2G
             reservations:
               memory: 512M

2.Update Limits on Running Containers dynamically:

Apply new limits without stopping the container:

bash
     docker update --memory 2G --memory-swap 4G <container_id>

3.Verify Status:

Relaunch workload and confirm memory headroom via docker stats.

Prevention & Long-Term Monitoring:

Set memory reservations alongside hard limits to ensure orchestrators place containers on nodes with adequate capacity.

Configuring JVM Native Memory and Cgroup-Aware Heap Tuning

Solution:

Root Cause: Unbounded JVM Native Memory and Off-Heap Overhead

Java applications inside containers frequently crash with exit code 137 when the Java Virtual Machine (JVM) native memory allocation (Metaspace, Thread Stacks, Garbage Collection tables, Direct ByteBuffers) pushes total process RAM usage past the container's cgroup memory limit, even if -Xmx (Max Heap) is set lower than the limit.

Diagnostic Verification:

1.Check if -Xmx matches or exceeds container memory limits.

2.Review JVM Native Memory Tracking (NMT) in logs or run:

bash
   jcmd <pid> VM.native_memory baseline

Step-by-Step Fix:

1.Enable Cgroup-Aware JVM Ergonomics:

Ensure Java 11+ or Java 8u191+ is used, which respects cgroup memory limits by default via -XX:+UseContainerSupport.

2.Calculate Heap Ceiling Correctly:

Set maximum heap (-Xmx) to no more than 70%-75% of total container memory to leave room for off-heap allocations:

bash
     JAVA_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:ActiveProcessorCount=4"

3.Limit Metaspace and Thread Stack Sizes:

Add explicit limits in JAVA_OPTS:

bash
     -XX:MaxMetaspaceSize=256m -Xss512k

4.Redeploy Container:

Restart container with updated environment flags.

Prevention & Long-Term Monitoring:

Always monitor total resident set size (RSS) rather than JVM heap usage alone.

Tracing Application-Level Memory Leaks and Heap Buffers

Solution:

Root Cause: Application-Level Memory Leak or Unbounded In-Memory Buffering

Software applications written in Python, Node.js, or Go may continuously accumulate uncollected references, cache objects, or buffer large file uploads entirely in RAM, steadily driving process memory up until the Linux cgroup killer intervenes.

Diagnostic Verification:

1.Monitor memory progression over time:

bash
   docker stats <container_id> --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"

2.Check application logs leading up to the exit for memory warnings or allocation errors.

Step-by-Step Fix:

1.Configure Node.js Max Old Space Size:

Set --max-old-space-size to ~75% of container RAM limit:

bash
     NODE_OPTIONS="--max-old-space-size=1536"

2.Configure Python Garbage Collection & Worker Recycles:

For Gunicorn/Uvicorn, force worker recycling to release leaked memory:

bash
     gunicorn --workers 4 --max-requests 1000 --max-requests-jitter 50 wsgi:app

3.Enable Streaming File Processing:

Refactor application code to stream large payloads or file uploads directly to disk/s3 instead of loading them into RAM.

Prevention & Long-Term Monitoring:

Integrate continuous memory profiling tools (e.g., pprof, clinic.js) into staging environments.

Configuring Swap Limit Capabilities and Linux Kernel Memory Settings

Solution:

Root Cause: Unconfigured Swap Memory and Host Cgroup Driver Misconfiguration

If container swap limits are not configured or swap accounts are disabled at the Linux kernel boot parameters level, containers hitting their memory cap are immediately killed instead of gracefully swapping pages to disk.

Diagnostic Verification:

1.Run docker info and check for warnings:

bash
   docker info | grep -i swap
   # Warning: WARNING: No swap limit support

Step-by-Step Fix:

1.Enable Cgroup Swap Accounting in GRUB:

Edit /etc/default/grub on the host Linux machine:

bash
     GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

2.Update GRUB and Reboot Host:

Execute:

bash
     sudo update-grub && sudo reboot

3.Set Swap Limits on Containers:

Run containers with proportional swap capacity:

bash
     docker run -d --memory="1g" --memory-swap="2g" my-app:latest

Prevention & Long-Term Monitoring:

Ensure production Kubernetes/Docker hosts have swap accounting enabled if swapping is permissible under workload SLAs.

Resolving Host Node Memory Exhaustion and Global OOM Kills

Solution:

Root Cause: Physical Host Node Memory Depletion

When total memory demand across all running containers and host daemons exceeds physical RAM and swap capacity, the host OS Linux kernel invokes the global OOM killer (mm/oom_kill.c). The kernel selects candidate processes based on oom_score and terminates them via SIGKILL (exit code 137).

Diagnostic Verification:

1.Check host system kernel logs using dmesg or journalctl:

bash
   dmesg -T | grep -i -E 'oom|killed process'

2.Look for lines like: Out of memory: Kill process <pid> (docker-container) score <score>.

Step-by-Step Fix:

1.Restrict Unlimited Containers:

Ensure all running containers on the host have explicit --memory limits so no single container can drain host memory.

2.Adjust OOM Score Adjustments:

Protect critical system containers or daemons from being targeted first:

bash
     docker run -d --oom-score-adj=-500 critical-app:latest

3.Add Host Swap Space:

Create a dedicated swap file on the host to buffer short memory spikes:

bash
     sudo fallocate -l 4G /swapfile
     sudo chmod 600 /swapfile
     sudo mkswap /swapfile
     sudo swapon /swapfile

Prevention & Long-Term Monitoring:

Implement host-level monitoring via Prometheus Node Exporter to alert when host memory utilization exceeds 85%.

Fixing Orchestrator Timeout Kills and Ungraceful SIGKILL Signals

Solution:

Root Cause: Forced SIGKILL Termination Following Shutdown Timeout

When Docker, Docker Swarm, or Kubernetes attempts to stop a container, it sends SIGTERM and waits for a graceful shutdown period (default 10 seconds). If the application fails to handle SIGTERM or takes too long to clean up resources, the daemon sends SIGKILL (signal 9), resulting in exit code 137.

Diagnostic Verification:

1.Inspect events log to determine if exit 137 was preceded by a stop command:

bash
   docker events --until 0m --filter event=die

2.Review application logs to check if SIGTERM was received but ignored.

Step-by-Step Fix:

1.Increase Stop Timeout in Docker Run / Compose:

Allow more time for graceful shutdown:

bash
     docker run --stop-timeout 30 my-app:latest

In docker-compose.yml:

yaml
     services:
       app:
         image: my-app:latest
         stop_grace_period: 30s

2.Handle SIGTERM inside Application Code:

Ensure main process (PID 1) catches SIGTERM signals and initiates graceful shutdown of active connections.

3.Use Tini or Dumb-Init Init System:

If app runs as PID 1 without signal handlers, add an init wrapper:

dockerfile
     RUN apk add --no-cache tini
     ENTRYPOINT ["/sbin/tini", "--"]
     CMD ["node", "server.js"]

Prevention & Long-Term Monitoring:

Always test process signal handling during integration testing.

Reconfiguring Desktop VM Allocation Settings (WSL2 / Hyper-V / macOS)

Solution:

Root Cause: Docker Desktop Virtual Machine Memory Cap

On Windows (WSL2 / Hyper-V) and macOS (Docker Desktop / Colima), Docker runs inside an isolated utility Virtual Machine. If the total memory consumption of all containers exceeds the VM's assigned ceiling, the VM kernel kills the container process with exit code 137, even if your physical host PC has plenty of free RAM.

Diagnostic Verification:

1.Check Docker Desktop resources tab or inspect WSL2 memory usage.

2.On Windows, check if .wslconfig restricts total WSL2 RAM allocation.

Step-by-Step Fix:

1.Update WSL2 Memory Limit (Windows):

Create or edit %USERPROFILE%\.wslconfig:

ini
     [wsl2]
     memory=8GB
     swap=2GB

Restart WSL via PowerShell:

powershell
     wsl --shutdown

2.Adjust Resources in Docker Desktop GUI:

Open Docker Desktop > Settings > Resources.

Slide Memory to allocate a higher portion of system RAM (e.g., 8 GB or 16 GB).

Click Apply & restart.

Prevention & Long-Term Monitoring:

Set appropriate memory limits on development containers to prevent dev VMs from exhausting host system memory.

Resolving Buildkit and Compiler Out-of-Memory Errors

Solution:

Root Cause: Resource-Intensive Build Pipeline Steps

Asset compilation tools (e.g., Webpack, Vite, Rustc, g++) consume massive spikes of RAM during parallel compilation threads, causing the build step container to hit cgroup limits and terminate with exit code 137.

Diagnostic Verification:

1.Review the build output log to locate the exact step where compilation failed:

bash
   DOCKER_BUILDKIT=1 docker build --progress=plain -t my-app .

2.Identify if node/npm or compiler processes exited with Signal 9.

Step-by-Step Fix:

1.Limit Parallel Compilation Workers:

Restrict parallel job slots in Dockerfile:

dockerfile
     # For Node.js / Webpack
     ENV NODE_OPTIONS="--max-old-space-size=4096"
     # For Make / GCC
     RUN make -j2
     # For Cargo / Rust
     ENV CARGO_BUILD_JOBS=2

2.Pass Memory Limits to Docker Build Daemon:

Execute build with higher RAM allocation:

bash
     docker build --memory=8g -t my-app .

Prevention & Long-Term Monitoring:

Use multi-stage Docker builds to keep build dependencies isolated from runtime images.

Increasing BuildKit Daemon Resource Limits and Garbage Collection

Solution:

Root Cause: BuildKit Daemon Storage and Worker Memory Starvation

When using Docker BuildKit or standalone buildx builders, the build container runs as a separate background daemon that can exhaust its allocated memory cache during large multi-architecture image builds.

Diagnostic Verification:

1.Inspect builder instance status:

bash
   docker buildx ls
   docker buildx inspect

Step-by-Step Fix:

1.Re-create Buildx Builder with Memory Limits:

Create a dedicated builder with custom driver options:

bash
     docker buildx create --name highmem-builder --driver-opt memory=8Gi --use
     docker buildx inspect --bootstrap

2.Prune Build Cache:

Clear stale build cache layers:

bash
     docker builder prune -f

Prevention & Long-Term Monitoring:

Configure automated build cache pruning in CI/CD runner maintenance cron jobs.

Expanding IPC Shared Memory (/dev/shm) Mount Capacities

Solution:

Root Cause: Default 64MB /dev/shm Shared Memory Exhaustion

By default, Docker allocates a minimal 64MB virtual memory mount to /dev/shm (POSIX shared memory). Applications relying on heavy inter-process communication—such as PyTorch, PySpark, Google Chrome / Selenium, or PostgreSQL—will crash with exit code 137 or SIGBUS once this 64MB space fills up.

Diagnostic Verification:

1.Check /dev/shm size inside running container:

bash
   docker exec -it <container_id> df -h /dev/shm

2.Look for No space left on device or POSIX shared memory failure messages in application logs.

Step-by-Step Fix:

1.Expand Shared Memory Size via CLI:

Pass the --shm-size flag on startup:

bash
     docker run -d --shm-size=2g pytorch/pytorch:latest

2.Expand Shared Memory in Docker Compose:

Update docker-compose.yml:

yaml
     services:
       ml-task:
         image: pytorch/pytorch:latest
         shm_size: '2gb'

3.Mount Host IPC Namespace (Alternative for IPC-heavy loads):

Use --ipc=host to share the host's /dev/shm directly:

bash
     docker run -d --ipc=host postgres:15

Prevention & Long-Term Monitoring:

Always allocate explicit --shm-size settings for machine learning, web scraping, and database containers.

Tuning Database Buffer Pools and Concurrent Worker Connections

Solution:

Root Cause: Database Memory Allocation Exceeding Cgroup Envelope

Database containers (PostgreSQL, MySQL, Redis) configured with high shared buffers, work memory, or max connection limits can rapidly exceed container cgroup bounds under peak query concurrency, triggering an OOM kill.

Diagnostic Verification:

1.Inspect database log files prior to crash for allocation errors.

2.Calculate total potential memory usage: shared_buffers + (work_mem * max_connections).

Step-by-Step Fix:

1.Tune PostgreSQL Memory Parameters:

Adjust postgresql.conf flags to keep total usage within container bounds:

ini
     # For a 2GB container
     shared_buffers = 512MB
     work_mem = 16MB
     maintenance_work_mem = 128MB
     max_connections = 100

2.Tune MySQL / MariaDB InnoDB Buffer Pool:

Set innodb_buffer_pool_size to ~50-60% of total container RAM:

ini
     [mysqld]
     innodb_buffer_pool_size = 1G

3.Restart Database Container:

Apply new configuration and monitor memory usage.

Prevention & Long-Term Monitoring:

Use connection poolers (e.g., PgBouncer) to restrict active connection counts and prevent unbounded memory spikes.

Related Diagnostic Guides

High Server Load But Low CPU: iowait & Zombie Process Tree

MySQL Too Many Connections Error - Connection Leak Diagnostic

Kubernetes Pod Stuck at CrashLoopBackOff - Log Analysis Tree

Nginx 502 Bad Gateway After Deployment - Upstream Diagnostic

Ubuntu Black Screen After Nvidia Driver Install - Secure Boot Diagnostic

Subscribe to StepDiag

Embed Guide on Your Website