Interactive Troubleshooter

Published: Oct 18, 2026

Nginx 502 Bad Gateway After Deployment - Upstream Diagnostic

Time: 15 mins Level: Beginner

By StepDiag Engineering Team

Reviewed by Ibraheem Taofeeq Opeyemi

Exhaustive diagnostic decision tree to isolate and resolve HTTP 502 Bad Gateway errors in Nginx reverse proxies following application code updates, web server rebuilds, or container deployments.

Full Diagnostic Tree & Step-by-Step Overview

What specific error signature appears in your Nginx log file (/var/log/nginx/error.log) when the 502 Bad Gateway is triggered?

connect() to unix:/var/run/... failed (111: Connection refused) or (2: No such file or directory)
connect() to unix:/var/run/... failed (13: Permission denied) while connecting to upstream
upstream sent too big header while reading response header from upstream
recv() failed (104: Connection reset by peer) or upstream prematurely closed connection

How is your application backend or process manager (PHP-FPM, Gunicorn, Node.js, Puma) configured to listen for requests?

Listening on a UNIX domain socket file (e.g., unix:/run/php/php-fpm.sock) that is missing or dead
Listening on a TCP loopback address and port (e.g., 127.0.0.1:9000 or 127.0.0.1:8000)
Deployed inside Docker/Kubernetes containers with mismatched internal port mappings or container names
Managed via Systemd service unit that crashes continuously in a bootloop after code deployment

What security sub-system or filesystem permission layer is blocking Nginx worker access to the socket?

Linux DAC socket file ownership (chown) or permission bits (chmod) restrict the Nginx user (www-data/nginx)
SELinux is operating in Enforcing mode on RHEL/Rocky/CentOS and blocking httpd_t domain transitions
Systemd ProtectSystem or PrivateTmp directives isolate socket files inside an isolated ephemeral mount namespace

What type of deployment change introduced the expanded response header or cookie payload?

Added OAuth2 / OIDC authentication tokens or large JWT session cookies during deployment
Upgraded application framework (e.g., Laravel, Django, FastApi) causing expanded debug or header stacks

What process state or resource exhaustion event causes the upstream application worker to terminate during handling?

PHP-FPM or Gunicorn worker killed by Linux OOM (Out Of Memory) killer due to memory exhaustion
Application throws an unhandled fatal execution exception or segmentation fault upon processing cold requests

Resolving Missing or Dead UNIX Domain Socket Paths

Solution:

Root Cause: Dead or Uncreated UNIX Domain Socket

Nginx returns 502 Bad Gateway with error (2: No such file or directory) when the fastcgi_pass or proxy_pass directive targets a UNIX socket path (unix:/run/php/php-fpm.sock) that does not exist on the filesystem. This occurs when the upstream service (PHP-FPM, Gunicorn, Puma) failed to start, crashed during deployment initialization, or created the socket file in a different directory path than Nginx expects.

Diagnostic Verification:

1.Check Nginx log output:

bash
   tail -n 25 /var/log/nginx/error.log

2.Verify whether the socket file physically exists on disk:

bash
   ls -la /run/php/ /var/run/

3.Query the operational status of the upstream daemon:

bash
   systemctl status php8.3-fpm # or gunicorn, uwsgi

Step-by-Step Fix:

1.Update Upstream Pool Configuration:

Open your application pool configuration file (e.g., /etc/php/8.3/fpm/pool.d/www.conf or Gunicorn systemd unit).

Confirm the listen property matches the exact path defined in Nginx:

ini
     listen = /run/php/php8.3-fpm.sock

2.Restart Upstream Application Daemon:

Execute:

bash
     systemctl restart php8.3-fpm

3.Verify Socket File Creation:

Confirm the socket file is active:

bash
     test -S /run/php/php8.3-fpm.sock && echo "Socket Active"

4.Test Nginx Configuration and Reload:

Test syntax and reload Nginx:

bash
     nginx -t && systemctl reload nginx

Prevention & Long-Term Monitoring:

Implement Systemd service restart directives (Restart=always and RestartSec=3s) inside upstream service unit files to recover automatically from deployment initialization crashes.

Fixing TCP Loopback Connection Refused and Port Binding Mismatches

Solution:

Root Cause: Upstream TCP Service Unbound or Port Mismatch

When Nginx is configured to pass traffic to an upstream via TCP loopback (proxy_pass http://127.0.0.1:8000;), a 111: Connection refused error indicates that no application daemon is actively listening on that specified IP address and port combination. This commonly happens after deployments when the application port changed, or the app bound exclusively to IPv6 ([::1]) while Nginx attempted an IPv4 (127.0.0.1) connection.

Diagnostic Verification:

1.Identify active listening TCP ports on the host machine:

bash
   ss -tulpn | grep -E '8000|9000|3000'

2.Verify if the process is bound to 127.0.0.1, 0.0.0.0, or ::1.

Step-by-Step Fix:

1.Adjust Application Binding Host:

Reconfigure your application server (Node.js, Gunicorn, Uvicorn) to explicitly bind to IPv4 loopback 127.0.0.1 or all interfaces 0.0.0.0 rather than localhost (which may resolve inconsistently to IPv6).

2.Align Nginx Upstream Directives:

Edit /etc/nginx/sites-available/your-app.conf and align port definitions:

nginx
     upstream backend_app {
         server 127.0.0.1:8000 max_fails=3 fail_timeout=10s;
     }
     server {
         location / {
             proxy_pass http://backend_app;
         }
     }

3.Restart Upstream and Nginx:

Execute:

bash
     systemctl restart your-app
     nginx -t && systemctl reload nginx

Prevention & Long-Term Monitoring:

Standardize application port environment variables across deployment scripts using explicit .env files.

Resolving Docker / Containerized Network Gateway Communication Breaks

Solution:

Root Cause: Docker Container Isolation or Network Bridge Misconfiguration

In containerized deployments, Nginx returns 502 Bad Gateway when it cannot route traffic across the virtual Docker bridge network to the application container. Common post-deployment causes include container name resolution failures, application containers failing healthchecks, or ports exposed internally in the Dockerfile but not published across the shared Docker bridge network.

Diagnostic Verification:

1.Inspect running Docker containers and port mappings:

bash
   docker ps

2.Check container application logs:

bash
   docker logs --tail 50 <app_container_name>

3.Verify internal network connectivity from the Nginx container:

bash
   docker exec -it nginx_container curl -i http://app_container:8000

Step-by-Step Fix:

1.Attach Containers to Shared Docker Network:

Ensure both Nginx and the application service belong to the same user-defined bridge network in docker-compose.yml:

yaml
     version: '3.8'
     services:
       web:
         image: nginx:alpine
         networks:
           - app-network
       app:
         build: .
         networks:
           - app-network
     networks:
       app-network:
         driver: bridge

2.Update Nginx Proxy Pass Service Name:

Use the exact Compose service name in Nginx location blocks:

nginx
     proxy_pass http://app:8000;

3.Redeploy Containers:

Execute:

bash
     docker compose up -d --force-recreate

Prevention & Long-Term Monitoring:

Implement explicit healthcheck definitions inside docker-compose.yml to prevent Nginx from routing traffic to initializing app containers.

Fixing Systemd Bootlooping Application Daemon Crashes

Solution:

Root Cause: Unhandled Startup Exception Triggering Systemd Crash Loop

A 111: Connection Refused error occurring immediately after code deployment often signals that the backend service started, encountered an unhandled execution exception (e.g., missing database migration, broken environment variable, invalid syntax), crashed, and is being repeatedly restarted by Systemd.

Diagnostic Verification:

1.Inspect Systemd journal logs for the application unit:

bash
   journalctl -u your-app.service -n 50 --no-pager

2.Look for stack traces such as Python ImportError, Node.js UnhandledPromiseRejection, or PHP Fatal Error.

Step-by-Step Fix:

1.Halt Automatic Restart Loop:

Temporarily stop the failing service to stabilize system logs:

bash
     systemctl stop your-app.service

2.Execute Application Entrypoint Manually:

Run the application executable directly as the application user to isolate runtime environment faults:

bash
     sudo -u appuser /var/www/app/venv/bin/gunicorn --bind 127.0.0.1:8000 wsgi:app

3.Resolve Missing Runtime Dependencies:

Fix missing environment keys or execute database migrations (php artisan migrate or python manage.py migrate).

4.Restart Service:

Execute:

bash
     systemctl start your-app.service

Prevention & Long-Term Monitoring:

Incorporate automated smoke tests and syntax verification checks into your CI/CD deployment pipeline before reloading Systemd daemons.

Fixing Discrepancies in UNIX Socket Ownership and Permission Bits

Solution:

Root Cause: File Permission Mismatch on UNIX Socket File

Nginx logs (13: Permission denied) while connecting to upstream when the Nginx worker process user (www-data, nginx, or nobody) lacks read/write permissions on the created socket file. During deployment resets, upstream socket files are frequently recreated with restrictive default permissions (e.g., owned by root:root with mode 0600), blocking the Nginx worker process.

Diagnostic Verification:

1.Check Nginx worker process execution user:

bash
   ps aux | grep nginx | grep worker

2.Check ownership and permission bits on the target socket file:

bash
   ls -la /run/php/php8.3-fpm.sock

Step-by-Step Fix:

1.Update Upstream Socket Owner Directives:

Edit your PHP-FPM pool (/etc/php/8.3/fpm/pool.d/www.conf) or Gunicorn configuration to set owner and mode explicitly:

ini
     listen.owner = www-data
     listen.group = www-data
     listen.mode = 0660

2.Verify Parent Directory Permissions:

Ensure Nginx worker user has execute (x) permissions across all parent directories leading to the socket:

bash
     namei -nom /run/php/php8.3-fpm.sock

3.Restart Upstream Service:

Execute:

bash
     systemctl restart php8.3-fpm

Prevention & Long-Term Monitoring:

Add file permission verification assertions to deployment automation scripts.

Reconfiguring SELinux Network Access Booleans and File Contexts

Solution:

Root Cause: SELinux Denial Blocking Network Socket Connections

On RHEL, Rocky Linux, AlmaLinux, and CentOS systems operating with SELinux in Enforcing mode, Nginx (httpd_t security domain) is blocked by default from initiating TCP connections to non-standard ports or reading socket files located outside approved web contexts (httpd_sys_content_t or var_run_t), returning (13: Permission denied).

Diagnostic Verification:

1.Check SELinux audit logs for denials:

bash
   ausearch -m avc -ts recent | grep nginx
   # Or check audit.log directly:
   grep "denied" /var/log/audit/audit.log | grep httpd_t

2.Test temporary permissive mode to confirm SELinux as the root cause:

bash
   setenforce 0

*(Re-test Nginx; if 502 disappears, SELinux policies require updating).* Re-enable with setenforce 1.

Step-by-Step Fix:

1.Enable HTTPD Network Connect Boolean:

Allow Nginx to connect to upstream network sockets:

bash
     setsebool -P httpd_can_network_connect 1

2.Apply Correct Security Context to Socket Directory:

If using custom socket paths (e.g., /var/www/sockets/):

bash
     semanage fcontext -a -t httpd_var_run_t "/var/www/sockets(/.*)?"
     restorecon -Rv /var/www/sockets

3.Reload Nginx:

Execute:

bash
     systemctl reload nginx

Prevention & Long-Term Monitoring:

Always configure persistent SELinux booleans (setsebool -P) rather than disabling SELinux globally on production systems.

Resolving Systemd PrivateTmp and Mount Namespace Isolation

Solution:

Root Cause: Systemd PrivateTmp Namespace Isolation

When upstream services (such as Gunicorn, Uvicorn, or PHP-FPM) write their UNIX domain socket to /tmp/app.sock while configured with PrivateTmp=true in their Systemd unit file, Linux creates an isolated mount namespace for that process (e.g., /tmp/systemd-private-X-gunicorn.service-Y/tmp/app.sock). Nginx cannot see or access this file from the global filesystem root, resulting in (2: No such file or directory) or permission denials.

Diagnostic Verification:

1.Inspect Systemd unit settings for PrivateTmp:

bash
   systemctl cat your-app.service | grep PrivateTmp

2.Check for isolated Systemd temporary directories:

Execute ls -la /tmp/systemd-private-* under root privileges.

Step-by-Step Fix:

1.Relocate Socket File Directory:

Change socket location to the standard runtime directory /run/ or /var/run/ (e.g., /run/your-app/app.sock).

2.Update Systemd Service Unit:

Edit unit file using override editor:

bash
     systemctl edit your-app.service

Add runtime directory definition:

ini
     [Service]
     RuntimeDirectory=your-app
     RuntimeDirectoryMode=0755

3.Update Nginx Configuration:

Point Nginx to unix:/run/your-app/app.sock.

4.Reload Systemd and Services:

Execute:

bash
     systemctl daemon-reload
     systemctl restart your-app
     systemctl reload nginx

Prevention & Long-Term Monitoring:

Never store persistent operational UNIX sockets inside /tmp; always use /run/ or /var/run/ with RuntimeDirectory directives.

Expanding Nginx FastCGI and Proxy Response Buffer Sizes

Solution:

Root Cause: Upstream Response Header Size Exceeding Nginx Buffer Allocation

When a backend application sends response HTTP headers (including large set-cookie headers, JWT authentication tokens, or debugging stacks) that exceed Nginx's default buffer limit (typically 4k or 8k), Nginx truncates the read buffer, drops the upstream connection, and logs upstream sent too big header while reading response header from upstream alongside a 502 Bad Gateway error.

Diagnostic Verification:

1.Check /var/log/nginx/error.log for the exact buffer overflow string:

bash
   grep "upstream sent too big header" /var/log/nginx/error.log

Step-by-Step Fix:

1.Expand Proxy Buffer Directives (For Reverse Proxy setups):

Edit /etc/nginx/sites-available/your-app.conf and append inside the target location / block:

nginx
     proxy_buffer_size 128k;
     proxy_buffers 4 256k;
     proxy_busy_buffers_size 256k;

2.Expand FastCGI Buffer Directives (For PHP-FPM setups):

Append inside the location ~ \.php$ block:

nginx
     fastcgi_buffer_size 128k;
     fastcgi_buffers 4 256k;
     fastcgi_busy_buffers_size 256k;

3.Validate and Reload Nginx:

Execute:

bash
     nginx -t && systemctl reload nginx

Prevention & Long-Term Monitoring:

Avoid storing large binary payloads inside HTTP session cookies; store state on backend Redis caches instead.

Tuning Framework Header Output and Nginx FastCGI Header Buffers

Solution:

Root Cause: Uncompressed Framework Debug Output and Header Accumulation

Upgrading framework versions or deploying code with verbose debug headers enabled (such as Symfony Profiler, Laravel Debugbar, or custom middleware) can inflate header counts exponentially, causing buffer overruns during upstream read cycles.

Diagnostic Verification:

1.Inspect headers returned directly from the backend bypassing Nginx:

bash
   curl -I -s --unix-socket /run/php/php8.3-fpm.sock http://localhost/api/test

2.Measure cumulative header byte sizes in the raw HTTP output.

Step-by-Step Fix:

1.Disable Verbose Framework Debugging in Production:

Set APP_DEBUG=false in your .env configuration file.

2.Increase Nginx Global Header Buffer Limits:

Open /etc/nginx/nginx.conf and set under the http block:

nginx
     http {
         large_client_header_buffers 4 32k;
         fastcgi_buffer_size 64k;
         fastcgi_buffers 8 64k;
     }

3.Reload Nginx and App Daemon:

Execute:

bash
     nginx -t && systemctl reload nginx

Prevention & Long-Term Monitoring:

Enforce production build environment checks in CI/CD pipelines to guarantee debug suites are purged before deployment.

Eliminating Memory Leak Fatalities and OOM Upstream Process Terminations

Solution:

Root Cause: Upstream Process Termination by Kernel OOM Killer

Nginx logs recv() failed (104: Connection reset by peer) when the upstream worker process processing a request is abruptly killed mid-execution by the Linux Out-Of-Memory (OOM) Killer due to server RAM exhaustion, or when process execution limits (max_requests) force an immediate worker exit without flushing TCP buffers.

Diagnostic Verification:

1.Check Linux kernel ring buffer for OOM killer invocations:

bash
   dmesg -T | grep -i -E 'oom|killed process'

2.Inspect Systemd journal for process exit codes:

bash
   journalctl -u php8.3-fpm.service -n 25 --no-pager

Step-by-Step Fix:

1.Increase Memory Allocation in Upstream Config:

For PHP-FPM (php.ini):

ini
     memory_limit = 512M

For Gunicorn workers: Increase server swap space or resize host instance RAM.

2.Adjust Process Manager Worker Count:

Prevent RAM exhaustion by lowering max worker counts in PHP-FPM pool (www.conf):

ini
     pm = dynamic
     pm.max_children = 20
     pm.start_servers = 5
     pm.min_spare_servers = 3
     pm.max_spare_servers = 10
     pm.max_requests = 500

3.Restart Upstream Service:

Execute:

bash
     systemctl restart php8.3-fpm

Prevention & Long-Term Monitoring:

Set up memory usage alerts via Prometheus/Grafana or Datadog to notify engineering teams before host RAM hits 90% utilization.

Debugging Unhandled Native Execution Crashes and Segmentation Faults

Solution:

Root Cause: Native Extension Segmentation Fault or Unhandled Application Exception

When an application worker process encounters a core segmentation fault (SIGSEGV or SIGABRT) inside compiled C/C++ extensions (e.g., ImageMagick, cURL, database driver bindings) or hits an unhandled fatal execution path during cold request initialization, the worker drops the socket connection instantly. Nginx receives an empty TCP FIN packet and immediately returns a 502 Bad Gateway to the browser.

Diagnostic Verification:

1.Inspect core dump logs or Systemd crash logs:

bash
   coredumpctl list

2.Check application-specific log files:

PHP: /var/log/php8.3-fpm.log

Python/Node: /var/log/your-app/error.log

3.Look for fatal strings such as SIGSEGV (Segmentation fault) or Address boundary error.

Step-by-Step Fix:

1.Isolate and Disable Faulty Extension:

If a specific extension triggered the crash (e.g., broken OPcache build or outdated PECL driver), disable it in configuration:

bash
     phpdismod xdebug imagick

2.Rebuild Application Virtual Environment:

Reinstall dependencies with fresh binary wheels:

bash
     pip install --no-cache-dir --force-reinstall -r requirements.txt

3.Restart Application Daemon:

Execute:

bash
     systemctl restart your-app

Prevention & Long-Term Monitoring:

Test extension updates in a staging environment matching production kernel and C-library (glibc) versions before deploying updates.

Related Diagnostic Guides

High Server Load But Low CPU: iowait & Zombie Process Tree

MySQL Too Many Connections Error - Connection Leak Diagnostic

Kubernetes Pod Stuck at CrashLoopBackOff - Log Analysis Tree

Docker Container Exits With Code 137: OOM Diagnostic

Ubuntu Black Screen After Nvidia Driver Install - Secure Boot Diagnostic

Subscribe to StepDiag

Embed Guide on Your Website