Heads up! Long post and lots of head bashing against the wall.
Context:
I have a written an a python app (Django). I have dockerized the deployment and the compose file has three containers, app, nginx and postgres. I'm currently trying to deploy a demo of it in a VPS running Debian 11. Information below has been redacted (IPs, Domain name, etc.)
Problem:
I keep running into 502 errors. Locally things work very well even with nginx (but running on 80). As I try to deploy this I'm trying to configure nginx the best I can and redirecting http traffic to https and ssl certs. The nginx logs simply say "connect() failed (111: Connection refused) while connecting to upstream, client: 1.2.3.4, server: demo.example.com, request: "GET / HTTP/1.1", upstream: "http://192.168.0.2:8020/", host: "demo.example.com"". I have tried just about everything.
What I've tried:
- Adding my server block configs to /etc/nginx/conf.d/default.conf
- Adding my server block configs to a new file in /etc/nginx/conf.d/app.conf and leaving default at out of box config.
- Tried putting the above config (default.conf and app.conf) in sites-available (/etc/nginx/sites-available/* not at the same time tho).
- Recreated /etc/nginx/nginx.conf by copy/pasting out of box nginx.conf and then adding server blocks directly in nginx.conf
- Running nginx -t inside of the nginx container (Syntax and config were "successful")
- Running nginx -T when recreated /etc/nginx/nginx.conf
- nginx -T when the server blocks where in /etc/nginx/conf.d/* lead me to think that since there were two server listen 80 blocks that I should ensure only one listen 80 block was being read by the container hence the recreated /etc/nginx/nginx.conf from above
- Restarted container each time a change was made.
- Changed the user block from nginx (no dice when using nginx as user) to www-data, root and nobody
- Deleted my entire docker data and redeployed everything a few times.
- Double checked the upstream block 1,000 times
- Confirmed upstream block container is running and on the right exposed port
Checked access.log and error.log but they were both empty (not sure why, tried cat and tail)
- Probably forgetting more stuff (6 hours deep in the same error loop by now)
How can you help:
Please take a look at the nginx.conf config below and see if you guys can spot a problem, PLEASE! This is my current /etc/nginx/nginx.conf
`
user www-data;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
upstream djangoapp {
server app:8020;
}
server {
listen 80;
listen [::]:80;
server_name demo.example.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name demo.example.com;
ssl_certificate /etc/letsencrypt/live/demo.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/demo.example.com/privkey.pem;
#ssl_protocols TLSv1.2 TLSv1.3;
#ssl_prefer_server_ciphers on;
location / {
proxy_pass http://djangoapp;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header Upgrade $http_upgrade;
#proxy_set_header Connection keep-alive;
proxy_redirect off;
}
location /static/ {
autoindex on;
alias /static/;
}
}
}
`
-
EDIT: I have also confirmed that both containers are connected to the same docker network (docker network inspect frontend)
-
EDIT 2: Solved my problem. See my comments to @chaospatterns. TLDR there was an uncaught exception in the app but it didn’t cause a crash with the container. Had to dig deep into logs to find it.
I don’t have an answer for you but I have one instead. When I attempted to do swarm my biggest challenge was shared storage. I was attempting to run a swarm with shared storage on a NAS. Literally could not run apps, ran into a ton of problems running stacks (NAS share tried SMB and NFS). How did you get around this problem?