Troubleshooting

Service Won’t Start (0/1 Replicas)

1. Check the service task errors:

   docker service ps dev_api --no-trunc

   Look at the ERROR column for the most recent task.

2. Common causes:

   • Image not found — The image doesn’t exist in Harbor. Push it first.
   • Port conflict — Another service is using the same port.
   • Config missing — The service references a Docker config that doesn’t exist.
   • Resource constraint — Node doesn’t have enough memory or CPU.
   • Health check failing — Container starts but health check fails, causing restart loop.

3. Check container logs (if it started briefly):

   docker service logs dev_api --tail 200

4. Force redeploy:

   docker service update --force dev_api

502 Bad Gateway

This means Nginx is running but can’t reach the backend service.

1. Check if the backend service is running:

   docker service ls | grep <service_name>

2. Check if the service is on the correct network:

   docker service inspect <service_name> --format '{{.Spec.TaskTemplate.Networks}}'

3. Check Nginx config references the correct service name and port:

   docker config inspect <nginx_config> --pretty

4. Test from inside the Nginx container:

   docker exec $(docker ps -q -f name=dev_nginx) curl -s http://<service_name>:<port>/

Container Restart Loop

The service keeps restarting — replicas show 0/1 or tasks show repeated failures.

# See the restart history
docker service ps dev_api --no-trunc

# Check logs from the failing container
docker service logs dev_api --tail 300

Common causes:

• Application crash on startup (bad config, missing env var, DB connection failure)
• Out of memory (OOM killed) — check docker inspect <container_id> for OOMKilled
• Health check timeout too aggressive

Can’t Pull Images from Harbor

# Check if you're logged in
docker login harbor.ayinza.dev

# Check if Harbor is reachable
curl -sk https://harbor.ayinza.dev/api/v2.0/ping

# Check Harbor services
docker stack services shared | grep harbor

If Harbor itself is down: Restart it:

docker stack deploy -c /home/kaks/stacks/shared-infrastructure.yml shared

WireGuard Tunnel Down

Symptoms: Services on one node can’t talk to services on the other node.

# Check WireGuard status
wg show

# Look for "latest handshake" — if it's more than 2 minutes ago, the tunnel may be down

# Restart WireGuard
systemctl restart wg-quick@wg0

# Verify connectivity
ping 10.10.0.1   # Azure (from Contabo)
ping 10.10.0.2   # Contabo (from Azure)

Caution

If WireGuard goes down, the Docker Swarm overlay network also breaks. Services on different nodes won’t be able to communicate until the tunnel is restored.

Node Shows “Down” in Swarm

docker node ls
# If a node shows "Down":

# 1. Check if the server itself is reachable
ping 4.180.181.86    # Azure
ping 84.247.134.135  # Contabo

# 2. SSH into the node and check Docker daemon
systemctl status docker

# 3. Restart Docker if needed
systemctl restart docker

# 4. Check WireGuard (nodes communicate over VPN)
wg show

Database Connection Issues

# Check if PostgreSQL container is running
docker ps -f name=postgresql

# Test connection from inside the container
docker exec $(docker ps -q -f name=dev_postgresql) pg_isready -U postgres

# Check PgBouncer (connection pooler)
docker service logs dev_pgbouncer --tail 50

# Check max connections
docker exec $(docker ps -q -f name=dev_postgresql) psql -U postgres -c "SHOW max_connections;"
docker exec $(docker ps -q -f name=dev_postgresql) psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"

Disk Full

1. Check what’s using space:

   df -h
   du -sh /var/lib/docker/ 2>/dev/null
   du -sh /data/* 2>/dev/null          # Contabo - Harbor data

2. Clean up Docker:

   # Remove unused images
   docker image prune -a
   
   # Remove unused volumes (CAREFUL — only removes truly unused ones)
   docker volume prune
   
   # Full cleanup
   docker system prune

3. Harbor garbage collection:

   • Log into Harbor UI → Administration → Garbage Collection → GC Now

4. Remove old database backups:

   ls -lah ~/sseris-*.sql.gz
   # Delete backups older than 7 days

SSL Certificate Expired

# Check expiry date
echo | openssl s_client -connect ayinza.dev:443 2>/dev/null | openssl x509 -noout -dates

# If using Let's Encrypt:
certbot renew

# After renewal, restart/reload the proxy:
docker service update --force dev_nginx

GitHub Actions Runner Not Picking Up Jobs

# Check runner status on Contabo
systemctl status actions.runner.*

# View runner logs
journalctl -u actions.runner.* --no-pager --since "1 hour ago"

# Restart the runner
systemctl restart actions.runner.*

General Debugging Checklist

1. Is the server reachable? — ping the IP
2. Is Docker running? — systemctl status docker
3. Is the WireGuard tunnel up? — wg show
4. Is the service running? — docker service ls | grep <name>
5. What do the logs say? — docker service logs <name> --tail 200
6. Is there disk space? — df -h
7. Is there memory? — free -h
8. Can the service reach its dependencies? — Test from inside the container