Monitoring & Health

Quick Health Check

Run these commands to get a snapshot of the cluster:

# Are all nodes healthy?
docker node ls

# Are all services running?
docker service ls | grep -v "1/1\|0/0"

# Any failed tasks?
docker stack ps dev --filter "desired-state=shutdown" --format "{{.Name}} {{.Error}}"
docker stack ps uat --filter "desired-state=shutdown" --format "{{.Name}} {{.Error}}"
docker stack ps shared --filter "desired-state=shutdown" --format "{{.Name}} {{.Error}}"

If docker service ls shows services with 0/1 replicas, those services are down.

Portainer Dashboard

Portainer provides a web UI for monitoring at:

URL	Access
https://portainer.ayinza.dev	Admin credentials

From Portainer you can:

• View all stacks, services, and containers
• See resource usage (CPU, memory) per container
• Read container logs
• Restart services
• View cluster node status

Checking Individual Services

Service Status

# See if a service is running
docker service ps dev_api

# Output columns: ID, NAME, IMAGE, NODE, DESIRED STATE, CURRENT STATE, ERROR
# Look for "Running" in CURRENT STATE

Service Logs

# Tail logs in real time
docker service logs dev_api --tail 200 -f

# Logs with timestamps
docker service logs dev_api --tail 200 -f -t

# Logs for a specific time range (last 30 minutes)
docker service logs dev_api --since 30m

Container Resource Usage

# Live resource stats for all containers on this node
docker stats

# Stats for a specific container
docker stats <container_id>

Server-Level Monitoring

Disk Usage

# Overall disk usage
df -h

# Docker-specific disk usage
docker system df

# Largest directories
du -sh /data/* 2>/dev/null | sort -rh | head -10   # Contabo
du -sh /var/lib/docker/volumes/* | sort -rh | head -10

Memory Usage

# Quick overview
free -h

# Detailed per-process memory
ps aux --sort=-%mem | head -20

CPU Usage

# Interactive process monitor
top

# Snapshot of top CPU consumers
ps aux --sort=-%cpu | head -20

Network

# Check open ports
ss -tlnp

# Check WireGuard tunnel
wg show

# Test connectivity between nodes
ping 10.10.0.1   # Azure
ping 10.10.0.2   # Contabo

Common Health Indicators

What to Check	Healthy	Unhealthy
docker node ls	Both nodes Ready + Active	Node shows Down or Drain
docker service ls	All services show 1/1	Any service shows 0/1
df -h	Disk usage < 80%	Disk usage > 90%
free -h	Free memory > 1 GB	Free memory < 500 MB
wg show	Shows peer with recent handshake	No peer or handshake > 5 min ago
Portainer UI	Loads and shows all services	Unreachable or shows errors

Setting Up Alerts (Recommended)

Currently there is no automated alerting. A basic monitoring setup could include:

1. Cron-based health check — Script that runs every 5 minutes, checks docker service ls for unhealthy services, and sends a notification
2. Disk space monitor — Alert when disk usage exceeds 80%
3. Uptime monitoring — External service (e.g., UptimeRobot) pinging key URLs

Simple Health Check Script

#!/bin/bash
# Save as /home/kaks/health-check.sh
# Add to cron: */5 * * * * /home/kaks/health-check.sh

UNHEALTHY=$(docker service ls --format "{{.Name}} {{.Replicas}}" | grep "0/")

if [ -n "$UNHEALTHY" ]; then
  echo "ALERT: Unhealthy services detected:"
  echo "$UNHEALTHY"
  # Add notification here (email, Slack webhook, etc.)
fi