Docker Services Troubleshooting

Version: 1.0
Last Updated: 2025-01-27
Owner: Technical Operations
Status: Active Guide

Overview

This guide helps resolve common Docker service issues when running the Zixly internal operations stack. It covers container startup failures, port conflicts, volume mounting problems, and service connectivity issues.

Common Docker Service Issues

Error: “Port already in use”

Cause: Another service is using the required port.

Solutions:

Check port usage:

# Check what's using port 5678
lsof -i :5678

# Check all Zixly ports
lsof -i :5678 -i :5432 -i :6379

Stop conflicting services:

# Stop other Docker containers
docker ps
docker stop CONTAINER_NAME

# Kill processes using ports
sudo kill -9 PID

Change port mapping: ```yaml

docker-compose.pipeline services.yml

ports:
- ‘5679:5678’ # Change host port ```

Error: “Container failed to start”

Cause: Container startup configuration or resource issues.

Solutions:

Check container logs:

# Check startup logs
docker-compose logs webhook-receiver
docker-compose logs trading-api
docker-compose logs pipeline-worker
docker-compose logs postgres
docker-compose logs redis

Check if required images exist:

# Verify generic images are built
docker images | grep -E "trading-api|arq-worker|webhook-receiver|pipeline-worker"

# If images missing, build them
cd /Users/colemorton/Projects/trading
docker-compose build

cd /Users/colemorton/Projects/zixly
docker-compose build webhook-receiver pipeline-worker

Verify Docker resources:

# Check Docker system info
docker system info

# Check available resources
docker system df

Check Docker daemon status:

# Restart Docker if needed
sudo systemctl restart docker

# Check Docker status
sudo systemctl status docker

Error: “Volume mount failed”

Cause: Docker volume permissions or path issues.

Solutions:

Check volume permissions:

# Check volume ownership
ls -la /var/lib/docker/volumes/

# Fix permissions if needed
sudo chown -R $USER:$USER /var/lib/docker/volumes/

Verify volume paths:

# Check if volumes exist
docker volume ls | grep zixly

# Inspect volume details
docker volume inspect zixly_pipeline services_data

Recreate volumes:

# Remove and recreate volumes
docker-compose -f docker-compose.pipeline services.yml down -v
docker-compose -f docker-compose.pipeline services.yml up -d

Service Connectivity Issues

pipeline services Service Not Accessible

Symptoms: Cannot access http://localhost:5678

Troubleshooting Steps:

Check container status:

# Check if container is running
docker-compose -f docker-compose.pipeline services.yml ps

# Check container health
docker inspect zixly-pipeline services | grep -A 5 "Health"

Test connectivity:

# Test from host
curl http://localhost:5678/healthz

# Test from container
docker exec zixly-pipeline services curl http://localhost:5678/healthz

Check port mapping:

# Verify port is mapped correctly
docker port zixly-pipeline services

# Check if port is listening
netstat -tlnp | grep 5678

Database Connection Issues

Symptoms: pipeline services cannot connect to PostgreSQL

Troubleshooting Steps:

Check PostgreSQL status:

# Check if PostgreSQL is running
docker exec zixly-pipeline services-postgres pg_isready -U pipeline services

# Check PostgreSQL logs
docker-compose -f docker-compose.pipeline services.yml logs postgres

Test database connectivity:

# Connect to database
docker exec -it zixly-pipeline services-postgres psql -U pipeline services -d pipeline services

# Test from pipeline services container
docker exec zixly-pipeline services ping postgres

Check network connectivity:

# Check Docker network
docker network ls
docker network inspect zixly-internal-network

Redis Connection Issues

Symptoms: pipeline services cannot connect to Redis

Troubleshooting Steps:

Check Redis status:

# Check if Redis is running
docker exec zixly-pipeline services-redis redis-cli ping

# Check Redis logs
docker-compose -f docker-compose.pipeline services.yml logs redis

Test Redis connectivity:

# Connect to Redis
docker exec -it zixly-pipeline services-redis redis-cli

# Test from pipeline services container
docker exec zixly-pipeline services ping redis

Performance and Resource Issues

High Resource Usage

Symptoms: Slow performance, high CPU/memory usage

Solutions:

Monitor resource usage:

# Check container resource usage
docker stats

# Check system resources
top
free -h
df -h

Optimize resource allocation:

# docker-compose.pipeline services.yml
deploy:
  resources:
    limits:
      memory: 2G
      cpus: '1.0'
    reservations:
      memory: 1G
      cpus: '0.5'

Clean up unused resources:

# Remove unused containers
docker container prune

# Remove unused images
docker image prune

# Remove unused volumes
docker volume prune

Slow Startup Times

Symptoms: Services take long time to start

Solutions:

Use pre-built images (Generic Image Strategy):

# Ensure generic images are built and available
docker images | grep -E "trading-api|arq-worker|webhook-receiver|pipeline-worker"

# If using generic images, startup should be ~1.7 seconds
# If building from source, startup takes 30-60 seconds

Check startup dependencies:

# Ensure proper dependency order
depends_on:
  - postgres
  - redis

Optimize image sizes:

# Check image sizes
docker images | grep -E "trading-api|arq-worker|webhook-receiver|pipeline-worker"

# Generic images should be:
# trading-api:latest      ~3.23GB
# arq-worker:latest       ~3.23GB
# webhook-receiver:latest ~508MB
# pipeline-worker:latest  ~517MB

Check system resources:

# Monitor startup process
docker-compose up --no-deps trading-api

Data Persistence Issues

Data Not Persisting

Symptoms: Data lost after container restart

Troubleshooting Steps:

Check volume mounting:

# Check if volumes are mounted
docker inspect zixly-pipeline services | grep -A 10 "Mounts"

# Check volume content
docker exec zixly-pipeline services ls -la /home/node/.pipeline services/

Verify volume configuration:

# Ensure volumes are properly configured
volumes:
  - pipeline services_data:/home/node/.pipeline services
  - postgres_data:/var/lib/postgresql/data
  - redis_data:/data

Check volume permissions:

# Check volume permissions
docker exec zixly-pipeline services ls -la /home/node/.pipeline services/

# Fix permissions if needed
docker exec zixly-pipeline services chown -R node:node /home/node/.pipeline services/

Backup and Recovery

Prevent Data Loss:

Regular backups:

# Backup pipeline services data
docker run --rm -v zixly_pipeline services_data:/data -v $(pwd):/backup alpine tar czf /backup/pipeline services-backup-$(date +%Y%m%d).tar.gz -C /data .

# Backup PostgreSQL data
docker exec zixly-pipeline services-postgres pg_dump -U pipeline services pipeline services > pipeline services-db-backup-$(date +%Y%m%d).sql

Test recovery procedures:

# Test backup restoration
docker run --rm -v zixly_pipeline services_data:/data -v $(pwd):/backup alpine tar xzf /backup/pipeline services-backup-YYYYMMDD.tar.gz -C /data

Network and Security Issues

Network Connectivity Problems

Symptoms: Containers cannot communicate with each other

Troubleshooting Steps:

Check Docker network:

# List Docker networks
docker network ls

# Inspect network configuration
docker network inspect zixly-internal-network

Test inter-container communication:

# Test from pipeline services to postgres
docker exec zixly-pipeline services ping postgres

# Test from pipeline services to redis
docker exec zixly-pipeline services ping redis

Check firewall settings:

# Check firewall status
sudo ufw status

# Check iptables rules
sudo iptables -L

Security Configuration

Best Practices:

Use non-root users:

# Run containers as non-root
user: '1000:1000'

Limit container privileges:

# Remove unnecessary privileges
security_opt:
  - no-new-privileges:true

Use secrets for sensitive data: ```yaml

Use Docker secrets instead of environment variables

secrets:
- db_password ```

Diagnostic Commands

Service Health Checks

# Check all service status
docker-compose -f docker-compose.pipeline services.yml ps

# Check service health
docker-compose -f docker-compose.pipeline services.yml exec pipeline services curl http://localhost:5678/healthz
docker-compose -f docker-compose.pipeline services.yml exec postgres pg_isready -U pipeline services
docker-compose -f docker-compose.pipeline services.yml exec redis redis-cli ping

Resource Monitoring

# Monitor resource usage
docker stats --no-stream

# Check disk usage
docker system df

# Monitor logs in real-time
docker-compose -f docker-compose.pipeline services.yml logs -f

Network Diagnostics

# Check network connectivity
docker exec zixly-pipeline services ping postgres
docker exec zixly-pipeline services ping redis

# Check port accessibility
netstat -tlnp | grep -E "(5678|5432|6379)"

Prevention and Best Practices

Regular Maintenance

Monitor service health:
- Set up health checks
- Monitor resource usage
- Check log files regularly
Keep services updated:
- Update Docker images regularly
- Apply security patches
- Monitor for vulnerabilities
Backup data regularly:
- Automated backup scripts
- Test backup restoration
- Store backups securely

Configuration Management

Use version control:
- Track Docker Compose files
- Document configuration changes
- Maintain change history
Environment separation:
- Separate dev/staging/prod environments
- Use environment-specific configurations
- Test changes in staging first

Getting Help

Self-Service Resources

Check this troubleshooting guide
Review Docker logs and status
Test network connectivity
Monitor resource usage

When to Contact Support

Contact support if:

Services won’t start after troubleshooting
Data corruption or loss occurs
Security issues are identified
Performance problems persist

Support Information

Primary Contact: Your Name (your_email@domain.com)
Documentation: This troubleshooting guide
Logs Location: Docker container logs and system logs

Document Version: 1.0
Last Updated: 2025-01-27
Owner: Zixly Technical Operations
Review Cycle: Monthly

Zixly Documentation

DevOps Automation Services for Brisbane Tech Companies

Docker Services Troubleshooting

Overview

Common Docker Service Issues

Error: “Port already in use”

docker-compose.pipeline services.yml

Error: “Container failed to start”

Error: “Volume mount failed”

Service Connectivity Issues

pipeline services Service Not Accessible

Database Connection Issues

Redis Connection Issues

Performance and Resource Issues

High Resource Usage

Slow Startup Times

Data Persistence Issues

Data Not Persisting

Backup and Recovery

Network and Security Issues

Network Connectivity Problems

Security Configuration

Use Docker secrets instead of environment variables

Diagnostic Commands

Service Health Checks

Resource Monitoring

Network Diagnostics

Prevention and Best Practices

Regular Maintenance

Configuration Management

Getting Help

Self-Service Resources

When to Contact Support

Support Information