Phase 2: Local Development & Testing

Phase: 2 of 6
Duration: 2 weeks (Weeks 5-6)
Status: 🎯 READY TO START
Prerequisites: Phase 1 Complete βœ…


Executive Summary

Phase 2 focuses on local development and testing of the self-hostable stack integration, specifically adding Plane project management alongside the existing pipeline services setup. This phase is 100% local and functional, enabling rapid development and testing without deployment complexity.

Strategic Importance: This phase establishes the core project management capability for Zixly’s internal operations while maintaining a local development environment for rapid iteration and testing.


Foundation Context:

Implementation Guidance:


Phase 2 Objectives

Primary Goals

  1. Local Docker Environment - Spin up both pipeline services and Plane locally with single command
  2. Plane Integration - Configure Plane for Zixly internal project management
  3. Plane Smoke Test - Pipeline Workflow to validate Plane connectivity and functionality
  4. Local Development Workflow - Streamlined development and testing process

Success Criteria


Implementation Plan

Week 5: Local Docker Environment Setup

5.1 Local Docker Compose Stack

File: docker-compose.local.yml

version: '3.8'

services:
  # PostgreSQL Database
  postgres:
    image: postgres:15-alpine
    container_name: zixly-postgres
    restart: unless-stopped
    environment:
      - POSTGRES_USER=zixly_admin
      - POSTGRES_PASSWORD=local_dev_password
      - POSTGRES_DB=zixly_main
    ports:
      - '5432:5432'
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-scripts:/docker-entrypoint-initdb.d
    networks:
      - zixly-local
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U zixly_admin -d zixly_main']
      interval: 30s
      timeout: 10s
      retries: 3

  # Redis Cache
  redis:
    image: redis:7-alpine
    container_name: zixly-redis
    restart: unless-stopped
    command: redis-server --appendonly yes --requirepass local_dev_password
    ports:
      - '6379:6379'
    volumes:
      - redis_data:/data
    networks:
      - zixly-local
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 30s
      timeout: 10s
      retries: 3

  # Pipeline Workflow Automation
  pipeline services:
    image: pipeline servicesio/pipeline services:latest
    container_name: zixly-pipeline services
    restart: unless-stopped
    environment:
      - pipeline services_HOST=localhost
      - pipeline services_PORT=5678
      - pipeline services_BASIC_AUTH_ACTIVE=true
      - pipeline services_BASIC_AUTH_USER=admin
      - pipeline services_BASIC_AUTH_PASSWORD=local_dev_password
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=pipeline services
      - DB_POSTGRESDB_USER=zixly_admin
      - DB_POSTGRESDB_PASSWORD=local_dev_password
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=168
    ports:
      - '5678:5678'
    volumes:
      - pipeline services_data:/home/node/.pipeline services
    depends_on:
      postgres:
        condition: service_healthy
    networks:
      - zixly-local
    healthcheck:
      test:
        ['CMD', 'wget', '--no-verbose', '--tries=1', '--spider', 'http://localhost:5678/healthz']
      interval: 30s
      timeout: 10s
      retries: 3

  # Plane Project Management
  plane:
    image: makeplane/plane:latest
    container_name: zixly-plane
    restart: unless-stopped
    environment:
      - DATABASE_URL=postgresql://zixly_admin:local_dev_password@postgres:5432/plane
      - REDIS_URL=redis://:local_dev_password@redis:6379
      - SECRET_KEY=local_plane_secret_key
      - WEB_URL=http://localhost:8000
      - NEXT_PUBLIC_API_BASE_URL=http://localhost:8000
    ports:
      - '8000:8000'
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - zixly-local
    healthcheck:
      test: ['CMD', 'wget', '--no-verbose', '--tries=1', '--spider', 'http://localhost:8000/health']
      interval: 30s
      timeout: 10s
      retries: 3

volumes:
  postgres_data:
    driver: local
  redis_data:
    driver: local
  pipeline services_data:
    driver: local

networks:
  zixly-local:
    driver: bridge
    ipam:
      config:
        - subnet: 172.20.0.0/16

5.2 Secure Credential Management

File: .env.local

# Local Development Environment
NODE_ENV=development

# Database Configuration
POSTGRES_USER=zixly_admin
POSTGRES_PASSWORD=local_dev_password
POSTGRES_DB=zixly_main

# pipeline services Configuration
pipeline services_BASIC_AUTH_ACTIVE=true
pipeline services_BASIC_AUTH_USER=admin
pipeline services_BASIC_AUTH_PASSWORD=local_dev_password
pipeline services_DB_NAME=pipeline services
pipeline services_DB_USER=zixly_admin
pipeline services_DB_PASSWORD=local_dev_password

# Plane Configuration
PLANE_SECRET_KEY=local_plane_secret_key
PLANE_DB_NAME=plane

# External Services (for testing)
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key
SMTP_HOST=smtp.office365.com
SMTP_PORT=587
SMTP_USER=ops@colemorton.com.au
SMTP_PASSWORD=your_email_password

File: .env.local.template

# Local Development Environment Template
# Copy this file to .env.local and update with your values

NODE_ENV=development

# Database Configuration
POSTGRES_USER=zixly_admin
POSTGRES_PASSWORD=your_local_db_password
POSTGRES_DB=zixly_main

# pipeline services Configuration
pipeline services_BASIC_AUTH_ACTIVE=true
pipeline services_BASIC_AUTH_USER=admin
pipeline services_BASIC_AUTH_PASSWORD=your_pipeline services_password
pipeline services_DB_NAME=pipeline services
pipeline services_DB_USER=zixly_admin
pipeline services_DB_PASSWORD=your_local_db_password

# Plane Configuration
PLANE_SECRET_KEY=your_plane_secret_key
PLANE_DB_NAME=plane

# External Services (for testing)
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key
SMTP_HOST=smtp.office365.com
SMTP_PORT=587
SMTP_USER=your_email@domain.com
SMTP_PASSWORD=your_email_password

5.3 Database Initialization

File: init-scripts/01-create-databases.sql

-- Create databases for local development
CREATE DATABASE pipeline services;
CREATE DATABASE plane;

-- Create service users
CREATE USER pipeline services_user WITH PASSWORD 'pipeline services_password';
CREATE USER plane_user WITH PASSWORD 'plane_password';

-- Grant permissions
GRANT ALL PRIVILEGES ON DATABASE pipeline services TO pipeline services_user;
GRANT ALL PRIVILEGES ON DATABASE plane TO plane_user;

-- Enable required extensions
\c pipeline services;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

\c plane;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

5.4 Local Development Scripts

File: scripts/start-local.sh

#!/bin/bash
# Start Zixly local development stack

set -euo pipefail

echo "πŸš€ Starting Zixly local development stack..."

# Check if .env.local exists
if [ ! -f ".env.local" ]; then
    echo "❌ .env.local file not found. Please copy .env.local.template to .env.local and configure."
    exit 1
fi

# Check if Docker is running
if ! docker info > /dev/null 2>&1; then
    echo "❌ Docker is not running. Please start Docker Desktop."
    exit 1
fi

# Create necessary directories
mkdir -p letsencrypt init-scripts

# Start the stack
echo "πŸ“¦ Starting Docker Compose stack..."
docker-compose -f docker-compose.local.yml up -d

# Wait for services to be healthy
echo "⏳ Waiting for services to be healthy..."
sleep 30

# Check service health
echo "πŸ” Checking service health..."

# Check PostgreSQL
if docker exec zixly-postgres pg_isready -U zixly_admin -d zixly_main > /dev/null; then
    echo "βœ… PostgreSQL is healthy"
else
    echo "❌ PostgreSQL health check failed"
fi

# Check Redis
if docker exec zixly-redis redis-cli ping > /dev/null; then
    echo "βœ… Redis is healthy"
else
    echo "❌ Redis health check failed"
fi

# Check pipeline services
if curl -f -s http://localhost:5678/healthz > /dev/null; then
    echo "βœ… pipeline services is healthy"
else
    echo "❌ pipeline services health check failed"
fi

# Check Plane
if curl -f -s http://localhost:8000/health > /dev/null; then
    echo "βœ… Plane is healthy"
else
    echo "❌ Plane health check failed"
fi

echo "πŸŽ‰ Zixly local stack is running!"
echo ""
echo "πŸ“‹ Service URLs:"
echo "  - pipeline services: http://localhost:5678"
echo "  - Plane: http://localhost:8000"
echo "  - PostgreSQL: localhost:5432"
echo "  - Redis: localhost:6379"
echo ""
echo "πŸ”§ Next steps:"
echo "  1. Configure Plane workspace and generate API token"
echo "  2. Import Plane smoke test workflow in pipeline services"
echo "  3. Run smoke test to validate integration"

File: scripts/stop-local.sh

#!/bin/bash
# Stop Zixly local development stack

echo "πŸ›‘ Stopping Zixly local stack..."

docker-compose -f docker-compose.local.yml down

echo "βœ… Zixly local stack stopped"

File: scripts/restart-local.sh

#!/bin/bash
# Restart Zixly local development stack

echo "πŸ”„ Restarting Zixly local stack..."

./scripts/stop-local.sh
sleep 5
./scripts/start-local.sh

File: scripts/clean-local.sh

#!/bin/bash
# Clean Zixly local development stack (removes all data)

echo "🧹 Cleaning Zixly local stack (this will remove all data)..."

read -p "Are you sure? This will delete all local data. (y/N): " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
    docker-compose -f docker-compose.local.yml down -v
    docker volume prune -f
    echo "βœ… Local stack cleaned"
else
    echo "❌ Clean cancelled"
fi

File: scripts/setup-local.sh

#!/bin/bash
# Setup Zixly local development environment

echo "πŸ”§ Setting up Zixly local development environment..."

# Make scripts executable
chmod +x scripts/start-local.sh
chmod +x scripts/stop-local.sh
chmod +x scripts/restart-local.sh
chmod +x scripts/clean-local.sh

# Create .env.local if it doesn't exist
if [ ! -f ".env.local" ]; then
    if [ -f ".env.local.template" ]; then
        cp .env.local.template .env.local
        echo "πŸ“ Created .env.local from template. Please update with your values."
    else
        echo "❌ .env.local.template not found. Please create .env.local manually."
        exit 1
    fi
fi

# Create necessary directories
mkdir -p letsencrypt init-scripts

echo "βœ… Local development environment setup complete!"
echo ""
echo "πŸ”§ Next steps:"
echo "  1. Update .env.local with your configuration"
echo "  2. Run ./scripts/start-local.sh to start the stack"

Week 6: Plane Configuration & Integration

6.1 Plane Setup

Initial Configuration Steps:

  1. Access Plane: http://localhost:8000
  2. Complete Setup Wizard:
    • Create admin account
    • Set up workspace: β€œZixly Internal Operations”
    • Configure team members
  3. Create Project Templates:
    • Client Onboarding Template
    • Support Ticket Template
    • Internal Operations Template
  4. Generate API Token:
    • Go to Settings β†’ API Tokens
    • Create token with full access
    • Store in pipeline services credentials

6.2 Plane Smoke Test Workflow

File: pipeline services-workflows/internal/plane-smoke-test.json

{
  "name": "Plane Smoke Test",
  "nodes": [
    {
      "parameters": {
        "httpMethod": "GET",
        "url": "http://plane:8000/api/workspaces/",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "httpHeaderAuth": {
          "name": "Authorization",
          "value": "Bearer "
        }
      },
      "id": "plane-workspaces-test",
      "name": "Test Plane Workspaces",
      "type": "pipeline services-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [240, 300]
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "strict"
          },
          "conditions": [
            {
              "id": "condition1",
              "leftValue": "=",
              "rightValue": 200,
              "operator": {
                "type": "number",
                "operation": "equals"
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "id": "check-response",
      "name": "Check Response Status",
      "type": "pipeline services-nodes-base.if",
      "typeVersion": 2,
      "position": [460, 300]
    },
    {
      "parameters": {
        "httpMethod": "GET",
        "url": "http://plane:8000/api/projects/",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "httpHeaderAuth": {
          "name": "Authorization",
          "value": "Bearer "
        }
      },
      "id": "plane-projects-test",
      "name": "Test Plane Projects",
      "type": "pipeline services-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [680, 200]
    },
    {
      "parameters": {
        "httpMethod": "POST",
        "url": "http://plane:8000/api/workspaces//projects/",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "httpHeaderAuth": {
          "name": "Authorization",
          "value": "Bearer "
        },
        "sendBody": true,
        "bodyContentType": "json",
        "jsonBody": "{\n  \"name\": \"Smoke Test Project\",\n  \"description\": \"Test project created by pipeline services smoke test\",\n  \"identifier\": \"ST\",\n  \"network\": 0,\n  \"icon_prop\": {\n    \"name\": \"package\",\n    \"color\": \"#3b82f6\"\n  }\n}"
      },
      "id": "create-test-project",
      "name": "Create Test Project",
      "type": "pipeline services-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [900, 200]
    },
    {
      "parameters": {
        "httpMethod": "POST",
        "url": "http://plane:8000/api/workspaces//projects//issues/",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "httpHeaderAuth": {
          "name": "Authorization",
          "value": "Bearer "
        },
        "sendBody": true,
        "bodyContentType": "json",
        "jsonBody": "{\n  \"name\": \"Smoke Test Issue\",\n  \"description\": \"Test issue created by pipeline services smoke test workflow\",\n  \"priority\": \"low\",\n  \"state\": \"backlog\"\n}"
      },
      "id": "create-test-issue",
      "name": "Create Test Issue",
      "type": "pipeline services-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [1120, 200]
    },
    {
      "parameters": {
        "httpMethod": "DELETE",
        "url": "http://plane:8000/api/workspaces//projects//",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "httpHeaderAuth": {
          "name": "Authorization",
          "value": "Bearer "
        }
      },
      "id": "cleanup-test-project",
      "name": "Cleanup Test Project",
      "type": "pipeline services-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [1340, 200]
    },
    {
      "parameters": {
        "authentication": "genericCredentialType",
        "genericAuthType": "smtp",
        "smtpAuth": {
          "user": "",
          "password": ""
        },
        "fromEmail": "",
        "toEmail": "ops@colemorton.com.au",
        "subject": "βœ… Plane Smoke Test - SUCCESS (Local)",
        "message": "Plane integration smoke test completed successfully in local environment:\n\n- Plane API connectivity: βœ…\n- Workspaces access: βœ…\n- Projects access: βœ…\n- Project creation: βœ…\n- Issue creation: βœ…\n- Cleanup: βœ…\n\nPlane is ready for production use with Pipeline Workflows.",
        "options": {}
      },
      "id": "send-success-email",
      "name": "Send Success Email",
      "type": "pipeline services-nodes-base.emailSend",
      "typeVersion": 2.1,
      "position": [1560, 200]
    },
    {
      "parameters": {
        "authentication": "genericCredentialType",
        "genericAuthType": "smtp",
        "smtpAuth": {
          "user": "",
          "password": ""
        },
        "fromEmail": "",
        "toEmail": "ops@colemorton.com.au",
        "subject": "❌ Plane Smoke Test - FAILED (Local)",
        "message": "Plane integration smoke test failed in local environment:\n\n- Error: \n- Status: \n\nPlease check Plane configuration and API credentials.",
        "options": {}
      },
      "id": "send-failure-email",
      "name": "Send Failure Email",
      "type": "pipeline services-nodes-base.emailSend",
      "typeVersion": 2.1,
      "position": [680, 400]
    }
  ],
  "connections": {
    "Test Plane Workspaces": {
      "main": [
        [
          {
            "node": "Check Response Status",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Check Response Status": {
      "main": [
        [
          {
            "node": "Test Plane Projects",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Send Failure Email",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Test Plane Projects": {
      "main": [
        [
          {
            "node": "Create Test Project",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Create Test Project": {
      "main": [
        [
          {
            "node": "Create Test Issue",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Create Test Issue": {
      "main": [
        [
          {
            "node": "Cleanup Test Project",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Cleanup Test Project": {
      "main": [
        [
          {
            "node": "Send Success Email",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "pinData": {},
  "settings": {
    "executionOrder": "v1"
  },
  "staticData": null,
  "tags": [],
  "triggerCount": 0,
  "updatedAt": "2025-01-27T00:00:00.000Z",
  "versionId": "1"
}

6.3 pipeline services Credentials Configuration

File: pipeline services-credentials/credentials-local.json

{
  "supabase": {
    "name": "Supabase",
    "type": "httpBasicAuth",
    "data": {
      "user": "your-supabase-user",
      "password": "your-supabase-password"
    }
  },
  "outlookSmtp": {
    "name": "Outlook SMTP",
    "type": "smtp",
    "data": {
      "host": "smtp.office365.com",
      "port": 587,
      "user": "ops@colemorton.com.au",
      "password": "your-email-password"
    }
  },
  "planeApi": {
    "name": "Plane API",
    "type": "httpHeaderAuth",
    "data": {
      "name": "Authorization",
      "value": "Bearer your-plane-api-token"
    }
  }
}

6.4 Local Development Documentation

File: docs/local-development/README.md

# Zixly Local Development Environment

## Quick Start

1. **Setup environment**:
   ```bash
   ./scripts/setup-local.sh
   ```
  1. Configure credentials:
    • Update .env.local with your values
    • Configure pipeline services credentials in the interface
  2. Start the stack:

    ./scripts/start-local.sh
    
  3. Access services:
    • pipeline services: http://localhost:5678
    • Plane: http://localhost:8000
    • PostgreSQL: localhost:5432
    • Redis: localhost:6379
  4. Configure Plane:
    • Complete setup wizard
    • Generate API token
    • Update pipeline services credentials
  5. Import smoke test workflow:
    • Import plane-smoke-test.json in pipeline services
    • Configure credentials
    • Execute workflow

Development Workflow

Daily Development

  1. Start stack: ./scripts/start-local.sh
  2. Make changes to workflows
  3. Test changes in pipeline services
  4. Stop stack: ./scripts/stop-local.sh

Clean Development

  1. Clean stack: ./scripts/clean-local.sh
  2. Start fresh: ./scripts/start-local.sh
  3. Reconfigure services

Debugging

Troubleshooting

Common Issues

  1. Port conflicts: Check if ports 5432, 5678, 6379, 8000 are available
  2. Docker not running: Start Docker Desktop
  3. Services not starting: Check logs for errors
  4. Database connection issues: Wait for PostgreSQL to be ready

Reset Everything

./scripts/clean-local.sh
./scripts/start-local.sh

Security Notes


---

## Success Criteria & Validation

### Technical Validation

| Component | Success Criteria | Validation Method |
|-----------|------------------|-------------------|
| **Docker Services** | Both pipeline services and Plane containers healthy | `docker-compose -f docker-compose.local.yml ps` |
| **Local Access** | Both services accessible via localhost | `curl http://localhost:5678` and `curl http://localhost:8000` |
| **Database** | Both pipeline services and Plane databases accessible | Connect to each service database |
| **Plane Smoke Test** | Workflow executes successfully | Run smoke test workflow in pipeline services |
| **Integration** | pipeline services can create projects/issues in Plane | Manual test via Pipeline Workflows |

### Performance Validation

| Metric | Target | Measurement |
|--------|--------|-------------|
| **Service Startup** | < 2 minutes | Time from `./scripts/start-local.sh` to both healthy |
| **Plane API Response** | < 500ms | API calls from pipeline services to Plane |
| **Memory Usage** | < 2GB total | `docker stats --no-stream` |
| **Disk Usage** | < 5GB | `df -h` on local machine |

---

## Local Development Benefits

### Advantages of Local Development

1. **Rapid Iteration** - No deployment delays
2. **Offline Development** - No internet dependency
3. **Easy Debugging** - Direct access to logs and containers
4. **Cost Effective** - No cloud infrastructure costs
5. **Version Control** - All changes tracked in git
6. **Team Collaboration** - Consistent local environment

### Development Workflow

1. **Start Development**:
   ```bash
   ./scripts/start-local.sh
  1. Make Changes:
    • Edit workflows in pipeline services
    • Modify Plane configuration
    • Update database schemas
  2. Test Changes:
    • Run smoke test workflow
    • Validate integration
    • Check logs for errors
  3. Commit Changes:

    git add .
    git commit -m "feat: add plane integration workflow"
    
  4. Stop Development:
    ./scripts/stop-local.sh
    

Security Considerations

Credential Management

  1. Environment Variables:
    • All sensitive data in .env.local (not committed)
    • Template file .env.local.template for reference
    • Strong passwords for local development
  2. Database Security:
    • Local PostgreSQL with restricted access
    • No external network exposure
    • Data encrypted at rest
  3. Network Security:
    • Services only accessible via localhost
    • No external port exposure
    • Internal Docker network isolation

Best Practices

  1. Never commit credentials:

    # Add to .gitignore
    .env.local
    .env.production
    
  2. Use strong passwords:

    # Generate secure passwords
    openssl rand -base64 32
    
  3. Regular credential rotation:

    • Change passwords monthly
    • Update API tokens regularly
    • Monitor access logs

Next Phase Preparation

Phase 3 Prerequisites

With local development complete, Phase 3 will focus on:

  1. Production Deployment - Deploy to DigitalOcean
  2. SSL/TLS Configuration - Traefik with Let’s Encrypt
  3. Monitoring Setup - Prometheus and Grafana
  4. Backup Systems - Automated backups

Local to Production Migration

The local development environment provides:


Conclusion

Phase 2 establishes a robust local development environment for Zixly’s self-hostable stack integration. This approach provides:

  1. Rapid Development - Local environment enables fast iteration
  2. Cost Efficiency - No cloud infrastructure costs during development
  3. Team Collaboration - Consistent local environment for all developers
  4. Production Readiness - Validated workflows ready for deployment
  5. Security - Secure credential management and network isolation

By completing Phase 2, Zixly has a fully functional local development environment with Plane integration, ready for production deployment in Phase 3.


Phase 2 Status: 🎯 READY FOR LOCAL DEVELOPMENT
Estimated Implementation Time: 20 hours over 2 weeks
Critical Path: Docker setup β†’ Plane configuration β†’ Smoke test β†’ Integration validation

Document Version: 1.0
Last Updated: 2025-01-27
Owner: Zixly Technical Architecture
Review Cycle: Daily during development