arbiter/docs/deployment.md

Deployment Guide

This guide covers deploying Arbiter in various environments.

Local Development

Prerequisites

• Python 3.12+
• Node.js 20+
• Docker and Docker Compose
• PostgreSQL 16 (or use Docker)
• Redis 7 (or use Docker)

Quick Start with Docker

Start all services with a single command:

# Start everything
docker compose up -d

# Run database migrations
docker compose run --rm migrate

# View logs
docker compose logs -f

Services:

• API: http://localhost:8000
• Dashboard: http://localhost:3000
• PostgreSQL: localhost:5432
• Redis: localhost:6379

Development without Docker

For faster iteration during development, run infrastructure in Docker and services locally:

# Start only database and Redis
docker compose up -d db redis

# Install Python dependencies
pip install -e ".[dev]"

# Run migrations
alembic upgrade head

# Terminal 1: Start API server
uvicorn src.arbiter.main:app --reload

# Terminal 2: Start worker
arq src.arbiter.worker.tasks.WorkerSettings

# Terminal 3: Start dashboard
cd dashboard && npm install && npm run dev

Production Deployment

Environment Configuration

Create a .env file with production values. See .env.example for all available options.

Required environment variables:

# Database (use strong credentials)
ARBITER_DATABASE_URL=postgresql+asyncpg://user:password@db-host:5432/arbiter

# Redis
ARBITER_REDIS_URL=redis://redis-host:6379/0

# LLM API key (at least one required)
OPENAI_API_KEY=sk-...
# or
ANTHROPIC_API_KEY=sk-ant-...

# Platform integration (at least one required)
ARBITER_GITHUB_TOKEN=ghp_...
ARBITER_GITHUB_WEBHOOK_SECRET=your-webhook-secret
# or
ARBITER_GITLAB_TOKEN=glpat-...
ARBITER_GITLAB_WEBHOOK_TOKEN=your-webhook-token

# Security
ARBITER_CORS_ORIGINS=["https://your-dashboard-domain.com"]

Production Checklist

Before deploying to production:

• Set strong database credentials
• Configure webhook secrets
• Set up SSL/TLS termination
• Configure CORS origins to match your dashboard domain
• Set appropriate rate limits
• Configure monitoring and alerting
• Set up log aggregation
• Plan for database backups
• Review cost control settings

Database Setup

Arbiter uses PostgreSQL with async support via asyncpg.

1. Create database:

CREATE USER arbiter WITH PASSWORD 'strong-password-here';
CREATE DATABASE arbiter OWNER arbiter;

2. Run migrations:

alembic upgrade head

3. (Optional) Enable connection pooling with PgBouncer for high-traffic deployments.

Redis Configuration

Redis is used for:

• Job queue (arq worker)
• LLM response caching
• Session management

Recommended Redis configuration for production:

# redis.conf
maxmemory 256mb
maxmemory-policy allkeys-lru
appendonly yes

For high availability, consider Redis Sentinel or a managed Redis service.

Webhook Setup

GitHub

1. Go to repository Settings > Webhooks > Add webhook
2. Configure:
   • Payload URL: https://your-domain.com/webhooks/github
   • Content type: application/json
   • Secret: Same value as ARBITER_GITHUB_WEBHOOK_SECRET
   • Events: Select "Pull requests" and "Issue comments"
3. Save and test delivery

GitLab

1. Go to project Settings > Webhooks
2. Configure:
   • URL: https://your-domain.com/webhooks/gitlab
   • Secret token: Same value as ARBITER_GITLAB_WEBHOOK_TOKEN
   • Triggers: Select "Merge request events" and "Comments"
3. Save and test

SSL/TLS

Arbiter should always be deployed behind HTTPS in production.

With nginx

server {
    listen 443 ssl http2;
    server_name arbiter.example.com;

    ssl_certificate /etc/ssl/certs/arbiter.crt;
    ssl_certificate_key /etc/ssl/private/arbiter.key;

    # API
    location /api/ {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Webhooks
    location /webhooks/ {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Dashboard
    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
    }
}

With Caddy

arbiter.example.com {
    handle /api/* {
        reverse_proxy localhost:8000
    }
    handle /webhooks/* {
        reverse_proxy localhost:8000
    }
    handle {
        reverse_proxy localhost:3000
    }
}

Container Orchestration

Docker Compose (Single Server)

For small deployments, use the provided Docker Compose configuration:

docker compose -f docker-compose.yml up -d
docker compose run --rm migrate

Kubernetes

Example deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: arbiter-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: arbiter-api
  template:
    metadata:
      labels:
        app: arbiter-api
    spec:
      containers:
      - name: api
        image: arbiter:latest
        ports:
        - containerPort: 8000
        envFrom:
        - secretRef:
            name: arbiter-secrets
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8000
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8000

Scaling Considerations

Horizontal Scaling

• API servers: Stateless, scale horizontally behind a load balancer
• Workers: Scale based on job queue depth; each worker processes up to ARBITER_WORKER_MAX_JOBS concurrent jobs
• Dashboard: Static files, can be served from CDN

Vertical Scaling

• Database: Increase connection pool size for more concurrent requests
• Redis: Increase memory for larger cache hit rates

Recommended Starting Configuration

Component	Replicas	Resources
API	2	512MB RAM, 0.5 CPU
Worker	2	1GB RAM, 1 CPU
Dashboard	1	128MB RAM
PostgreSQL	1	1GB RAM
Redis	1	256MB RAM

Monitoring

Prometheus Metrics

The API exposes Prometheus metrics at /metrics:

• arbiter_reviews_total — Total reviews processed
• arbiter_review_duration_seconds — Review processing time
• arbiter_llm_requests_total — LLM API calls
• arbiter_llm_cache_hits_total — Cache hit rate
• arbiter_cost_usd_total — Total LLM costs

Health Endpoints

• GET /health — Basic liveness check
• GET /health/live — Kubernetes liveness probe
• GET /health/ready — Readiness check (database + Redis)

Alerting Recommendations

Set up alerts for:

• High error rate on webhook endpoints
• Worker queue depth exceeding threshold
• LLM cost exceeding daily budget
• Database connection pool exhaustion
• Redis memory usage

Environment Variables Reference

All environment variables are prefixed with ARBITER_ unless noted.

Database

Variable	Description	Default
DATABASE_URL	PostgreSQL connection URL	postgresql+asyncpg://arbiter:arbiter@localhost:5432/arbiter
DATABASE_POOL_SIZE	Connection pool size	5
DATABASE_MAX_OVERFLOW	Max overflow connections	10

Redis

Variable	Description	Default
REDIS_URL	Redis connection URL	redis://localhost:6379/0
REDIS_MAX_CONNECTIONS	Max connection pool size	10

LLM Configuration

Variable	Description	Default
DEFAULT_MODEL	Default LLM model	gpt-4o
LLM_TIMEOUT	Request timeout (seconds)	60
LLM_MAX_RETRIES	Retry attempts	3
MAX_TOKENS_PER_REVIEW	Token budget per review	50000
MAX_COST_PER_REVIEW_USD	Cost limit per review	0.50
CACHE_TTL_HOURS	Response cache TTL	24

Note: LLM API keys are read directly (not prefixed):

• OPENAI_API_KEY
• ANTHROPIC_API_KEY

GitHub Integration

Variable	Description	Default
GITHUB_TOKEN	API token for fetching diffs and posting comments	-
GITHUB_WEBHOOK_SECRET	Webhook HMAC secret	-
GITHUB_BASE_URL	API base URL (for GitHub Enterprise)	https://api.github.com

GitLab Integration

Variable	Description	Default
GITLAB_TOKEN	API token for fetching diffs and posting comments	-
GITLAB_WEBHOOK_TOKEN	Webhook verification token	-
GITLAB_BASE_URL	Instance URL	https://gitlab.com

API Settings

Variable	Description	Default
API_TITLE	OpenAPI title	Arbiter API
API_VERSION	API version string	0.5.0
CORS_ORIGINS	Allowed CORS origins (JSON array)	["http://localhost:3000"]
API_RATE_LIMIT_PER_MINUTE	Rate limit per client	60

Worker Settings

Variable	Description	Default
WORKER_MAX_JOBS	Max concurrent jobs	10
WORKER_JOB_TIMEOUT	Job timeout (seconds)	300
WORKER_RETRY_ATTEMPTS	Retry count on failure	3

Feature Flags

Variable	Description	Default
POST_COMMENTS	Post review comments to PRs	true
UPDATE_STATUS	Update commit status checks	true
FOLLOWUP_ENABLED	Enable follow-up question handling	true
FOLLOWUP_CONFIDENCE_THRESHOLD	Minimum confidence to respond	0.5
FOLLOWUP_MAX_TOKENS_PER_RESPONSE	Token limit for follow-ups	2000

Integration Settings

Variable	Description	Default
INTEGRATION_TIMEOUT	API request timeout (seconds)	30
INTEGRATION_MAX_RETRIES	Retry attempts	3
STATUS_CHECK_CONTEXT	Commit status context name	arbiter