Telemetry Migration Guide

Migrating from standalone Vitals service (4 separate services) to an integrated module within the Core API.

Current State (Standalone)

vitals-project/
├── services/
│   ├── api/              ← separate Go binary, port 4000
│   ├── audit-worker/     ← separate Go binary, Redis consumer
│   ├── analytics-worker/ ← separate Go binary, Redis consumer
│   └── proxy-discovery/  ← separate Go binary, IP detection
├── internal/
│   ├── handlers/         ← HTTP handlers (Fiber)
│   ├── middleware/        ← 7-processor compliance pipeline
│   ├── analytics/        ← analytics service
│   ├── media/            ← media session service
│   ├── audit/            ← audit service
│   ├── staff/            ← staff tracking service
│   ├── privacy/          ← CCPA exclusion management
│   ├── geo/              ← MaxMind GeoIP lookup
│   ├── storage/
│   │   ├── clickhouse/   ← ClickHouse connection
│   │   └── postgres/     ← Vitals PG connection (sqlc generated)
│   ├── config/           ← standalone config
│   └── utils/            ← crypto, enrichment, JSON
├── migrations/
│   ├── clickhouse/       ← ClickHouse migrations
│   ├── postgres_v2/      ← Vitals PG migrations
│   └── postgres/         ← legacy (ignore)
└── cmd/                  ← CLI tools (migrate, reset, etc.)

4 separate services, own databases, own config, own auth.

Target State (Integrated)

core-api/
├── internal/
│   ├── ... (existing Core API packages)
│   │
│   └── telemetry/                 ← new package
│       ├── handlers/
│       │   ├── media.go           ← COPY from vitals handlers/media.go
│       │   ├── analytics.go       ← COPY from vitals handlers/analytics.go
│       │   ├── audit.go           ← COPY from vitals handlers/audit.go
│       │   ├── dashboard.go       ← COPY from vitals handlers/dashboard.go
│       │   └── admin.go           ← COPY from vitals handlers/admin.go
│       ├── middleware/
│       │   ├── compliance.go      ← COPY (adapt to Core API middleware chain)
│       │   ├── audit_processor.go
│       │   ├── security_processor.go
│       │   ├── actor_processor.go
│       │   ├── request_processor.go
│       │   └── compliance_policy.go
│       ├── services/
│       │   ├── analytics.go       ← COPY from vitals analytics/service.go
│       │   ├── media.go           ← COPY from vitals media/service.go
│       │   ├── audit.go           ← COPY from vitals audit/service.go
│       │   └── privacy.go         ← COPY from vitals privacy/exclusions.go
│       ├── storage/
│       │   ├── clickhouse.go      ← COPY from vitals storage/clickhouse/
│       │   └── telemetry_pg.go    ← COPY from vitals storage/postgres/
│       ├── geo/
│       │   └── init.go            ← COPY from vitals geo/init.go
│       ├── workers/
│       │   ├── audit_worker.go    ← REFACTOR: goroutine, not separate binary
│       │   └── analytics_worker.go ← REFACTOR: goroutine, not separate binary
│       └── config.go              ← MERGE into Core API config
│
├── migrations/
│   ├── ... (existing Core API migrations)
│   ├── telemetry_clickhouse/       ← COPY from vitals migrations/clickhouse/
│   └── telemetry_postgres/        ← COPY from vitals migrations/postgres_v2/

1 App Runner service (Core API), workers as goroutines, shared auth.

Migration Steps

Phase 1: Copy & Restructure

1.1 Copy packages into Core API repo

# Create telemetry package structure
mkdir -p internal/telemetry/{handlers,middleware,services,storage,geo,workers}

# Copy handlers (adapt import paths)
cp vitals/internal/handlers/media.go      internal/telemetry/handlers/
cp vitals/internal/handlers/analytics.go  internal/telemetry/handlers/
cp vitals/internal/handlers/audit.go      internal/telemetry/handlers/
cp vitals/internal/handlers/dashboard.go  internal/telemetry/handlers/
cp vitals/internal/handlers/admin.go      internal/telemetry/handlers/

# Copy middleware pipeline
cp vitals/internal/middleware/*.go         internal/telemetry/middleware/

# Copy services
cp vitals/internal/analytics/service.go   internal/telemetry/services/analytics.go
cp vitals/internal/media/service.go       internal/telemetry/services/media.go
cp vitals/internal/audit/service.go       internal/telemetry/services/audit.go
cp vitals/internal/privacy/exclusions.go  internal/telemetry/services/privacy.go

# Copy storage
cp vitals/internal/storage/clickhouse/connection.go  internal/telemetry/storage/clickhouse.go
cp vitals/internal/storage/postgres/*.go             internal/telemetry/storage/

# Copy geo
cp vitals/internal/geo/init.go            internal/telemetry/geo/

# Copy utils (merge into Core API utils or keep separate)
cp vitals/internal/utils/*.go             internal/telemetry/utils/

# Copy migrations
cp -r vitals/migrations/clickhouse/       migrations/telemetry_clickhouse/
cp -r vitals/migrations/postgres_v2/      migrations/telemetry_postgres/

1.2 Update import paths

All files: change vitals/internal/... → core-api/internal/telemetry/...

1.3 Copy CLI tools

cp -r vitals/cmd/migrate-clickhouse/   cmd/migrate-telemetry-clickhouse/
cp -r vitals/cmd/migrate-postgres/     cmd/migrate-telemetry-postgres/

Phase 2: Adapt Integration Points

2.1 Config — Merge into Core API config

Before (standalone):

// vitals/internal/config/config.go
type Config struct {
    Port            string
    Host            string
    DatabaseURL     string  // Vitals PG
    RedisURL        string
    ClickhouseURL   string
    JWTSecret       string  // own JWT
    EncryptionKey   string
    Environment     string
    TrustedProxyCIDRs string
    MaxmindAccountID  string
    MaxmindLicenseKey string
}

After (merged):

// core-api/internal/config/config.go
type Config struct {
    // ... existing Core API config ...

    // Telemetry databases (separate from main PG)
    TelemetryDatabaseURL string `env:"TELEMETRY_DATABASE_URL"`
    ClickhouseURL       string `env:"CLICKHOUSE_URL"`

    // Telemetry services
    EncryptionKey       string `env:"TELEMETRY_ENCRYPTION_KEY"`
    MaxmindAccountID    string `env:"MAXMIND_ACCOUNT_ID"`
    MaxmindLicenseKey   string `env:"MAXMIND_LICENSE_KEY"`

    // Reuse from Core API (no duplication):
    // - RedisURL (same Redis)
    // - JWTSecret (same auth)
    // - Environment (same env)
}

Drop: Port, Host, TrustedProxyCIDRs (Core API handles these), JWTSecret (reuse Core API's).

2.2 Auth — Reuse Core API's JWT middleware

Before:

// Vitals had its own JWT validation (old standalone project)
app.Use(vitalsMiddleware.JWTAuth(config.JWTSecret))

After:

// Telemetry routes use Core API's auth middleware
telemetryGroup := app.Group("/v1", coreMiddleware.RequireAuth())
telemetryGroup.Post("/media/events", telemetryHandlers.TrackMediaEvent)
telemetryGroup.Post("/analytics/track", telemetryHandlers.TrackAnalytics)
// ...

// Admin routes use Core API's admin middleware
adminGroup := app.Group("/v1/admin", coreMiddleware.RequireRole("admin", "superadmin"))
adminGroup.Post("/geo/update", telemetryHandlers.UpdateGeo)
// ...

2.3 Router — Mount Telemetry routes on Core API router

// core-api/internal/router/router.go
func SetupRoutes(app *fiber.App, ..., telemetryHandlers *telemetry.Handlers) {
    // ... existing Core API routes ...

    // Telemetry routes (same API server)
    media := app.Group("/v1/media")
    media.Post("/events", telemetryHandlers.TrackMediaEvent)
    media.Get("/bandwidth/stats", telemetryHandlers.GetBandwidthStats)
    media.Get("/sessions/stats", telemetryHandlers.GetSessionStats)

    app.Post("/v1/analytics/track", telemetryHandlers.TrackAnalytics)
    app.Post("/v1/pose/frames", telemetryHandlers.TrackPoseFrames)
    app.Post("/v1/audit/ingest", telemetryHandlers.IngestAudit)
    app.Post("/v1/errors/report", telemetryHandlers.ReportError)

    admin := app.Group("/v1/admin", requireAdmin)
    admin.Post("/geo/update", telemetryHandlers.UpdateGeo)
    admin.Get("/geo/status", telemetryHandlers.GeoStatus)
    admin.Get("/privacy/exclusions", telemetryHandlers.ListExclusions)
    admin.Post("/privacy/exclusions/sync", telemetryHandlers.SyncExclusions)
    admin.Get("/privacy/exclusions/check", telemetryHandlers.CheckExclusion)

    dashboard := app.Group("/dashboard", requireAdmin)
    dashboard.Get("/", telemetryHandlers.DashboardHome)
    // ...
}

2.4 Workers — Convert to goroutines

Before (separate binaries):

// services/audit-worker/main.go
func main() {
    config := config.Load()
    redis := connectRedis(config.RedisURL)
    pg := connectPG(config.DatabaseURL)
    // blocks forever, consuming from Redis queue
    worker.Run(redis, pg)
}

After (goroutines in Core API startup):

// cmd/api/main.go
func main() {
    // ... existing Core API setup ...

    // Start Telemetry workers as background goroutines
    auditWorker := telemetry.NewAuditWorker(redisClient, telemetryPG, geoDB)
    analyticsWorker := telemetry.NewAnalyticsWorker(redisClient, clickhouseConn)

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    go auditWorker.Run(ctx)
    go analyticsWorker.Run(ctx)

    // ... start HTTP server ...
}

Phase 3: Drop Unnecessary Components

Component	Action	Reason
services/api/main.go	Drop	Core API is the server now
services/audit-worker/main.go	Drop	Converted to goroutine
services/analytics-worker/main.go	Drop	Converted to goroutine
services/proxy-discovery/	Drop	App Runner/Cloudflare handles trusted proxy detection
internal/config/config.go	Drop	Merged into Core API config
internal/middleware/staff_processor.go	Evaluate	May overlap with Core API's existing staff tracking
migrations/postgres/ (v1)	Drop	Legacy, use postgres_v2/ only
tools/	Drop	Dev tools, not needed in production
docs/ (vitals internal)	Drop	Replaced by docs/telemetry/
scripts/	Evaluate	Keep setup-geodb.sh, generate-keys.sh; drop platform-specific deploy scripts
data/geo/	Keep	GeoLite2 database files needed at runtime

Phase 4: Add New Features (from docs)

These are in our docs but not in the existing vitals code:

Feature	Schema	Handler Changes
media_buffering_events table	New ClickHouse migration	Add insert in buffering_start/buffering_end handler
pose_tracking_frames table	New ClickHouse migration	New handler for POST /v1/pose/frames
Video performance fields on media_sessions	Alter table migration	Update heartbeat/session_start handlers to accept new fields
ttfb_ms, video_load_time_ms, etc.	Part of above	Frontend sends, handler stores

Phase 5: Database Setup

# Telemetry ClickHouse (ClickHouse Cloud, accessed over HTTPS)
# Run migrations
go run cmd/migrate-telemetry-clickhouse/main.go

# Telemetry PostgreSQL (AWS RDS instance in private subnet)
# Run migrations
go run cmd/migrate-telemetry-postgres/main.go

Environment variables to add to Core API service:

TELEMETRY_DATABASE_URL=postgresql://...      # Telemetry PG (TimescaleDB)
CLICKHOUSE_URL=clickhouse://...             # ClickHouse
TELEMETRY_ENCRYPTION_KEY=...                # PII encryption key
MAXMIND_ACCOUNT_ID=...                      # MaxMind API
MAXMIND_LICENSE_KEY=...                     # MaxMind API

Checklist

• [ ] Copy packages into internal/telemetry/
• [ ] Update all import paths
• [ ] Merge config into Core API config
• [ ] Replace JWT auth with Core API's auth middleware
• [ ] Mount routes on Core API router
• [ ] Convert audit-worker to goroutine
• [ ] Convert analytics-worker to goroutine
• [ ] Drop proxy-discovery service
• [ ] Drop standalone API server
• [ ] Add new ClickHouse migrations (buffering_events, pose_tracking_frames, video perf fields)
• [ ] Create Telemetry PG service in AWS (RDS instance in private subnet)
• [ ] Create ClickHouse Cloud service (configure access credentials)
• [ ] Run all migrations
• [ ] Verify audit log immutability
• [ ] Verify CCPA exclusion sync
• [ ] Verify media event tracking end-to-end
• [ ] Remove old standalone vitals services