Media Events Specification

Layer 2 feature, not yet implemented. See index.md for the architecture and rationale; api.md for the request envelope and auth.

This document defines all video/exercise media events that the Patient Portal sends to Telemetry API via POST /v1/media/events. Server-side these events drive aggregation into Postgres (media_session_metrics + media_buffering_events, both monthly partitioned per P41) and feed the specialist's per-patient session view in the Clinic app.

Consent gate. Every event in this doc requires the analytics per-purpose consent flag for the patient in the org. Telemetry API rejects ingest with 403 if the flag is not active. There is no consent ladder — see index.md → Consent.

Event Lifecycle

Patient opens exercise
  │
  ├── session_start         ← TTFB, load time, initial connection info
  │
  ├── play                  ← playback begins
  │     │
  │     ├── heartbeat       ← every 10s: position, buffering totals, bitrate, quality
  │     ├── heartbeat
  │     ├── buffering_start ← video stalls (separate buffering event)
  │     ├── buffering_end   ← video resumes
  │     ├── heartbeat
  │     ├── quality_change  ← ABR switches resolution/bitrate
  │     ├── heartbeat
  │     │
  │     ├── pause           ← user pauses
  │     ├── play            ← user resumes
  │     ├── seek            ← user scrubs to position
  │     │
  │     ├── milestone       ← 25%, 50%, 75%, 95% watched
  │     │
  │     ├── error           ← playback error
  │     │
  │     └── heartbeat
  │
  └── session_end           ← final state: completed or abandoned

Event Types

session_start

Fired when the player initializes and loads the video resource.

json

{
  "event": "session_start",
  "session_id": "uuid",
  "media_id": "hashed-exercise-id",
  "media_type": "video",
  "timestamp": "2026-02-17T10:00:00.000Z",
  "data": {
    "total_duration_seconds": 120.5,
    "ttfb_ms": 340,
    "video_load_time_ms": 1200,
    "cdn_response_time_ms": 280,
    "connection_type": "wifi",
    "effective_bandwidth": 12.5,
    "rtt_ms": 45,
    "initial_bitrate": 2500000,
    "initial_resolution": "720p"
  }
}

Key metrics captured:

Field	Type	Description	Source
`ttfb_ms`	uint	Time from request to first byte received	`PerformanceResourceTiming.responseStart - requestStart`
`video_load_time_ms`	uint	Time from click to first frame rendered	Custom: click timestamp → `canplay` event
`cdn_response_time_ms`	uint	CDN edge response time	`PerformanceResourceTiming.responseStart - connectEnd`
`connection_type`	string	Network type	`navigator.connection.effectiveType`
`effective_bandwidth`	float	Estimated download speed (Mbps)	`navigator.connection.downlink`
`rtt_ms`	uint	Round-trip time estimate	`navigator.connection.rtt`

play

Fired when playback starts or resumes after pause.

json

{
  "event": "play",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:01.200Z",
  "data": {
    "position_seconds": 0.0,
    "is_resume": false
  }
}

pause

Fired when user pauses playback.

json

{
  "event": "pause",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:01:15.000Z",
  "data": {
    "position_seconds": 45.2
  }
}

seek

Fired when user scrubs/jumps to a position.

json

{
  "event": "seek",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:01:20.000Z",
  "data": {
    "from_seconds": 45.2,
    "to_seconds": 10.0
  }
}

heartbeat

Fired every 10 seconds during active playback. This is the primary data source for ongoing performance monitoring. Telemetry API aggregates heartbeats server-side and writes a single media_session_metrics row per session at session_end.

json

{
  "event": "heartbeat",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:30.000Z",
  "data": {
    "position_seconds": 29.5,
    "watched_duration_seconds": 29.5,
    "completion_percent": 24.5,

    "buffering_count": 1,
    "buffering_duration_ms": 800,

    "current_bitrate": 2500000,
    "avg_bitrate": 2400000,
    "peak_bitrate": 3200000,
    "bitrate_switches": 1,
    "current_resolution": "720p",
    "resolution_switches": 0,

    "dropped_frames": 2,
    "total_frames": 750,

    "connection_type": "wifi",
    "effective_bandwidth": 11.2,
    "rtt_ms": 48,

    "error_count": 0,
    "error_types": []
  }
}

Frontend implementation notes:

buffering_count / buffering_duration_ms: cumulative for the session, not per-heartbeat
dropped_frames / total_frames: from HTMLVideoElement.getVideoPlaybackQuality()
current_bitrate: from the player's ABR controller (HLS.js: hls.bandwidthEstimate, dash.js: getQualityFor('video'))
avg_bitrate: running average computed by frontend
resolution_switches: increment on quality_change events

buffering_start

Fired when the video stalls (buffer underrun). Telemetry API persists one row in media_buffering_events per buffering occurrence (monthly-partitioned, P41).

json

{
  "event": "buffering_start",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:23.100Z",
  "data": {
    "position_seconds": 23.1,
    "bitrate_before": 2500000,
    "resolution_before": "720p",
    "cdn_response_time_ms": 1200,
    "connection_type": "wifi",
    "effective_bandwidth": 3.2
  }
}

buffering_end

Fired when playback resumes after stall. Updates the corresponding media_buffering_events row (joined by (session_id, buffering_start_position_seconds)).

json

{
  "event": "buffering_end",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:25.900Z",
  "data": {
    "position_seconds": 23.1,
    "duration_ms": 2800,
    "bitrate_after": 1500000,
    "resolution_after": "480p",
    "recovered": true
  }
}

quality_change

Fired when ABR (Adaptive Bitrate) switches video quality.

json

{
  "event": "quality_change",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:26.000Z",
  "data": {
    "position_seconds": 23.1,
    "from_bitrate": 2500000,
    "to_bitrate": 1500000,
    "from_resolution": "720p",
    "to_resolution": "480p",
    "reason": "bandwidth_decrease"
  }
}

Reasons: bandwidth_decrease, bandwidth_increase, buffer_low, user_manual, initial

milestone

Fired when the patient reaches 25%, 50%, 75%, or 95% of the video.

json

{
  "event": "milestone",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:01:30.000Z",
  "data": {
    "milestone_percent": 50,
    "position_seconds": 60.25,
    "elapsed_real_seconds": 90.0
  }
}

error

Fired when a playback error occurs.

json

{
  "event": "error",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:45.000Z",
  "data": {
    "error_code": "MEDIA_ERR_NETWORK",
    "error_message": "Failed to load video segment 5",
    "position_seconds": 45.0,
    "is_fatal": false,
    "recovery_action": "retry_segment"
  }
}

Error codes:

Code	Description
`MEDIA_ERR_ABORTED`	User or system aborted download
`MEDIA_ERR_NETWORK`	Network error during download
`MEDIA_ERR_DECODE`	Video decoding failed
`MEDIA_ERR_SRC_NOT_SUPPORTED`	Format not supported on device
`HTTP_403`	CDN rejected request (expired token, geo block)
`HTTP_404`	Video file not found
`TIMEOUT`	Request timed out
`DRM_ERROR`	DRM license acquisition failed

session_end

Fired when the session ends (user closes, video completes, or session times out).

json

{
  "event": "session_end",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:02:30.000Z",
  "data": {
    "final_position_seconds": 120.5,
    "watched_duration_seconds": 118.0,
    "completion_percent": 97.9,
    "status": "completed",

    "buffering_count": 2,
    "buffering_duration_ms": 3600,
    "bitrate_switches": 3,
    "resolution_switches": 2,
    "dropped_frames": 5,
    "total_frames": 3615,
    "error_count": 1,
    "error_types": ["MEDIA_ERR_NETWORK"]
  }
}

Status values: completed (≥95% watched), abandoned (user left), error (fatal error ended session)

Frontend Integration

Minimal Implementation

typescript

// Initialize on video mount
const sessionId = crypto.randomUUID();

// Session start — measure loading performance
const clickTime = performance.now();

videoElement.addEventListener('canplay', () => {
  const loadTime = performance.now() - clickTime;
  const timing = performance.getEntriesByName(videoUrl)[0];

  telemetry.track({
    event: 'session_start',
    session_id: sessionId,
    media_id: hashedExerciseId,
    media_type: 'video',
    data: {
      total_duration_seconds: videoElement.duration,
      ttfb_ms: timing?.responseStart - timing?.requestStart,
      video_load_time_ms: Math.round(loadTime),
      cdn_response_time_ms: timing?.responseStart - timing?.connectEnd,
      connection_type: navigator.connection?.effectiveType ?? 'unknown',
      effective_bandwidth: navigator.connection?.downlink,
      rtt_ms: navigator.connection?.rtt,
    }
  });
});

// Heartbeat — every 10 seconds during playback
const heartbeatInterval = setInterval(() => {
  if (videoElement.paused) return;
  const quality = videoElement.getVideoPlaybackQuality?.();

  telemetry.track({
    event: 'heartbeat',
    session_id: sessionId,
    data: {
      position_seconds: videoElement.currentTime,
      watched_duration_seconds: totalWatchedTime,
      completion_percent: (videoElement.currentTime / videoElement.duration) * 100,
      buffering_count: bufferCount,
      buffering_duration_ms: totalBufferMs,
      dropped_frames: quality?.droppedVideoFrames ?? 0,
      total_frames: quality?.totalVideoFrames ?? 0,
      current_bitrate: hlsPlayer?.bandwidthEstimate,
      // ... other metrics
    }
  });
}, 10_000);

// Buffering detection
videoElement.addEventListener('waiting', () => {
  bufferStartTime = performance.now();
  telemetry.track({ event: 'buffering_start', session_id: sessionId, data: { ... } });
});

videoElement.addEventListener('playing', () => {
  if (bufferStartTime) {
    const duration = performance.now() - bufferStartTime;
    bufferCount++;
    totalBufferMs += duration;
    telemetry.track({ event: 'buffering_end', session_id: sessionId, data: { duration_ms: duration, ... } });
    bufferStartTime = null;
  }
});

Browser API Compatibility

Metric	API	Support
TTFB	`PerformanceResourceTiming`	All modern browsers
Connection type	`navigator.connection`	Chrome, Edge, Opera (NOT Safari/Firefox)
Dropped frames	`getVideoPlaybackQuality()`	Chrome, Edge, Firefox (NOT Safari)
Bandwidth estimate	`navigator.connection.downlink`	Chrome, Edge, Opera
RTT	`navigator.connection.rtt`	Chrome, Edge, Opera

Fallback strategy: When an API is unavailable, send null. The backend handles nullable fields. Partial data is better than no data.

Specialist & support queries (Postgres, RLS-scoped)

All reads flow through Core API. The aggregator persists per-session totals into media_session_metrics (one row per session) and per-buffering occurrences into media_buffering_events (monthly-partitioned). Queries are org-scoped by RLS — the specialist sees only their org's data.

"Why is this patient's video not loading?"

sql

SELECT
    session_id, media_id, started_at,
    ttfb_ms, video_load_time_ms,
    buffering_count, total_buffering_duration_ms,
    avg_bitrate, peak_resolution,
    connection_type, error_count
FROM media_session_metrics
WHERE patient_id = $1
  AND started_at > now() - INTERVAL '7 days'
ORDER BY started_at DESC
LIMIT 20;

"Which exercises buffer the most for this clinic?"

sql

SELECT
    media_id,
    count(*) AS buffer_events,
    avg(duration_ms) AS avg_buffer_duration,
    percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_buffer_duration
FROM media_buffering_events
WHERE started_at > now() - INTERVAL '7 days'
GROUP BY media_id
ORDER BY buffer_events DESC
LIMIT 20;

Materialized views (media_session_weekly_agg per clinic) refresh nightly to serve cohort dashboards without scanning millions of rows on every page load. Cross-tenant queries (e.g. "which country has the worst video performance across the platform") are not exposed today — readers are clinic-scoped per the processor rule. If/when Tier 3 fires (cross-tenant analytical workload), those queries move to ClickHouse via the swap-point interfaces — see index.md → Scaling roadmap.

Video health dashboards

The Clinic app surfaces:

Per-patient view (specialist)

Recent sessions with completion %, total watch time, buffer count
Per-session detail: TTFB, load time, buffer events, error events
Replay link (signed S3 URL) to scrub through the recorded session

Per-clinic cohort view (clinic admin)

Avg TTFB / load time / completion rate across the clinic this week
Top buffering exercises
Device/connection breakdown for the clinic's patients
Adherence funnel (assigned → started → completed)

Alerting (operational)

Server-side jobs evaluate per-org thresholds and create automations that surface in the clinic-app inbox:

TTFB > 3000ms for >20% of clinic's sessions in 1h → "CDN issue affecting patients" automation
Error rate > 10% for a specific exercise → "Content quality issue" automation
Buffer rate spikes 3× above the clinic's 7-day baseline → "Network degradation" automation

Media Events Specification ​

Event Lifecycle ​

Event Types ​

session_start ​

play ​

pause ​

seek ​

heartbeat ​

buffering_start ​

buffering_end ​

quality_change ​

milestone ​

error ​

session_end ​

Frontend Integration ​

Minimal Implementation ​

Browser API Compatibility ​

Specialist & support queries (Postgres, RLS-scoped) ​

"Why is this patient's video not loading?" ​

"Which exercises buffer the most for this clinic?" ​

Video health dashboards ​

Per-patient view (specialist) ​

Per-clinic cohort view (clinic admin) ​

Alerting (operational) ​

Media Events Specification

Event Lifecycle

Event Types

session_start

play

pause

seek

heartbeat

buffering_start

buffering_end

quality_change

milestone

error

session_end

Frontend Integration

Minimal Implementation

Browser API Compatibility

Specialist & support queries (Postgres, RLS-scoped)

"Why is this patient's video not loading?"

"Which exercises buffer the most for this clinic?"

Video health dashboards

Per-patient view (specialist)

Per-clinic cohort view (clinic admin)

Alerting (operational)