Media Events Specification

This document defines all video/exercise media events that the frontend sends to Telemetry. These events feed the media_sessions and media_buffering_events ClickHouse tables and enable the Video Health Dashboard for troubleshooting patient playback issues.

Event Lifecycle

Patient opens exercise
  │
  ├── session_start         ← TTFB, load time, initial connection info
  │
  ├── play                  ← playback begins
  │     │
  │     ├── heartbeat       ← every 10s: position, buffering totals, bitrate, quality
  │     ├── heartbeat
  │     ├── buffering_start ← video stalls (separate buffering event)
  │     ├── buffering_end   ← video resumes
  │     ├── heartbeat
  │     ├── quality_change  ← ABR switches resolution/bitrate
  │     ├── heartbeat
  │     │
  │     ├── pause           ← user pauses
  │     ├── play            ← user resumes
  │     ├── seek            ← user scrubs to position
  │     │
  │     ├── milestone       ← 25%, 50%, 75%, 95% watched
  │     │
  │     ├── error           ← playback error
  │     │
  │     └── heartbeat
  │
  └── session_end           ← final state: completed or abandoned

Event Types

session_start

Fired when the player initializes and loads the video resource.

{
  "event": "session_start",
  "session_id": "uuid",
  "media_id": "hashed-exercise-id",
  "media_type": "video",
  "timestamp": "2026-02-17T10:00:00.000Z",
  "data": {
    "total_duration_seconds": 120.5,
    "ttfb_ms": 340,
    "video_load_time_ms": 1200,
    "cdn_response_time_ms": 280,
    "connection_type": "wifi",
    "effective_bandwidth": 12.5,
    "rtt_ms": 45,
    "initial_bitrate": 2500000,
    "initial_resolution": "720p"
  }
}

Key metrics captured:

Field	Type	Description	Source
ttfb_ms	uint	Time from request to first byte received	PerformanceResourceTiming.responseStart - requestStart
video_load_time_ms	uint	Time from click to first frame rendered	Custom: click timestamp → canplay event
cdn_response_time_ms	uint	CDN edge response time	PerformanceResourceTiming.responseStart - connectEnd
connection_type	string	Network type	navigator.connection.effectiveType
effective_bandwidth	float	Estimated download speed (Mbps)	navigator.connection.downlink
rtt_ms	uint	Round-trip time estimate	navigator.connection.rtt

play

Fired when playback starts or resumes after pause.

{
  "event": "play",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:01.200Z",
  "data": {
    "position_seconds": 0.0,
    "is_resume": false
  }
}

pause

Fired when user pauses playback.

{
  "event": "pause",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:01:15.000Z",
  "data": {
    "position_seconds": 45.2
  }
}

seek

Fired when user scrubs/jumps to a position.

{
  "event": "seek",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:01:20.000Z",
  "data": {
    "from_seconds": 45.2,
    "to_seconds": 10.0
  }
}

heartbeat

Fired every 10 seconds during active playback. This is the primary data source for ongoing performance monitoring. Updates the media_sessions row via ReplacingMergeTree.

{
  "event": "heartbeat",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:30.000Z",
  "data": {
    "position_seconds": 29.5,
    "watched_duration_seconds": 29.5,
    "completion_percent": 24.5,

    "buffering_count": 1,
    "buffering_duration_ms": 800,

    "current_bitrate": 2500000,
    "avg_bitrate": 2400000,
    "peak_bitrate": 3200000,
    "bitrate_switches": 1,
    "current_resolution": "720p",
    "resolution_switches": 0,

    "dropped_frames": 2,
    "total_frames": 750,

    "connection_type": "wifi",
    "effective_bandwidth": 11.2,
    "rtt_ms": 48,

    "error_count": 0,
    "error_types": []
  }
}

Frontend implementation notes:

• buffering_count / buffering_duration_ms: cumulative for the session, not per-heartbeat
• dropped_frames / total_frames: from HTMLVideoElement.getVideoPlaybackQuality()
• current_bitrate: from the player's ABR controller (HLS.js: hls.bandwidthEstimate, dash.js: getQualityFor('video'))
• avg_bitrate: running average computed by frontend
• resolution_switches: increment on quality_change events

buffering_start

Fired when the video stalls (buffer underrun). Creates a row in media_buffering_events.

{
  "event": "buffering_start",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:23.100Z",
  "data": {
    "position_seconds": 23.1,
    "bitrate_before": 2500000,
    "resolution_before": "720p",
    "cdn_response_time_ms": 1200,
    "connection_type": "wifi",
    "effective_bandwidth": 3.2
  }
}

buffering_end

Fired when playback resumes after stall. Updates the corresponding media_buffering_events row.

{
  "event": "buffering_end",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:25.900Z",
  "data": {
    "position_seconds": 23.1,
    "duration_ms": 2800,
    "bitrate_after": 1500000,
    "resolution_after": "480p",
    "recovered": true
  }
}

quality_change

Fired when ABR (Adaptive Bitrate) switches video quality.

{
  "event": "quality_change",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:26.000Z",
  "data": {
    "position_seconds": 23.1,
    "from_bitrate": 2500000,
    "to_bitrate": 1500000,
    "from_resolution": "720p",
    "to_resolution": "480p",
    "reason": "bandwidth_decrease"
  }
}

Reasons: bandwidth_decrease, bandwidth_increase, buffer_low, user_manual, initial

milestone

Fired when the patient reaches 25%, 50%, 75%, or 95% of the video.

{
  "event": "milestone",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:01:30.000Z",
  "data": {
    "milestone_percent": 50,
    "position_seconds": 60.25,
    "elapsed_real_seconds": 90.0
  }
}

error

Fired when a playback error occurs.

{
  "event": "error",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:00:45.000Z",
  "data": {
    "error_code": "MEDIA_ERR_NETWORK",
    "error_message": "Failed to load video segment 5",
    "position_seconds": 45.0,
    "is_fatal": false,
    "recovery_action": "retry_segment"
  }
}

Error codes:

Code	Description
MEDIA_ERR_ABORTED	User or system aborted download
MEDIA_ERR_NETWORK	Network error during download
MEDIA_ERR_DECODE	Video decoding failed
MEDIA_ERR_SRC_NOT_SUPPORTED	Format not supported on device
HTTP_403	CDN rejected request (expired token, geo block)
HTTP_404	Video file not found
TIMEOUT	Request timed out
DRM_ERROR	DRM license acquisition failed

session_end

Fired when the session ends (user closes, video completes, or session times out).

{
  "event": "session_end",
  "session_id": "uuid",
  "timestamp": "2026-02-17T10:02:30.000Z",
  "data": {
    "final_position_seconds": 120.5,
    "watched_duration_seconds": 118.0,
    "completion_percent": 97.9,
    "status": "completed",

    "buffering_count": 2,
    "buffering_duration_ms": 3600,
    "bitrate_switches": 3,
    "resolution_switches": 2,
    "dropped_frames": 5,
    "total_frames": 3615,
    "error_count": 1,
    "error_types": ["MEDIA_ERR_NETWORK"]
  }
}

Status values: completed (≥95% watched), abandoned (user left), error (fatal error ended session)

Frontend Integration

Minimal Implementation

// Initialize on video mount
const sessionId = crypto.randomUUID();

// Session start — measure loading performance
const clickTime = performance.now();

videoElement.addEventListener('canplay', () => {
  const loadTime = performance.now() - clickTime;
  const timing = performance.getEntriesByName(videoUrl)[0];

  telemetry.track({
    event: 'session_start',
    session_id: sessionId,
    media_id: hashedExerciseId,
    media_type: 'video',
    data: {
      total_duration_seconds: videoElement.duration,
      ttfb_ms: timing?.responseStart - timing?.requestStart,
      video_load_time_ms: Math.round(loadTime),
      cdn_response_time_ms: timing?.responseStart - timing?.connectEnd,
      connection_type: navigator.connection?.effectiveType ?? 'unknown',
      effective_bandwidth: navigator.connection?.downlink,
      rtt_ms: navigator.connection?.rtt,
    }
  });
});

// Heartbeat — every 10 seconds during playback
const heartbeatInterval = setInterval(() => {
  if (videoElement.paused) return;
  const quality = videoElement.getVideoPlaybackQuality?.();

  telemetry.track({
    event: 'heartbeat',
    session_id: sessionId,
    data: {
      position_seconds: videoElement.currentTime,
      watched_duration_seconds: totalWatchedTime,
      completion_percent: (videoElement.currentTime / videoElement.duration) * 100,
      buffering_count: bufferCount,
      buffering_duration_ms: totalBufferMs,
      dropped_frames: quality?.droppedVideoFrames ?? 0,
      total_frames: quality?.totalVideoFrames ?? 0,
      current_bitrate: hlsPlayer?.bandwidthEstimate,
      // ... other metrics
    }
  });
}, 10_000);

// Buffering detection
videoElement.addEventListener('waiting', () => {
  bufferStartTime = performance.now();
  telemetry.track({ event: 'buffering_start', session_id: sessionId, data: { ... } });
});

videoElement.addEventListener('playing', () => {
  if (bufferStartTime) {
    const duration = performance.now() - bufferStartTime;
    bufferCount++;
    totalBufferMs += duration;
    telemetry.track({ event: 'buffering_end', session_id: sessionId, data: { duration_ms: duration, ... } });
    bufferStartTime = null;
  }
});

Browser API Compatibility

Metric	API	Support
TTFB	PerformanceResourceTiming	All modern browsers
Connection type	navigator.connection	Chrome, Edge, Opera (NOT Safari/Firefox)
Dropped frames	getVideoPlaybackQuality()	Chrome, Edge, Firefox (NOT Safari)
Bandwidth estimate	navigator.connection.downlink	Chrome, Edge, Opera
RTT	navigator.connection.rtt	Chrome, Edge, Opera

Fallback strategy: When an API is unavailable, send null. The backend handles nullable fields. Partial data is better than no data.

Troubleshooting Queries

These ClickHouse queries answer common support questions.

"Why is this patient's video not loading?"

-- Recent sessions for a specific actor
SELECT
    session_id, media_id, started_at,
    ttfb_ms, video_load_time_ms,
    buffering_count, buffering_duration_ms,
    current_bitrate, current_resolution,
    connection_type, effective_bandwidth, rtt_ms,
    error_count, error_types,
    device_type, browser_family, os_family,
    country_code
FROM media_sessions FINAL
WHERE actor_hash = 'sha256-of-patient-id'
  AND started_at > now() - INTERVAL 7 DAY
ORDER BY started_at DESC
LIMIT 20;

"Which country has the worst video performance?"

SELECT
    country_code,
    count() AS sessions,
    avg(ttfb_ms) AS avg_ttfb,
    avg(video_load_time_ms) AS avg_load_time,
    avg(buffering_count) AS avg_buffers,
    avg(buffering_duration_ms) AS avg_buffer_ms,
    avg(avg_bitrate) AS avg_bitrate,
    countIf(status = 'completed') / count() AS completion_rate,
    countIf(error_count > 0) / count() AS error_rate
FROM media_sessions FINAL
WHERE started_at > now() - INTERVAL 30 DAY
GROUP BY country_code
ORDER BY avg_ttfb DESC;

"Which exercises buffer the most?"

SELECT
    media_id,
    count() AS buffer_events,
    avg(duration_ms) AS avg_buffer_duration,
    quantile(0.95)(duration_ms) AS p95_buffer_duration,
    countIf(recovered = 0) AS abandoned_buffers
FROM media_buffering_events
WHERE timestamp > now() - INTERVAL 7 DAY
GROUP BY media_id
ORDER BY buffer_events DESC
LIMIT 20;

"Is the problem device-specific?"

SELECT
    device_type, browser_family, os_family,
    count() AS sessions,
    avg(ttfb_ms) AS avg_ttfb,
    avg(buffering_duration_ms) AS avg_buffer_ms,
    avg(dropped_frames) AS avg_dropped,
    countIf(error_count > 0) / count() AS error_rate
FROM media_sessions FINAL
WHERE started_at > now() - INTERVAL 7 DAY
GROUP BY device_type, browser_family, os_family
ORDER BY avg_buffer_ms DESC;

"What's happening right now?" (Real-time)

SELECT
    count() AS active_sessions,
    avg(buffering_count) AS avg_buffers,
    countIf(error_count > 0) AS sessions_with_errors,
    avg(current_bitrate) / 1000000 AS avg_bitrate_mbps
FROM media_sessions FINAL
WHERE status = 'active'
  AND last_heartbeat_at > now() - INTERVAL 2 MINUTE;

Video Health Dashboard

The metrics feed a dashboard with these panels:

Overview Metrics

• Active sessions now — real-time count
• Avg TTFB — last 24h, with trend
• Avg Load Time — click to first frame
• Buffer Rate — % of sessions with ≥1 buffer event
• Error Rate — % of sessions with errors
• Completion Rate — % of sessions reaching 95%+

Breakdown Charts

• Performance by country — map/table with TTFB, buffering, bitrate per country
• Performance by device — mobile vs desktop vs tablet
• Performance by connection — wifi vs 4g vs 3g
• Performance by browser — Chrome vs Safari vs Firefox
• Buffering heatmap — at which second in the video do buffers cluster?

Alerts (Automation Triggers)

• TTFB > 3000ms for >20% of sessions in 1h → CDN issue alert
• Error rate > 10% for a specific exercise → content issue alert
• Buffer rate spikes 3x above 7-day baseline → network degradation alert