CoreVideo

Introduction

CoreVideo integrates the Zoom Meeting SDK into OBS Studio with no screen capture or virtual camera required. It receives raw I420 video and 48 kHz PCM audio from any meeting participant, performs optional hardware-accelerated color conversion, and pushes frames into OBS as native sources.

The ZoomObsEngine child process hosts all SDK access in isolation, communicating with the plugin over JSON IPC (named pipes on Windows, Unix domain sockets on macOS/Linux) with frame data flowing through named shared memory. The plugin contains no Zoom SDK linkage.

Joining a meeting is driven by the built-in Zoom Control dock (or the TCP/OSC APIs). Once in a meeting, sources can follow a fixed participant, the active speaker, a ZoomISO-style spotlight slot, or the active screen share — and switch automatically with a configurable failover participant if the primary leaves.

A ZoomReconnectManager automatically re-joins after engine crashes, network drops, or unexpected disconnects, with exponential back-off and a cancel-at-any-time UI in the dock.

For Zoom App Marketplace compliance and attributed joins, ZoomOAuthManager uses the CoreVideo HTTPS broker to run Zoom Public Client OAuth with PKCE and receive a short-lived broker token through the bundled corevideo://oauth/callback helper. Before each join, the plugin asks the broker for a short-lived Meeting SDK JWT; the broker validates the signed-in Zoom access token before minting the SDK JWT. The plugin also exposes TCP and OSC control APIs for external production automation, including source assignment, meeting control, and ISO recording.

Features

Requirements

Architecture: System Overview

All Zoom SDK access lives exclusively in ZoomObsEngine. The plugin communicates with it through ZoomEngineClient — a singleton that launches the engine process, owns the IPC channels, and distributes roster events and frame notifications to registered sources. The plugin contains no Zoom SDK headers or linkage.

Meeting joining is handled by the Zoom Control dock (ZoomDock) or the TCP/OSC APIs. On unexpected disconnects the ZoomReconnectManager automatically re-joins with exponential back-off. Sources each hold an AssignmentMode that controls which feed they subscribe to: a fixed participant, the active speaker, a spotlight slot, or the active screen share.

Architecture: Plugin Components

All Zoom SDK access lives in ZoomObsEngine. The plugin-side hierarchy centers on ZoomEngineClient (IPC singleton) and ZoomSource (reads frames from ShmRegion). New additions: ZoomDock (join UI), ZoomReconnectManager (auto-reconnect), and HwVideoPipeline (hardware I420→NV12).

Architecture: ZoomEngineClient

ZoomEngineClient is the plugin-side singleton that manages the entire engine process lifecycle. It launches ZoomObsEngine as a child process, connects the two IPC channels, runs a dedicated reader thread for incoming events, and dispatches per-source frame/audio callbacks. All Zoom SDK state (auth, meeting, roster, active speaker) is owned by the engine; the client tracks it locally by processing the event stream.

Lifecycle

Roster, Active Speaker, and Spotlight

The engine sends a participants event whenever the roster changes and an active_speaker event when the speaking participant changes. ZoomEngineClient updates its internal roster and active-speaker ID from these events, then fires all registered RosterCallback functions. ZoomSource instances react based on their AssignmentMode: active-speaker sources run the debounce algorithm; spotlight sources send a subscribe_spotlight command when spotlight slots change.

Deferred Join (Pending Until Authenticated)

Calls to join() before the engine has authenticated store the meeting credentials in m_join_pending / m_pending_* fields. When the auth_ok event arrives, send_join_locked() is called automatically so operators can trigger join at any time.

Monitor Thread

A dedicated monitor_loop() thread watches for engine health issues (stalled reader, unexpected process exit) and sets the state to Failed, which triggers ZoomReconnectManager if recovery is configured.

Architecture: IPC Engine

ZoomObsEngine runs as a separate process on all platforms. Control messages use newline-delimited JSON over platform IPC channels. Frame data flows through named shared memory to avoid copying large buffers. ZoomEngineClient in the plugin abstracts all IPC details.

Architecture: Video Pipeline

Architecture: Audio Pipeline

Audio processing now lives entirely in the engine process. The engine writes PCM chunks to a per-source shared memory region and sends an audio event. ZoomSource reads the chunk from m_audio_shm and pushes it to OBS. Stereo sources use an internal m_stereo_buf to interleave channels before output.

Architecture: Screen Share Pipeline

Screen share handling now lives entirely inside ZoomObsEngine. The engine's IMeetingShareCtrlEvent handler detects share-start events, attaches an IZoomSDKRenderer to the share source, and writes I420 frames to a dedicated shared memory region. The plugin receives frame events on the share source UUID and forwards them to OBS exactly like participant video.

Flow: Authentication

Published CoreVideo builds do not ship Meeting SDK secrets in the OBS plugin. After Zoom sign-in, the plugin asks the CoreVideo broker for a short-lived Meeting SDK JWT. The broker validates the signed-in Zoom access token and signs the SDK JWT server-side. The engine is not started on module load; it is launched the first time a join is requested.

Flow: Zoom OAuth PKCE

ZoomOAuthManager implements broker-backed OAuth 2.0 with PKCE (Proof Key for Code Exchange, S256) for user-level attributed meeting joins and Zoom App Marketplace compliance. The desktop plugin opens the broker start URL, receives a short-lived broker token through corevideo://oauth/callback, and never stores a Zoom app secret.

Setup (one-time per machine)

Authorization Flow

Meeting SDK JWT Fetch (before each join)

Security Notes

Flow: Meeting Join & Capture

Joining is centralized in ZoomDock (or the TCP/OSC control APIs) — not in individual sources. Sources subscribe to the already-joined meeting.

Assignment Modes

Each ZoomSource has an AssignmentMode property that controls which feed it subscribes to. This is the ZoomISO 3.0 model: sources are reusable roles rather than fixed participant bindings.

Failover Participant

In Participant mode, a failover_participant_id (0 = disabled) can be configured alongside the primary participant. When the engine sends a participants event showing the primary ID has left, ZoomSource automatically re-subscribes to the failover ID. When the primary rejoins, it switches back.

Spotlight Tracking

In Spotlight Slot mode, the roster callback checks ParticipantInfo::spotlight_index on every roster update. The source sends a subscribe_spotlight(uuid, slot) to the engine, which resolves the participant in that slot and sets up the SDK renderer. This mirrors the ZoomISO "Spotlight 1/2/3" output model.

Active Speaker Mode

When Follow active speaker is enabled on a Zoom Participant source, the plugin continuously monitors who is speaking and switches the video subscription (and optionally the isolated audio feed) to the new speaker. A two-timer debounce prevents rapid camera cuts caused by brief interruptions.

The current build also includes a central Active Speaker Director in the Zoom Control dock and a dedicated CoreVideo Active Speaker OBS source. The director tracks the raw Zoom speaker, candidate speaker, directed speaker, last directed speaker, and manual take/release state. The dedicated source follows the directed speaker and uses a two-slot handoff: the current participant stays visible while the next participant warms on a hidden slot, then the source cuts only after a valid frame is available.

Debounce Algorithm

Every time ZoomEngineClient dispatches a roster callback (triggered by an active_speaker or participants IPC event), the source runs on_active_speaker_changed(). Two independent timers must both be satisfied before a switch is committed:

• Sensitivity guard — the candidate speaker must hold the floor continuously for at least speaker_sensitivity_ms (default 500 ms). A new candidate resets this clock.
• Hold guard — after any switch, no further switch may occur for at least speaker_hold_ms (default 2 000 ms), regardless of who speaks.

The actual delay is max(hold_remain, sense_remain). If the delay is zero the switch fires immediately on the calling thread; otherwise a detached background thread sleeps for the delay and then posts a UI task via obs_queue_task(OBS_TASK_UI, …) to re-evaluate on the OBS UI thread.

Switch Sequence

Timing Parameters

The Zoom Control dock exposes these values through the Active Speaker Director controls. Operators can also manually take a participant to air and release that supersede when automatic direction should resume.

Safety Mechanisms

• Liveness flag — speaker_alive is a shared_ptr<atomic<bool>> captured by value in every in-flight lambda. When the source is destroyed, speaker_alive → false is stored before deletion. The deferred UI task checks the flag before touching ZoomSource, preventing use-after-free.
• Supersede logic — if a new candidate arrives while one is already pending, pending_speaker_id and candidate_since are updated to the newer speaker, restarting the sensitivity clock. Stale callbacks from the previous candidate see spk != pending_speaker_id and bail out silently.
• Final verification — immediately before calling do_speaker_switch(), the code re-checks ZoomParticipants::active_speaker_id() == spk so a switch never fires for a speaker who stopped talking during the hold period.
• UI-thread commitment — all state mutations (participant_id, last_switch_time, video subscription) happen on the OBS UI thread via obs_queue_task, avoiding data races with the properties panel.

Audio Isolation Interaction

When Isolate Audio is also enabled, every speaker switch triggers a new subscribe command to the engine with the updated participant ID and the isolate_audio flag, so the engine adjusts the audio feed to follow the same participant as the video.

UI Behaviour

Enabling active speaker mode in the Properties panel automatically disables the Participant dropdown (not relevant while following the speaker) and enables the Sensitivity and Hold sliders. The participant list still shows live roster entries with talking (●) and video ([video]) indicators, and updates when the Refresh button is pressed.

Director TCP Controls

Auto-Reconnect

ZoomReconnectManager automatically re-joins the meeting after engine crashes, network drops, SDK errors, or host-ended meetings. It is configurable per-session and shows a live countdown in the Zoom Control dock.

Reconnect Policy

Recovery Triggers

Cancellation Safety

A monotonically increasing generation counter ensures stale timer wakeups never fire a retry that was already cancelled or superseded. Calling cancel() (via the dock or API) bumps the generation; the timer thread sees the mismatch and discards the pending retry.

Explicit leave() calls set the m_user_leaving flag and call clear_session(), preventing any recovery attempt after a deliberate disconnect.

Zoom Control Dock

ZoomDock is a dockable Qt widget added to OBS on plugin load. Access it via Tools → Zoom Control or drag it to any dock position in OBS. It is the primary UI for meeting management.

CoreVideo runs as normal OBS UI: the Zoom Control dock manages meeting state and raw media while Zoom Participant sources render inside OBS scenes.

Meeting Control Bar

Session Persistence

The dock saves last_meeting_id, last_display_name, and last_was_webinar to the plugin settings file on each successful join. These values are restored and pre-filled in the dock on next OBS launch.

Recovery Panel

Shown automatically when MeetingState::Recovering. Displays:

• Recovery status message with attempt count
• Live countdown timer to next retry (refreshes every second)
• Cancel Recovery button — calls ZoomReconnectManager::cancel()

Routing Section

The dock no longer embeds the full output table. Its routing section opens the dedicated Zoom Output Manager dock, which is the primary assignment surface for saving, loading, deleting profiles, and reviewing requested resolution, observed signal, frame rate, audio mode, and isolated-audio state.

Hardware Video Acceleration

When built with -DCOREVIDEO_HW_ACCEL=ON, each ZoomSource owns a HwVideoPipeline that converts incoming I420 frames to NV12 using FFmpeg hardware acceleration. This offloads color-space conversion from the CPU to the GPU/media engine.

A per-source hw_accel_override property (−1 = use global setting) allows different acceleration modes per source. HwVideoPipeline builds a dynamic FFmpeg filter graph on the first frame and rebuilds it automatically on resolution change. On any error it falls back to the CPU path and sets a m_broken flag to avoid repeated failures.

TCP Control API

Listens on 127.0.0.1:19870 (configurable). Each request and response is a single-line compact JSON terminated by \n. If a token is configured, every request must include "token":"<value>". Token comparison uses constant-time equality to prevent timing attacks.

Participant object

{ "id": 123, "name": "Alice", "has_video": true, "is_talking": false, "is_muted": false }

Output object

{ "source": "Zoom Participant 1", "participant_id": 123, "active_speaker": false,
  "assignment_mode": "participant", "video_resolution": "1080p",
  "observed_width": 1920, "observed_height": 1080, "observed_fps": 29.97,
  "isolate_audio": true, "audio_channels": "stereo" }

Meeting states

OSC Control API

Listens on 127.0.0.1:19871 UDP (configurable). Accepts standard OSC 1.0 datagrams. Replies are sent back to the originating host/port. Supported argument types: i (int32), f (float32), s (string), T / F (true/false booleans).

Incoming Addresses

Reply Addresses (sent by plugin)

Output Profiles

The ZoomOutputProfile namespace persists named output configurations as JSON files under the OBS plugin config directory:

obs-studio/plugin_config/obs-zoom-plugin/profiles/<name>.json

Each profile stores all current output assignments (source name, participant ID, active speaker flag, audio isolation, audio mode). Use the Zoom Output Manager dock (or OBS → Tools to focus it) to save, load, and delete profiles interactively, or call the ZoomOutputProfile API directly from code.

The Output Manager mirrors the dock assignment controls and remains useful for named profile save/load workflows.

Profile JSON format

[
  {
    "source": "Zoom Participant 1",
    "display_name": "Camera A",
    "participant_id": 123,
    "active_speaker": false,
    "isolate_audio": true,
    "audio_channels": "stereo"
  }
]

IPC Protocol Reference

All messages are UTF-8 JSON terminated by \n. Frame data is transferred through named shared memory (prefix ZoomObsPlugin_).

Plugin → Engine

Engine → Plugin

Installation

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DZOOM_SDK_DIR=third_party/zoom-sdk \
  -DCMAKE_PREFIX_PATH="/path/to/obs-studio;/path/to/Qt6"

cmake --build build --config Release

cmake --install build --prefix "/path/to/obs-studio"

Configuration

Settings Dialog (Tools → Zoom Plugin Settings)

SDK Credentials

OAuth Settings

Control & Network

Meeting Control Dock

Zoom Participant Source Properties

Each Zoom Participant source keeps its own assignment mode, audio routing, requested resolution, and failover behavior.

Zoom Participant Audio Source

Zoom Interpretation Audio Source

Building from Source

Directory Layout

CoreVideo/
├── CMakeLists.txt
├── buildspec/
│   ├── macos.cmake
│   └── windows.cmake
├── cmake/
│   └── CoreVideoOAuthCallback-Info.plist.in  # macOS OAuth helper bundle plist
├── data/locale/en-US.ini
├── docs/                                     # GitHub Pages documentation
│   ├── index.html
│   ├── ZOOM_MARKETPLACE_OAUTH.md             # OAuth PKCE setup guide
│   └── policies/                             # Security & privacy policy documents
├── engine/src/                               # ZoomObsEngine process (owns ALL SDK access)
│   ├── main.cpp                              # IPC loop, SDK auth/join, roster/spotlight tracking
│   ├── engine-video.cpp/h                    # IZoomSDKRenderer → named shared memory (I420)
│   └── engine-audio.cpp/h                    # SDK audio → named shared memory (PCM)
└── src/                                      # OBS plugin (no SDK linkage)
    ├── plugin-main.cpp                       # Module load/unload, dock, Tools menu, SIGPIPE
    ├── zoom-source.*                         # Participant source (ShmRegion, AssignmentMode,
    │                                         #   HwVideoPipeline, failover, hotkeys, placeholder)
    ├── zoom-engine-client.*                  # Singleton: IPC, engine launch, roster, callbacks,
    │                                         #   spotlight/screenshare subscribe, monitor thread
    ├── zoom-oauth.*                          # ZoomOAuthManager: broker PKCE, SDK JWT fetch, token storage
    ├── oauth-callback-helper.cpp             # Windows: CoreVideoOAuthCallback.exe entry point
    ├── oauth-callback-helper-macos.mm        # macOS: CoreVideoOAuthCallback.app entry point
    ├── zoom-dock.*                           # Qt dockable panel: CvStatusDot, CvBanner,
    │                                         #   token-type selector, recovery countdown
    ├── zoom-reconnect.*                      # Auto-reconnect manager (exponential back-off)
    ├── zoom-types.h                          # Shared enums + ZoomJoinAuthTokens struct
    ├── cv-style.h                            # CoreVideo Qt stylesheet (dark theme, button roles)
    ├── cv-widgets.*                          # CvStatusDot (animated dot), CvBanner (notice strip)
    ├── hw-video-pipeline.*                   # FFmpeg I420→NV12 (CUDA/VAAPI/VideoToolbox/QSV)
    ├── zoom-audio-delegate.*                 # Mixed / isolated SDK audio → OBS
    ├── zoom-audio-router.*                   # Central SDK audio fan-out dispatcher
    ├── zoom-auth.*                           # SDK JWT auth + observable auth state
    ├── zoom-meeting.*                        # Meeting state machine
    ├── zoom-participants.*                   # Roster, active speaker, spotlight callbacks
    ├── zoom-participant-audio-source.*       # Per-participant audio OBS source
    ├── zoom-interpretation-audio-source.*    # Language interpretation channel OBS source
    ├── zoom-video-delegate.*                 # Video frames, resolution, loss mode, preview
    ├── zoom-share-delegate.*                 # Screen share frames → OBS source
    ├── zoom-output-manager.*                 # Source registry + runtime reconfiguration
    ├── zoom-output-profile.*                 # Named JSON profile persistence
    ├── zoom-output-dialog.*                  # Qt Output Manager dock widget
    ├── zoom-control-server.*                 # TCP JSON control API (port 19870) + oauth_callback
    ├── zoom-osc-server.*                     # UDP OSC control API (port 19871)
    ├── zoom-settings.*                       # Broker URL, OAuth tokens, local ports + reconnect persistence
    ├── zoom-settings-dialog.*                # Qt Settings dialog (Zoom sign-in + local settings)
    ├── zoom-credentials.h.in                 # Embedded SDK credentials (CMake-generated)
    ├── obs-zoom-version.h.in                 # Plugin version string (CMake-generated)
    ├── engine-ipc.h                          # IPC constants + cross-platform pipe/socket helpers
    └── obs-utils.*                           # OBS helper functions

Platform Matrix

Auto ISO Recording

CoreVideo can record assigned participant media as separate ISO files while optionally starting the main OBS program recording. ISO recording is owned by the OBS plugin process: the engine remains minimal and only publishes raw I420 video and PCM audio through the existing IPC and shared-memory path.

OBS ISO Recorder Panel

Open Tools → Zoom ISO Recorder to manage ISO recording from a separate OBS dock. The panel lets operators choose an output folder, set or test the FFmpeg executable, choose whether to also control OBS program recording, start/stop ISO recording, and monitor active ISO sessions with source, participant, resolution, frame counts, audio chunk counts, and output file paths.

The panel uses the same ZoomIsoRecorder backend as the TCP and OSC APIs and persists its folder, FFmpeg path, and program-recording toggle in OBS global settings.

Assignments resolve the participant IDs. Each assigned Zoom source can feed an FFmpeg writer for video/audio ISO files while OBS continues to own the program output.

Start Recording

{
  "cmd": "iso_recording_start",
  "output_dir": "C:/Recordings/CoreVideo",
  "ffmpeg_path": "ffmpeg",
  "record_program": true
}

Check Status

{ "cmd": "iso_recording_status" }

Stop Recording

{ "cmd": "iso_recording_stop" }

OSC Controls

See the Core Plugin Functionality guide for complete TCP examples, OSC examples, troubleshooting notes, and the plugin media pipeline diagrams.

Modern UI (Phase 1)

CoreVideo applies its own scoped Qt stylesheet and custom widgets to the plugin UI, giving it a consistent dark-theme appearance without interfering with OBS's global QApplication stylesheet.

CoreVideo Stylesheet (cv-style.h)

Applied widget-level via cv_stylesheet(). Covers group boxes, buttons (with role variants), line edits, combo boxes, tables, and scroll bars.

CvStatusDot

A QWidget subclass that draws a coloured status indicator using QPainter rather than a Unicode bullet character. In transitional states (Joining, Leaving, Recovering) a QTimer drives a sinusoidal pulse glow animation.

CvBanner

A compact QFrame-derived notice strip. Supports three kinds: Info (blue), Warning (amber), Error (red). An optional underlined action button on the right emits actionClicked(). Used in ZoomDock as a first-run prompt to configure SDK credentials.