Realtime media

Sellable product

This capability is granted by an API key scoped to the `` product (product face). See the product reference below.

Real-time conversational media primitives as a shared capability. v1 ships SEMANTIC TURN-DETECTION: given a window of the caller's recent speech (16 kHz mono PCM16), it returns the probability the speaker has finished their turn — a learned semantic endpointer that replaces naive silence/energy VAD (which cuts users off mid-sentence). Stateless + synchronous (~12 ms CPU inference); the caller (voice-gateway / aakhara) feeds a rolling audio window and gates barge-in / response-start on the result. Later waves add VAD de-stub, backchannel/filler cues, and echo/PLC — same capability.

• Group: Voice & AI
• Contract: contracts/realtime-media/v1/openapi.yaml
• Public base: https://api.vagarylabs.com (the consolidated API gateway — one host, per-brand sibling api.<zone>)
• Auth: a product API key (vgk_…) issued from the console — Authorization: Bearer vgk_…
• Product face (customer-keyed):
  • https://api.vagarylabs.com/product/v1/realtime/vad

Endpoints

Method	Path	Summary
POST	/v1/turn/detect	Semantic turn-detection over a window of recent caller speech (Smart-Turn v3)
POST	/v1/turn/detect_semantic	Semantic (text) turn-detection over the running transcript (livekit EOU transformer)
POST	/v1/vad/analyze	Acoustic voice-activity over a window of recent caller audio (pipecat-harvested Silero VAD)
POST	/v1/denoise	Real-time speech noise-suppression over a window of caller audio (DTLN-harvested denoiser)
POST	/v1/echo/cancel	Acoustic echo cancellation over aligned near-end (mic) + far-end (reference) PCM16 audio
POST	/v1/plc/conceal	Packet-loss concealment — reconstruct a PCM16 stream, synthesizing the lost frames
POST	/v1/ambience/mix	Mix a low background-ambience bed under OUTBOUND agent speech (PCM16). Flag-gated
GET	/health	liveness + model-loaded state
GET	/metrics	Prometheus exposition (fleet observability SDK — fail-open)

Schemas

TurnDetectRequest

Field	Type	Description
audio_base64	string	base64 of raw mono PCM16 (signed 16-bit little-endian) at sample_rate. Most-recent speech.
sample_rate	integer	input sample rate in Hz. Resampled to 16000 if different (linear).
organization_id	string	product-tier end-customer tenant (I6); optional in v1 (accounting/label only, no store).

TurnDetectResponse

Field	Type	Description
is_complete	boolean	probability > 0.5 — the speaker has likely finished their turn
probability	number	sigmoid turn-completion probability in [0,1]
inference_ms	number	wall-clock inference latency
samples	integer	input sample count (post-resample

SemanticTurnDetectRequest

Field	Type	Description
chat_ctx	array	recent conversation turns (only user/assistant roles are scored; the last MAX_TURNS are kept). Adjacent same-role turns are combined to match the model's training data.
organization_id	string	product-tier end-customer tenant (I6); optional in v1 (accounting/label only, no store).

SemanticTurnDetectResponse

Field	Type	Description
eou_probability	number	end-of-utterance probability in [0,1] (higher ⇒ turn complete)
is_complete	boolean	eou_probability > threshold — the speaker has likely finished their turn
inference_ms	number	wall-clock inference latency
turns	integer	number of user/assistant turns scored (post-filter
backend	string	the turn-detection backend that produced this result

VADAnalyzeRequest

Field	Type	Description
audio_base64	string	base64 of raw mono PCM16 (signed 16-bit little-endian). Recent caller audio window.
sample_rate	integer	input sample rate in Hz. 8000 and 16000 are fed to Silero directly; any other rate is resampled to 16000 (linear) first. Must contain at least one VAD frame (512 samples @16k / 256 @8k).
organization_id	string	product-tier end-customer tenant (I6); optional in v1 (accounting/label only, no store).

VADAnalyzeResponse

Field	Type	Description
state	string	terminal VAD hysteresis state after consuming the window.
is_speech	boolean	true when state != quiet (voice activity present)
frames_analyzed	integer	number of fixed VAD frames consumed from the window
sample_rate	integer	sample rate the window was analyzed at (post-resample)
inference_ms	number	wall-clock analyze latency

DenoiseRequest

Field	Type	Description
audio_base64	string	base64 of raw mono PCM16 (signed 16-bit little-endian) at sample_rate. Recent caller audio window.
sample_rate	integer	input sample rate in Hz. DTLN is 16 kHz-fixed: the window is resampled to 16 kHz for denoising and the cleaned audio is resampled back to this rate. Must contain at least one 512-sample denoise block at 16 kHz.
organization_id	string	product-tier end-customer tenant (I6); optional in v1 (accounting/label only, no store — never read by the handler).

DenoiseResponse

Field	Type	Description
audio_base64	string	base64 of the denoised mono PCM16 (signed 16-bit little-endian), at sample_rate (round-tripped to the input rate).
sample_rate	integer	sample rate of the returned audio (equals the input rate)
frames_processed	integer	number of 512/128 overlap-add blocks processed at 16 kHz
inference_ms	number	wall-clock denoise latency

---

Generated by scripts/gen-capability-docs.py from contracts/realtime-media/v1/openapi.yaml — the contract IS the source of truth; edit the contract, not this page.