Eval harness

Sellable product

This capability is granted by an API key scoped to the `` product (product face). See the product reference below.

Evaluation & scoring as a shared capability. POST /v1/eval grades a single candidate synchronously (deterministic assertions and/or an LLM rubric — no store). /v1/datasets stores named org-scoped eval-case sets; /v1/runs batch-evaluates a dataset asynchronously (poll for the pass-rate). The deterministic assertion grader (contains/regex/equals/json-valid/json-path/json-schema/cosine-similarity/latency-cost budgets — the promptfoo assertion taxonomy, reimplemented in Python) is the authoritative gating verdict; the LLM-rubric judge is advisory (a live-LLM grader is never on a must-pass CI path — it flakes).

• Group: Voice & AI
• Contract: contracts/eval-harness/v1/openapi.yaml
• Public base: https://api.vagarylabs.com (the consolidated API gateway — one host, per-brand sibling api.<zone>)
• Auth: a product API key (vgk_…) issued from the console — Authorization: Bearer vgk_…
• Product face (customer-keyed):
  • https://api.vagarylabs.com/product/v1/eval/run

Endpoints

Method	Path	Summary
POST	/v1/eval	Grade a single candidate against deterministic assertions and/or an LLM rubric (synchronous, no store)
POST	/v1/simulate	Run a LoopTalk actor↔adversary conversation and score the transcript (synchronous, no store)
POST	/v1/datasets	Create/replace a named org-scoped set of eval cases
GET	/v1/datasets/{dataset_id}	Fetch a stored dataset (org-scoped)
POST	/v1/runs	Start an asynchronous batch eval over a dataset
GET	/v1/runs/{run_id}	Fetch a run's status, aggregate pass-rate, and per-case results (org-scoped)
POST	/v1/qa/calls	Automated QA grading of a REAL production call — the pre-release→production extension
GET	/v1/qa/records/{record_id}	One graded QA record
POST	/v1/qa/records/{record_id}/to-case	Promote a graded PRODUCTION call into a permanent regression CASE — the closed loop
GET	/v1/qa/trend	Per-COHORT QA trend over a window — turns per-call scores into a signal
GET	/health	liveness + store/judge readiness
GET	/metrics	Prometheus exposition (fleet observability SDK — fail-open)

Schemas

Assertion

A deterministic grading assertion (promptfoo-adapted taxonomy). type selects the check; value is its expected argument; optional weight (score weighting), threshold (cosine/latency/cost), path (json-path) and schema (json-schema) refine it.

Field	Type	Description
type	string
value	object	expected argument (string
weight	number	weight of this assertion in the aggregate score
threshold	number	cosine-similarity min / latency (ms) max / cost (usd) max
path	string	json-path expression ($.a.b[0]) for the json-path assertion
schema	object	JSON Schema for the json-schema assertion

EvalRequest

Field	Type	Description
organization_id	string	product-tier end-customer tenant (I6); optional in v1 (accounting/label — no store on /v1/eval).
candidate	string	the output under evaluation (an LLM response or a call transcript)
reference	string	optional known-good answer
assertions	array	deterministic assertions — the authoritative gating verdict when present.
rubric	string	optional natural-language grading rubric; when set the LLM-rubric judge (via provider-gateway) is run and returned as judge (advisory when assertions are also present, authoritative when alone).
context	object	optional eval-call context for budget assertions (e.g. {latency_ms, cost_usd})
judge_model	string	override the provider-gateway model for the rubric judge

PerAssertion

Field	Type	Description
type	string
passed	boolean
weight	number
score	number	1.0/0.0 for boolean checks; the metric for cosine
message	string

JudgeResult

Field	Type	Description
score	number	judge confidence in [0,1]
verdict	string
rationale	string
model	string
outcome	string	error = judge call/parse failed (fail-closed)

EvalResponse

Field	Type	Description
score	number	weighted pass-fraction of the deterministic assertions
verdict	string
passed	integer
total	integer
per_assertion	array
judge	object	present only when a rubric was supplied (advisory unless assertions were absent).

Participant

a LoopTalk conversation participant. The ACTOR (the customer flow under test) may be a SCRIPTED persona (deterministic script, no LLM) or a persona-driven LLM; the ADVERSARY is a persona-driven LLM. A participant needs at least a persona or a script.

Field	Type	Description
persona	string	system prompt driving this participant's LLM turns
opening	string	verbatim first utterance (skips the LLM for this party's turn 0)
script	array	deterministic utterances in order (the customer-flow-as-a-script actor); the sim ends when exhausted
model	string	override the provider-gateway model for this participant's turns

SimulateRequest

Field	Type	Description
organization_id	string	product-tier end-customer tenant (I6); optional in v1 (accounting/label — no store on /v1/simulate).
actor	object
adversary	object
max_turns	integer	total utterances to run; clamped to [1, EVAL_HARNESS_MAX_SIM_TURNS] (default cap 20) to bound spend.
first	string	who opens the conversation
assertions	array	deterministic assertions over the rendered transcript — the gating verdict when present.
rubric	string	optional LLM-rubric grading the transcript (advisory unless assertions absent)
reference	string	optional known-good transcript passed to the rubric judge
context	object	optional context for budget assertions (e.g. {latency_ms, cost_usd})
judge_model	string	override the provider-gateway model for the rubric judge

TranscriptTurn

Field	Type	Description
turn	integer	0-based utterance index
role	string
content	string

---

Generated by scripts/gen-capability-docs.py from contracts/eval-harness/v1/openapi.yaml — the contract IS the source of truth; edit the contract, not this page.