Dedup index
Sellable product
This capability is granted by an API key scoped to the `` product (product face). See the product reference below.
Near-duplicate detection capability — a durable, org-scoped, multi-worker-COHERENT 64-bit SimHash index. One check-and-store API + a per-org Hamming threshold, so products stop each re-rolling a per-worker in-memory dedup index (which split-brains across workers, letting duplicates through). The near-dup scan is serialized per org (advisory lock) against ONE shared store so decisions are coherent across every worker/replica. I4: own Postgres store, never a product Mongo.
- Group: Content & media
- Contract:
contracts/dedup-index/v1/openapi.yaml - Public base:
https://api.vagarylabs.com(the consolidated API gateway — one host, per-brand siblingapi.<zone>) - Auth: a product API key (
vgk_…) issued from the console —Authorization: Bearer vgk_… - Product face (customer-keyed):
https://api.vagarylabs.com/product/v1/dedup/check
Endpoints
| Method | Path | Summary |
|---|---|---|
POST | /v1/dedup/check | Compute a SimHash, find near-duplicates in the org's index, and (optionally) store it |
GET | /v1/dedup/stats | Fingerprint count + effective Hamming threshold for the caller's org |
PUT | /v1/dedup/threshold | Set the org's Hamming threshold (persisted to the shared per-org row — worker-coherent) |
GET | /v1/dedup/fingerprints/{article_id} | The stored SimHash fingerprint for one article (or 404) |
GET | /health | liveness |
GET | /metrics | Prometheus |
Schemas
CheckRequest
| Field | Type | Description |
|---|---|---|
organization_id | string | product-tier tenant (I6) |
article_id | string | identifier (platform canonical_url) |
text | string | the content to fingerprint |
threshold | integer | per-request Hamming override; omit to use the org threshold |
store | boolean | persist the fingerprint (false = check-only) |
CheckResult
| Field | Type | Description |
|---|---|---|
fingerprint | string | unsigned 64-bit SimHash as a decimal string (exceeds JS safe-int / signed BSON) |
is_duplicate | boolean | |
near_duplicates | array |
Stats
| Field | Type | Description |
|---|---|---|
fingerprint_count | integer | |
hamming_threshold | integer |
ThresholdRequest
| Field | Type | Description |
|---|---|---|
organization_id | string | product-tier tenant (I6) |
threshold | integer |
ThresholdResult
| Field | Type | Description |
|---|---|---|
hamming_threshold | integer |
Fingerprint
| Field | Type | Description |
|---|---|---|
article_id | string | |
fingerprint | string |
Error
| Field | Type | Description |
|---|---|---|
error | string |
Generated by scripts/gen-capability-docs.py from contracts/dedup-index/v1/openapi.yaml — the contract IS the source of truth; edit the contract, not this page.