Implement Edge Cookie identity system with KV graph and integration tests

[ChristianPavilonis]

Summary

• Implement the Edge Cookie (EC) identity system — a publisher-owned, first-party cookie-based identity mechanism that replaces the previous Synthetic ID concept. EC enables consent-gated identity generation, a KV-backed identity graph with CAS concurrency, partner sync (pixel, batch, pull), an identify endpoint, and auction bidstream decoration with partner EIDs.
• Add 7 integration test scenarios covering the full EC lifecycle: generation, consent withdrawal, identify edge cases, concurrent partner syncs, and batch sync flows.
• Rename "Synthetic ID" → "Edge Cookie" across the entire codebase, configuration, and documentation.

Changes

File / Area	Change
crates/trusted-server-core/src/ec/ (new module)	Full EC implementation: mod.rs (lifecycle orchestration), generation.rs (HMAC-SHA256), cookies.rs (cookie read/write), consent.rs (consent gating), device.rs (device signal derivation, bot gate), kv.rs + kv_types.rs (KV identity graph with CAS), partner.rs (partner registry + admin), sync_pixel.rs (pixel sync), batch_sync.rs (S2S batch sync), pull_sync.rs (background pull sync), identify.rs (identity lookup with CORS), eids.rs (EID encoding), finalize.rs (response finalization middleware), admin.rs (admin endpoints)
crates/integration-tests/	7 new EC lifecycle test scenarios in tests/common/ec.rs and tests/frameworks/scenarios.rs; updated integration.rs test runner and Viceroy config fixtures
crates/trusted-server-adapter-fastly/src/main.rs	New EC endpoint routes namespaced under /_ts/api/v1/ and /_ts/admin/; send_to_client() pattern with background pull sync dispatch
crates/trusted-server-core/src/auction/	endpoints.rs and formats.rs updated to decorate bid requests with partner EIDs from the KV identity graph (user.id, user.eids, user.consent)
crates/trusted-server-core/src/integrations/prebid.rs	Bidder deduplication fix; blank auction EC header handling
crates/js/lib/src/integrations/prebid/index.ts	JS-side Prebid integration updates for EC headers
crates/trusted-server-core/src/synthetic.rs	Deleted — replaced by the EC module
crates/trusted-server-core/src/cookies.rs	Slimmed down; EC-specific cookie logic moved to ec/cookies.rs
crates/trusted-server-core/src/settings.rs + settings_data.rs	EC configuration fields, partner config, consent config migration
crates/trusted-server-core/src/consent/	Updated consent module for EC-specific consent checks; KV consent storage
crates/trusted-server-core/src/http_util.rs	New HTTP helpers for EC endpoints (CORS, request parsing)
crates/trusted-server-core/src/proxy.rs	EC header propagation in proxy layer
crates/trusted-server-core/src/publisher.rs	Publisher config updates for EC
crates/trusted-server-core/src/constants.rs	New EC-related constants
docs/guide/ec-setup-guide.md (new)	End-to-end EC setup documentation
docs/guide/edge-cookies.md (new)	EC concept overview
docs/guide/api-reference.md	API reference updated with all EC endpoints
docs/guide/configuration.md	Configuration docs updated for EC settings
docs/guide/synthetic-ids.md	Deleted — replaced by EC docs
docs/ (various)	Documentation updates reflecting Synthetic ID → Edge Cookie rename
docs/internal/superpowers/specs/2026-03-24-ssc-technical-spec-design.md	EC technical spec updated with schema extensions, device signals, bot gate
fastly.toml	New KV store bindings for EC
trusted-server.toml	EC configuration section added

Closes

Closes #532
Closes #533
Closes #535
Closes #536
Closes #537
Closes #538
Closes #539
Closes #540
Closes #541
Closes #542
Closes #543
Closes #544
Closes #611
Closes #612

Follow-up

Follow-up: The pull sync mechanism (ec/pull_sync.rs) currently passes ec_id and client_ip as query parameters to partner endpoints. While connections are HTTPS-only, these values can appear in partner server access logs and CDN logs. A future improvement should migrate to POST body or hash/encrypt identifiers before transmission. Tracked separately.

Test plan

• cargo test --workspace
• cargo clippy --workspace --all-targets --all-features -- -D warnings
• cargo fmt --all -- --check
• JS tests: cd crates/js/lib && npx vitest run
• JS format: cd crates/js/lib && npm run format
• Docs format: cd docs && npm run format
• Other: 7 EC lifecycle integration test scenarios (full lifecycle, consent withdrawal, identify edge cases, concurrent partner syncs, batch sync)

Checklist

• Changes follow CLAUDE.md conventions
• No unwrap() in production code — use expect("should ...")
• Uses tracing macros (not println!)
• New code has tests
• No secrets or credentials committed

[aram356]

Summary

Comprehensive EC identity subsystem: HMAC-based ID generation, KV identity graph with CAS concurrency, partner sync (pixel, batch, pull), identify endpoint, and auction bidstream decoration. The architecture is well-designed — clean separation of concerns, strong input validation, and solid concurrency model. A few cleartext logging issues and a docs inconsistency need addressing before merge.

Blocking

🔧 wrench

• Cleartext EC ID logging: Full EC IDs are logged at warn!/error!/trace! levels in 5 locations across ec/mod.rs (lines 128, 211, 296), sync_pixel.rs (line 91), and pull_sync.rs (line 91). The log_id() truncation helper was introduced in this PR for exactly this purpose but is not used consistently. See inline comments for fixes.
• Stale env var in docs: TRUSTED_SERVER__EDGE_COOKIE__SECRET_KEY still appears in configuration.md (lines 40, 956) and error-reference.md (line 154). The PR renamed the TOML section from [edge_cookie] to [ec] and updated some env var references but missed these. Users following the docs will set a variable that is silently ignored, resulting in a startup failure from the placeholder passphrase. Should be TRUSTED_SERVER__EC__PASSPHRASE.

❓ question

• HttpOnly omitted from EC cookie (ec/cookies.rs:11): Intentionally omitted for a hypothetical future JS use case. Is there a concrete plan? The identify endpoint already exposes the EC ID. If not, HttpOnly would be cheap XSS defense-in-depth.

Non-blocking

🤔 thinking

• Uppercase EC ID rejection in batch sync (batch_sync.rs:209): is_valid_ec_id rejects uppercase hex, but batch sync validates before normalizing (line 225). Partners submitting uppercase EC IDs get invalid_ec_id instead of normalization.

♻️ refactor

• _pull_enabled index lacks CAS (partner.rs:564): Read-modify-write without generation markers. Concurrent partner registrations can overwrite each other's index updates. Self-healing via fallback, but inconsistent with the CAS discipline used elsewhere.

🌱 seedling

• Integration tests don't verify identify response shape: FullLifecycle and ConcurrentPartnerSyncs only assert uids.<partner>. The ec, consent, degraded, eids, and cluster_size fields from the API reference are never checked.
• Pixel sync failure paths untested end-to-end: ts_reason=no_ec, no_consent, domain mismatch redirects are unit-tested but not covered in integration tests.

📝 note

• trusted-server.toml ships banned placeholder: passphrase = "trusted-server" is rejected by reject_placeholder_secrets. Works in CI via env override, but new contributors will hit a confusing startup error. A TOML comment would help.

⛏ nitpick

• Eid/Uid missing Deserialize: openrtb.rs types derive Serialize only. If the auction ever needs to parse EIDs from provider responses, Deserialize will be needed.

CI Status

• cargo fmt: PASS
• clippy: PASS
• cargo test: PASS
• vitest: PASS
• integration tests: FAIL
• CodeQL: FAIL (likely related to cleartext logging findings above)

[prk-Jr]

Summary

Comprehensive EC identity subsystem: HMAC-based ID generation, KV identity graph with CAS concurrency, partner sync (pixel, batch, pull), identify endpoint, and auction bidstream decoration. The architecture is well-designed with clean separation of concerns, strong concurrency discipline, and solid security choices throughout. Four new blocking findings join the existing ones — three are input-validation gaps and one is a PII leak in the pull sync URL builder that should be resolved before merge.

Blocking

🔧 wrench

• PII leak: client IP forwarded to partners — pull_sync.rs:273: build_pull_request_url appends the raw user IP as the ip query parameter on every outbound pull-sync request, exposing it in partner access logs. Contradicts the privacy-preserving design intent and is likely a GDPR Article 5 concern. Remove the ip parameter or gate it behind an explicit per-partner allow_ip_forwarding config flag.
• Pull sync response body unbounded — pull_sync.rs:325: take_body_bytes() with no size cap before JSON parsing. A malicious partner can exhaust WASM memory. Batch sync and admin endpoints both have MAX_BODY_SIZE guards; this path needs one too (64 KiB is generous for a {"uid":"..."} response).
• Whitespace-only UIDs bypass validation — batch_sync.rs:217 and sync_pixel.rs:41: is_empty() passes " " and "\t", which get stored as garbage EID values in the KV graph. Replace with trim().is_empty() at both sites.
• rand::thread_rng() WASM compatibility unverified — generation.rs:52: requires getrandom with the wasi feature on wasm32-wasip1. Native cargo test passing does not prove the WASM build is safe; integration tests are already failing so this may not have been caught. Switch to OsRng or add getrandom = { version = "0.2", features = ["wasi"] } explicitly.

Non-blocking

🤔 thinking

• process_mappings UpsertResult branches untested — batch_sync.rs:232: NotFound, ConsentWithdrawn, and Stale have zero unit test coverage. Stale is documented as "counted as accepted" with no regression test.
• Shared error messages make pull sync validation tests non-isolating — partner.rs:152,165: both missing-URL and missing-token conditions return the identical error string, so the tests asserting on substrings pass even if the wrong branch fires. Use distinct messages per condition.

♻️ refactor

• ec_consent_granted has no tests — consent.rs:20: the entry-point gate for all EC creation has no #[cfg(test)] section. Add smoke tests for granted and denied paths.
• KV tombstone path never exercised in finalize tests — finalize.rs:149: all finalize tests pass kv: None, so the tombstone write and the cookie-EC ≠ active-EC case in withdrawal_ec_ids are untested.
• Missing HMAC prefix stability test — generation.rs:228: no test asserts that the same IP + passphrase always produces the same 64-char hash prefix, which is the core deduplication guarantee for the KV graph.
• normalize_ec_id duplicated in integration tests — integration-tests/tests/common/ec.rs:374: reimplements normalize_ec_id_for_kv from core. Re-export and use the canonical implementation.

⛏ nitpick

• Use saturating_sub for consistency — kv.rs:605: the subtraction is safe due to the guard above but inconsistent with saturating_sub used throughout the rest of the module.
• log_id should encapsulate the … suffix — mod.rs:51: every call site manually appends "…" in the format string; move it inside the function.

Praises 👍

• CAS-based optimistic concurrency (kv.rs): bounded retries, graceful ItemPreconditionFailed handling, and MAX_CAS_RETRIES preventing infinite loops — textbook correct for a single-writer KV model.
• Constant-time API key comparison (partner.rs): subtle::ConstantTimeEq for timing attack prevention on key lookups. Many implementations miss this entirely.
• KvMetadata fast-path consent check (kv_types.rs): mirroring ok/country/cluster_size in metadata to avoid streaming the full KV body is an excellent performance optimisation with the right constraint test.
• evaluate_known_browser with OnceLock-cached hash table (device.rs): pre-hashing fingerprints once and caching via OnceLock is the right WASM-compatible lazy-init pattern.
• HMAC stability explicitly documented (generation.rs:30-33): noting that the output format is a "stable contract" that would invalidate all existing identities if changed is exactly the kind of correctness annotation that prevents future breakage.

CI Status

• cargo fmt: PASS
• cargo clippy: PASS
• cargo test: PASS
• vitest: PASS
• integration tests: FAIL
• CodeQL: FAIL (cleartext logging — covered in prior review)

[aram356]

Summary

This PR lands a large, well-partitioned Edge Cookie identity subsystem, and the prior blocking feedback (PII logging via log_id(), whitespace UID checks, pull-sync response size cap, dropping raw client IP from pull-sync query strings, constant-time auth comparisons) has been materially addressed. Blocking issues remaining are: a CI-red integration test (deterministic, root cause in the test client, not the server), an unverified rand::thread_rng() on wasm32-wasip1 that CI cannot catch because no wasm build runs, two new CodeQL alerts on ec/kv.rs riding the pre-existing settings-taint false-positive flow, one untested backend-state machine in batch_sync, and two questions on the auction / partner surfaces.

Blocking

🔧 wrench

• CI red — test_ec_lifecycle_fastly deterministic failure: the test client omits Sec-Fetch-Dest: document and Accept: text/html, so is_navigation_request() correctly returns false and EC never generates (crates/integration-tests/tests/common/ec.rs:93).
• rand::thread_rng() on wasm32-wasip1 unverified; CI builds no wasm target (crates/trusted-server-core/src/ec/generation.rs:52). Add a wasm build to CI and/or switch to an explicit getrandom with the wasi feature.
• Two new CodeQL alerts on ec/kv.rs:637 and :789 ride the same settings.reject_placeholder_secrets() taint flow that already produces noise on main. Both are false positives but block the CodeQL gate. Fix with a scoped suppression or a repo-level CodeQL config filter on rust/cleartext-logging for settings-derived values.
• UpsertResult::{NotFound, ConsentWithdrawn, Stale} untested in batch_sync::process_mappings (crates/trusted-server-core/src/ec/batch_sync.rs:231-244). Prior reviewer flagged this; still unresolved.

❓ question

• Auction user.id = "" on the no-consent path is emitted into the OpenRTB bid request JSON (crates/trusted-server-core/src/auction/endpoints.rs:60-77, formats.rs:186). Has this been validated against the live Prebid Server target? If not, change UserInfo.id to Option<String> with skip_serializing_if.
• update_pull_enabled_index race self-heal path untested (crates/trusted-server-core/src/ec/partner.rs:552-604). The documented fallback at partner.rs:350 is the only thing keeping the index correct under concurrent upserts — please add a concurrency test or file a tracked follow-up.

Non-blocking

🤔 thinking

• MAX_CAS_RETRIES = 3, no backoff — likely to starve under contention on hot prefixes; consider exponential backoff + jitter (crates/trusted-server-core/src/ec/kv.rs:300-339).
• HashMap<String, KvPartnerId> non-deterministic serialization — breaks future hash-based dedup and byte-diffing of stored values; use BTreeMap or IndexMap (crates/trusted-server-core/src/ec/kv_types.rs:59).
• seen_domains drop-newest eviction freezes long-lived ECs on their first 50 domains, defeating the last timestamp; switch to LRU or document (crates/trusted-server-core/src/ec/kv.rs:627-642).

♻️ refactor

• EID gating failures log at debug! — bump to warn! so ad-stack anomalies surface in production (crates/trusted-server-core/src/auction/endpoints.rs:80).
• ec_consent_granted() is a 1-line pass-through — inline or document the sealing-point intent (crates/trusted-server-core/src/ec/consent.rs:20-22).
• Unvalidated domain strings in EC graph writes — add a 255-byte length cap and hostname-shape check at the write boundary (crates/trusted-server-core/src/ec/kv.rs update_last_seen, kv_types.rs).

🌱 seedling

• No integration test for the GPC / consent-denied identify path. Once the Sec-Fetch-Dest fix lands, add ec_full_lifecycle_with_gpc sending Sec-GPC: 1 and asserting /_ts/api/v1/identify returns 403 / no EID payload (crates/integration-tests/tests/frameworks/scenarios.rs:501-565).
• No wasm-target build in CI — root cause of how the rand::thread_rng() concern reached this point unnoticed. Adding cargo build -p trusted-server-adapter-fastly --target wasm32-wasip1 would catch an entire class of deps-pin regressions.

📌 out of scope

• Pull-sync EC ID still travels as a URL query parameter — acknowledged in the PR description as deferred. Please file the follow-up issue and reference it from the commit that ships this PR so it is not lost (crates/trusted-server-core/src/ec/pull_sync.rs:260-263).

CI Status

• cargo fmt: PASS
• cargo clippy: PASS
• cargo test (native): PASS
• vitest: PASS
• integration tests: FAIL — test_ec_lifecycle_fastly (see blocking #1)
• CodeQL: FAIL — 15 high alerts; 13 are pre-existing settings.reject_placeholder_secrets() taint-flow false positives already present on main, 2 are new on ec/kv.rs riding the same flow (see blocking #3)

[aram356]

🔧 wrench — EcTestClient::get() never sends Sec-Fetch-Dest: document, so the CI test_ec_lifecycle_fastly failure ("organic GET / should set ts-ec cookie" at scenarios.rs:507) is deterministic.

Commit f02b538a restricted EC generation to navigation requests via is_navigation_request() in http_util.rs:72-95. That helper prefers Sec-Fetch-Dest: document and falls back to Accept: text/html — deliberately not */*. reqwest defaults to Accept: */* and omits Sec-Fetch-Dest, so is_navigation_request() returns false, EC never generates, and the scenario panics.

The fallback is a legitimate security property — fix the test, not the helper.

Fix:

pub fn get(&self, path: &str) -> TestResult<Response> {
    let builder = self
        .client
        .get(format!("{}{path}", self.base_url))
        .header("sec-fetch-dest", "document");
    // ...
}

Apply the same default to get_with_headers() (callers that want a non-navigation request can override).

[aram356]

🔧 wrench — rand::thread_rng() on wasm32-wasip1 is unverified and CI never builds the wasm target.

cargo test --workspace runs natively, so the whole test suite proves nothing about the Fastly Compute build. rand 0.8's thread_rng() pulls getrandom; on wasm32-wasip1 this requires the wasi backend to be selected. If any transitive dep pins getrandom with the js feature or defaults, EC ID generation fails at runtime with getrandom::Error::UNEXPECTED — every organic request errors and no cookies are ever written. Prior reviewer (prk-Jr) flagged this; the "address all blocking findings" commit did not change the call site.

Fix (pick one):

1. Add cargo build -p trusted-server-adapter-fastly --release --target wasm32-wasip1 to CI and a Viceroy smoke test that exercises generate_ec_id().
2. Switch to an explicit getrandom call with getrandom = { version = "0.2", features = ["wasi"] } in the core crate's Cargo.toml, and document the feature requirement.

Option 1 is stronger because it prevents regression across the whole binary, not just this call site.

[aram356]

🔧 wrench — This line and ec/kv.rs:789 are the two new CodeQL rust/cleartext-logging alerts blocking CI. Both are false positives — they log log_id(ec_id) (truncated) and routine config, not secrets. CodeQL is tracing the settings → log-argument taint flow that already produces pre-existing noise on main.

The PR still lands the CodeQL job red. Options to clear the gate:

1. Targeted suppression on both sites: // lgtm[rust/cleartext-logging] with a comment explaining the taint-flow false positive.
2. Repository-level filter in .github/codeql/codeql-config.yml that excludes rust/cleartext-logging for values derived from settings validation helpers. Cleaner long-term since it also clears the existing alerts on main.
3. Break the taint flow by pre-formatting the config values into a local that does not flow from settings.reject_placeholder_secrets().

Option 2 is the correct permanent fix; option 1 is acceptable as a short-term unblock if scoped narrowly.

[aram356]

🔧 wrench — UpsertResult::{NotFound, ConsentWithdrawn, Stale} branches still have zero unit coverage.

Prior reviewer flagged this; the "address all blocking findings" commit did not resolve it. The existing process_mappings test at batch_sync.rs:416-432 only exercises UpsertResult::Written. Of the four branches this match dispatches, three are untested — exactly the branches most likely to regress silently as the KV layer evolves.

Add three small cases to the MockWriter-based test that yield each UpsertResult variant and assert the corresponding HTTP status / per-mapping result. ~20 LoC per branch.

[aram356]

❓ question — When consent is denied, ec_id is set to "" at lines 60-64 and passed into UserInfo.id at formats.rs:186. The x-ts-ec header is correctly omitted for empty IDs at formats.rs:317-319, but the empty-string user.id still ships in the OpenRTB bid request JSON.

OpenRTB 2.6 §3.2.20 defines user.id as an "Exchange-specific ID for the user", and Prebid Server plus several SSPs reject empty strings as invalid. Has this path been validated end-to-end against the live Prebid Server the deployment targets?

If not, the right type-level fix is to change UserInfo.id to Option<String> with #[serde(skip_serializing_if = "Option::is_none")] so the key is dropped from the JSON entirely on the no-consent path.

[aram356]

🤔 thinking — HashMap<String, KvPartnerId> serializes keys in arbitrary order. CAS correctness is unaffected (opaque bytes), but stored values will differ byte-for-byte across equivalent writes, which breaks future hash-based dedup and makes stored-value diffing miserable. Consider BTreeMap or indexmap::IndexMap for deterministic output.

[aram356]

🤔 thinking — When seen_domains hits MAX_SEEN_DOMAINS, the newer domain is silently dropped — so a long-lived EC is frozen on its first 50 domains forever, which defeats the purpose of last timestamps. Either document the drop-newest policy in the const comment on kv_types.rs:20 or switch to LRU eviction keyed on KvDomainVisit::last.

[aram356]

♻️ refactor — When gate_eids_by_consent() drops EIDs unexpectedly the message is logged at debug!, so operators will not see it in production. Bump to warn!: this is the kind of ad-stack anomaly that should surface immediately if revenue drops.

[aram356]

♻️ refactor — ec_consent_granted() is a 1-line pass-through to crate::consent::allows_ec_creation(). Either inline it at the two call sites or add a doc comment explaining the sealing-point intent so the next cleanup pass doesn't remove it.

[aram356]

♻️ refactor — domain is written into the EC graph unvalidated and uncapped. A compromised or buggy partner can push arbitrary-length strings (10 KB+) and arbitrary bytes into seen_domains keys and origin_domain. Add a 255-byte length cap and a lightweight hostname-shape check at the write boundary; the same treatment should apply to origin_domain in kv_types.rs.