perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write

[chopratejas]

Summary

Audit-driven perf PR. Three independent hot-path fixes targeting concurrent-request throughput. Each is bench-measured below; the proxy hot path benefits from all three at once.

Stacked on top of #292. Audit notes for the full perf landscape (Tier 1/2/3) are in the commit body — this PR ships only Tier 1 (highest impact, surgical changes).

The three changes

1. PyO3 GIL release on heavy Rust compute

SmartCrusher.crush, smart_crush_content, crush_array_json, compact_document_json, and DiffCompressor.compress/compress_with_stats used to hold the GIL across the entire Rust call. A 100 ms compress() blocked every other Python thread for 100 ms — multi-worker uvicorn deployments serialized through SmartCrusher.

Wrap each compute call in py.allow_threads(|| ...). &str Python inputs are copied to owned String first (PyO3 ties them to GIL hold).

Verified end-to-end: 4 Python threads × 20 crushes — single-thread baseline 829 ms; 4-thread parallel 826 ms → 4.01× speedup, perfect parallel scaling.

2. CcrStore: Mutex<HashMap> → DashMap-backed sharded

Single Mutex was the dominant bottleneck under multi-worker load. Replace with DashMap (sharded concurrent map) + a small Mutex<VecDeque> for FIFO eviction order only.

A/B bench in crates/headroom-core/benches/ccr_store.rs (200 mixed put/get × N threads):

Threads	DashMap	Legacy Mutex	Speedup
1	63 µs	71 µs	1.1×
2	98 µs	194 µs	2.0×
4	178 µs	707 µs	4.0×
8	342 µs	1267 µs	3.7×

Legacy degrades ~linearly with thread count (one Mutex serializes everything); DashMap stays near-flat per-thread.

3. Single-serialize the lossy CCR payload

The lossy crush_array path was serializing the full array twice — once for the hash, once for the store write — each one allocating Value::Array(items.to_vec()) and deep-cloning the entire tree. For a 50-item dict array that's MBs of allocator pressure per call.

Refactor: canonical_array_json serializes &[Value] directly (same bytes, no wrapper allocation, no tree clone), call ONCE, hash + store reuse the same bytes. Hash format stable — all 17 parity fixtures still match byte-for-byte.

Test plan

• cargo test --workspace — 479 lib + 14 integration + rest, all green
• cargo clippy --workspace -- -D warnings clean
• cargo fmt --check clean
• make ci-precheck-python — 185 tests pass
• New ccr.rs concurrent-stress test (8 threads × 200 puts/gets, every key readable)
• parity-run smart_crusher — 17/17 fixtures match (proves single-serialize is hash-stable)
• End-to-end Python GIL-release verification (script in commit body)
• Bench A/B for store (committed at benches/ccr_store.rs for regression visibility)

Dependencies added

• dashmap v6 (mature, widely used in tokio/linkerd; sharded concurrent HashMap)

What's NOT in this PR

Tier-2 / Tier-3 items deferred (full audit in PR9 commit body):

• EmbeddingScorer's Mutex<TextEmbedding> pool (need to verify ORT internal threadpool first)
• Redundant JSON parse in classify_cell + try_parse_json_container
• Lossless wins still allocate items.to_vec() even when caller consumes .compacted instead
• simd-json on the proxy hot path (when transforms get wired into the proxy)
• TCP nodelay / keepalive on reqwest