test(client): add fast-check model-based property tests and retry bound analysis

[KyleAMathews]

Summary

Fixes twelve bugs in ShapeStream's retry, error-handling, caching, and subscription paths — all uncovered by fast-check property-based testing (model-based + micro-target) and AST-based static analysis. Every bug is a concrete end-user hazard: infinite retry loops, stale data delivery, stack-frame leaks, cache-key collisions, dropped notifications, lost snapshots, wrong-row corruption, and deadlocks.

Root Cause

Every bug in this PR is an edge case in an error-handling or reconciliation path that only manifests under a specific sequence of events. Hand-written tests don't naturally explore adversarial sequences like "409 with same handle, then 409 with no handle, then 200 with empty body" or "addSnapshot(mark=1, xmax=5); removeSnapshot(1); addSnapshot(mark=1, xmax=10); reject(xid=5)". These bugs survived because they required rare permutations — a proxy stripping a header, two schema columns differing only in underscore count, a subscriber pushed over the must-refetch/up-to-date boundary at just the wrong instant.

PBT shakes them out mechanically.

Approach

Two complementary verification strategies that catch bugs without a human having to imagine the failure mode:

Model-based property testing (test/model-based.test.ts)

fc.asyncModelRun with 21 command types drives adversarial server response sequences. A simple model tracks { consecutiveErrors, terminated }, and each command predicts the model's state change before asserting the real ShapeStream matches.

FetchGate pattern: a controllable mock fetch that blocks each request until the test provides a response — turn-based coordination where the test controls what the server returns while ShapeStream drives when it fetches.

Global invariants after every command:

• URL length stays bounded (catches suffix growth)
• isUpToDate / lastSyncedAt consistency
• After 409, retry URL differs from pre-409 URL (catches identity loops)

Micro-target PBT (test/pbt-micro.test.ts, new)

Twelve narrow TARGETs, one per suspected bug site. Each TARGET's opening comment documents the invariant under test, then runs fc.assert with 500 runs in CI (2000-run soak available via PBT_MICRO_RUNS=2000). Runs against the dedicated vitest.pbt.config.ts (skips the real-Electric globalSetup). bin/pbt-soak.sh wraps it in a fresh-seed loop for overnight soaks.

Static analysis via AST walking

Seven rule types that mechanically detect structural bug patterns:

Rule	What it catches
unbounded-retry-loop	Recursive calls in catch blocks without detectable bounds
conditional-409-cache-buster	409 handlers where createCacheBuster() is conditional or missing
parked-tail-await	await this.#method(); return patterns that park stack frames
error-path-publish	#publish/#onMessages calls inside catch blocks or error status handlers
shared-instance-field	Mutable fields written before async boundaries and read by other methods
ignored-response-transition	Non-delegate states returning { action: 'ignored' }
protocol-literal-drift	Near-miss string literals for Electric protocol params

Each rule was RED/GREEN verified.

Bugs Fixed

From model-based tests + static analysis

1. Conditional cache buster on 409. Same-handle (and no-handle) 409s produced identical retry URLs, causing infinite CDN-cached retry loops. createCacheBuster() is now unconditional on every 409 in both #requestShape and #fetchSnapshotWithRetry.

2. Parked stack frame in #start retry. await this.#start(); return kept the caller's frame alive for the whole recursive chain. Switched to return this.#start() so the frame is released via promise chaining.

3. Missing EXPERIMENTAL_LIVE_SSE_QUERY_PARAM in protocol params. canonicalShapeKey wasn't stripping the deprecated param, so clients using it landed on a different cache key than clients on current live_sse. Added to the strip list; static analysis now enforces that all internal *_QUERY_PARAM exports live in the list.

4. Publishing 409 body to subscribers. The 409 handler was parsing e.json and calling #publish(messages409) before retrying, delivering stale rotation frames. Replaced with a synthetic must-refetch control message so the Shape still resets its cache on 409 rotation, without leaking the raw payload.

From test/pbt-micro.test.ts (new micro-target PBTs)

5. canonicalShapeKey collapsed duplicate custom params. Used URLSearchParams.set() in the copy loop, so ?tag=a&tag=b became ?tag=b. Two genuinely distinct shapes shared a cache key, producing cross-shape cache poisoning via expiredShapesCache/upToDateTracker. Switched to append().

6. Shape#process clobbered shouldNotify. Three sites assigned shouldNotify = this.#updateShapeStatus(...) instead of OR-accumulating. Any sequence where a change message followed a status-change message silently dropped the change's notification. Changed to OR-accumulate, and must-refetch now tracks hadData before clearing so subscribers still see the reset.

7. SubsetParams GET serialization dropped limit=0 / offset=0. Falsy checks. Switched to explicit !== undefined guards. Limit-0 is a valid "probe" shape request.

8. Shape#requestedSubSnapshots non-canonical JSON dedup. Keyed on bigintSafeStringify, which preserves insertion order — so {a:1, b:2} and {b:2, a:1} re-executed the same snapshot twice on rotation. Added canonicalBigintSafeStringify to helpers.ts that recursively sorts object keys.

9. snakeToCamel collided multi-underscore columns. A run of n underscores collapsed to a single camelCase boundary, so user_id and user__id (distinct columns) decoded to the same app key — wrong-row corruption at the ColumnMapper layer. snakeToCamel now preserves (n-1) literal underscores for a run of n, and camelToSnake's boundary regex was widened to ([a-z_])([A-Z]) so the round-trip is injective.

10. Shape#reexecuteSnapshots silently swallowed errors. Caught and discarded errors from stream.requestSnapshot on shape rotation, so failed sub-snapshot re-execution was invisible. Errors are now collected and the first is surfaced via #error + #notify, plus the pre-step #awaitUpToDate throw is caught and surfaced the same way.

11. SnapshotTracker stale reverse indexes on re-add. xmaxSnapshots and snapshotsByDatabaseLsn were populated on addSnapshot but never cleaned up on removeSnapshot, and re-adding with the same mark left stale entries pointing at the old (xmax, database_lsn). A later shouldRejectMessage eviction loop would walk the stale index and wrongly delete the current snapshot — duplicate change messages would slip through the filter. Tracked databaseLsn on each entry and added #detachFromReverseIndexes that runs on both add (before overwriting) and remove.

12. Shape#awaitUpToDate deadlocked on terminally-errored stream. The helper polled stream.isUpToDate via setInterval(check, 10) but never observed stream.error, so calling requestSnapshot after the stream had terminally failed hung forever. The helper now checks #error / stream.error up front, subscribes to the stream's onError, and settles the internal promise via reject on any terminal error path.

Key Invariants

1. Every 409 handler unconditionally calls createCacheBuster() before retrying (static analysis + PBT).
2. Every recursive call in a catch block has a detectable retry bound.
3. No await this.#method(); return in recursive methods.
4. No #publish or #onMessages calls in error handling paths.
5. All internal *_QUERY_PARAM constants appear in ELECTRIC_PROTOCOL_QUERY_PARAMS.
6. Consecutive-error counter resets only on proven success (200+data, 200 up-to-date, 204).
7. canonicalShapeKey is stable under protocol-param noise and distinct under origin/pathname/custom-param changes (PBT).
8. Shape#process notifies subscribers on every data change, even when interleaved with status-change messages (PBT).
9. snakeToCamel ∘ camelToSnake is identity; distinct db columns never collide on the same app key (PBT).
10. SnapshotTracker reverse indexes are consistent with activeSnapshots after every operation (PBT with 500–2000 runs).
11. Shape#awaitUpToDate settles on any terminal outcome — up-to-date OR error — in bounded time.

Non-goals

• Refactoring #start/#requestShape into iterative loops (structural change, separate PR).
• Fixing the pgArrayParser double-push bug (pre-existing, unreachable for valid PostgreSQL arrays).
• Testing network-level failures or backoff timing (model tests focus on response sequences).
• The wake-detection queueMicrotask timing issue (pre-existing, requires deeper investigation).
• Log-mode and subset__* collision in canonicalShapeKey — TARGET 10 pins current behavior with deterministic assertions; fixing them is a separate scope decision.

Verification

cd packages/typescript-client

# Unit suite (no Electric server required)
pnpm vitest run --config vitest.unit.config.ts          # 319 tests

# Micro-target PBT suite (12 TARGETs)
PBT_MICRO_RUNS=500 pnpm vitest run --config vitest.pbt.config.ts test/pbt-micro.test.ts  # 44 tests

# Soak runner (fresh seed each iteration, default 1 hour budget)
bin/pbt-soak.sh

# Static analysis tests
pnpm vitest run --config vitest.unit.config.ts test/static-analysis.test.ts  # 10 tests

# Typecheck + lint
pnpm exec tsc --noEmit && pnpm exec eslint src test

Verified: 319 unit tests, 44 PBT tests at 500 runs each (plus 2000-run soak), 62 column-mapper/snapshot-tracker tests, typecheck clean, eslint clean, 10 static-analysis rules clean.

Files changed

File	Change
src/client.ts	Unconditional cache buster on 409; return this.#start(); synthetic must-refetch on 409 instead of raw publish; EXPERIMENTAL_LIVE_SSE_QUERY_PARAM in strip list; canonicalShapeKey uses append(); SubsetParams GET uses !== undefined
src/shape.ts	Shape#process OR-accumulates shouldNotify + tracks hadData on must-refetch; #reexecuteSnapshots surfaces errors via #error+#notify; #awaitUpToDate settles on terminal stream/error states; requestSnapshot dedup uses canonical stringify
src/column-mapper.ts	snakeToCamel preserves (n-1) underscores; camelToSnake boundary regex widened to ([a-z_])([A-Z]) for injective round-trip
src/snapshot-tracker.ts	Reverse-index cleanup via #detachFromReverseIndexes on both add-before-overwrite and remove; tracks databaseLsn per entry
src/helpers.ts	Added canonicalBigintSafeStringify (recursive key-sorted stringify)
src/constants.ts	EXPERIMENTAL_LIVE_SSE_QUERY_PARAM in protocol strip list
test/model-based.test.ts	11 response factories + FetchGate + 21 commands + global invariants; new update/delete/mixed-batch 200 responses with lsn/op_position/txids headers
test/pbt-micro.test.ts	New. 12 micro-target PBTs (44 individual tests)
test/static-analysis.test.ts	Tests for all 7 AST-based rules including protocol param completeness
bin/lib/shape-stream-static-analysis.mjs	AST analysis: unbounded retry, 409 cache buster, tail-position await, error-path publish, protocol literal drift, shared-instance-field, ignored-response-transition
bin/pbt-soak.sh	New. Fresh-seed soak runner with configurable budget
vitest.pbt.config.ts	New. Dedicated config for PBT suites — skips Electric globalSetup
vitest.unit.config.ts	Added model-based test to include list
SPEC.md	Cross-references L6 fetchSnapshotWithRetry PBT from unconditional-409-cache-buster invariant; updated loop-back site line numbers
.changeset/add-model-based-property-tests.md	Updated to cover all 12 bugs in one entry
package.json	Added fast-check devDependency

---

🤖 Generated with Claude Code

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@electric-sql/react@4089

npm i https://pkg.pr.new/@electric-sql/client@4089

npm i https://pkg.pr.new/@electric-sql/y-electric@4089

commit: 13b26ac

[codecov]

Codecov Report

❌ Patch coverage is 89.41176% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.20%. Comparing base (690e25a) to head (13b26ac).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/typescript-client/src/shape.ts	80.43%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4089      +/-   ##
==========================================
+ Coverage   88.78%   89.20%   +0.41%     
==========================================
  Files          25       25              
  Lines        2471     2520      +49     
  Branches      622      635      +13     
==========================================
+ Hits         2194     2248      +54     
+ Misses        275      270       -5     
  Partials        2        2              

Flag	Coverage Δ
packages/experimental	87.73% <ø> (ø)
packages/react-hooks	86.48% <ø> (ø)
packages/start	82.83% <ø> (ø)
packages/typescript-client	94.30% <89.41%> (+0.43%)	⬆️
packages/y-electric	56.05% <ø> (ø)
typescript	89.20% <89.41%> (+0.41%)	⬆️
unit-tests	89.20% <89.41%> (+0.41%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[claude]

Claude Code Review

Summary

One new commit since iteration 4: 13b26ac5 ("fix(client): suppress empty-rows notification on must-refetch"). It resolves the Important CI failures flagged in iteration 4. The PR is now clean.

---

What's Working Well

must-refetch notification semantics are now correct and coherent. The hadData intermediate notify introduced in 2939f5ed caused subscribers to observe a brief empty-rows state during rotation — pointless re-renders and a broken integration test. The new commit removes the notify entirely, letting N1 (no notify while syncing) handle the transition naturally. The implementation in shape.ts:257-264, the SPEC.md update (N3 removed, must-refetch prose updated), and the PBT deterministic test are all consistent.

---

Issues Found

Critical (Must Fix)

None.

Important (Should Fix)

None. The two integration-test timeouts (should resync from scratch on a shape rotation) that were flagged in iteration 4 are addressed by 13b26ac5 — the intermediate empty-rows notification that caused those failures is now removed.

Suggestions (Nice to Have)

buildTailPositionAwaitReport JSDoc still claims two patterns; only one is implemented

File: packages/typescript-client/bin/lib/shape-stream-static-analysis.mjs:978-980

The implementation only checks followedByBareReturn (line 1013-1016). Pattern 2 (last statement in a block with no explicit return) is documented in the JSDoc but not wired up. Not a bug, but the comment creates a false expectation. Either trim the JSDoc to pattern 1 only, or add the stmtIndex === block.statements.length - 1 guard. Carried from iteration 3 — only remaining open item.

---

Issue Conformance

No linked issue; PR description remains comprehensive and self-documenting. Acceptable.

---

Previous Review Status

Issue	Status
isStaticControlMessagePublish too broad (Important)	Resolved (iteration 3, confirmed via diff)
Changeset missing original 4 bugs (Important)	Resolved (iteration 3)
CI failures: should resync from scratch on a shape rotation (Important)	Resolved (13b26ac5)
Misleading test titles in snakeCamelMapper PBT (Suggestion)	Resolved (iteration 3)
waitUntilSettled fragility maxYields=50 (Suggestion)	Resolved (iteration 3)
PR description command count (Suggestion)	Resolved (iteration 3)
Dead userFacingParams entries (Suggestion)	Resolved (iteration 3)
buildTailPositionAwaitReport JSDoc pattern 2 (Suggestion)	Still open

---

Review iteration: 5 | 2026-04-10

[balegas]

Suggestion: tighten isStaticControlMessagePublish

The error-path-publish exception in shape-stream-static-analysis.mjs:507-511 is currently too permissive:

function isStaticControlMessagePublish(callExpr) {
  const [firstArg] = callExpr.arguments
  if (!firstArg || !ts.isArrayLiteralExpression(firstArg)) return false
  return firstArg.elements.every((el) => ts.isObjectLiteralExpression(el))
}

It fires for any static array of object literals, so a future edit like:

// inside a catch block
await this.#publish([{ offset: '1_0', value: {...}, headers: { operation: 'insert' }, key: 'k' }])

would bypass the rule — exactly the shape of Bug 4 this rule is meant to catch.

Consider requiring a headers.control string literal so the exception only fires for genuine synthetic control messages (like the must-refetch publish added here):

function isStaticControlMessagePublish(callExpr) {
  const [firstArg] = callExpr.arguments
  if (!firstArg || !ts.isArrayLiteralExpression(firstArg)) return false
  return firstArg.elements.length > 0 && firstArg.elements.every((el) => {
    if (!ts.isObjectLiteralExpression(el)) return false
    const headersProp = el.properties.find(
      (p) => ts.isPropertyAssignment(p) && ts.isIdentifier(p.name) && p.name.text === 'headers'
    )
    if (!headersProp || !ts.isObjectLiteralExpression(headersProp.initializer)) return false
    return headersProp.initializer.properties.some(
      (p) => ts.isPropertyAssignment(p) && ts.isIdentifier(p.name) && p.name.text === 'control'
        && ts.isStringLiteral(p.initializer)
    )
  })
}

Without this, the rule's signal for Bug 4 will erode over time.

[netlify]

✅ Deploy Preview for electric-next ready!

Name	Link
🔨 Latest commit	154180f
🔍 Latest deploy log	https://app.netlify.com/projects/electric-next/deploys/69d941f9e9b26900089c51ef
😎 Deploy Preview	https://deploy-preview-4089--electric-next.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

This PR has been released! 🚀

The following packages include changes from this PR:

• @electric-sql/client@1.5.15

Thanks for contributing to Electric!