Skip to content

Sy 3833 strongly type schematics#2190

Open
emilbon99 wants to merge 642 commits intosy-3990-server-side-metadata-importexportfrom
sy-3833-strongly-type-schematics
Open

Sy 3833 strongly type schematics#2190
emilbon99 wants to merge 642 commits intosy-3990-server-side-metadata-importexportfrom
sy-3833-strongly-type-schematics

Conversation

@emilbon99
Copy link
Copy Markdown
Contributor

@emilbon99 emilbon99 commented Apr 7, 2026

Pull Request

Go to the Preview tab and select the appropriate template for your pull request:

Greptile Summary

This PR strongly types the schematic data model by replacing the opaque Data msgpack.EncodedJSON blob in v54 with explicit Nodes, Edges, Props, Legend, and Authority fields — both server-side (Go) and client-side (TypeScript v6 state). Node/edge/props data is now owned by the server; the console Redux slice retains only UI state (selection, control status, toolbar, viewport). Action-based mutation (SetNodePosition, AddNode, SetEdge, SetProps, etc.) now drives server-side state transitions with real-time fan-out via the sy_schematic_set signal channel.

  • P1: extractLegend in migrate.go writes to a nil Colors map and calls color.MustFromHex (panics on any malformed hex string) — any schematic with legend colors that triggers migration will crash the server on upgrade.

Confidence Score: 4/5

Not safe to merge until the nil map + MustFromHex panics in extractLegend are fixed — these will crash the server during first upgrade migration of any schematic with legend colors.

The architectural restructuring is well-designed and the generated code is correct throughout. A single P1 bug in migrate.go will cause a runtime panic in the migration path for existing schematics with legend colors. All other files are clean.

core/pkg/service/schematic/migrate.go — extractLegend writes to nil Colors map and calls color.MustFromHex on unvalidated user data.

Vulnerabilities

No security concerns identified beyond the migration-time panic from color.MustFromHex on unvalidated input (addressed as P1 inline). Auth/authz, injection vectors, and secrets handling are unchanged from prior versions.

Important Files Changed

Filename Overview
core/pkg/service/schematic/migrate.go Migration helper with nil map panic and MustFromHex panic risk in extractLegend; extractViewport is dead code
core/pkg/service/schematic/types.gen.go Generated: strongly-typed Schematic struct replacing opaque Data blob; Legend, Node, Edge, Handle, Segment all defined
core/pkg/service/schematic/actions.gen.go Generated: Action discriminated union and ReduceAll for schematic mutations; unknown types no-op (forward compat)
core/pkg/service/schematic/actions.go Clean action handlers; AddNode/SetProps correctly lazy-init Props map; delete on nil map in RemoveNode is safe
core/pkg/service/schematic/writer.go Dispatch applies ReduceAll atomically via ChangeErr; nil-guard on actionObserver prevents panics in test mode
core/pkg/service/schematic/writer_test.go Good Ginkgo coverage: Create, Rename, Dispatch (position/sequence/edge), nonexistent-key error, Copy with ontology verification
client/ts/src/schematic/client.ts Clean TS client; dispatch forwards sessionKey for self-dedup; all request/response Zod-validated
client/ts/src/schematic/actions.gen.ts Generated TS discriminated union actions; reduceAll wraps in immer produce; null handling differs intentionally between AddNode and SetProps
client/ts/src/schematic/types.gen.ts Generated TS zod schemas; nodes/edges use nullishToEmpty, authority defaults to 1, newZ correctly makes key/snapshot optional
console/src/schematic/remote.ts Returns ZERO_STATE from remote (server data intentionally discarded), enforcing v6 server/UI state separation
console/src/schematic/types/v6.ts v6: nodes/edges/props/authority/snapshot moved server-side; client state is UI-only (selection, viewport, toolbar, editable)
console/src/schematic/types/migrations.spec.ts Comprehensive migration tests v0-v5 to v6; explicitly asserts server-side fields absent from migrated state
pluto/src/schematic/queries.ts Flux API; ACTION_LISTENER enables real-time collaboration via sy_schematic_set; augmentWithEdgeSegments keeps edge routing consistent on node moves
schemas/schematic.oracle Complete Oracle schema driving TS/Go/protobuf codegen; actions declared inline on Schematic, AddNode props optional vs SetProps required drives null-handling split

Sequence Diagram

sequenceDiagram
    participant C as Console
    participant SC as Schematic Client (TS)
    participant SV as Schematic Service (Go)
    participant DB as gorp DB
    participant SIG as Signals Channel
    participant C2 as Other Clients

    C->>SC: dispatch(key, actions, sessionKey)
    Note over SC: augmentWithEdgeSegments adds<br/>SetProps for moved edges
    SC->>SV: POST /schematic/dispatch
    SV->>DB: ChangeErr(ReduceAll(state, actions))
    DB-->>SV: updated Schematic
    SV->>SIG: Publish ScopedAction {key, sessionKey, actions}
    SIG-->>C: sy_schematic_set channel event
    Note over C: ACTION_LISTENER: skip if sessionKey == client.key
    SIG-->>C2: sy_schematic_set channel event
    C2->>C2: reduceAll(current, changed.actions)
Loading

Reviews (1): Last reviewed commit: "updated python client channel tests" | Re-trigger Greptile

Greptile also left 2 inline comments on this PR.

emilbon99 and others added 30 commits March 24, 2026 14:38
- Fix MigrationConfig to pass DB default codec (msgpack) instead of table
  override, preventing codec transition from crashing on upgrades with
  existing msgpack data
- Add inputCodec/outputCodec to TypedMigration for future schema change
  migrations that need version-specific codecs
- Change TypedMigration to single TransformFunc (removes auto/post split)
- Implement SkipRawFields/ReadRawField for raw binary field access
- Fix Oracle marshal plugin recursive helper error returns
- Build Oracle migrate plugin that generates migrate.gen.go for codec
  transitions, triggered by @go migrate annotation
- Wire generated migration functions into all 13 gorp entry services
- Fix service compilation errors from half-wired branch state

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Build snapshot package: creates versioned copies of all .oracle files
  preserving directory structure, with FileLoader that redirects imports
  to snapshot directory for isolated analysis
- Build oracle migrate command: loads old snapshot, analyzes current
  schemas, diffs, generates frozen types for changed types, creates
  new snapshot, then runs oracle sync
- Extend migrate plugin to detect schema changes by comparing old/new
  field lists and generate frozen types in migrations sub-package
- Handle circular dependency: unwrap parent-package type aliases to
  underlying primitives in frozen types
- Test: add description field to Workspace, oracle migrate generates
  WorkspaceV2 frozen type with correct uuid.UUID types

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Generate developer transform template in parent package (not sub-package)
  so it can reference both frozen types (via import) and current types
- Template maps migrations.WorkspaceV2 -> Workspace with TODO for new fields
- Fix snapshot FileLoader to handle subdirectory imports (arc/graph, etc.)
- Fix snapshot Create to preserve directory structure
- Filter parent-package imports from frozen types to prevent circular deps
- Test: filled in Workspace transform with Description default, compiles clean

Still needed: frozen codec generation, updated migrate.gen.go registration

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generate bounds checks before every field read in binary decoders. When
data is shorter than expected (old binary data with fewer fields), the
decoder returns early with remaining fields at zero values instead of
panicking.

This enables additive schema changes (field added at end) without frozen
types or frozen codecs. The current type and codec handle both old and
new binary data. Only field removals/type changes need frozen types.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implement Avro-style schema resolution that transforms binary data from
one field layout to another without frozen Go types. Fields are matched
by name between old and new layouts. Nested structs resolve recursively.

The resolver handles every encoding pattern the marshal plugin produces:
fixed-size primitives, length-prefixed strings/bytes/JSON, length-prefixed
nested structs, arrays, maps, soft/hard optional fields.

Tests verify: field copy, field addition, field removal, field reordering,
optional fields, recursive nested struct resolution, and arrays.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NewSchemaResolution(name, oldLayout, newLayout) creates a migration that
iterates all entries and resolves their binary data from one field layout
to another. Uses the generic byte resolver. No frozen Go types needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BuildLayout walks an Oracle resolved type tree and produces a
gorp.FieldLayout tree that describes the binary encoding format.
Handles all encoding patterns: primitives, enums, aliases, distinct
types, structs (with recursion detection), arrays, maps, optional
fields, and JSON/record fields.

This is the bridge between Oracle's schema representation and gorp's
schema-driven byte resolver.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rewrite migrate plugin to generate migrate.gen.go with both codec
  transition AND schema resolution migrations in the chain
- Schema resolution embeds FieldLayout literals directly in generated Go
  code, no frozen types needed
- Separate migrate plugin from sync registry (migrate owns migrate.gen.go,
  sync owns types.gen.go and codec.gen.go)
- Generate transform template for developer to set defaults after
  schema resolution
- Add layout-to-Go-literal serializer for embedding in generated code
- Test: Workspace with added description field generates correct migration
  chain with embedded old/new layouts. All tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix resolver to handle nested type changes inside arrays and maps,
  not just top-level struct fields. resolveArrayField and resolveMapField
  recursively resolve elements when their layouts differ.
- Fix schema change detection to use deep layout comparison instead of
  shallow field name matching. Changes to nested types at ANY depth
  (e.g., types.Param inside Arc) are now detected.
- Wire developer's transform function into migration chain via
  TypedMigration after SchemaResolution.
- Fix template to conditionally import context only when schema changes
  exist.
- Export LayoutsEqual for use by Oracle plugin.

Migration chain for schema changes is now:
  1. CodecTransition (msgpack -> binary)
  2. SchemaResolution (old layout -> new layout, byte-level)
  3. TypedMigration (developer sets defaults)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- LayoutsEqual now recursively compares Element (array), Key and Value
  (map) layouts, not just Fields. Fixes nested type changes inside
  arrays/maps not being detected.
- Fix layout builder to match marshal plugin's optionality logic: only
  set Optional/HardOptional when the codec actually uses a presence flag.
  Single ? on non-nilable types (like uuid?) has no presence flag.
- Add test for nested struct change inside array (verifies recursive
  element resolution works).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Only create a new schema snapshot when this is the first run (no
existing snapshot) or when migration files were actually written.
Running oracle migrate with no schema changes no longer creates
redundant snapshot copies.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace NewSchemaResolution + NewTypedMigration with single
  NewSchemaEvolution[E] that does schema resolution + optional developer
  transform in one step. One migration, one concept.
- Transform template now panics if not edited, forcing the developer to
  set defaults for new fields. Clear instructions on how to implement.
- Fix recursive type handling in resolver: copy raw bytes for cyclic
  structs instead of destroying data with empty resolution.
- Remove unused context import from migrate.gen.go template.
- Remove HasSchemaChanges from template data.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Show which template files need developer attention with ✏️ marker.
Helps developers know exactly which files to edit after running
oracle migrate.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add integration tests that encode with v1 codec, run schema resolver,
  decode with v2 codec, and verify data integrity. Proves the full
  pipeline works end-to-end.
- Add OpenTable integration test: write v1 binary data, open table with
  SchemaEvolution migration, verify entries are migrated with correct
  field values and developer-set defaults.
- Fix migrateOldPrefixKeys to use DB's default codec (msgpack) for
  computing old prefix instead of table's custom codec. Custom codecs
  that only handle their entry type panicked when passed a string for
  prefix computation. This also fixes the pre-existing structpb test
  failure.
- All 203 gorp tests now pass (0 failures).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add 27 layout builder tests using real Arc schema files. Verifies
  correct layout generation for all 35 nested types including recursive
  Param/Type, struct extension (Program extends IR + Output), arrays,
  maps, optional fields, and enums.
- Add nested struct change in array integration test: simulates adding
  a field to a type nested inside an array (the Arc/Param case).
- Add multiple sequential migration test: chains two SchemaEvolution
  migrations (v1→v2→v3) and verifies both run correctly.
- All 205 gorp tests pass. All 27 layout builder tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the runtime schema resolver approach with versioned codecs.
For each schema change, Oracle generates:
  - v{N}_types.gen.go: frozen Go type (WorkspaceV2)
  - v{N}_codec.gen.go: frozen binary codec (WorkspaceV2Codec)
  - v{N}_migrate.go: developer transform template (panic if not edited)
  - migrate.gen.go: TypedMigration[WorkspaceV2, Workspace] with versioned codecs

The frozen codec is generated by the marshal plugin's code generation
(new public GenerateCodecFile function), ensuring the frozen codec uses
the exact same encoding logic as the current codec. Type assertions
reference the frozen type (WorkspaceV2, not Workspace).

gorp stays encoding-agnostic: only TypedMigration with inputCodec/
outputCodec. No FieldLayout, Resolve, or SchemaEvolution needed for
the core migration path.

Migration chain: CodecTransition (msgpack->binary) then
TypedMigration[OldType, CurrentType](oldCodec, currentCodec, transform)

All 205 gorp tests pass. Core compiles clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add NameOverrides to marshal plugin's codeBuilder. When generating
  frozen codecs, nested Oracle-defined types are resolved as local
  versioned names (ParamV2 instead of types.Param). This makes frozen
  codecs self-contained for any schema change, including breaking
  changes to deeply nested types.
- Add buildNameOverrides: walks the entry type's dependency tree and
  maps every Oracle-defined struct to a versioned name.
- Add renderFrozenTypes: generates frozen type definitions for the entry
  AND all nested types, using overrides for type references.
- Add filterParentImports: prevents circular dependencies by filtering
  imports that reference the parent package.
- Export GenerateCodecFile from marshal plugin for migrate plugin reuse.
- Name overrides propagated to sub-codeBuilders for recursive helpers.

The versioned codec approach now handles:
- Simple types (Workspace: 4 fields, all primitives)
- Complex types with nested structs (via flatten + name overrides)
- Breaking changes at any nesting depth (frozen types for all affected)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Name override system works: 24 types discovered and mapped for Arc.
Frozen types generated with namespace-prefixed names (GraphNodeV3 vs
IrNodeV3) to avoid collisions. Name overrides propagated through
codeBuilder for versioned helper function names.

Known issues blocking compilation:
- Entry type name in registration uses un-overridden name (ArcV3 vs ArcArcV3)
- Frozen codec generated against resolution.Table types, not frozen Go
  types. Field access code uses original types (ir.Functions) but frozen
  type has versioned types ([]IrFunctionV3). Need deeper integration
  of name overrides into all codeBuilder code paths, not just goTypeName.
- StatusDetails reference unresolved (generic type parameter substitution)

The architecture (versioned codecs + flatten + name overrides) is correct.
The marshal plugin's codeBuilder needs more comprehensive override support
for array/map element types, struct field access, and make() calls.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix DistinctForm handling in resolveTypeWithOverrides (ir.Functions etc.)
- Fix namespace context per-type in collectFrozenTypes
- Add alias/distinct override propagation as second pass in buildNameOverrides
- Fix entry versioned name in migration registration (ArcArcV3 not ArcV3)
- Array aliases (Functions Function[]) now resolve to []IrFunctionV3

23 of 24 frozen types compile correctly. 2 remaining edge cases:
- Generic type parameter substitution (Status<StatusDetails> -> Details field)
- telem.TimeStamp distinct type casting in codec

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@emilbon99 emilbon99 changed the base branch from rc to sy-3990-server-side-metadata-importexport April 13, 2026 18:39
emilbon99 and others added 20 commits April 13, 2026 15:10
)

Replace map[T]bool and map[T]struct{} with set.Set, and improve set semantics
* SY-3964:  Add dot-based module syntax to arc (e.g. time.now(), error.panic{})

* SY-3964: Fix LSP completions for module qualified syntax like `time.` and `math.` with tests for filtering, TextEdit, and a guard test that catches missing KindFunction on new STL members

* SY-3964: Fix non-callable output param names on math.pow, string.len, string.concat, string.equal and mark state/string plumbing as internal with a guard test

* SY-3964: Analyzer, compiler, and runtime guard against `()` misuse in flow statements with a clear error instead of crashing the core

* SY-3964: Add tests
* [docs] preserve sidebar scroll position across page navigations
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants