Sy 3833 strongly type schematics#2190
Open
emilbon99 wants to merge 642 commits intosy-3990-server-side-metadata-importexportfrom
Open
Sy 3833 strongly type schematics#2190emilbon99 wants to merge 642 commits intosy-3990-server-side-metadata-importexportfrom
emilbon99 wants to merge 642 commits intosy-3990-server-side-metadata-importexportfrom
Conversation
…ynnax into sy-3816-oracle-migrations
- Fix MigrationConfig to pass DB default codec (msgpack) instead of table override, preventing codec transition from crashing on upgrades with existing msgpack data - Add inputCodec/outputCodec to TypedMigration for future schema change migrations that need version-specific codecs - Change TypedMigration to single TransformFunc (removes auto/post split) - Implement SkipRawFields/ReadRawField for raw binary field access - Fix Oracle marshal plugin recursive helper error returns - Build Oracle migrate plugin that generates migrate.gen.go for codec transitions, triggered by @go migrate annotation - Wire generated migration functions into all 13 gorp entry services - Fix service compilation errors from half-wired branch state Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Build snapshot package: creates versioned copies of all .oracle files preserving directory structure, with FileLoader that redirects imports to snapshot directory for isolated analysis - Build oracle migrate command: loads old snapshot, analyzes current schemas, diffs, generates frozen types for changed types, creates new snapshot, then runs oracle sync - Extend migrate plugin to detect schema changes by comparing old/new field lists and generate frozen types in migrations sub-package - Handle circular dependency: unwrap parent-package type aliases to underlying primitives in frozen types - Test: add description field to Workspace, oracle migrate generates WorkspaceV2 frozen type with correct uuid.UUID types Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Generate developer transform template in parent package (not sub-package) so it can reference both frozen types (via import) and current types - Template maps migrations.WorkspaceV2 -> Workspace with TODO for new fields - Fix snapshot FileLoader to handle subdirectory imports (arc/graph, etc.) - Fix snapshot Create to preserve directory structure - Filter parent-package imports from frozen types to prevent circular deps - Test: filled in Workspace transform with Description default, compiles clean Still needed: frozen codec generation, updated migrate.gen.go registration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generate bounds checks before every field read in binary decoders. When data is shorter than expected (old binary data with fewer fields), the decoder returns early with remaining fields at zero values instead of panicking. This enables additive schema changes (field added at end) without frozen types or frozen codecs. The current type and codec handle both old and new binary data. Only field removals/type changes need frozen types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implement Avro-style schema resolution that transforms binary data from one field layout to another without frozen Go types. Fields are matched by name between old and new layouts. Nested structs resolve recursively. The resolver handles every encoding pattern the marshal plugin produces: fixed-size primitives, length-prefixed strings/bytes/JSON, length-prefixed nested structs, arrays, maps, soft/hard optional fields. Tests verify: field copy, field addition, field removal, field reordering, optional fields, recursive nested struct resolution, and arrays. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NewSchemaResolution(name, oldLayout, newLayout) creates a migration that iterates all entries and resolves their binary data from one field layout to another. Uses the generic byte resolver. No frozen Go types needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BuildLayout walks an Oracle resolved type tree and produces a gorp.FieldLayout tree that describes the binary encoding format. Handles all encoding patterns: primitives, enums, aliases, distinct types, structs (with recursion detection), arrays, maps, optional fields, and JSON/record fields. This is the bridge between Oracle's schema representation and gorp's schema-driven byte resolver. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rewrite migrate plugin to generate migrate.gen.go with both codec transition AND schema resolution migrations in the chain - Schema resolution embeds FieldLayout literals directly in generated Go code, no frozen types needed - Separate migrate plugin from sync registry (migrate owns migrate.gen.go, sync owns types.gen.go and codec.gen.go) - Generate transform template for developer to set defaults after schema resolution - Add layout-to-Go-literal serializer for embedding in generated code - Test: Workspace with added description field generates correct migration chain with embedded old/new layouts. All tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix resolver to handle nested type changes inside arrays and maps, not just top-level struct fields. resolveArrayField and resolveMapField recursively resolve elements when their layouts differ. - Fix schema change detection to use deep layout comparison instead of shallow field name matching. Changes to nested types at ANY depth (e.g., types.Param inside Arc) are now detected. - Wire developer's transform function into migration chain via TypedMigration after SchemaResolution. - Fix template to conditionally import context only when schema changes exist. - Export LayoutsEqual for use by Oracle plugin. Migration chain for schema changes is now: 1. CodecTransition (msgpack -> binary) 2. SchemaResolution (old layout -> new layout, byte-level) 3. TypedMigration (developer sets defaults) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- LayoutsEqual now recursively compares Element (array), Key and Value (map) layouts, not just Fields. Fixes nested type changes inside arrays/maps not being detected. - Fix layout builder to match marshal plugin's optionality logic: only set Optional/HardOptional when the codec actually uses a presence flag. Single ? on non-nilable types (like uuid?) has no presence flag. - Add test for nested struct change inside array (verifies recursive element resolution works). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Only create a new schema snapshot when this is the first run (no existing snapshot) or when migration files were actually written. Running oracle migrate with no schema changes no longer creates redundant snapshot copies. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace NewSchemaResolution + NewTypedMigration with single NewSchemaEvolution[E] that does schema resolution + optional developer transform in one step. One migration, one concept. - Transform template now panics if not edited, forcing the developer to set defaults for new fields. Clear instructions on how to implement. - Fix recursive type handling in resolver: copy raw bytes for cyclic structs instead of destroying data with empty resolution. - Remove unused context import from migrate.gen.go template. - Remove HasSchemaChanges from template data. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Show which template files need developer attention with ✏️ marker. Helps developers know exactly which files to edit after running oracle migrate. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add integration tests that encode with v1 codec, run schema resolver, decode with v2 codec, and verify data integrity. Proves the full pipeline works end-to-end. - Add OpenTable integration test: write v1 binary data, open table with SchemaEvolution migration, verify entries are migrated with correct field values and developer-set defaults. - Fix migrateOldPrefixKeys to use DB's default codec (msgpack) for computing old prefix instead of table's custom codec. Custom codecs that only handle their entry type panicked when passed a string for prefix computation. This also fixes the pre-existing structpb test failure. - All 203 gorp tests now pass (0 failures). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add 27 layout builder tests using real Arc schema files. Verifies correct layout generation for all 35 nested types including recursive Param/Type, struct extension (Program extends IR + Output), arrays, maps, optional fields, and enums. - Add nested struct change in array integration test: simulates adding a field to a type nested inside an array (the Arc/Param case). - Add multiple sequential migration test: chains two SchemaEvolution migrations (v1→v2→v3) and verifies both run correctly. - All 205 gorp tests pass. All 27 layout builder tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the runtime schema resolver approach with versioned codecs.
For each schema change, Oracle generates:
- v{N}_types.gen.go: frozen Go type (WorkspaceV2)
- v{N}_codec.gen.go: frozen binary codec (WorkspaceV2Codec)
- v{N}_migrate.go: developer transform template (panic if not edited)
- migrate.gen.go: TypedMigration[WorkspaceV2, Workspace] with versioned codecs
The frozen codec is generated by the marshal plugin's code generation
(new public GenerateCodecFile function), ensuring the frozen codec uses
the exact same encoding logic as the current codec. Type assertions
reference the frozen type (WorkspaceV2, not Workspace).
gorp stays encoding-agnostic: only TypedMigration with inputCodec/
outputCodec. No FieldLayout, Resolve, or SchemaEvolution needed for
the core migration path.
Migration chain: CodecTransition (msgpack->binary) then
TypedMigration[OldType, CurrentType](oldCodec, currentCodec, transform)
All 205 gorp tests pass. Core compiles clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add NameOverrides to marshal plugin's codeBuilder. When generating frozen codecs, nested Oracle-defined types are resolved as local versioned names (ParamV2 instead of types.Param). This makes frozen codecs self-contained for any schema change, including breaking changes to deeply nested types. - Add buildNameOverrides: walks the entry type's dependency tree and maps every Oracle-defined struct to a versioned name. - Add renderFrozenTypes: generates frozen type definitions for the entry AND all nested types, using overrides for type references. - Add filterParentImports: prevents circular dependencies by filtering imports that reference the parent package. - Export GenerateCodecFile from marshal plugin for migrate plugin reuse. - Name overrides propagated to sub-codeBuilders for recursive helpers. The versioned codec approach now handles: - Simple types (Workspace: 4 fields, all primitives) - Complex types with nested structs (via flatten + name overrides) - Breaking changes at any nesting depth (frozen types for all affected) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Name override system works: 24 types discovered and mapped for Arc. Frozen types generated with namespace-prefixed names (GraphNodeV3 vs IrNodeV3) to avoid collisions. Name overrides propagated through codeBuilder for versioned helper function names. Known issues blocking compilation: - Entry type name in registration uses un-overridden name (ArcV3 vs ArcArcV3) - Frozen codec generated against resolution.Table types, not frozen Go types. Field access code uses original types (ir.Functions) but frozen type has versioned types ([]IrFunctionV3). Need deeper integration of name overrides into all codeBuilder code paths, not just goTypeName. - StatusDetails reference unresolved (generic type parameter substitution) The architecture (versioned codecs + flatten + name overrides) is correct. The marshal plugin's codeBuilder needs more comprehensive override support for array/map element types, struct field access, and make() calls. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix DistinctForm handling in resolveTypeWithOverrides (ir.Functions etc.) - Fix namespace context per-type in collectFrozenTypes - Add alias/distinct override propagation as second pass in buildNameOverrides - Fix entry versioned name in migration registration (ArcArcV3 not ArcV3) - Array aliases (Functions Function[]) now resolve to []IrFunctionV3 23 of 24 frozen types compile correctly. 2 remaining edge cases: - Generic type parameter substitution (Status<StatusDetails> -> Details field) - telem.TimeStamp distinct type casting in codec Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…3-strongly-type-schematics
…ithub.com/synnaxlabs/synnax into sy-3833-strongly-type-schematics
* SY-3964: Add dot-based module syntax to arc (e.g. time.now(), error.panic{})
* SY-3964: Fix LSP completions for module qualified syntax like `time.` and `math.` with tests for filtering, TextEdit, and a guard test that catches missing KindFunction on new STL members
* SY-3964: Fix non-callable output param names on math.pow, string.len, string.concat, string.equal and mark state/string plumbing as internal with a guard test
* SY-3964: Analyzer, compiler, and runtime guard against `()` misuse in flow statements with a clear error instead of crashing the core
* SY-3964: Add tests
* [docs] preserve sidebar scroll position across page navigations
…3-strongly-type-schematics
…ithub.com/synnaxlabs/synnax into sy-3833-strongly-type-schematics
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Go to the
Previewtab and select the appropriate template for your pull request:Greptile Summary
This PR strongly types the schematic data model by replacing the opaque
Data msgpack.EncodedJSONblob in v54 with explicitNodes,Edges,Props,Legend, andAuthorityfields — both server-side (Go) and client-side (TypeScript v6 state). Node/edge/props data is now owned by the server; the console Redux slice retains only UI state (selection, control status, toolbar, viewport). Action-based mutation (SetNodePosition,AddNode,SetEdge,SetProps, etc.) now drives server-side state transitions with real-time fan-out via thesy_schematic_setsignal channel.extractLegendinmigrate.gowrites to a nilColorsmap and callscolor.MustFromHex(panics on any malformed hex string) — any schematic with legend colors that triggers migration will crash the server on upgrade.Confidence Score: 4/5
Not safe to merge until the nil map + MustFromHex panics in extractLegend are fixed — these will crash the server during first upgrade migration of any schematic with legend colors.
The architectural restructuring is well-designed and the generated code is correct throughout. A single P1 bug in migrate.go will cause a runtime panic in the migration path for existing schematics with legend colors. All other files are clean.
core/pkg/service/schematic/migrate.go — extractLegend writes to nil Colors map and calls color.MustFromHex on unvalidated user data.
Vulnerabilities
No security concerns identified beyond the migration-time panic from
color.MustFromHexon unvalidated input (addressed as P1 inline). Auth/authz, injection vectors, and secrets handling are unchanged from prior versions.Important Files Changed
Sequence Diagram
sequenceDiagram participant C as Console participant SC as Schematic Client (TS) participant SV as Schematic Service (Go) participant DB as gorp DB participant SIG as Signals Channel participant C2 as Other Clients C->>SC: dispatch(key, actions, sessionKey) Note over SC: augmentWithEdgeSegments adds<br/>SetProps for moved edges SC->>SV: POST /schematic/dispatch SV->>DB: ChangeErr(ReduceAll(state, actions)) DB-->>SV: updated Schematic SV->>SIG: Publish ScopedAction {key, sessionKey, actions} SIG-->>C: sy_schematic_set channel event Note over C: ACTION_LISTENER: skip if sessionKey == client.key SIG-->>C2: sy_schematic_set channel event C2->>C2: reduceAll(current, changed.actions)Reviews (1): Last reviewed commit: "updated python client channel tests" | Re-trigger Greptile