Skip to content

feat(extensions): add conversation-compiler extension#28

Open
ehfazrezwan wants to merge 15 commits intodevfrom
feature/conversation-compiler
Open

feat(extensions): add conversation-compiler extension#28
ehfazrezwan wants to merge 15 commits intodevfrom
feature/conversation-compiler

Conversation

@ehfazrezwan
Copy link
Copy Markdown
Owner

Summary

  • Adds the conversation-compiler NeuralScape extension implementing Karpathy's "LLM Wiki" + coleam00's "Claude Memory Compiler" pattern
  • Flush engine: Extracts facts, decisions, preferences, and patterns from conversation turns via Gemini, stores in NeuralScape, and writes daily logs to Obsidian vault
  • Compiler: Synthesizes daily logs into session summaries, project pages, decision records, and research articles with idempotent re-compilation support
  • Lint tool: 7 health checks (broken links, orphans, stale pages, missing cross-refs, LLM-powered contradictions, data gaps, index drift)
  • Query tool: Index-guided vault retrieval with optional file-back to vault
  • API routes: 5 endpoints at /v1/extensions/conversation-compiler/ (flush, compile, query, lint, status)
  • ARQ integration: 2 new worker tasks + periodic auto-compile cron job
  • 51 unit tests covering writer, extraction parsing, lint checks, schemas, config, and extension class — all passing alongside existing 247 tests

Depends on

  • feature/extension-interface (merged into this branch)

Test plan

  • All 298 tests pass (uv run python -m pytest tests/ --ignore=tests/test_async_pipeline.py -v)
  • Manually test flush endpoint with a real conversation turn
  • Verify daily log files created in Obsidian vault
  • Test compile endpoint and verify article generation
  • Test lint endpoint on populated vault
  • Verify auto-compile cron fires after configured hour

…fest

Define the interface all NeuralScape extensions must implement:
- ExtensionManifest (Pydantic model with name, version, description, hooks)
- NeuralscapeExtension (runtime-checkable Protocol with startup, shutdown,
  on_event, and get_routes methods)
Define canonical event types (conversation_turn, session_start,
session_end, memory_stored, compile_requested) as an enum with
corresponding Pydantic payload models for type-safe event dispatch.
ExtensionRegistry handles:
- Auto-discovery from extensions/ subdirectories
- Explicit registration via NEURALSCAPE_EXTENSIONS env var
- Startup/shutdown lifecycle with graceful failure handling
- Route mounting at /v1/extensions/<name>/
- Event broadcasting to extensions matching manifest.hooks
Integrate ExtensionRegistry into main.py:
- Discover and start extensions during lifespan startup
- Mount extension routes onto the app
- Shutdown extensions during lifespan teardown
- Add GET /v1/extensions endpoint to list registered extensions
- Add POST /v1/extensions/events endpoint for external event posting
Cover the extension protocol, step-by-step creation guide, lifecycle
(discovery → startup → events → shutdown), route mounting, event
listening, configuration via env vars, and a skeleton example.
30 tests covering registration, lifecycle (startup/shutdown), event
dispatch, route mounting, discovery (local + env var), graceful failure
handling, /v1/extensions listing endpoint, event schemas, and protocol
compliance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant