RFC: Ebitengine Debug Protocol (EDP) — Engine Observability for AI Agents

[myzhan]

RFC: Ebitengine Debug Protocol (EDP) — Engine Observability for AI Agents

Summary

This proposal introduces the Ebitengine Debug Protocol (EDP), a built-in HTTP JSON API that exposes engine runtime state to external tools. The primary motivation is to enable AI coding agents (e.g., LLM-based copilots) to observe and verify the correctness of game code they generate — effectively giving AI "eyes" into a running Ebitengine game.

Motivation

The Problem

AI coding assistants are increasingly capable of generating game code, but they face a fundamental limitation: they cannot see the result. After generating Ebitengine code, an AI agent can verify that the code compiles, but has no way to answer:

• Is the sprite rendered at the correct position?
• Is the FPS stable or did the code introduce a performance regression?
• Does pressing Space actually make the character jump?
• Are draw calls being batched efficiently, or did a change break atlas packing?

This is analogous to a blind programmer writing UI code — the compiler says it's fine, but the visual result is entirely unknown.

The Opportunity

Ebitengine's architecture is uniquely well-suited for this kind of observability:

1. Centralized state with thread-safe APIs — TPS, FPS, window properties, and input state are already accessible via concurrent-safe public functions.
2. The Game interface is a natural interception point — Update() / Draw() / Layout() can be wrapped without modifying user code.
3. Go's standard library — net/http and goroutines make it trivial to run an HTTP server alongside the game loop with zero external dependencies.
4. Existing debug infrastructure — internal/debug, imageDumper, DebugInfo, and ebitenutil.DebugPrint show that the engine already values debuggability.

Prior Art

Engine	Tool	Protocol	Standalone?
Unreal Engine	Remote Control API	HTTP/WebSocket JSON	✅
Unity	Remote Profiler	Private binary TCP	❌ (needs Editor)
Godot	Remote Debugger	Private TCP	❌ (needs Editor)
Babylon.js	Inspector	In-browser DOM	✅
Ebitengine (proposed)	EDP	HTTP JSON	✅

No existing game engine provides a debug protocol specifically designed for AI agent consumption. EDP would be the first.

Design

Architecture: Two-Layer Observability

EDP operates at two layers to provide comprehensive coverage:

┌─────────────────────────────────────────────────────┐
│  User Game Code (Game interface implementation)      │
├─────── User-Layer Observation ──────────────────────┤
│  exp/debug.GameWrapper                               │
│  Wraps Game interface to intercept Update/Draw       │
│  → Screenshot capture, pause/resume, input sim       │
├─────────────────────────────────────────────────────┤
│  ebiten Public API (run.go, image.go, input.go)      │
├─────── Engine-Layer Observation ────────────────────┤
│  internal/debugger (statistics collector)            │
│  Instrumented in:                                    │
│  → internal/ui/context.go (frame timing)             │
│  → internal/graphicscommand/commandqueue.go          │
│    (draw calls, command merging, vertex counts)      │
├─────────────────────────────────────────────────────┤
│  internal/graphicsdriver (Metal/OpenGL/DirectX)      │
└─────────────────────────────────────────────────────┘

User-layer (exp/debug/wrapper.go): Wraps the Game interface. Detects issues in user game logic — wrong rendering, broken input handling, etc.

Engine-layer (internal/debugger): Instruments engine internals. Detects issues in the engine itself — frame scheduling anomalies, draw call explosion, command merging failures, etc.

Protocol Design

EDP uses HTTP JSON (no external dependencies). Two access patterns are supported:

1. REST endpoints (GET) — Simple curl-friendly access for quick queries
2. JSON-RPC (POST to /debug/rpc) — Structured access for programmatic use

Protocol Domains

Domain	Methods	Description
Runtime	getState, setTPS, pause, resume	FPS, TPS, tick, graphics library, GPU memory
Window	getState	Size, position, fullscreen, focus, vsync, scale factor
Input	getState, simulateKey	Pressed keys, cursor position, mouse buttons, wheel
Engine	getFrameStats, getRecentFrames, getCumulativeStats	Per-frame draw calls, Update/Draw timing, vertices, command merging, frame skips
Screenshot	GET /debug/screenshot	Capture current screen as PNG

Example Interactions

# Get runtime state
curl http://localhost:9222/debug/runtime.getState
# → {"actualFPS":59.94,"actualTPS":60.01,"targetTPS":60,"tick":3842,...}

# Capture screenshot
curl http://localhost:9222/debug/screenshot -o screenshot.png

# Get engine-level frame stats
curl http://localhost:9222/debug/engine.getFrameStats
# → {"frame":{"tick":3842,"updateCount":1,"drawCalls":12,"mergedDrawCalls":8,
#     "vertices":480,"updateDurationNs":45000,"drawDurationNs":890000,...},
#    "cumulative":{"totalFrames":3842,"totalDrawCalls":46104,...}}

# Pause game logic
curl http://localhost:9222/debug/runtime.pause

# JSON-RPC style
curl -X POST http://localhost:9222/debug/rpc \
  -d '{"id":1,"method":"Runtime.getState"}'

User-Facing API

Enabling EDP requires one line of code and zero configuration:

package main

import (
    "github.com/hajimehoshi/ebiten/v2"
    "github.com/hajimehoshi/ebiten/v2/exp/debug"
)

func main() {
    game := &MyGame{}
    debugGame := debug.Enable(game, &debug.Options{Port: 9222})
    ebiten.RunGame(debugGame)
}

Zero-Cost When Disabled

• The internal/debugger collector checks enabled.Load() (atomic bool) at the top of every recording function. When disabled, the overhead is a single atomic load per call — effectively zero.
• The exp/debug package is opt-in. If not imported, no HTTP server is started and no Game wrapping occurs.
• Engine-layer instrumentation points are always compiled in but gated by the atomic check, avoiding the need for build tags for basic usage.

Implementation

New Files

File	Purpose
exp/debug/server.go	HTTP server with JSON-RPC and REST endpoints
exp/debug/domains.go	Protocol domain handlers (Runtime, Window, Input, Engine)
exp/debug/wrapper.go	GameWrapper that wraps Game interface for user-layer observation
exp/debug/image.go	Helper to convert raw RGBA pixels to image.Image for PNG encoding
internal/debugger/debugger.go	Engine-level statistics collector with ring buffer history

Modified Files

File	Change
internal/ui/context.go	Added debugger.BeginFrame/EndFrame in updateFrame, Update/Draw timing in updateFrameImpl
internal/graphicscommand/commandqueue.go	Added debugger.RecordDrawCall/RecordMergedDrawCall/RecordVerticesAndIndices in EnqueueDrawTrianglesCommand

Key Design Decisions

1. HTTP instead of WebSocket for MVP: Simpler, works with curl, no external dependencies. WebSocket can be added later for event push (e.g., real-time FPS streaming).

2. exp/debug package location: Placed under exp/ to signal experimental status, consistent with exp/textinput.

3. internal/debugger vs extending internal/debug: Created a separate package to avoid conflating compile-time debug logging (internal/debug, gated by build tags) with runtime statistics collection (internal/debugger, gated by atomic bool).

4. Screenshot via channel synchronization: ReadPixels must be called from the Draw goroutine. The wrapper uses an atomic flag + channel to request a screenshot from the HTTP handler and wait for the Draw goroutine to fulfill it.

5. Ring buffer for frame history: Fixed-size (120 frames ≈ 2 seconds at 60 FPS) to bound memory usage while providing enough history for trend analysis.

Future Work

The following features are intentionally deferred from the MVP but are natural extensions:

Near-Term

• WebSocket event push: Real-time streaming of FPS, input events, and frame stats
• Semantic scene snapshots: Instead of raw pixels, return structured descriptions of draw calls (position, source image, transform) — enabling AI to "understand" the scene without vision models
• Input replay: Accept a sequence of timed input events and replay them, returning state snapshots at specified ticks — enabling automated testing of game logic
• Audio state: Expose audio.Player state (playing, position, volume) via a registered player registry

Long-Term

• Assertion API: AI agents submit assertions ("FPS > 55", "object at (320, 240)") and receive pass/fail results
• State diff: Save a baseline state, make changes, and diff — enabling AI to verify that a code change had the intended effect
• Hot reload integration: Detect code changes, rebuild, and automatically verify via assertions
• GPU-level instrumentation: Expose graphics driver errors, shader compilation failures, and resource lifecycle events

Compatibility

• No breaking changes: All modifications are additive. Existing code compiles and runs identically.
• No new external dependencies: Uses only Go standard library (net/http, encoding/json, image/png).
• Platform support: Works on all platforms where Ebitengine runs (the HTTP server runs in a goroutine alongside the game loop).
• Build tag compatibility: Engine-layer instrumentation is always compiled but runtime-gated, avoiding interaction with existing ebitenginedebug build tag.

Open Questions

1. Package location: Should this live under exp/debug or directly under debug? The exp/ prefix signals instability but may discourage adoption.

2. Default port: 9222 is chosen as a nod to Chrome DevTools Protocol. Is there a risk of conflict?

3. Security: The HTTP server binds to all interfaces by default. Should it default to localhost only? Should there be an authentication mechanism?

4. Performance budget: The atomic bool check adds ~1ns per instrumentation point. Is this acceptable for the engine's performance goals, even when EDP is not enabled?

5. internal/debugger naming: The name debugger may conflict with Go's runtime/debug in developer mental models. Alternatives: internal/edp, internal/enginestats, internal/observe.

References

• Chrome DevTools Protocol
• Unreal Engine Remote Control API
• Godot Remote Debugger
• OpenAI Gymnasium (AI-environment interaction paradigm)

[hajimehoshi]

Though this seems interesting, I don't want to talk with an AI in this discussion forum

[myzhan]

Inspired by playright and chrome debugger protocol, I make claude to write a demo in ebiten, maybe it worth a try?

[hajimehoshi]

Feel free to experiment, but I won't be merging this.

[kendfss]

had you considered using functional options instead of a struct?