Open
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub. |
- Add result saving to JSON files in results/ directory - Add formatted metrics display in console output - Reduce default run counts for faster iteration during development - Fix Circom v2 placeholder stats to include all TimingStats fields - Refactor Noir provider to parse bb gates JSON output directly - Fix capture group types to use number[][] for multi-group patterns
- Implement collect-results.ts to aggregate individual benchmark files into comparison.json with hardware specs and tool versions - Add Table 3 (Noir Backend Performance) to LaTeX output with ACIR opcodes, gates, prove/verify times, and proof size - Add Table 4 (Scaling Analysis) showing metrics across input lengths - Add helper functions for formatting and pattern extraction - Generate both markdown and LaTeX outputs with siunitx formatting
Display a formatted table summary when collecting benchmark results, showing the exact regex pattern and test input string used for each benchmark. This improves visibility into benchmark runs without needing to open JSON/Markdown output files. Changes: - Add PatternMetadata interface for pattern display data - Load regex patterns from graph JSON files - Add sample input strings for all benchmark patterns - Display results grouped by pattern with console.table() - Escape control characters (\r\n) for terminal readability 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Move ptau URL/filename from hardcoded constants to benchmark.json - Auto-calculate maxConstraints from filename (e.g., pot18 = 2^18) - Upgrade default from pot16 (65K) to pot18 (262K constraints) - Add PtauConfig type definition - Update providers to use async getMaxConstraints()
- Rename ScalingDataPoint fields to distinguish v1/v2 Circom data - Add circomV1Constraints and circomV1ProveMs optional fields - Update console summary to show all three providers with clear labels - Update Markdown tables with "Circom v1 (DFA)", "Circom v2 (NFA)", "Noir v2 (UltraHonk)" headers - Update LaTeX tables with explicit provider names - Include v1 Circom data in scaling analysis table
- Separate circuit size and proving time into distinct tables for clarity - Add pattern definitions table with regex and sample inputs - Add escaping helpers for markdown and LaTeX special characters - Upgrade PTAU from pot18 to pot21 for larger circuits - Fix unnecessary field deletion in circom-v2 provider
Measure actual peak RSS for each benchmark phase using /usr/bin/time, replacing file-size-based estimation with accurate runtime measurements. - Add memory.ts utility with macOS/Linux platform detection - Add MemoryStats and PhaseMemory interfaces to types - Update all providers with memory tracking per phase - Rename executeMs to witnessGenMs in Noir for consistency - Add --no-memory CLI flag to disable profiling - Update result collection and output generation - Add memory tables to Markdown and LaTeX outputs - Add unit tests for memory utilities - Document memory profiling in README
… circuits Add complexity-based (simple/medium/complex/v2-only) and feature-based classification to pattern definitions. Create decomposed regex configs and generate Circom v2 + Noir v2 circuits for all new patterns.
…nfig Test all 15 patterns against v1 DFA compiler. Only fixed_range_quantifier fails (Accept Nodes Error). Update worktree.ts to use commit hash from config instead of hardcoded main. Add v1 provider sample inputs for all new patterns.
Rewrite generate-outputs.ts to support complexity-grouped tables, V1 vs V2 comparison, V1 compatibility analysis, V2-only patterns, summary statistics by complexity level, and separate Noir details. Also adds 512-byte input length to benchmark config.
…atterns Add sample_haystacks for 11 new benchmark patterns and regenerate Noir circuit test functions from the sample data.
… and fix input fallback - collect-results.ts was loading v1-compatibility.json as a benchmark result, causing undefined pattern name error - circom-v2 provider now falls back to pattern.sampleInput instead of hardcoded 'b' for new patterns - Same fallback fix applied to circom-v1 provider
The 11 new benchmark pattern circuits were generated but not added to noir/src/templates/circuits/mod.nr, causing nargo to fail with "Could not resolve" errors during Noir benchmarks.
…shutdown Replace per-process SIGINT/SIGTERM handlers with a centralized AbortController. All Bun.spawn calls now receive the abort signal, benchmark loops check isAborted() to break early, and the first Ctrl+C triggers graceful unwinding through finally blocks.
… at all sizes Replace hardcoded sample inputs with a configurable scaling system that fills each target input length with regex-matching content. Three strategies: repeat (tile template), extend (grow match portion), and pad-with-match (anchored template + filler). Each pattern in patterns.json now declares inputTemplate and scalingStrategy. All three providers use generateScaledInput() instead of static lookup tables. Output reports include scaling methodology, actual content lengths, and per-pattern strategy tables. Also improves memory measurement: separates command stderr from /usr/bin/time output for reliable RSS parsing, records data from non-zero exits (common with nested shell wrapping), and adds --no-memory flag support via BENCH_NO_MEMORY env var.
Use accurate compiler backend names (DFA/NFA) instead of version numbers (v1/v2) throughout benchmark configs, output generation, and type definitions for clarity in academic context.
Increase minRuns to 5 and maxRuns to 10 for better statistical significance. Add missing generateScaledInput imports to circom and noir providers. Add docs/ to gitignore.
Add compilerCommitHash and toolVersions fields to individual benchmark result files for reproducibility. Rename all provider identifiers from v1/v2 to DFA/NFA to better describe the algorithm difference. - Extend BenchmarkProvider interface with getCommitHash() and getToolVersions() - Move BenchmarkResult type from bench.ts to shared types.ts - Rename providers: circom-v1→circom-dfa, circom-v2→circom-nfa, noir-v2→noir-nfa - Update all type fields, config keys, CLI args, scripts, and README - Add compilerVersions aggregation to collect-results.ts
Remove ~120 lines of dead code: isOk, isErr, unwrap, unwrapOr, map, flatMap from errors.ts; runHyperfineBatch, measureSync, hashFile, proveAndVerify from utils. None had any callers in the codebase.
…code Create two new shared modules (exec.ts, project.ts) and centralize duplicated functions across 7 files (~19 duplicate instances removed): - execAsync/execCommand: unified in utils/exec.ts with configurable options (useNvm, errorFactory) to preserve each call site's behavior - wrapCommandWithNvm/getNvmWrappedCommand: unified with bashWrap option - getProjectRoot/getGitCommitHash: extracted to utils/project.ts - defaultMemoryStats: removed local copy from collect-results.ts - getToolVersions: Circom sync version centralized in hardware.ts - measureWitnessGeneration: promoted to shared utility in snarkjs.ts Net reduction: ~300 lines. No behavioral changes.
…teStats Variance was dividing by n (population variance) instead of n-1 (sample variance). With minRuns=5 this underestimates stddev by ~12%. Added n>1 guard to avoid division by zero on single measurements.
Validate git ref against a strict pattern and quote it in the shell command to prevent injection via a malicious benchmark.json value.
Rename remaining v1/v2 references in generate-outputs.ts comments and reductionPct parameter names to use DFA/NFA terminology, and update the module docstring in src/index.ts.
The PTAU file download performed no integrity verification after fetching a ~2GB file. Add streaming SHA256 verification for both fresh downloads and cached files, with automatic re-download on mismatch.
Replace the abstract class with direct interface implementation since all methods were abstract except an empty cleanup() that every provider overrides. Also consolidate NoirCircuitInput and MemoryStats into types.ts.
…benchmarks-package feat(benchmarks): add benchmarking suite for DFA vs NFA comparison
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Replaces the old Rust compiler and extensive Circom tests with a new Bun/TypeScript scripts suite (gen-regex, gen-inputs), shared utilities, and Jest-based tests.
circom/scripts/gen-regex.ts,noir/scripts/gen-regex.ts,noir/scripts/gen-inputs.ts.utils/*) for logging, file ops, subprocess, and types.tsconfig.json,jest.config.ts,jest.setup.ts,bun.lock, andpackage.json.scripts/__tests__for script workflows and utilities.packages/compiler/*).packages/circom/tests/*.Written by Cursor Bugbot for commit 3d37c31. This will update automatically on new commits. Configure here.
Summary by CodeRabbit
New Features
Infrastructure & Build
Documentation
Tests
Chores