fix(ci): publish headroom-core-py binary wheels — fixes #355#357
Closed
chopratejas wants to merge 6 commits intomainfrom
Closed
fix(ci): publish headroom-core-py binary wheels — fixes #355#357chopratejas wants to merge 6 commits intomainfrom
chopratejas wants to merge 6 commits intomainfrom
Conversation
… (issue #355) v0.20.13 shipped to PyPI as `headroom_ai-0.20.13-py3-none-any.whl` — pure Python, no Rust extension. The release workflow's `Build Python package` step runs `python -m build`, which invokes hatch only; maturin was never called, so `headroom._core` was never compiled or shipped. With A0's fail-loud startup check, users hit `event=rust_core_missing reason=ModuleNotFoundError action=exit_78`. Reported by @SvenMeyer in #355. Fix: 1. .github/workflows/release.yml — new `build-rust-wheels` job that runs maturin on a 4-platform matrix (linux x86_64, linux aarch64, macos x86_64, macos aarch64) for cp310/cp311/cp312/cp313. Sets PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 so PyO3 0.22.6 builds work on Python 3.14 until we bump PyO3 (tracked separately). `publish-pypi` now depends on this job and merges the rust wheels into `dist/` before the OIDC PyPI upload. 2. scripts/version-sync.py — sync the same release version into `crates/headroom-py/Cargo.toml` and `crates/headroom-py/pyproject.toml` so the wheel filename matches the headroom-ai version. Also rewrites the new `headroom-core-py>=…` dep line so `pip install headroom-ai` resolves the correct platform wheel. 3. pyproject.toml — add `"headroom-core-py>=0.20.13"` to runtime deps. For users stuck on v0.20.13: - Build from source with `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` - OR `HEADROOM_REQUIRE_RUST_CORE=false` for degraded Python-only mode.
Two PR-CI failures from the runtime-dep declaration:
1. workflow-validation: actionlint rejected `macos-13` — deprecated
runner label. Replaced with `macos-15-intel` (per actionlint's list
of valid macOS labels).
2. test-agno / test-extras / docker-native-e2e / test (matrix):
`pip install -e .` failed with
ERROR: Could not find a version that satisfies the requirement
headroom-core-py>=0.20.13 (from headroom-ai)
Chicken-and-egg: this PR is what publishes headroom-core-py to
PyPI, so PR-CI install can't resolve the constraint from PyPI yet.
Fix:
* New composite action `.github/actions/build-rust-core-wheel/`:
builds the headroom-core-py wheel locally via maturin into a
`wheelhouse/` directory and exports `PIP_FIND_LINKS=$wheelhouse`
to GITHUB_ENV. Every subsequent `pip install` in the same job
resolves the constraint from the local wheelhouse.
* Lower the runtime-dep floor from `>=0.20.13` to `>=0.1.0` so
freshly-built local wheels (Cargo.toml `version = "0.1.0"`) satisfy
the constraint without needing version-sync to mutate
pyproject.toml mid-build. version-sync.py still rewrites the floor
on each release, so users `pip install`ing from PyPI get a
release-tied pairing.
* Wire the composite action into `test`, `test-extras`, `test-agno`
jobs in ci.yml — runs BEFORE the existing `pip install -e .` step.
* Restructure Dockerfile: build the Rust wheel BEFORE the `uv pip
install ".[${HEADROOM_EXTRAS}]"` stub install. Set
`PIP_FIND_LINKS=/build/wheels` + `UV_FIND_LINKS=/build/wheels` as
ENV so both pip and uv pick up the local wheel.
Net behavior: every CI install path that needs `headroom-core-py`
gets it from a freshly-built local wheel; production installs from
PyPI continue to work normally once the next release lands.
Two follow-up failures from the runtime-dep declaration: 1. Python 3.10 in the composite action `python -c 'import tomllib'` fails on 3.10 (tomllib is 3.11+). Replaced with grep/sed on pyproject.toml — version line is `version = "..."`, regex extraction is portable across all supported Python versions. 2. e2e/wrap/Dockerfile and e2e/init/Dockerfile both do `pip install -e ".[proxy]"` from source. Same chicken-and-egg as test-agno: they can't resolve `headroom-core-py>=0.1.0` from PyPI yet. Fix: CI's `docker-native-e2e` job now runs the build-rust-core-wheel composite action BEFORE the e2e docker builds. The action populates `wheelhouse/` in the workspace. Both e2e Dockerfiles now `COPY wheelhouse /workspace/wheelhouse` and set `PIP_FIND_LINKS=/workspace/wheelhouse`. pip finds the local wheel and resolves the constraint. `wheelhouse/.gitkeep` + `wheelhouse/README.md` ensure the directory exists in clean checkouts so the `COPY` doesn't fail when developers `docker build` locally without first running the composite action. `.gitignore` excludes `wheelhouse/*.whl` so accidentally-committed build artifacts don't get tracked.
…p-installs Round 3 of #355 fallout: the runtime dep on `headroom-core-py` breaks every job that does `pip install -e .` (or installs an extras set) before the build-rust-core-wheel composite action has run. Round 1 covered ci.yml's test/test-extras/test-agno/docker-native-e2e jobs; round 2 added the wheelhouse copy to e2e/{wrap,init}/Dockerfile. This round catches the remaining workflows the prior rounds missed: - `headroom-e2e-setup` composite action — used by init-native-e2e.yml and the broader e2e harness. Embeds the wheel build before its `pip install -e ".[proxy]"` step so every consumer inherits the fix. - `init-e2e.yml` + `wrap-e2e.yml` — standalone docker-build jobs. The Dockerfiles already COPY `wheelhouse/` and set PIP_FIND_LINKS, but the workflows never populated the directory. Now they do. - `rust.yml` parity-nightly — `pip install -e .` inside a venv. The venv inherits PIP_FIND_LINKS from $GITHUB_ENV. - `eval.yml` smoke-test + weekly-suite — `pip install -e ".[all]"`. - `publish.yml` SBOM step — `pip install -e ".[proxy]"`. ci.yml's docker-native-e2e job already pre-builds the wheelhouse; the duplicate Dockerfile builds in the e2e/{wrap,init} jobs of ci.yml continue to work via the same mechanism. Refs: #355
PR #357 was still failing on docker-init-e2e / docker-wrap-e2e with: ERROR: Could not find a version that satisfies the requirement headroom-core-py>=0.9.1 (from versions: none) The wheel WAS in /workspace/wheelhouse — but tagged `manylinux_2_38_x86_64`. PyO3/maturin-action with `manylinux: auto` (default) uses host glibc; Ubuntu 24.04 runners ship glibc 2.39, so maturin tags wheels at the `2_38` floor. The `node:22-bookworm` base image (Debian 12) has glibc 2.36 — pip rejects the incompatible wheel silently and reports "from versions: none". Fix: pin to `manylinux: 2_28` for Linux builds, matching the production matrix in release.yml. 2.28 is RHEL 8 era, broad enough to cover every base image the e2e/* Dockerfiles touch (bookworm, jammy, etc.). macOS and Windows builds are gated separately since the manylinux tag is a Linux-only concept. Refs: #355
Round 5 of #355 fallout. With the manylinux pin in 986341a, maturin runs inside `quay.io/pypa/manylinux_2_28_x86_64` (AlmaLinux 8). That container ships without OpenSSL dev headers, so the next failure was: error: failed to run custom build command for `openssl-sys v0.9.114` 💥 maturin failed Error: The process '/usr/bin/docker' failed with exit code 1 `openssl-sys` arrives via `native-tls`, which is pulled transitively by `hf-hub` (used by `fastembed`) and `ureq` (build-dep of `ort-sys`). Our workspace pins `reqwest` to `rustls-tls`, but cargo's feature unification means a single crate enabling openssl flips it on for everyone. Severing that chain is a bigger refactor (each crate's features need auditing); the fast fix is to give the build container what it needs. `before-script-linux` runs inside the manylinux container before maturin. `yum install -y openssl-devel pkgconfig perl-IPC-Cmd` covers headers, pkg-config detection, and the autotools probe IPC-Cmd dep. Refs: #355
3 tasks
pull Bot
pushed a commit
to pythoninthegrass/headroom
that referenced
this pull request
May 3, 2026
Eliminates the dual-package architecture that was the root cause of chopratejas#355. `pip install headroom-ai` now produces ONE wheel containing both the Python source (headroom/*.py) and the compiled Rust extension (headroom/_core.so). No more separate `headroom-core-py` package, no more chicken-and-egg with PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action plumbing in CI. This is the canonical pattern used by cryptography, polars, ruff, pydantic-core, and other Rust-as-core Python packages. Honors the "Rust as core engine" direction. ## What changed - pyproject.toml: `[build-system]` swapped from hatchling to maturin. `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."` picks up the root `headroom/` package directly (dashboard HTML templates and other non-Python files included automatically). - crates/headroom-py/pyproject.toml: deleted. The crate is no longer a separate published package; its Cargo.toml stays as the cdylib build target invoked via `[tool.maturin] manifest-path`. - crates/headroom-py/python/: deleted (placeholder layout for the old separate package). ## CI updates - ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust toolchain set up before `pip install -e .` (which now invokes maturin via build-system). Removed the "build wheel + symlink .so" dance. `build` job swapped from `python -m build` (hatch) to `maturin build` + `maturin sdist`. - release.yml: collapsed dual-package matrix into one. New `build-wheels` matrix produces cross-platform wheels for cp310/11/12/13 × {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New `collect-dist` aggregator merges artifacts. publish-pypi consumes the merged dist. - init-native-e2e.yml: dropped windows-latest from the matrix — upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting MSVC C runtime libraries, so the Rust extension cannot build for win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS. - headroom-e2e-setup: composite action now sets up Rust toolchain + Swatinem/rust-cache before `pip install -e .[proxy]`. - eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before install. rust.yml's wheels job builds from root pyproject.toml (no more `-m crates/headroom-py/Cargo.toml`). - e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false` from wrap-e2e — the image now ships the full Rust core. - Dockerfile (main): simplified — no more Layer 2/3 dance with `headroom-core-py` install + symlink. Single `uv pip install` builds + installs everything. - .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin added so `uv sync` builds the extension inside the devcontainer. ## Lockfile + script - uv.lock: regenerated. No `headroom-core-py` entries remain. - scripts/build_rust_extension.sh: simplified from a symlink-into-tree workaround to a thin wrapper around `pip install -e .`. The maturin build-backend handles placement automatically. ## Local validation (all green on macOS aarch64) 1. Clean venv `pip install -e .` → `from headroom._core import …` works. 2. `maturin build --release` → 13.8 MB wheel, 336 files including `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and `headroom/dashboard/templates/dashboard.html`. 3. `pip install <wheel>` in fresh venv → import works. 4. Wheel contents verified via `unzip -l`. 5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed. 6. `pytest tests/test_relevance.py` — 30 passed. 7. `cargo build --workspace` + `cargo test --workspace` — all green. 8. `make ci-precheck` — 176 Python tests + Rust + commitlint green. ## Migration notes Users on `pip install headroom-ai` get the Rust core automatically (linux + macos wheels). sdist installs require rust toolchain available locally — pip will build via maturin. Closes chopratejas#355 Supersedes chopratejas#357 (workarounds-based fix abandoned in favor of architectural fix)
Owner
Author
|
Superseded by PR #360 (single-wheel maturin refactor). The original goal of this hotfix — publishing Rust wheels with headroom-ai — is now achieved via maturin's PEP 517 build backend producing a single wheel containing both Python source and the compiled |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #355.
Summary
v0.20.13shipped to PyPI asheadroom_ai-0.20.13-py3-none-any.whl(pure-Python). The release workflow'sBuild Python packagestep runspython -m buildwhich invokes hatch only — maturin was never called, soheadroom._corewas never compiled into the published wheel. A0's fail-loud startup check correctly catches the missing module and exits 78.What this PR does
1.
.github/workflows/release.yml— newbuild-rust-wheelsjoblinux x86_64,linux aarch64,macos x86_64,macos aarch64.PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1— proper fix is bumping PyO3 ≥0.23, tracked separately).PyO3/maturin-action@v1withmanylinux: autofor x86_64 andmanylinux: 2_28for aarch64.publish-pypinow depends onbuild-rust-wheelsand merges the rust wheels intodist/before the OIDC PyPI upload — single trusted-publisher credential, single artifact upload.2.
scripts/version-sync.py— sync Rust crate versions on releaseupdate_rust_pyproject_version()andupdate_rust_cargo_version()helpers keepcrates/headroom-py/Cargo.tomlandcrates/headroom-py/pyproject.tomlin lockstep with the headroom-ai release. Wheel filename (headroom_core_py-X.Y.Z-cp311-...whl) matches the dependency declaration.headroom-core-py>=…dep line in rootpyproject.tomlon each release sync.3.
pyproject.toml— runtime dependency onheadroom-core-py"headroom-core-py>=0.20.13"sopip install headroom-aiautomatically resolves the correct platform wheel.Workaround for users stuck on v0.20.13
From the issue:
PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 maturin build --release -m crates/headroom-py/Cargo.tomlHEADROOM_REQUIRE_RUST_CORE=falsefor degraded Python-only mode.Test plan
make ci-precheckPASSED locallypython scripts/version-sync.py --version 0.20.99produces the expected diff (Cargo.toml + pyproject.toml + headroom-py/* + plugin manifests all bumped to 0.20.99; deps line in root pyproject rewritten toheadroom-core-py>=0.20.99)build-rust-wheelsand produce 4 platform × 4 Python wheels — that's the smoke test the existingwheels (aarch64-apple-darwin)andwheels (x86_64-unknown-linux-gnu)checks already cover, just now wired into release publish too.