Fix missing stats when migrating from 1.13.0 to 1.14.x#937
Fix missing stats when migrating from 1.13.0 to 1.14.x#937Theelx wants to merge 2 commits intoh2oai:mainfrom
Conversation
JiwaniZakir
left a comment
There was a problem hiding this comment.
The migration block in get_experiments_info (utils.py, around line 1655) has no error handling around the SqliteDict read. If charts.db is corrupted or was only partially written when the old version was stopped mid-run, a SqliteDict exception will propagate up and break experiment listing for all experiments in the loop, not just the one being migrated. Wrapping the migration in a try/except with a log warning would make the failure isolated and recoverable.
Additionally, the condition if not logs and os.path.exists(old_db_path) means the migration will be re-attempted on every call to get_experiments_info for any experiment whose diskcache is currently empty but still has an old charts.db present. While cache.add is idempotent (it won't overwrite existing keys), this repeated SqliteDict open/read on every page load is unnecessary overhead. A lightweight sentinel key written to the diskcache after a successful migration (e.g., cache.set("__migrated__", True)) would let you skip the old DB entirely on subsequent calls without relying on the cache being non-empty.
The requires-python widening from ==3.10.* to >=3.10, <=3.13 in pyproject.toml is a meaningful change that goes beyond the migration fix — if this wasn't intentional or hasn't been tested against 3.11–3.13, it should be split into a separate PR.
Hello! I noticed that when I upgraded from 1.13.0 to 1.14.4 earlier today, the statistics and charts for completed experiments were blank. In order to fix that, I re-added sqlitedict as a dependency and added a short migration step from sqlitedict to diskcache. Additionally, I noticed that when I tried to run uv on this repo, it gave me an error because my pyenv version is 3.12.3 but this repo only allowed 3.10.x. I have run this codebase, including normal training, grid search, and most of the features, on 3.11.9 and 3.12.3 successfully without any issues. I've tested it on Ubuntu 22.04 and Linux Mint 22.1 (based on Ubuntu 24.04). I don't have access to any Windows or Mac machines, otherwise I'd have tested on those, however there shouldn't be any issues with only slightly loosening the CPython version requirements. 3.13 and 3.14 have a bunch of weird changes with the C-API and aren't widely supported, but <=3.12 is quite widely supported so it should help the popularity of this project if we let more versions install it.