Introduction
The 2024 OME-NGFF Challenge was conceived at the OME Annual Community Meeting in Dundee in May 2024. The stated goals of Challenge were:
- Advancing the status of OME-Zarr specification proposals — Requests for Comment (RFC) — especially the adoption of the Zarr V3 specification
- Collecting the URLs of converted examples of OME-Zarr from multiple modalities, use cases, and institutions, all hosted by each participant themselves
- Testing the level of maturity of OME-Zarr conversion tools, image viewers, and metadata in general
The inspiration for the Challenge came from presentations at the OME Meeting that demonstrated the level of adoption of OME-Zarr and the relative ease of conversion for some community members, but highlighted the often poor findability of the data and the need for driving the OME-Zarr specification towards version 1.0.
A lofty goal of 1 Petabyte (PB) of data was set, mostly to make clear the scale of what could be.
Work on the challenge started in earnest in July 2024. All progress is tracked in the ome2024-ngff-challenge repository. It was agreed that the deadline for submission would be such that the results of the OME-NGFF Challenge could be presented at the 2024 Global BioImaging Meeting.
Process
The Challenge was run via a series of virtual meetings open to all which were coordinated on the Image.sc Forums, with all notes and lists of participants available. Initial meetings including that at OME2024 focused on defining the scope of changes that would be made to the OME-Zarr format. A Python-based tool to convert data to this format was built. Submissions were then collected in simple CSV files with a Zarr URL per row.
Results
At the outset, it was unclear how much OME-Zarr data was available, if it could be converted to Zarr V3, and how many organisations would participate. In the end, we are pleased with the more than 500 Terabytes of data which have been made available, across a wide range of modalities.
To make the collection as accessible as possible, the OME-NGFF Challenge Viewer, parses submissions from contributors and provides a single view across all Zarr files. Data can be searched by key-word, filtered and sorted by various metadata and browsed with thumbnails, all generated on the fly. Links allow opening of datasets with the OME NGFF Validator for metadata validation and viewing. Please let us know on image.sc or GitHub if you have any issues or ideas.
Looking back, perhaps the most rewarding outcome of the Challenge is that with a remarkably modest investment of time and cloud resources, we have almost inadvertently prototyped a federated bioimage data system based on OME-Zarr, the largest one we know of. We hope future challenges will continue to push the state-of-the-art forward.
Acknowledgements
Coordination of the challenge has been supported by the German National Research Data Initiative (NFDI) and the Chan Zuckerberg Initiative: