scientific data management hpc

**Scientific Data Management and Provenance in HPC** is the **discipline of organizing, storing, describing, and tracking the lineage of large-scale simulation and experimental datasets produced by supercomputers — ensuring that terabyte-to-exabyte datasets are Findable, Accessible, Interoperable, and Reusable (FAIR) through standardized formats, metadata schemas, and provenance tracking systems that allow scientific results to be reproduced, validated, and built upon years after their production**. **The HPC Data Challenge** Frontier generates ~20 TB/day from climate simulations. A single NWChem quantum chemistry run produces 500 GB of checkpoint files. Without systematic management, these datasets become orphaned, undocumented, and irreproducible within months. Funding agencies (DOE, NSF, NIH) now mandate data management plans (DMPs). **FAIR Data Principles** - **Findable**: unique persistent identifier (DOI, Handle), searchable metadata, registered in data catalog. - **Accessible**: downloadable via standard protocols (HTTP, HTTPS, Globus), with authentication where necessary. - **Interoperable**: community-standard formats (NetCDF, HDF5), controlled vocabularies, linked metadata. - **Reusable**: provenance documented (who ran, when, with what code version), license specified (CC-BY, open data). **Standard File Formats** - **HDF5 (Hierarchical Data Format 5)**: groups (directories) + datasets (n-dimensional arrays) + attributes (metadata), supports parallel I/O via MPI-IO (HDF5 parallel), chunking + compression (BLOSC, GZIP, ZSTD), self-describing format. - **NetCDF-4** (built on HDF5): CF (Climate and Forecast) conventions for atmospheric/ocean data, coordinate variables, standard_name vocabulary, used by all major climate models (WRF, CESM, MPAS). - **ADIOS2**: I/O middleware designed for extreme-scale HPC, supports staging (data in transit processing), BP5 format with compression, used by fusion and combustion codes. - **Zarr**: cloud-native chunked array format (cloud object storage), emerging alternative to HDF5. **Parallel I/O Best Practices** - **Collective I/O** (MPI-IO): aggregate writes from multiple ranks into large sequential I/O operations (avoids small-file overhead on Lustre). - **Subfiling**: each node writes to local file, merged in postprocessing (avoids MPI-IO overhead for write-once data). - **Checkpointing frequency**: balance between checkpoint overhead and expected loss from failure (Young's formula: optimal interval = √(2 × MTBF × t_checkpoint)). **Provenance and Workflow Tracking** - **PROV-DM (W3C standard)**: entity-activity-agent model for provenance representation. - **Nextflow / Snakemake**: workflow managers that automatically capture provenance (which script, which inputs, which outputs, timestamps, checksums). - **DVC (Data Version Control)**: Git-based data versioning (track large files via content hash, store in remote object storage). - **MLflow**: experiment tracking for ML workflows (parameters, metrics, artifacts). **Data Repositories** - **ESnet Globus**: high-speed data transfer (100 Gbps) between DOE facilities, with access control. - **NERSC HPSS**: long-term tape archive for permanent preservation. - **Zenodo / Figshare**: academic data publication with DOI assignment. - **LLNL Data Store / ALCF Petrel**: facility-specific data portals. Scientific Data Management is **the institutional infrastructure that transforms petabyte simulation outputs from temporary files into permanent scientific assets — ensuring that the trillion CPU-hour investments of exascale computing yield reproducible, reusable scientific knowledge that compounds across generations of researchers**.