**Treat a run as an immutable experimental claim.** Official W&B documentation defines a run as the atomic record of one computation. In practice, one run should represent one coherent execution attempt with a stable ID, purpose, configuration, state, metric history, summaries, and input/output relationships. Projects group related runs so teams can filter, compare, and review them. A human-readable run name is useful for navigation but should not be the primary identity because names need not be unique and may change. Store the stable run ID with scheduler job IDs, commit or source digest, model output metadata, and decision records. If an external system promotes a model, it should record the exact run and artifact version—not “the latest good experiment.” | Object or field | Engineering role | What to record | Failure if omitted | |---|---|---|---| | Project | Cohort boundary | Product/model, lifecycle stage, ownership | Unrelated runs become incomparable | | Run ID | Stable execution identity | ID plus scheduler/request linkage | Resume and audit ambiguity | | Run config | Declared inputs | Hyperparameters, model/data references, policies | Curves cannot be reconstructed | | Metric history | Time-varying observations | Value, step axis, units, phase | Misaligned or misleading comparison | | Run summary | Final/aggregate result | Best/final metrics and validity status | Dashboard sorting selects wrong result | | Artifact input | Versioned dependency | Dataset, features, base model, calibration | Hidden input drift | | Artifact output | Versioned result | Checkpoint, evaluation, export package | “Best model” cannot be located exactly | | Tags/job type/notes | Searchable context | Baseline, train/eval, incident, hypothesis | Institutional context stays in chat | | Report or review record | Decision narrative | Cohort, plots, caveats, approver | Selection rationale disappears | **Experiment tracking is not automatically reproducibility.** Logging loss and accuracy cannot reconstruct a run if the dataset snapshot, preprocessing code, dependency environment, randomization policy, base model, hardware-sensitive behavior, and exact command are unknown. The platform stores evidence supplied by the workflow; it cannot infer omitted semantics. Define a reproducibility contract before instrumentation. A useful run record can be modeled as $$R=(I,C,D,E,M,A,S)$$ where $I$ is identity, $C$ is code/configuration, $D$ is versioned data lineage, $E$ is the execution environment, $M$ is metric history with semantics, $A$ is input/output artifacts, and $S$ is state plus validity. A missing component may be acceptable for exploratory work, but it should be deliberate and visible. Do not log secrets, private keys, tokens, credentials, raw personal information, controlled design data, or proprietary samples merely because configuration and media logging are convenient. Classify fields and artifacts, allowlist what may leave the process, and choose deployment/storage controls that match organizational requirements. **Initialize runs through a small owned wrapper.** Direct SDK calls scattered across training scripts produce inconsistent projects, names, metric schemas, tags, resume behavior, and error handling. A team wrapper can validate required metadata, redact prohibited fields, standardize environment capture, and provide a fallback when tracking is unavailable. ```python from dataclasses import asdict import os import wandb def start_run(cfg, *, run_id: str, source_revision: str): safe_config = asdict(cfg) safe_config.pop("access_token", None) safe_config.update({ "source/revision": source_revision, "data/train_artifact": cfg.train_artifact, "data/eval_artifact": cfg.eval_artifact, "repro/seed_policy": "rank-offset-v1", "runtime/scheduler_job": os.getenv("JOB_ID", "local"), }) return wandb.init( project="accelerator-model-training", id=run_id, config=safe_config, job_type="train", tags=[cfg.stage, cfg.architecture], notes=cfg.hypothesis, ) ``` The exact resume arguments and failure policy should be defined for the approved SDK version rather than copied blindly. Decide whether restarting a failed scheduler job continues the same logical run, creates a child/attempt run, or creates a new run linked through custom metadata. Accidental merging can hide failed attempts; accidental duplication can make one experiment appear statistically replicated. A robust identity tuple can include $$I=(project,run\_id,attempt,job\_id,source\_digest)$$ with an explicit uniqueness rule. Persist it outside the worker process before training starts. If a spot/preemptible job restarts, the orchestrator—not an ad hoc timestamp—should decide identity. **Configuration needs values and meaning.** Log training hyperparameters, architecture, optimizer, scheduler, precision, gradient accumulation, sequence/image dimensions, augmentation policy, checkpoint source, dataset artifact versions, split definitions, evaluation protocol, seed policy, and relevant runtime settings. Keep config reasonably flat and queryable, but preserve enough structure to avoid ambiguous names. Do not mutate input config silently after initialization. If the runtime derives effective batch size, learning-rate scaling, number of updates, token budget, or actual device count, log both requested and effective values. For example, $$B_{effective}=B_{device}\times N_{devices}\times N_{accumulation}$$ is more useful than a lone `batch_size=8`. Record whether the value counts examples, sequences, tokens, wafers, simulation cases, or another unit. Separate configuration from observed state. Requested GPU model is config; actual assigned GPU model and driver/runtime versions are environment observations. Intended dataset is config; resolved artifact digest is lineage. This distinction helps detect deployment drift. **Metric schemas are contracts.** A metric name should have a stable definition, units, aggregation, population, phase, and step axis. `loss` is ambiguous: it may mean per-microbatch training loss, epoch average, validation objective, cross-entropy, regularized total, or one worker’s local result. Prefer names such as `train/loss_total`, `eval/accuracy_top1`, `system/tokens_per_second`, and `chip/power_w` with documented semantics. Explicitly log the axis used for comparison: optimizer update, microbatch, epoch, examples, tokens, simulated cycles, or wall time. When runs use different accumulation, batch size, or early stopping, comparing by raw logging index creates false conclusions. ```python for update, batch in enumerate(loader): metrics = train_one_update(batch) tokens_seen += metrics.tokens run.log({ "train/update": update, "train/tokens_seen": tokens_seen, "train/loss_total": metrics.loss, "perf/tokens_per_second": metrics.tokens_per_second, "optimizer/learning_rate": metrics.learning_rate, }) run.summary["validity/status"] = "passed" run.summary["eval/accuracy_top1_final"] = evaluate(model) ``` Log all values that belong to one step together when possible. Independently logged values with unclear step handling can form misleading charts. Monotonic explicit axes are especially important when jobs resume, validation occurs sparsely, or several processes log concurrently. **Logging frequency is a systems decision.** High-frequency scalar calls, media uploads, histograms, tables, checkpoints, and system telemetry consume CPU, network bandwidth, local buffering, backend ingestion, and storage. Measure instrumentation overhead on representative training and simulation jobs. An approximate outbound telemetry rate is $$B_{log}\approx f_{log}(P_{payload}+P_{protocol})+B_{media}+B_{artifact}$$ where $f_{log}$ is scalar logging frequency. Batching several related metrics reduces per-call overhead. Downsample high-rate signals after preserving local raw telemetry when required. Never let best-effort observability block a safety-critical or expensive long-running workload without a defined reason. Set separate policies for exploratory, tuning, and release runs. Exploratory jobs may log rich diagnostics temporarily; sweeps need lean schemas; release candidates need complete provenance and retained evaluation evidence. Storage retention should reflect these classes. **Artifacts connect runs into lineage.** W&B Artifacts can represent versioned inputs and outputs: raw or processed datasets, feature sets, checkpoints, evaluation bundles, calibration data, exported models, and reports. A training run can declare a dataset artifact as input and log a checkpoint artifact as output; an evaluation run then consumes the exact checkpoint and test-data versions. ```python with wandb.init(project="accelerator-model-training", job_type="train") as run: dataset = run.use_artifact("training-corpus:approved") dataset_dir = dataset.download() checkpoint_path = train(dataset_dir) model_artifact = wandb.Artifact( name="decoder-checkpoint", type="model", metadata={ "format": "safetensors", "architecture": "decoder-v3", "validation_policy": "release-gates-v2", }, ) model_artifact.add_file(checkpoint_path) run.log_artifact(model_artifact) ``` Aliases such as `latest`, `approved`, or `production` are movable references, not immutable evidence. A deployment manifest should retain the resolved artifact version or digest. Before promotion, verify file checksums, expected inventory, format, model signature, preprocessing contract, license/usage metadata, and evaluation linkage. Artifact versioning does not solve data governance by itself. Define who may create, move, approve, delete, and consume versions; where payloads are stored; how retention and legal holds work; and what happens when source data must be removed. Large artifacts need lifecycle rules, deduplication awareness, and egress/cost monitoring. **Lineage should reflect transformations, not just final models.** A preprocessing run consumes raw data and produces a processed dataset. Training consumes the processed dataset and base model, producing checkpoints. Evaluation consumes a checkpoint and frozen test set, producing an evaluation artifact. Optimization/quantization consumes an approved model and calibration set, producing a deployable package. Benchmarking consumes that package plus hardware/software configuration and produces latency, throughput, energy, and accuracy evidence. This graph supports impact analysis. If a source dataset or preprocessing version is invalidated, identify descendants rather than searching filenames. If an exported model behaves unexpectedly, trace backward to checkpoint, training run, data, and source revision. **Distributed training needs one clear logging topology.** If every rank independently creates the same logical run and logs global metric names, records may duplicate, conflict, or become nondeterministic. A common policy is rank-zero ownership of the primary run after metrics are reduced across workers. Other ranks can write local diagnostics to separate files or intentionally distinct worker runs grouped under the job. Record world size, rank topology, host/device inventory, communication backend, precision, sharding/parallelism strategy, effective batch and token counts, restart count, and scheduler identity. A throughput metric must state whether it is per-device or global and whether it includes data loading, evaluation, checkpointing, or only steady-state kernels. For fault-tolerant jobs, align checkpoint completion with tracking state. Do not mark an artifact complete before its file is durable. On resume, verify that the tracker step, optimizer step, scheduler state, RNG state, data-loader position, and checkpoint generation agree. A pretty continuous curve can conceal a repeated or skipped data segment. **Sweeps automate execution; they do not validate the experiment.** Official W&B guidance supports search methods including grid, random, and Bayesian approaches, with agents that can run across machines. Define the search space, objective name and direction, resource bounds, early-termination policy, base configuration, and program entry point under version control. For a grid over parameters $H_1,\ldots,H_k$, the number of combinations is $$N_{grid}=\prod_{i=1}^{k}|H_i|$$ before seeds or folds multiply cost. Continuous parameters and conditional architecture choices often make random or model-based search more practical, but the search method cannot rescue a leaking validation set or unstable metric. Prevent test-set overfitting. Sweeps should optimize a validation objective; the held-out test set should be used under a predetermined final evaluation policy. If the same test metric guides hundreds of choices, it is no longer an unbiased estimate of generalization. Record failed, pruned, preempted, invalid, and out-of-memory trials rather than deleting them. Missingness can be informative: one configuration region may fail systematically. Define whether infeasible runs receive a penalty, are excluded, or trigger a constrained analysis. Compare sweep results with uncertainty. Repeat promising configurations across seeds and, where appropriate, data folds or hardware conditions. The top observed run among many noisy trials is subject to selection bias. Preserve the full candidate cohort and final selection rule. **Dashboards support analysis when cohorts are valid.** Filter by data version, code revision, model family, validity status, hardware class, and evaluation protocol before comparing. A plot combining incompatible runs can be visually persuasive and scientifically wrong. Use parallel-coordinate plots, parameter-importance views, scalar tables, and custom charts as hypothesis tools, not causal proof. Parameter importance in an adaptive sweep can reflect sampling policy and correlations. Validate conclusions with controlled follow-up experiments. Reports should capture the cohort query, metric definitions, artifact versions, charts, caveats, rejected alternatives, decision, and reviewer. If an interactive report can change as filters or aliases move, export or otherwise preserve the decision-time identity set according to governance requirements. **System metrics explain, but do not replace, workload metrics.** GPU utilization, memory allocation, power, temperature, CPU use, storage, and network observations help diagnose regressions. Their sampling frequency and meaning may differ across platforms. Low GPU utilization could indicate input starvation, synchronization, small kernels, communication, compilation, or deliberate latency optimization. For AI-chip experiments, pair model metrics with hardware-aware metrics: achieved throughput, latency distribution, energy or power, memory footprint, communication volume, kernel mix, compile time, and utilization under a stated batch/sequence/workload. Record the hardware and software stack necessary to interpret them. A useful deployment objective may be constrained rather than scalar: $$\text{maximize quality}\quad\text{subject to}\quad p99\ latency\le L_{max},\; memory\le M_{max},\; power\le P_{max}$$ If a sweep optimizes a weighted score, retain the component metrics and weights. Otherwise a change in scaling can reverse rankings without any model improvement. **Define failure semantics.** Runs can terminate successfully, fail, be killed, preempt, time out, or remain stale. Add an application-level validity field because process exit alone does not establish scientific validity. A run may finish normally while data checks failed, evaluation was incomplete, NaNs occurred, or the wrong artifact was resolved. Use structured status such as `passed`, `failed-data-check`, `failed-numerics`, `incomplete-eval`, `infrastructure-failure`, and `cancelled`. Log the first invalidating condition and preserve diagnostic artifacts within privacy limits. Selection queries should require the approved validity state. Tracking outages should have a defined policy. Options include failing before expensive work begins when auditability is mandatory, buffering locally and syncing later, or continuing with a local manifest and marking the run incomplete. Do not silently drop telemetry and still promote the output. **Security begins before `run.log`.** API credentials belong in an approved secret manager or workload identity path, not config, source, notebooks, or artifacts. Redact environment variables and command lines. Review automatic code, system, console, and metadata capture against policy. Use least-privilege projects/entities and separate development from controlled release areas. Define access for contractors, service accounts, CI, sweep agents, and production systems. Rotate credentials, monitor access, and remove stale identities. Confirm region, storage, encryption, backup, deletion, retention, and private-network requirements for the organization’s chosen deployment. Treat logged media and tables as data export. A single sample image, prompt, waveform, wafer map, netlist-derived feature, or text row can expose sensitive information. Prefer synthetic/redacted examples and aggregate statistics where detailed samples are unnecessary. **Control cost and lifecycle.** Total retained storage can be approximated as $$S_{total}\approx N_r(S_{history}+S_{logs}+S_{media})+\sum_j S_{artifact,j}$$ where $N_r$ is run count. Large sweeps multiply history and checkpoint volume quickly. Log only checkpoints with a declared purpose, apply retention tiers, and distinguish recoverable caches from records that support a release decision. Measure ingestion volume, artifact growth, API query load, dashboard performance, and egress. Archive or delete under approved policy rather than relying on manual cleanup. A failed sweep of hundreds of runs can cost more in logs and checkpoints than compute estimates assumed. **Preserve portability.** The training loop should not depend on the tracker to compute correct gradients or produce a model. Put instrumentation behind an interface; retain machine-readable local configuration, metrics summaries, manifests, and checksums; and periodically test export/query paths. A minimal independent run manifest might contain stable ID, timestamps, command, source digest, environment lock digest, requested/effective config, resolved input artifact digests, output checksums, metric summary, validity, and parent/child relationships. W&B can be the primary collaboration interface while the manifest remains a durable contract with CI, registry, deployment, and audit systems. Avoid using mutable web URLs as the only reference in tickets or model cards. Store stable IDs and resolved versions. Verify that a new SDK or backend release does not change step handling, resume behavior, media encoding, artifact resolution, or automatic capture in a way that affects the organization’s contract. **Instrument frameworks intentionally.** Automatic integrations can log gradients, parameter histograms, checkpoints, and media with little code, but defaults may be too expensive or too revealing. Review frequency, naming, worker behavior, and storage before enabling in large training. High-dimensional histograms can overwhelm the signal needed for a decision. Start with loss, task metrics, optimizer state summaries, throughput, memory, and selected diagnostics tied to a hypothesis. Add richer telemetry temporarily to investigate a failure, then return to a controlled baseline schema. For notebooks, explicitly finish runs or use context management so repeated cell execution does not leak state into an unintended run. For services handling many requests, decide whether a run represents process lifetime, model build, evaluation batch, or request cohort; creating a training-style run per inference request is usually the wrong abstraction. **Review experiments through gates.** A candidate should not be selected solely because one summary metric is highest. Example gates include: 1. Required identity/config fields present and schema-valid. 2. Code and environment identity resolved. 3. Approved dataset and base-model artifact versions used. 4. Data-quality and leakage checks passed. 5. Training completed without invalid numerics. 6. Evaluation protocol and population match the comparison cohort. 7. Repeated-seed uncertainty is acceptable. 8. Latency, memory, power, robustness, and safety constraints pass. 9. Output artifact checksums and format validation pass. 10. Reviewer records decision and immutable identities. Automate objective gates in CI or workflow orchestration and log their outputs as structured evidence. Keep human review for tradeoffs and caveats that cannot be reduced to a scalar. **Common anti-patterns undermine trustworthy tracking.** Watch for these: - Reusing one run for unrelated attempts because the chart looks continuous. - Encoding all metadata in a clever run name instead of structured config. - Logging mutable dataset paths without version/digest resolution. - Comparing runs across different splits, preprocessing, or metric definitions. - Using `latest` artifact aliases in a deployment manifest. - Letting every distributed rank log duplicate global metrics. - Logging validation at epoch number while training logs optimizer step, then plotting both as if aligned. - Uploading every checkpoint and full media batch with no retention policy. - Deleting failed trials and creating survivorship bias. - Choosing a sweep winner on the repeatedly inspected test set. - Storing tokens, user data, prompts, or proprietary assets in config or tables. - Depending on network tracking for training correctness. - Treating system utilization as proof of model efficiency. - Assuming a completed run is valid without explicit gates. - Recording a dashboard screenshot instead of the run/artifact cohort that generated it. **Validate the integration itself.** Create a small deterministic canary experiment and assert that required config fields, metric axes, summary fields, artifact input/output edges, checksums, and final state are correct. Kill and resume it to test identity policy. Disconnect tracking to test fallback behavior. Run two distributed workers to confirm ownership. Attempt to log prohibited keys and ensure redaction blocks them. Query the completed record programmatically and compare it with the local manifest. Download a versioned artifact into a clean environment and verify its digest and inventory. Reproduce the evaluation from recorded inputs. These checks turn observability plumbing into a tested part of the ML platform. ```flowchart Define the experimental question, decision owner, comparison cohort, and success constraints → Define run identity, attempt/resume semantics, project, job type, tags, and validity states → Specify an allowlisted config schema and secret/sensitive-data redaction → Resolve code, environment, dataset, base-model, and preprocessing identities before execution → Initialize one logical run with stable external job linkage → Log metrics with explicit names, units, aggregation, and monotonic step axes → Batch/downsample telemetry to a measured overhead budget → Record distributed topology and reduce global metrics before rank-zero logging → Declare versioned artifacts as run inputs and log durable outputs only after completion → Mark data, numerical, evaluation, and infrastructure failures explicitly → For sweeps, freeze objective, search space, budget, validity rules, and held-out-test policy → Compare only schema-compatible cohorts and quantify repeated-run uncertainty → Validate accuracy, latency, throughput, memory, power, robustness, and governance gates → Preserve decision narrative with stable run and artifact identities → Promote an immutable artifact version, never a mutable dashboard label → Apply access, retention, deletion, cost, and export policies → Periodically replay a canary run and verify the full provenance chain ``` **An operational runbook closes the loop.** Assign owners for SDK wrapper changes, project creation, artifact schemas, sweep templates, security review, retention, incident response, and backend upgrades. Publish approved metric/config naming conventions and example queries. Version these contracts like APIs. Monitor stale runs, ingestion failures, duplicate IDs, missing required fields, artifact upload failures, excessive media, and unauthorized project creation. During an incident, preserve local logs and manifests, identify affected runs/artifacts, prevent promotion, and document whether records can be repaired or must be invalidated. The durable design uses a provenance-and-decision lens. W&B provides runs, projects, configurations, metric histories, artifacts, sweeps, and collaborative analysis surfaces; the engineering organization supplies stable semantics, versioned inputs, identity policy, governance, and validation. When those layers are combined, experiment tracking becomes more than visualization: it becomes a tested evidence chain from hypothesis through computation to an auditable model or system decision.