Drift detection definition and system boundary. Drift detection identifies statistically and operationally meaningful change between a reference distribution or relationship and current production behavior. Data drift changes the distribution of inputs or predictions; label drift changes outcome prevalence; concept drift changes the relationship P(Y|X), so the same feature pattern no longer implies the same target. Covariate shift, prior-probability shift, seasonality, instrumentation changes, policy changes, and genuine behavioral evolution require different responses. Detection matters because model accuracy can decay silently while software remains available. A production definition names the data owners and consumers, source contracts, event or snapshot identity, schemas and compatibility policy, timestamps and time zones, freshness objective, correctness invariants, volume and growth envelope, retention and deletion rules, access boundary, residency, recovery point and recovery time, and the evidence required for release. Data is not trustworthy merely because a job completed: completeness, uniqueness, validity, referential integrity, timeliness, distribution, provenance, and reconciliation must be measured at the consumer boundary.

Architecture, semantics, and machine-learning relevance. Univariate numerical features can use Kolmogorov-Smirnov distance, Wasserstein distance, Jensen-Shannon divergence after careful binning, or PSI; categorical features can use frequency divergence and rare-category checks; high-dimensional or embedding data can use classifier two-sample tests, MMD, cluster movement, neighborhood statistics, or learned summaries. ADWIN, Page-Hinkley, CUSUM, and related sequential techniques target online change. Concept drift generally needs labels, causal proxies, residuals, calibration evidence, or controlled experiments. Every method needs a reference, window, sample policy, multiplicity strategy, and threshold tied to impact. The end-to-end system separates control-plane decisions from data-plane work. The control plane stores definitions, schedules, schemas, lineage, policy, metadata, credentials, quotas, and deployment state; the data plane moves records through connectors, queues, compute, storage, indexes, caches, and serving interfaces. Immutable object storage, transactional metadata, idempotent writers, explicit checkpoints, and versioned contracts make retries and recovery understandable. Partitioning, clustering, compression, column pruning, predicate pushdown, vectorized execution, caching, and locality reduce bytes moved, which often matters more than peak arithmetic. For machine learning, every feature and label must be reconstructable as of an event time and a processing time. Training-serving skew appears when offline transformations, online feature logic, defaults, joins, or freshness differ. A defensible lineage chain binds raw source versions, transformation code, environment, feature definitions, label windows, split policy, training run, model artifact, evaluation, deployment, and production telemetry. Point-in-time joins prevent future information from leaking into historical examples, while late labels and backfills remain explicit.

Implementation and failure modes. Monitor critical features and slices, not only a single global score. Preserve reference datasets and preprocessing versions, compare like-for-like populations, enforce minimum samples, calculate uncertainty and effect size, correct or triage multiple comparisons, and visualize the changed region. Separate short-term alerts from slower trend review. Use champion replays and sensitivity analysis to determine whether a shift changes predictions or outcomes before retraining. For seasonality, compare against aligned historical periods or a modeled expected distribution. Large samples make tiny harmless shifts significant; small samples hide important changes. PSI thresholds copied without context, arbitrary bins, correlated features, post-selection, missingness encoded as a value, logging changes, bot traffic, promotion campaigns, and upstream policy changes create misleading alerts. A model may remain robust under data drift, or suffer concept drift without obvious marginal feature shift. Retraining on the newest window can worsen rare classes, forget stable behavior, or learn contamination. Distributed data systems fail partially: a producer retries after a timeout, one partition lags, a worker dies after an external write, a schema changes mid-run, clocks disagree, an object becomes visible before its catalog commit, or a downstream service accepts only part of a batch. Designs therefore use stable record identifiers, deduplication, atomic or transactional publication, bounded retries with jitter, dead-letter or quarantine paths, backpressure, watermarks or cutoffs, replayable sources, checksummed artifacts, and reconciliation. Exactly-once is an end-to-end property of source, processor, state, and sink, not a label inherited from one component.

Verification, operations, security, and governance. Use synthetic shifts with known magnitude and location, historical incident replays, no-change seasonal controls, subgroup tests, delayed labels, and end-to-end alert drills. Report detection delay, false-alarm rate, missed changes, stability under resampling, compute cost, affected traffic, prediction sensitivity, and downstream metric change. Root-cause analysis links the alert to source, transformation, feature, deployment, and product changes. Operations track input and output rows or events, bytes, lag, freshness, watermark, queue depth, job duration, task skew, spill, shuffle, cache hit rate, storage requests, query latency, concurrency, retries, duplicates, rejected records, schema changes, data-quality failures, lineage gaps, cost, energy, and service-level objective burn. Alerts point to an owned action and avoid unbounded cardinality. Runbooks cover replay, backfill, bad-data isolation, credential rotation, dependency loss, regional recovery, rollback, and consumer communication; each path is exercised with production-like permissions and scale. Security starts with data classification and least-privilege identities for people, workloads, and automation. Transport and stored data are encrypted; secrets are short-lived; sensitive fields are tokenized, masked, or minimized; row, column, and object policies are tested; administrative and query activity is audited; and retention and deletion propagate through replicas, caches, backups, indexes, and derived datasets. Governance assigns stewards, approves contract and purpose changes, records lineage and quality exceptions, reviews vendors and open-source dependencies, and preserves evidence without exposing protected values. Verification combines unit tests for transformations, contract and schema-compatibility tests, property and metamorphic tests, golden datasets, differential queries against a trusted implementation, fault injection, replay and idempotency tests, load and soak tests, skewed-key tests, late and out-of-order inputs, corrupted files, permission failures, checkpoint restoration, backup recovery, regional failover, and end-to-end reconciliation. Performance tests use representative cardinality, file sizes, partitions, concurrency, selectivity, compression, and hardware rather than toy rows.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect width="760" height="470" fill="#0d1117"/>
  <defs>
    <marker id="ddBlueArrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto">
      <path d="M0 0L10 5L0 10Z" fill="#60a5fa"/>
    </marker>
    <marker id="ddGreenArrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto">
      <path d="M0 0L10 5L0 10Z" fill="#34d399"/>
    </marker>
    <marker id="ddRedArrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto">
      <path d="M0 0L10 5L0 10Z" fill="#f87171"/>
    </marker>
    <filter id="ddAlertGlow" x="-80%" y="-80%" width="260%" height="260%">
      <feGaussianBlur stdDeviation="4" result="blur"/>
      <feMerge><feMergeNode in="blur"/><feMergeNode in="SourceGraphic"/></feMerge>
    </filter>
  </defs>

  <text x="380" y="31" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Drift Detection — Find Meaningful Change Before It Hurts</text>
  <text x="380" y="53" fill="#8b98a5" font-size="12.5" text-anchor="middle">feature shift can appear immediately; model-quality drift is confirmed later when outcomes and labels arrive</text>

  <!-- Reference and rolling production distributions -->
  <g transform="translate(24 83)">
    <text x="270" y="-8" fill="#93c5fd" font-size="10.5" font-weight="700" text-anchor="middle">REFERENCE DISTRIBUTION vs ROLLING PRODUCTION WINDOWS</text>
    <rect width="540" height="133" rx="12" fill="#0a1421" stroke="#60a5fa" stroke-width="1.6"/>

    <!-- Baseline -->
    <g transform="translate(15 18)">
      <text x="52" y="0" fill="#93c5fd" font-size="8.5" font-weight="700" text-anchor="middle">TRAIN REFERENCE</text>
      <path d="M0 84V10M0 84H105" stroke="#3a4453"/>
      <path d="M4 82C18 80 26 68 35 44S51 10 63 36 80 75 101 81" fill="#60a5fa" fill-opacity=".12" stroke="#60a5fa" stroke-width="2.2"/>
      <text x="52" y="99" fill="#8b98a5" font-size="7.5" text-anchor="middle">feature x · μ₀, σ₀</text>
    </g>

    <!-- Window 1 stable -->
    <g transform="translate(132 18)">
      <text x="52" y="0" fill="#6ee7b7" font-size="8.5" font-weight="700" text-anchor="middle">W₁ · STABLE</text>
      <path d="M0 84V10M0 84H105" stroke="#3a4453"/>
      <path d="M4 82C18 80 26 68 35 44S51 10 63 36 80 75 101 81" fill="none" stroke="#60a5fa" stroke-width="1.5" stroke-dasharray="4 3"/>
      <path d="M4 82C18 79 27 66 36 43S52 12 64 37 81 75 101 81" fill="#34d399" fill-opacity=".1" stroke="#34d399" stroke-width="2.2"/>
      <text x="52" y="99" fill="#8b98a5" font-size="7.5" text-anchor="middle">production ≈ reference</text>
    </g>

    <!-- Window 2 warning -->
    <g transform="translate(249 18)">
      <text x="52" y="0" fill="#fbbf24" font-size="8.5" font-weight="700" text-anchor="middle">W₂ · SHIFTING</text>
      <path d="M0 84V10M0 84H105" stroke="#3a4453"/>
      <path d="M4 82C18 80 26 68 35 44S51 10 63 36 80 75 101 81" fill="none" stroke="#60a5fa" stroke-width="1.5" stroke-dasharray="4 3"/>
      <path d="M4 82C24 81 38 73 50 51S70 13 82 40 92 73 103 80" fill="#f59e0b" fill-opacity=".11" stroke="#f59e0b" stroke-width="2.2"/>
      <text x="52" y="99" fill="#8b98a5" font-size="7.5" text-anchor="middle">location + shape change</text>
    </g>

    <!-- Window 3 persistent alert -->
    <g transform="translate(366 18)">
      <text x="52" y="0" fill="#fca5a5" font-size="8.5" font-weight="700" text-anchor="middle">W₃ · DRIFT</text>
      <path d="M0 84V10M0 84H105" stroke="#3a4453"/>
      <path d="M4 82C18 80 26 68 35 44S51 10 63 36 80 75 101 81" fill="none" stroke="#60a5fa" stroke-width="1.5" stroke-dasharray="4 3"/>
      <path d="M4 82C31 82 48 76 62 54S82 15 94 41 100 70 103 80" fill="#f87171" fill-opacity=".12" stroke="#f87171" stroke-width="2.3"/>
      <text x="52" y="99" fill="#8b98a5" font-size="7.5" text-anchor="middle">persistent covariate shift</text>
    </g>

    <!-- Legend -->
    <g transform="translate(478 22)">
      <path d="M0 10h18" stroke="#60a5fa" stroke-dasharray="4 3"/><text x="23" y="13" fill="#8b98a5" font-size="7.5">reference P₀(X)</text>
      <path d="M0 31h18" stroke="#f87171" stroke-width="2"/><text x="23" y="34" fill="#8b98a5" font-size="7.5">current Pt(X)</text>
      <rect y="49" width="48" height="25" rx="5" fill="#211318" stroke="#f87171"/>
      <text x="24" y="60" fill="#fca5a5" font-size="7.2" font-weight="700" text-anchor="middle">COVARIATE</text>
      <text x="24" y="70" fill="#8b98a5" font-size="6.8" text-anchor="middle">P(X) changed</text>
    </g>
  </g>

  <!-- Statistical change score with persistence -->
  <g transform="translate(24 248)">
    <text x="270" y="-9" fill="#fbbf24" font-size="10.5" font-weight="700" text-anchor="middle">CHANGE STATISTIC · KS / PSI / MMD · REQUIRE PERSISTENCE</text>
    <rect width="540" height="116" rx="11" fill="#17150f" stroke="#f59e0b" stroke-width="1.6"/>
    <path d="M37 87V14M37 87H516" stroke="#3a4453"/>
    <path d="M37 58H516" stroke="#f59e0b" stroke-width="1.3" stroke-dasharray="6 4"/>
    <path d="M37 37H516" stroke="#f87171" stroke-width="1.3" stroke-dasharray="6 4"/>
    <g fill="#8b98a5" font-size="7.8">
      <text x="32" y="61" text-anchor="end">warn</text><text x="32" y="40" text-anchor="end">alert</text>
    </g>

    <path d="M43 79L94 76 145 78 196 69 247 62 298 51 349 31 400 27 451 34 502 25" fill="none" stroke="#fbbf24" stroke-width="2.7"/>
    <g fill="#fbbf24">
      <circle cx="43" cy="79" r="3"/><circle cx="94" cy="76" r="3"/><circle cx="145" cy="78" r="3"/><circle cx="196" cy="69" r="3"/><circle cx="247" cy="62" r="3"/><circle cx="298" cy="51" r="3"/>
    </g>
    <g fill="#f87171" stroke="#fecaca" filter="url(#ddAlertGlow)">
      <circle cx="349" cy="31" r="5"/><circle cx="400" cy="27" r="5"/><circle cx="451" cy="34" r="5"/><circle cx="502" cy="25" r="6"/>
    </g>
    <path d="M349 17H502" stroke="#f87171" stroke-width="2"/>
    <text x="425" y="13" fill="#fca5a5" font-size="8.2" font-weight="700" text-anchor="middle">3+ consecutive windows → ALERT</text>
    <g fill="#8b98a5" font-size="7.8" text-anchor="middle">
      <text x="43" y="101">t₀</text><text x="145" y="101">t₂</text><text x="247" y="101">t₄</text><text x="349" y="101">t₆</text><text x="451" y="101">t₈</text><text x="502" y="101">t₉</text>
    </g>
    <text x="270" y="111" fill="#8b98a5" font-size="7.5" text-anchor="middle">thresholds are calibrated on expected noise, seasonality, sample size, and false-alert cost</text>
  </g>

  <!-- Delayed labels distinguish feature drift from actual model degradation -->
  <g transform="translate(24 390)">
    <text x="270" y="-9" fill="#c4b5fd" font-size="10.5" font-weight="700" text-anchor="middle">DELAYED LABELS CONFIRM WHETHER MODEL QUALITY CHANGED</text>
    <path d="M0 40H540" stroke="#3a4453"/>
    <g stroke="#3a4453" stroke-dasharray="3 3">
      <path d="M112 0v52M227 0v52M342 0v52M457 0v52"/>
    </g>
    <!-- Predictions happen now -->
    <path d="M9 22h36v-9h10v9h47v-9h10v9h105v-9h10v9h105v-9h10v9h105v-9h10v9h73" fill="none" stroke="#60a5fa" stroke-width="2"/>
    <!-- Labels arrive later -->
    <path d="M9 34H177q28 0 45-17t43 17h47q28 0 45-17t43 17h130" fill="none" stroke="#a78bfa" stroke-width="2"/>
    <!-- Error rate rises -->
    <path d="M9 46C113 46 195 45 264 44S365 36 421 22 488 15 530 13" fill="none" stroke="#f87171" stroke-width="2.5"/>
    <g fill="#8b98a5" font-size="7.7">
      <text x="7" y="10">prediction</text><text x="7" y="34">labels +7d</text><text x="485" y="10" fill="#fca5a5">error rate ↑</text>
    </g>
    <rect x="379" y="30" width="112" height="20" rx="10" fill="#211318" stroke="#f87171"/>
    <text x="435" y="43" fill="#fca5a5" font-size="7.8" font-weight="700" text-anchor="middle">PERFORMANCE DRIFT</text>
  </g>

  <!-- Human/operational response path -->
  <g transform="translate(587 83)">
    <text x="74" y="-8" fill="#6ee7b7" font-size="10.5" font-weight="700" text-anchor="middle">SAFE RESPONSE</text>
    <rect width="148" height="349" rx="12" fill="#101b18" stroke="#34d399" stroke-width="1.6"/>

    <g transform="translate(13 17)">
      <circle cx="17" cy="18" r="14" fill="#211b0d" stroke="#f59e0b"/>
      <text x="17" y="22" fill="#fbbf24" font-size="11" font-weight="700" text-anchor="middle">1</text>
      <text x="40" y="13" fill="#fbbf24" font-size="8.5" font-weight="700">INVESTIGATE</text>
      <text x="40" y="26" fill="#8b98a5" font-size="7.2">pipeline bug?</text>
      <text x="40" y="37" fill="#8b98a5" font-size="7.2">seasonality? segment?</text>
    </g>
    <path d="M30 58V82" stroke="#f59e0b" stroke-width="2" marker-end="url(#ddBlueArrow)"/>
    <g transform="translate(13 87)">
      <circle cx="17" cy="18" r="14" fill="#142b44" stroke="#60a5fa"/>
      <text x="17" y="22" fill="#93c5fd" font-size="11" font-weight="700" text-anchor="middle">2</text>
      <text x="40" y="13" fill="#93c5fd" font-size="8.5" font-weight="700">RECALIBRATE</text>
      <text x="40" y="26" fill="#8b98a5" font-size="7.2">or retrain only with</text>
      <text x="40" y="37" fill="#8b98a5" font-size="7.2">representative labels</text>
    </g>
    <path d="M30 128V152" stroke="#60a5fa" stroke-width="2" marker-end="url(#ddBlueArrow)"/>
    <g transform="translate(13 157)">
      <circle cx="17" cy="18" r="14" fill="#211936" stroke="#a78bfa"/>
      <text x="17" y="22" fill="#c4b5fd" font-size="11" font-weight="700" text-anchor="middle">3</text>
      <text x="40" y="13" fill="#c4b5fd" font-size="8.5" font-weight="700">SHADOW TEST</text>
      <text x="40" y="26" fill="#8b98a5" font-size="7.2">candidate receives live</text>
      <text x="40" y="37" fill="#8b98a5" font-size="7.2">traffic, serves no users</text>
    </g>
    <path d="M30 198V222" stroke="#a78bfa" stroke-width="2" marker-end="url(#ddGreenArrow)"/>
    <g transform="translate(13 227)">
      <circle cx="17" cy="18" r="14" fill="#153127" stroke="#34d399"/>
      <text x="17" y="22" fill="#6ee7b7" font-size="11" font-weight="700" text-anchor="middle">4</text>
      <text x="40" y="13" fill="#6ee7b7" font-size="8.5" font-weight="700">PROMOTE</text>
      <text x="40" y="26" fill="#8b98a5" font-size="7.2">only if offline + online</text>
      <text x="40" y="37" fill="#8b98a5" font-size="7.2">guardrails improve</text>
    </g>

    <path d="M13 285h122" stroke="#315044"/>
    <g transform="translate(13 300)">
      <path d="M0 0h122v33H0Z" fill="#211318" stroke="#f87171"/>
      <path d="M9 24L18 7l9 17Z" fill="none" stroke="#f87171"/>
      <text x="18" y="21" fill="#fca5a5" font-size="10" font-weight="700" text-anchor="middle">!</text>
      <text x="70" y="13" fill="#fca5a5" font-size="7.8" font-weight="700" text-anchor="middle">NO BLIND AUTO-SWAP</text>
      <text x="70" y="25" fill="#8b98a5" font-size="7" text-anchor="middle">retain rollback + champion</text>
    </g>
  </g>

  <text x="380" y="458" fill="#6b7684" font-size="11" text-anchor="middle">A drift alert is evidence to investigate—not proof of root cause and not permission to replace a model automatically.</text>
</svg>

Selection and practical application. Use simple two-sample tests for interpretable single features, multivariate tests for interactions, sequential detectors for low-latency streams, and labeled performance monitoring for concept drift. Drift detection supports fraud, demand forecasting, recommender systems, sensors, autonomous systems, credit, medical workflows, ads, language models, and any production environment where population or behavior changes. Selection is an architectural decision, not a tool popularity contest. Teams compare semantics, access patterns, latency and freshness, consistency, durability, scale, operational maturity, ecosystem, portability, governance, recovery, staffing, and total lifecycle cost. A faster engine can make the complete system worse if it increases small files, weakens lineage, duplicates state, hides fallbacks, or transfers complexity to every consumer. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.