data center
**A data center is the physical and operational system that turns electrical power, network connectivity, and cooling capacity into dependable computing.** For artificial intelligence, it is the machine around the chips: servers hold GPUs and CPUs, leaf-and-spine networks join thousands of accelerators, storage feeds training data, and mechanical and electrical plants remove heat while keeping every rack available. The useful unit is therefore not one fast accelerator but a cluster that can sustain synchronized work without being limited by power, temperature, failed links, or stalled data pipelines.
**AI has changed rack density and facility design.** Conventional enterprise racks often operated well below 20 kW, while modern accelerated racks commonly require 40–100 kW and emerging configurations can go higher. Air alone becomes difficult at those heat fluxes. Direct-to-chip cold plates, coolant distribution units, rear-door heat exchangers, and immersion systems move heat into liquid with far greater volumetric capacity. Facility teams must coordinate chip temperature, coolant chemistry, condensation margin, pumps, heat rejection, and service procedures.
| AI system | Accelerator organization | Scale-up or scale-out fabric | Memory and facility implication |
|---|---|---|---|
| NVIDIA DGX H100 | 8 H100 GPUs per system | NVLink/NVSwitch inside; InfiniBand or RoCE outside | 640 GB HBM3 per system; dense liquid-ready clusters |
| NVIDIA GB200 NVL72 | 72 Blackwell GPUs with Grace CPUs | NVLink rack-scale domain plus scale-out network | Very high rack power and liquid cooling required |
| AMD MI300X platform | Commonly 8 MI300X accelerators | Infinity Fabric locally; Ethernet/InfiniBand scale-out | Large HBM capacity favors memory-heavy models |
| Google TPU v5p pod | Thousands of TPU chips in pod topology | Proprietary high-speed inter-chip interconnect | Co-designed compute, network, software, and cooling |
| AWS Trainium cluster | Trainium accelerators in EC2 infrastructure | NeuronLink and Elastic Fabric Adapter | Cloud-managed scaling and distributed training |
**Compute nodes are organized around locality.** GPUs within a server communicate over a scale-up fabric whose bandwidth and latency are much better than the data-center network. Servers then communicate over scale-out links. Parallel training software maps tensor, pipeline, data, and expert parallelism onto those levels. If a collective operation crosses slow or oversubscribed paths unnecessarily, thousands of expensive GPUs wait at barriers. Topology-aware placement and failure handling are performance features.
```svg
```
**Leaf-and-spine networking provides predictable path capacity.** Each server-facing leaf connects upward to multiple spine switches, allowing equal-cost paths between racks. InfiniBand emphasizes lossless transport and collective offload; Ethernet clusters commonly use RoCE with priority flow control, congestion notification, careful buffering, and telemetry. Rail-optimized designs connect each GPU or NIC plane consistently. Optical transceivers, cables, switches, and NICs must be provisioned as a system, not as an afterthought.
**Training storage is a pipeline rather than a disk cabinet.** Object stores retain durable corpora, parallel file systems feed checkpoints and large datasets, and local NVMe caches absorb repeated reads and shuffle traffic. Metadata service can bottleneck millions of small files even when bulk bandwidth is ample. Data preprocessing consumes CPUs and memory bandwidth. Checkpoints need enough write throughput that failure recovery does not dominate productive time, and restore behavior must be tested at cluster scale.
**Power delivery starts at utility interconnection and ends at silicon voltage rails.** Substations, switchgear, transformers, uninterruptible power systems, generators, busways, rack power shelves, and board regulators form a chain. Each conversion loses energy and introduces a failure mode. Large AI campuses require hundreds of megawatts and long utility lead times. Operators increasingly coordinate workload scheduling with power availability, renewable generation, grid constraints, and battery systems.
**Power usage effectiveness (PUE) is facility energy divided by IT energy.** A PUE of 1.2 means 0.2 units support cooling and distribution for every unit delivered to IT. Hyperscale sites target values below 1.2 under favorable conditions, but annual average, climate, utilization, water strategy, and measurement boundary matter. A low PUE does not guarantee low carbon emissions or efficient model training; chip utilization and electricity source remain decisive.
**Cooling design follows heat from junction to atmosphere.** Thermal interface material moves heat to cold plates or heatsinks; facility coolant carries it to a CDU; secondary loops reach chillers, cooling towers, or dry coolers. Warm-water systems may avoid compressor energy and enable heat reuse. Redundancy must cover pumps and controls, and quick-disconnects must resist leakage. Maintenance requires isolation valves and enough aisle space to replace a failed node without disturbing neighbors.
**Reliability is expressed through fault domains.** Dual utility feeds, UPS paths, generators, network planes, and storage replicas reduce single points of failure, but excessive redundancy wastes capital. Software should tolerate failed accelerators, links, and nodes through checkpointing and elastic job recovery. A rack-level coolant or busway event can remove dozens of nodes together, so schedulers and placement policies must understand correlated failures.
**Hyperscalers and colocation providers optimize different layers.** Google, Microsoft, AWS, and Meta co-design facilities, networks, servers, accelerators, and orchestration at enormous scale. Colocation operators such as Equinix provide power, cooling, security, and connectivity to many tenants. Enterprises may use cloud capacity for elasticity, colocation for controlled deployments, and on-premises systems for data locality. The right model depends on utilization, capital, energy access, compliance, and operational skill.
**Security spans concrete walls and firmware roots of trust.** Sites control personnel, cages, cameras, media handling, and supply access. Networks segment management, storage, tenant, and training traffic. Secure boot, measured firmware, device attestation, encryption, and key management protect servers. Decommissioning must sanitize drives and accelerator memory. Availability controls must also resist malicious commands that could overload power or cooling.
**Operations depend on telemetry and disciplined change control.** Sensors report inlet temperature, coolant pressure, flow, power, fan speed, packet loss, optics health, storage latency, and accelerator errors. DCIM and cluster observability correlate facility and application events. Canary deployments, maintenance windows, spare pools, and documented rollback reduce blast radius. Predictive models help only when sensor calibration and maintenance records are trustworthy.
**Economics are dominated by utilized output over asset life.** Accelerators, networking, buildings, power agreements, and cooling plant create large fixed costs. Idle GPUs are expensive even if PUE is excellent. Operators track tokens, training steps, or useful jobs per megawatt-hour and per invested USD, alongside uptime. Cluster design is successful when infrastructure continuously delivers useful synchronized compute within thermal, electrical, reliability, and environmental limits.