ray
**Ray** is the **unified Python framework for distributed computing that scales any Python function or class from a laptop to a cluster of thousands of machines** — providing the infrastructure backbone for distributed LLM training, large-scale hyperparameter search, model serving with Ray Serve, and data preprocessing with Ray Data across the AI engineering ecosystem.
**What Is Ray?**
- **Definition**: An open-source distributed computing framework from UC Berkeley (and Anyscale) that provides simple primitives (@ray.remote) for parallelizing Python code across cores and machines, alongside high-level libraries (Ray Train, Ray Tune, Ray Serve, Ray Data) for AI/ML workloads.
- **Core Abstraction**: Any Python function decorated with @ray.remote becomes a "remote function" that can execute on any core or machine in the Ray cluster — the cluster appears as a pool of compute resources addressable from a single Python script.
- **Design Philosophy**: "Make distributed computing as easy as multiprocessing" — write Python, scale to cluster without learning new APIs, data formats, or paradigms.
- **Ecosystem**: Ray is the infrastructure chosen by Anyscale (managed Ray), used by OpenAI for RL training, used by Uber, Shopify, and Spotify for ML platform infrastructure.
**Why Ray Matters for AI**
- **Distributed LLM Training**: Ray Train wraps PyTorch DDP/FSDP/DeepSpeed — launch multi-node training with a single script, automatic fault tolerance, checkpoint management.
- **Hyperparameter Optimization**: Ray Tune implements 20+ search algorithms (ASHA, PBT, Bayesian) — parallelizes HPO across hundreds of GPUs, stopping bad trials early and allocating more resources to promising ones.
- **Production Model Serving**: Ray Serve provides model composition, dynamic batching, streaming responses, and autoscaling — powers serving pipelines combining embedding, retrieval, reranking, and generation.
- **Data Preprocessing**: Ray Data processes training datasets in parallel across CPU and GPU workers — with streaming to prevent memory bottlenecks.
- **Reinforcement Learning**: Ray RLlib implements 30+ RL algorithms (PPO, SAC, DQN) with distributed rollout — the standard framework for large-scale RL experiments.
**Core Ray Primitives**
**Remote Functions (stateless parallel tasks)**:
import ray
ray.init() # Start Ray (local or connect to cluster)
@ray.remote
def embed_document(text: str) -> list[float]:
return embedding_model.encode(text)
# Launch 1000 parallel embedding jobs
futures = [embed_document.remote(doc) for doc in documents]
embeddings = ray.get(futures) # Collect results
**Remote Classes (stateful actors)**:
@ray.remote(num_gpus=1)
class ModelServer:
def __init__(self, model_path: str):
self.model = load_model(model_path) # GPU resident
def predict(self, inputs: list) -> list:
return self.model(inputs)
server = ModelServer.remote("llama-3-8b") # Starts actor on a GPU worker
result = ray.get(server.predict.remote(batch))
**Ray Train (Distributed Training)**:
from ray.train.torch import TorchTrainer
from ray.train import ScalingConfig
def train_fn(config):
model = MyModel()
optimizer = AdamW(model.parameters())
# Standard PyTorch training loop — Ray handles distribution
for batch in train_loader:
loss = model(batch)
loss.backward()
optimizer.step()
trainer = TorchTrainer(
train_loop_per_worker=train_fn,
scaling_config=ScalingConfig(num_workers=8, use_gpu=True) # 8 GPU workers
)
result = trainer.fit()
**Ray Tune (Hyperparameter Search)**:
from ray import tune
from ray.tune.schedulers import ASHAScheduler
def train_fn(config):
model = MyModel(lr=config["lr"], hidden=config["hidden"])
# ... training loop with tune.report(loss=val_loss) ...
tuner = tune.Tuner(
train_fn,
param_space={"lr": tune.loguniform(1e-5, 1e-2), "hidden": tune.choice([128, 256, 512])},
tune_config=tune.TuneConfig(
scheduler=ASHAScheduler(metric="loss", mode="min"),
num_samples=100 # Try 100 configurations, stop bad ones early
)
)
results = tuner.fit()
**Ray Serve (Production Serving)**:
from ray import serve
@serve.deployment(num_replicas=3, ray_actor_options={"num_gpus": 1})
class LLMServe:
def __init__(self):
self.model = load_llm()
async def __call__(self, request):
data = await request.json()
return self.model.generate(data["prompt"])
serve.run(LLMServe.bind()) # Deploy with autoscaling
**Ray vs Alternatives**
| Framework | Strength | Weakness |
|-----------|---------|---------|
| Ray | Python-native, full ML lifecycle | Younger ecosystem than Spark |
| Dask | Pandas compatibility | Less ML-specific tooling |
| Spark | Enterprise scale, SQL | JVM overhead, Java API |
| Celery | Task queuing | No data-parallel computing |
| Kubernetes | Container orchestration | No Python-native compute |
Ray is **the Python-native distributed computing platform purpose-built for the AI era** — by treating clusters as a pool of Python functions and actors rather than JVM processes, Ray enables AI researchers and engineers to scale training, tuning, serving, and data processing workflows from laptop to cloud with minimal code changes and maximum ecosystem integration.