performance optimization
**Performance optimization** for ML systems encompasses **systematic approaches to improving speed, efficiency, and resource utilization** — profiling to identify bottlenecks, applying targeted optimizations like vectorization, batching, caching, and GPU tuning, enabling faster training, lower inference latency, and reduced costs.
**Why Performance Matters**
- **User Experience**: Faster responses improve satisfaction.
- **Cost**: Efficient code uses fewer resources.
- **Scale**: Optimization enables handling more load.
- **Iteration Speed**: Faster training means more experiments.
- **Competitive**: Speed is often a differentiator.
**Golden Rule: Profile First**
**Never Optimize Without Data**:
```python
# Python profiling
import cProfile
cProfile.run("main()", sort="cumtime")
# Line-by-line profiling
# pip install line_profiler
@profile
def my_function():
# code here
pass
# Run: kernprof -l -v script.py
```
**Memory Profiling**:
```python
# pip install memory_profiler
from memory_profiler import profile
@profile
def my_function():
large_list = [x for x in range(1000000)]
return sum(large_list)
```
**GPU Profiling**:
```bash
# NVIDIA tools
nvidia-smi dmon -s u # Utilization over time
nsys profile python train.py # Detailed trace
```
**Common Bottlenecks & Solutions**
**Slow Loops**:
```python
# ❌ Slow: Python loop
result = []
for x in data:
result.append(x * 2)
# ✅ Fast: Vectorized with NumPy
result = data * 2
# ✅ Fast: List comprehension (for non-numeric)
result = [x * 2 for x in data]
```
**Memory Issues**:
```python
# ❌ Bad: Load entire file
with open("huge_file.csv") as f:
data = f.readlines() # All in memory
# ✅ Good: Generator/streaming
def read_chunks(file_path, chunk_size=1000):
with open(file_path) as f:
while True:
chunk = list(itertools.islice(f, chunk_size))
if not chunk:
break
yield chunk
```
**I/O Bottlenecks**:
```python
# ❌ Sequential requests
results = []
for url in urls:
results.append(requests.get(url))
# ✅ Concurrent requests
import asyncio
import aiohttp
async def fetch_all(urls):
async with aiohttp.ClientSession() as session:
tasks = [session.get(url) for url in urls]
return await asyncio.gather(*tasks)
```
**LLM-Specific Optimizations**
**Quantization**:
```python
# Load in 4-bit for faster inference
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16
)
model = AutoModelForCausalLM.from_pretrained(
"model-name",
quantization_config=bnb_config
)
```
**Batching**:
```python
# ❌ Process one at a time
for prompt in prompts:
response = llm.generate(prompt)
# ✅ Batch process
responses = llm.generate(prompts, batch_size=16)
```
**Response Caching**:
```python
from functools import lru_cache
import hashlib
@lru_cache(maxsize=10000)
def cached_llm_call(prompt_hash):
return llm.generate(unhash(prompt_hash))
def call_with_cache(prompt):
h = hashlib.sha256(prompt.encode()).hexdigest()
return cached_llm_call(h)
```
**Streaming**:
```python
# Stream for perceived speed
for chunk in llm.generate(prompt, stream=True):
print(chunk, end="", flush=True)
```
**GPU Optimization**
**Maximize Utilization**:
```python
# Check current utilization
nvidia-smi --query-gpu=utilization.gpu --format=csv
# Increase batch size until GPU is ~80-90% utilized
# Too low utilization = wasted GPU capacity
# Use mixed precision
with torch.autocast("cuda"):
output = model(input)
```
**Memory Management**:
```python
# Clear cache when needed
torch.cuda.empty_cache()
# Delete unused tensors
del large_tensor
# Use gradient checkpointing
model.gradient_checkpointing_enable()
```
**Data Loading**:
```python
# Use multiple workers for data loading
dataloader = DataLoader(
dataset,
batch_size=32,
num_workers=8, # Parallel loading
pin_memory=True, # Faster GPU transfer
prefetch_factor=2
)
```
**Optimization Checklist**
```
□ Profile before optimizing
□ Identify actual bottleneck (CPU, GPU, I/O, memory)
□ Apply targeted fix
□ Measure improvement
□ Check for regressions
□ Document changes
□ Repeat until goals met
```
**Tools Summary**
```
Purpose | Tool
----------------|---------------------------
Python profile | cProfile, line_profiler
Memory profile | memory_profiler, tracemalloc
GPU profile | nvidia-smi, nsys, PyTorch profiler
Web/API | locust, k6
Benchmarking | pytest-benchmark, timeit
```
Performance optimization is **a systematic discipline, not guesswork** — measuring before optimizing ensures effort is focused on actual bottlenecks, leading to real improvements rather than premature optimization that adds complexity without benefit.