controlnet
**ControlNet and Image Conditioning**
**What is ControlNet?**
ControlNet adds spatial conditioning to diffusion models, allowing precise control over generated images using edge maps, poses, depth maps, and more.
**Control Types**
| Control | Input | Use Case |
|---------|-------|----------|
| Canny Edge | Edge detection | Preserve structure |
| Pose | OpenPose skeleton | Character poses |
| Depth | Depth map | 3D-aware generation |
| Segmentation | Semantic masks | Layout control |
| Normal Map | Surface normals | Lighting/texture |
| Scribble | Hand-drawn lines | Sketch to image |
| LineArt | Line drawings | Illustration style |
**Basic Usage**
```python
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import cv2
import numpy as np
# Load ControlNet
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny")
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
controlnet=controlnet
)
# Prepare control image
image = cv2.imread("input.jpg")
edges = cv2.Canny(image, 100, 200)
# Generate
result = pipe(
prompt="a detailed architectural rendering",
image=edges,
num_inference_steps=30
).images[0]
```
**Multi-ControlNet**
Combine multiple controls:
```python
controlnets = [
ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny"),
ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-depth")
]
pipe = StableDiffusionControlNetPipeline.from_pretrained(
model_id,
controlnet=controlnets
)
result = pipe(
prompt="...",
image=[edge_image, depth_image],
controlnet_conditioning_scale=[1.0, 0.8]
)
```
**IP-Adapter**
Control generation with reference images:
```python
# Use reference image to guide style/content
pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models")
result = pipe(
prompt="a dog in the park",
ip_adapter_image=reference_image # Style reference
).images[0]
```
**Use Cases**
| Use Case | Controls |
|----------|----------|
| Architecture | Canny + Depth |
| Character design | Pose + Reference |
| Product visualization | Depth + Segmentation |
| Before/after edits | Canny (preserve structure) |
**Best Practices**
- Match control strength to desired fidelity
- Preprocess control images consistently
- Combine controls for more precise output
- Use lower strength for creative freedom