ControlNet and Image Conditioning

What is ControlNet? ControlNet adds spatial conditioning to diffusion models, allowing precise control over generated images using edge maps, poses, depth maps, and more.

Control Types

Basic Usage

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import cv2
import numpy as np

# Load ControlNet
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny")
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet
)

# Prepare control image
image = cv2.imread("input.jpg")
edges = cv2.Canny(image, 100, 200)

# Generate
result = pipe(
    prompt="a detailed architectural rendering",
    image=edges,
    num_inference_steps=30
).images[0]

Multi-ControlNet Combine multiple controls:

controlnets = [
    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny"),
    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-depth")
]

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    model_id,
    controlnet=controlnets
)

result = pipe(
    prompt="...",
    image=[edge_image, depth_image],
    controlnet_conditioning_scale=[1.0, 0.8]
)

IP-Adapter Control generation with reference images:

# Use reference image to guide style/content
pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models")
result = pipe(
    prompt="a dog in the park",
    ip_adapter_image=reference_image  # Style reference
).images[0]

Use Cases

Best Practices

• Match control strength to desired fidelity
• Preprocess control images consistently
• Combine controls for more precise output
• Use lower strength for creative freedom