Residual Networks Resnet

# Residual Networks (ResNet)

## Introduction & Motivation

ResNet: learn residual functions instead of direct mappings. Enable training of very deep networks. Applications: deep learning foundation, transfer learning.

Motivation: Solve vanishing gradient problem with residual connections.

Applications: Image classification, transfer learning, backbone for detection/segmentation.

---

## Core Concepts & Theory

### Residual Blocks

Learn incremental changes.

### Skip Connections

Bypass for gradient flow.

### Bottleneck Architecture

1×1 → 3×3 → 1×1 convolutions.

### Shortcut Variants

Identity vs projection.

---

## Mathematical Formulation

Residual Block:
$$y = F(x) + x$$

Bottleneck Residual:
$$y = ext{Conv1x1}( ext{Conv3x3}( ext{Conv1x1}(x))) + x$$

Deep Networks:
$$ ext{Depth: 50, 101, 152 layers}$$

---

## Advanced Theory & Extensions

### ResNeXt

Group convolutions.

### Wide ResNet

Increased width.

### Improved ResNet

Better initialization.

---

## Computational Considerations

Parameters: 50 layers ≈ 25M params.

Computation: O(H·W·C²·K²).

Inference: Real-time on GPU.

---

## Practical Implementation Strategies

### Batch Normalization

After convolution.

### Activation Function

ReLU typical.

### Residual Path

Identity or projection.

---

## Benchmark Datasets & Evaluation

ImageNet: Classification benchmark.

COCO: Detection and segmentation.

Transfer tasks: Fine-tuning experiments.

---

## Key Challenges & Limitations

### Deep Networks

Optimization still challenging.

### Skip Connection Design

When to use projection.

### Initialization

Careful setup needed.

---

## Hyperparameter Tuning

Block depth: 50, 101, 152.

Bottleneck ratio: 4 typical.

Learning rate: 0.01.

---

## Real-World Applications & Case Studies

ImageNet: State-of-the-art results.

Transfer Learning: Excellent backbone.

Detection: YOLO, Faster R-CNN.

---

## Integration with Other Methods

ResNet + FPN for detection; + attention for vision transformer.

---

## Summary & Key Takeaways

ResNet enables training of very deep networks.

Principles:
1. Residual: Learn increments.
2. Skip connections: Gradient flow.
3. Bottleneck: Parameter efficiency.
4. Scaling: Depth or width.
5. Versatility: Detection, segmentation backbone.

---

## Appendix: Practical Labs

### Lab 1: Residual Block

import numpy as np

def residual_block(x, W1, W2, W3):
 """Residual block: y = F(x) + x"""
 # Convolutions
 h = x @ W1
 h = np.maximum(h, 0) # ReLU
 h = h @ W2
 h = np.maximum(h, 0)
 h = h @ W3
 
 # Residual connection
 y = h + x
 return y

np.random.seed(42)
x = np.random.randn(10, 64)
W1 = np.random.randn(64, 64)
W2 = np.random.randn(64, 64)
W3 = np.random.randn(64, 64)
y = residual_block(x, W1, W2, W3)
assert y.shape == x.shape
print("✓ Residual block working")

### Lab 2: Bottleneck Block

import numpy as np

def bottleneck_block(x, W_down, W_mid, W_up, reduction=4):
 """Bottleneck block: 1x1 -> 3x3 -> 1x1"""
 # Dimension reduction
 h = x @ W_down # D → D/r
 h = np.maximum(h, 0)
 
 # Main computation
 h = h @ W_mid
 h = np.maximum(h, 0)
 
 # Dimension restoration
 h = h @ W_up # D/r → D
 
 # Residual
 y = h + x
 return y

np.random.seed(42)
x = np.random.randn(10, 64)
W_down = np.random.randn(64, 16)
W_mid = np.random.randn(16, 16)
W_up = np.random.randn(16, 64)
y = bottleneck_block(x, W_down, W_mid, W_up)
assert y.shape == x.shape
print("✓ Bottleneck block working")

### Lab 3: Skip Connection Types

import numpy as np

def identity_skip(x, W):
 """Identity skip connection"""
 y = x @ W + x
 return y

def projection_skip(x, W_main, W_proj):
 """Projection skip for dimension mismatch"""
 h = x @ W_main
 x_proj = x @ W_proj
 y = h + x_proj
 return y

np.random.seed(42)
x = np.random.randn(10, 64)
W = np.random.randn(64, 64)
W_proj = np.random.randn(64, 128)

y_identity = identity_skip(x, W)
assert y_identity.shape == x.shape
print("✓ Skip connections working")

### Lab 4: Depth Comparison

import numpy as np

def estimate_depth_parameters(depths=[50, 101, 152]):
 """Estimate parameters for different depths"""
 results = []
 for depth in depths:
 # Simplified: base params + residual blocks
 params = 23e6 + (depth - 3) * 0.2e6 # Rough estimate
 results.append({'depth': depth, 'params': params / 1e6})
 return results

depths = estimate_depth_parameters()
print(f"✓ Depth analysis: {depths}")

---