Convolutional Neural Networks Filters Pooling Architecture Design
# Convolutional Neural Networks: Filters, Pooling & Architecture Design
## Introduction & Motivation
Convolutional neural networks: operate on grid-like data (images). Convolutional layer: learn local features via filters. Pooling: spatial downsampling; reduce dimensionality. Architecture: stacking layers; LeNet → AlexNet → ResNets. Applications: image classification, detection, segmentation; dominant in computer vision.
Motivation: Images: local structure important. Convolutions capture local patterns; weight sharing reduces parameters.
Applications: Image classification, object detection, segmentation.
---
## Core Concepts & Theory
### Convolutional Layer
Apply filter across input; learn local feature detectors.
### Pooling
Spatial downsampling; max or average pooling.
### Receptive Field
Region of input affecting output; grows with depth.
---
## Mathematical Formulation
Convolution (2D):
$$y[i,j] = b + \sum_{k,l} w[k,l] \cdot x[i+k, j+l]$$
Pooling (max):
$$y[i,j] = \max_{k,l \in ext{window}} x[i+k, j+l]$$
Receptive field:
$$RF_l = RF_{l-1} + (K_l - 1) \cdot \prod_{i=1}^{l-1} S_i$$
where K_l = kernel size, S_i = stride at layer i.
---
## Advanced Theory & Extensions
### Dilated Convolutions
Sparse sampling; increase receptive field without parameters.
### Grouped Convolutions
Separate channel groups; reduce computation.
### Depthwise Separable
Depthwise (channel-wise) + pointwise; efficient.
---
## Computational Considerations
Conv layer: O(H·W·K²·C_in·C_out) for H×W input, K×K filter.
Pooling: O(H·W·pool_size²) linear; efficient.
Memory: Feature maps: O(H·W·C) per layer.
---
## Practical Implementation Strategies
### Architecture Design
Stack conv + pooling; gradually reduce spatial, increase channels.
### Skip Connections
ResNets: bypass layers; enable deep networks.
### Normalization
Batch norm after conv; stabilizes training.
---
## Benchmark Datasets & Evaluation
ImageNet: Standard; ResNet-50 baseline ~76% top-1.
CIFAR-10: Small; simple CNNs competitive.
MNIST: Toy; any reasonable CNN ~99%+.
---
## Key Challenges & Limitations
### Vanishing Activations
Deep networks; activations → 0; residuals help.
### Overfitting
Many parameters; data augmentation critical.
### Computational Cost
Large models slow; pruning, distillation help.
---
## Hyperparameter Tuning
Kernel size: 3×3 standard; 1×1 for efficiency.
Stride: 1 typical; 2 for downsampling.
Padding: Same-padding preserve spatial dimensions.
---
## Real-World Applications & Case Studies
Image Classification: ResNet/EfficientNet standard.
Object Detection: Faster R-CNN, YOLO; backbone + detection heads.
Segmentation: U-Net, DeepLab; encoder-decoder architecture.
---
## Integration with Other Methods
CNN + Batch Norm → standard combination.
CNN + Attention → channel/spatial attention modules.
---
## Summary & Key Takeaways
Convolutional neural networks via filters, pooling, and skip connections enable efficient local feature learning on grid-structured data.
Principles:
1. Convolution: local feature detection via filters.
2. Pooling: spatial downsampling; invariance.
3. Receptive field: grows with depth; controls context.
4. Skip connections: enable deep architectures.
5. Efficiency: weight sharing reduces parameters.
---
---
## Appendix: Practical Labs
### Lab 1: 2D Convolution
import torch
import torch.nn.functional as F
import numpy as np
def conv2d_manual(image, kernel, bias=0):
"""Manual 2D convolution"""
h, w = image.shape
k = kernel.shape[0]
output = torch.zeros(h - k + 1, w - k + 1)
for i in range(h - k + 1):
for j in range(w - k + 1):
patch = image[i:i+k, j:j+k]
output[i, j] = (patch * kernel).sum() + bias
return output
# Test
np.random.seed(42)
image = torch.randn(5, 5)
kernel = torch.randn(3, 3)
output = conv2d_manual(image, kernel)
assert output.shape == (3, 3), "Output shape correct"
assert torch.isfinite(output).all(), "All finite"
print("✓ 2D convolution working")
if __name__ == "__main__":
print("Lab 1: Conv2D - PASSED")### Lab 2: Pooling Operations
import numpy as np
def max_pool2d(image, pool_size=2):
"""Max pooling 2D"""
h, w = image.shape
ph, pw = pool_size, pool_size
output = np.zeros((h // ph, w // pw))
for i in range(h // ph):
for j in range(w // pw):
pool = image[i*ph:(i+1)*ph, j*pw:(j+1)*pw]
output[i, j] = pool.max()
return output
# Test
np.random.seed(42)
image = np.random.randn(8, 8)
output = max_pool2d(image, pool_size=2)
assert output.shape == (4, 4), "Output shape correct"
assert np.isfinite(output).all(), "All finite"
print("✓ Max pooling working")
if __name__ == "__main__":
print("Lab 2: Pooling - PASSED")### Lab 3: Receptive Field Computation
import numpy as np
def compute_receptive_field(layer_config):
"""Compute receptive field for architecture"""
# layer_config: list of (kernel_size, stride, padding)
rf = 1
stride = 1
for kernel, s, pad in layer_config:
rf = rf + (kernel - 1) * stride
stride = stride * s
return rf
# Test
config = [(3, 1, 1), (3, 2, 1), (3, 1, 1)]
rf = compute_receptive_field(config)
assert rf > 1, "RF should increase with depth"
print("✓ Receptive field computation working")
if __name__ == "__main__":
print("Lab 3: ReceptiveField - PASSED")### Lab 4: CNN Forward Pass
import torch
import torch.nn as nn
import numpy as np
class SimpleCNN(nn.Module):
def __init__(self, num_classes=10):
super().__init__()
self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
self.fc = nn.Linear(64 * 8 * 8, num_classes)
def forward(self, x):
x = self.pool(torch.relu(self.conv1(x)))
x = self.pool(torch.relu(self.conv2(x)))
x = x.view(x.size(0), -1)
x = self.fc(x)
return x
# Test
np.random.seed(42)
model = SimpleCNN(num_classes=10)
x = torch.randn(4, 3, 32, 32)
output = model(x)
assert output.shape == (4, 10), "Output shape correct"
print("✓ CNN forward pass working")
if __name__ == "__main__":
print("Lab 4: ForwardPass - PASSED")