Pooling Operations Max Average Attention Pooling
# Pooling Operations: Max, Average & Attention Pooling
## Introduction & Motivation
Pooling operations: downsample spatial dimensions. Max pooling: preserve strong activations. Average pooling: aggregate information. Attention pooling: weighted aggregation. Applications: feature extraction, dimensionality reduction, spatial invariance.
Motivation: Reduce computation; preserve important features. Provide translation invariance; robustness.
Applications: CNNs, sequence models, feature extraction.
---
## Core Concepts & Theory
### Max Pooling
Select maximum in window; non-smooth.
### Average Pooling
Average values in window; smooth aggregation.
### Attention Pooling
Learned weights; adaptive aggregation.
---
## Mathematical Formulation
Max pooling:
$$y = \max_{i \in W} x_i$$
where W = pooling window.
Average pooling:
$$y = \frac{1}{|W|} \sum_{i \in W} x_i$$
Attention pooling:
$$y = \sum_i \alpha_i x_i \quad ext{where} \quad \alpha_i = \frac{\exp(\beta^T x_i)}{\sum_j \exp(\beta^T x_j)}$$
---
## Advanced Theory & Extensions
### Stochastic Pooling
Randomly select in window.
### Learnable Pooling
Parametrized aggregation.
### Multi-Scale Pooling
Multiple scales combined; SPP-Net.
---
## Computational Considerations
Max/Average: O(W²) per spatial location.
Attention: O(N · D) where N = positions, D = dimension.
SPP: O(multiple scales) parallel computation.
---
## Practical Implementation Strategies
### Stride and Padding
Control output size; typical stride = pool size.
### Overlap vs. Non-Overlap
Overlapping improves performance; higher compute.
### Global Average Pooling
Reduce to single value per channel; fully convolutional.
---
## Benchmark Datasets & Evaluation
ImageNet: Max pooling standard; stride 2, size 2.
CIFAR-10: Average pooling competitive.
Video Understanding: Temporal pooling variants.
---
## Key Challenges & Limitations
### Information Loss
Max pooling discards most values; lossy.
### Gradient Flow
Non-differentiable maxima; straight-through estimators.
### Scale Sensitivity
Different image sizes; adaptive pooling.
---
## Hyperparameter Tuning
Pool size: 2x2 standard; 3x3 sometimes.
Stride: Equal to pool size typical; overlap possible.
Padding: Usually 0; preserve boundaries.
---
## Real-World Applications & Case Studies
Image Classification: Max pooling standard; ResNets.
Object Detection: ROI pooling; spatial normalization.
Sequence Models: Global average pooling; remove length.
---
## Integration with Other Methods
Pooling + Convolution → CNN hierarchy.
Pooling + Attention → hybrid aggregation.
---
## Summary & Key Takeaways
Pooling operations via max, average, and attention aggregation provide efficient spatial downsampling and feature extraction.
Principles:
1. Max: preserve strong features.
2. Average: smooth aggregation.
3. Attention: learned weighting.
4. Global: reduce to summary.
5. Adaptive: handle variable sizes.
---
---
## Appendix: Practical Labs
### Lab 1: Max Pooling
import numpy as np
def max_pooling_2d(x, pool_size=2, stride=None):
"""2D max pooling"""
if stride is None:
stride = pool_size
B, C, H, W = x.shape
H_out = (H - pool_size) // stride + 1
W_out = (W - pool_size) // stride + 1
output = np.zeros((B, C, H_out, W_out))
for b in range(B):
for c in range(C):
for i in range(H_out):
for j in range(W_out):
h_start = i * stride
w_start = j * stride
window = x[b, c, h_start:h_start+pool_size, w_start:w_start+pool_size]
output[b, c, i, j] = window.max()
return output
# Test
np.random.seed(42)
x = np.random.randn(2, 3, 8, 8)
y = max_pooling_2d(x, pool_size=2, stride=2)
assert y.shape == (2, 3, 4, 4), "Output shape"
assert y.max() <= x.max(), "Max pooling property"
print("✓ Max pooling working")
if __name__ == "__main__":
print("Lab 1: MaxPooling - PASSED")### Lab 2: Average Pooling
import numpy as np
def average_pooling_2d(x, pool_size=2, stride=None):
"""2D average pooling"""
if stride is None:
stride = pool_size
B, C, H, W = x.shape
H_out = (H - pool_size) // stride + 1
W_out = (W - pool_size) // stride + 1
output = np.zeros((B, C, H_out, W_out))
for b in range(B):
for c in range(C):
for i in range(H_out):
for j in range(W_out):
h_start = i * stride
w_start = j * stride
window = x[b, c, h_start:h_start+pool_size, w_start:w_start+pool_size]
output[b, c, i, j] = window.mean()
return output
# Test
np.random.seed(42)
x = np.random.randn(2, 3, 8, 8)
y = average_pooling_2d(x, pool_size=2, stride=2)
assert y.shape == (2, 3, 4, 4), "Output shape"
print("✓ Average pooling working")
if __name__ == "__main__":
print("Lab 2: AveragePooling - PASSED")### Lab 3: Global Average Pooling
import numpy as np
def global_average_pooling(x):
"""Global average pooling: reduce spatial dims"""
B, C, H, W = x.shape
output = x.mean(axis=(2, 3)) # (B, C)
return output
# Test
np.random.seed(42)
x = np.random.randn(4, 64, 8, 8)
y = global_average_pooling(x)
assert y.shape == (4, 64), "Output shape: (B, C)"
print("✓ Global average pooling working")
if __name__ == "__main__":
print("Lab 3: GlobalAveragePooling - PASSED")### Lab 4: Attention Pooling
import numpy as np
def attention_pooling(x, attention_weights=None):
"""Attention pooling: weighted aggregation"""
if attention_weights is None:
# Learned attention weights (simple case)
B, C, H, W = x.shape
scores = np.random.randn(B, H * W)
attention_weights = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)
# Reshape for multiplication
B, C, H, W = x.shape
x_flat = x.reshape(B, C, -1) # (B, C, H*W)
# Weighted sum
output = (x_flat * attention_weights[:, np.newaxis, :]).sum(axis=2) # (B, C)
return output
# Test
np.random.seed(42)
x = np.random.randn(4, 64, 8, 8)
y = attention_pooling(x)
assert y.shape == (4, 64), "Output shape"
print("✓ Attention pooling working")
if __name__ == "__main__":
print("Lab 4: AttentionPooling - PASSED")