Attention Mechanisms Self-Attention Scaled Dot-Product
# Attention Mechanisms: Self-Attention & Scaled Dot-Product
## Introduction & Motivation
Attention mechanisms: focus on relevant parts of input. Self-attention: query-key-value interaction; all-to-all connections. Scaled dot-product: efficient attention computation. Multi-head attention: multiple representation subspaces. Applications: transformers, sequence transduction, machine translation, NLP.
Motivation: Sequential models limited by context window. Attention enables long-range dependencies without recurrence.
Applications: NLP, vision transformers, sequence models.
---
## Core Concepts & Theory
### Attention Function
Query attends to key-value pairs; weighted sum of values.
### Scaled Dot-Product
Normalization by √d_k; computational efficiency.
### Multi-Head Attention
Parallel attention heads; diverse representation.
---
## Mathematical Formulation
Scaled dot-product attention:
$$ ext{Attention}(Q, K, V) = ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}}
ight)V$$
where Q = query, K = key, V = value.
Multi-head attention:
$$ ext{MultiHead}(Q, K, V) = ext{Concat}( ext{head}_1, \ldots, ext{head}_h)W^O$$
$$ ext{head}_i = ext{Attention}(QW_i^Q, KW_i^K, VW_i^V)$$
---
## Advanced Theory & Extensions
### Additive Attention (Bahdanau)
Learnable alignment score; non-linear.
### Multiplicative Attention
Bilinear scoring; efficient dot-product.
### Sparse Attention
Reduce O(n²) via local/strided patterns.
---
## Computational Considerations
Attention: O(n² d) quadratic in sequence length.
Multi-head: O(h · n² d) where h = number heads.
Sparse: O(n d log n) or O(n d) locality dependent.
---
## Practical Implementation Strategies
### Positional Encoding
Inject position information; absolute/relative.
### Attention Masking
Prevent future attention; causal masking.
### Scaling Factor
√d_k prevents saturation; improves gradient flow.
---
## Benchmark Datasets & Evaluation
Machine Translation: WMT benchmarks; BLEU standard.
Question Answering: SQuAD; exact match and F1.
Text Generation: GLUE benchmark suite.
---
## Key Challenges & Limitations
### Quadratic Complexity
O(n²) memory and compute; limits sequence length.
### Positional Bias
Learned vs. sinusoidal; empirical tradeoffs.
### Attention Collapse
All heads may converge; diversity loss.
---
## Hyperparameter Tuning
Number of heads: 8-16 typical; dimension divisible.
Head dimension: 64-128; total 512-1024.
Attention dropout: 0.0-0.1; regularization.
---
## Real-World Applications & Case Studies
Machine Translation: Transformers standard; seq2seq.
Language Models: GPT; autoregressive generation.
Visual Features: Vision transformers; patch embedding.
---
## Integration with Other Methods
Attention + RNN → hybrid; combines attention + recurrence.
Attention + CNN → fusion; visual feature refinement.
---
## Summary & Key Takeaways
Attention mechanisms via scaled dot-product and multi-head architectures enable efficient sequence modeling and long-range dependency capture.
Principles:
1. Query-key-value: interaction framework.
2. Scaled dot-product: normalized attention.
3. Multi-head: parallel representations.
4. Positional encoding: position awareness.
5. Masking: causal constraints.
---
---
## Appendix: Practical Labs
### Lab 1: Scaled Dot-Product Attention
import numpy as np
def scaled_dot_product_attention(Q, K, V, mask=None):
"""Compute scaled dot-product attention"""
d_k = Q.shape[-1]
# Scores: Q @ K^T / sqrt(d_k)
scores = np.matmul(Q, K.T) / np.sqrt(d_k)
# Mask (optional)
if mask is not None:
scores = np.where(mask, scores, -1e9)
# Softmax
weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True))
weights = weights / weights.sum(axis=-1, keepdims=True)
# Weighted sum of values
output = np.matmul(weights, V)
return output, weights
# Test
np.random.seed(42)
Q = np.random.randn(4, 8) # 4 queries, dim 8
K = np.random.randn(4, 8) # 4 keys, dim 8
V = np.random.randn(4, 8) # 4 values, dim 8
output, weights = scaled_dot_product_attention(Q, K, V)
assert output.shape == (4, 8), "Output shape"
assert weights.shape == (4, 4), "Weight shape"
assert np.allclose(weights.sum(axis=-1), 1), "Weights sum to 1"
print("✓ Scaled dot-product attention working")
if __name__ == "__main__":
print("Lab 1: Attention - PASSED")### Lab 2: Multi-Head Attention
import numpy as np
def multi_head_attention(Q, K, V, num_heads=4):
"""Multi-head attention"""
d_model = Q.shape[-1]
d_k = d_model // num_heads
# Split into heads
heads_output = []
for h in range(num_heads):
start = h * d_k
end = (h + 1) * d_k
Q_h = Q[:, start:end]
K_h = K[:, start:end]
V_h = V[:, start:end]
# Attention per head
scores = np.matmul(Q_h, K_h.T) / np.sqrt(d_k)
weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True))
weights = weights / weights.sum(axis=-1, keepdims=True)
head_out = np.matmul(weights, V_h)
heads_output.append(head_out)
# Concatenate
output = np.concatenate(heads_output, axis=-1)
return output
# Test
np.random.seed(42)
Q = np.random.randn(4, 8)
K = np.random.randn(4, 8)
V = np.random.randn(4, 8)
output = multi_head_attention(Q, K, V, num_heads=4)
assert output.shape == (4, 8), "Output shape"
print("✓ Multi-head attention working")
if __name__ == "__main__":
print("Lab 2: MultiHeadAttention - PASSED")### Lab 3: Positional Encoding
import numpy as np
def positional_encoding(seq_length, d_model):
"""Sinusoidal positional encoding"""
position = np.arange(seq_length)[:, np.newaxis]
div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
pe = np.zeros((seq_length, d_model))
pe[:, 0::2] = np.sin(position * div_term)
pe[:, 1::2] = np.cos(position * div_term)
return pe
# Test
np.random.seed(42)
pe = positional_encoding(10, 8)
assert pe.shape == (10, 8), "PE shape"
assert not np.allclose(pe[0], pe[1]), "Different positions differ"
print("✓ Positional encoding working")
if __name__ == "__main__":
print("Lab 3: PositionalEncoding - PASSED")### Lab 4: Causal Masking
import numpy as np
def create_causal_mask(seq_length):
"""Create causal (lower triangular) mask"""
mask = np.tril(np.ones((seq_length, seq_length))) == 1
return mask
def apply_causal_mask(scores, mask):
"""Apply causal mask to attention scores"""
scores = np.where(mask, scores, -1e9)
return scores
# Test
np.random.seed(42)
mask = create_causal_mask(5)
assert mask.shape == (5, 5), "Mask shape"
assert mask[0, 0] == True, "Diagonal masked"
assert mask[0, 1] == False, "Future masked"
scores = np.random.randn(5, 5)
masked_scores = apply_causal_mask(scores, mask)
assert masked_scores[0, 1] <= -1e8, "Future masked out"
print("✓ Causal masking working")
if __name__ == "__main__":
print("Lab 4: CausalMask - PASSED")