Attention Mechanisms Self-Attention Scaled Dot-Product

# Attention Mechanisms: Self-Attention & Scaled Dot-Product

## Introduction & Motivation

Attention: dynamically weight context by relevance. Scaled dot-product attention: query-key-value triple. Self-attention: attend within sequence; enables long-range dependencies. Applications: NLP, vision, multimodal models.

Motivation: RNNs struggle with long-range dependencies (vanishing gradients). Attention provides direct paths; parallelizable.

Applications: Machine translation, question answering, image captioning, visual reasoning.

---

## Core Concepts & Theory

### Query-Key-Value

Query: what to look for. Key: what each position offers. Value: information to aggregate.

### Scaled Dot-Product Attention

Similarity: dot product of query-key. Scale by √d_k; softmax; aggregate values.

### Multi-Head Attention

Multiple representation subspaces; captures diverse patterns.

---

## Mathematical Formulation

Scaled dot-product attention:
$$ ext{Attention}(Q, K, V) = ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}} ight)V$$

Multi-head attention:
$$ ext{MultiHead}(Q, K, V) = ext{Concat}( ext{head}_1, \ldots, ext{head}_h) W^O$$

where head_i = Attention(QW_i^Q, KW_i^K, VW_i^V).

---

## Advanced Theory & Extensions

### Masked Attention

Prevent attending to future positions; autoregressive generation.

### Relative Position Bias

Bias attention by relative distances; improves locality.

### Linear Attention

Approximate softmax attention; O(n) complexity.

---

## Computational Considerations

Attention: O(n² d) where n=sequence length, d=dimension.

Multi-head: O(n² d); linear in d due to head aggregation.

Memory: O(n²) for attention weights.

---

## Practical Implementation Strategies

### Dropout on Attention Weights

Prevent overfitting; random attention zeroing.

### Layer Normalization

Normalize inputs/outputs; stable training.

### Attention Visualization

Inspect learned attention patterns; interpretability.

---

## Benchmark Datasets & Evaluation

BLEU (Machine Translation): Attention baseline strong.

SQuAD (QA): Attention-based models excel.

COCO (Captioning): Attention over image regions.

---

## Key Challenges & Limitations

### Quadratic Complexity

O(n²) limits very long sequences.

### Position Awareness

Attention permutation-invariant; needs position encoding.

### Information Bottleneck

Single attention head may be insufficient.

---

## Hyperparameter Tuning

Number of heads: 4-16; typically 8.

Attention dropout: 0.1-0.3.

Scaling factor: √d_k (fixed); alternatively learnable.

---

## Real-World Applications & Case Studies

Machine Translation: Seq2Seq with attention; standard.

Visual QA: Attend over image regions by question.

Code Understanding: Attention for cross-references.

---

## Integration with Other Methods

Attention + LSTM → hybrid seq2seq (Bahdanau).

Attention + Transformer → full attention-based architecture.

---

## Summary & Key Takeaways

Attention mechanisms via scaled dot-product enable dynamic context weighting, with multi-head variants capturing diverse representations across sequence positions.

Principles:
1. Query-key-value: decompose input into three roles.
2. Softmax attention weights: probability distribution over positions.
3. Scaled by √d_k: prevent vanishing gradients.
4. Multi-head: diverse subspace representations.
5. Parallel computation: efficient vs sequential RNNs.

---

---

## Appendix: Practical Labs

### Lab 1: Scaled Dot-Product Attention

import torch
import torch.nn.functional as F
import numpy as np

def scaled_dot_product_attention(Q, K, V, mask=None):
 """Compute scaled dot-product attention"""
 d_k = Q.size(-1)
 
 # Attention scores
 scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(d_k)
 
 if mask is not None:
 scores = scores.masked_fill(mask == 0, float('-inf'))
 
 # Attention weights
 weights = F.softmax(scores, dim=-1)
 
 # Weighted sum of values
 output = torch.matmul(weights, V)
 
 return output, weights

# Test
Q = torch.randn(8, 10, 64) # [batch, seq_len, d_k]
K = torch.randn(8, 10, 64)
V = torch.randn(8, 10, 64)

output, weights = scaled_dot_product_attention(Q, K, V)

print(f"Attention output shape: {output.shape}, Weights shape: {weights.shape}")
assert output.shape == V.shape, "Should preserve value dimensions"
assert weights.shape == (8, 10, 10), "Should be attention matrix"
print("✓ Scaled attention working")

if __name__ == "__main__":
 print("Lab 1: Attention - PASSED")

### Lab 2: Multi-Head Attention

import torch
import torch.nn as nn

class MultiHeadAttention(nn.Module):
 def __init__(self, d_model=512, num_heads=8):
 super().__init__()
 self.num_heads = num_heads
 self.d_k = d_model // num_heads
 
 self.W_q = nn.Linear(d_model, d_model)
 self.W_k = nn.Linear(d_model, d_model)
 self.W_v = nn.Linear(d_model, d_model)
 self.W_o = nn.Linear(d_model, d_model)
 
 def forward(self, Q, K, V):
 batch_size = Q.size(0)
 
 # Linear projections
 Q = self.W_q(Q).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
 K = self.W_k(K).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
 V = self.W_v(V).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
 
 # Scaled dot-product attention
 scores = torch.matmul(Q, K.transpose(-2, -1)) / (self.d_k ** 0.5)
 weights = torch.softmax(scores, dim=-1)
 output = torch.matmul(weights, V)
 
 # Concatenate heads
 output = output.transpose(1, 2).contiguous().view(batch_size, -1, 512)
 output = self.W_o(output)
 
 return output

# Test
mha = MultiHeadAttention(d_model=512, num_heads=8)
Q = torch.randn(8, 10, 512)
K = torch.randn(8, 10, 512)
V = torch.randn(8, 10, 512)

output = mha(Q, K, V)

print(f"MHA output shape: {output.shape}")
assert output.shape == (8, 10, 512), "Should preserve dimensions"
print("✓ Multi-head attention working")

if __name__ == "__main__":
 print("Lab 2: MHA - PASSED")

### Lab 3: Masked Attention

import torch
import torch.nn.functional as F
import numpy as np

def masked_attention(Q, K, V, mask=None):
 """Attention with optional masking (causal, padding)"""
 d_k = Q.size(-1)
 
 scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(d_k)
 
 # Apply mask (0 = mask out, 1 = keep)
 if mask is not None:
 scores = scores.masked_fill(mask == 0, float('-inf'))
 
 weights = F.softmax(scores, dim=-1)
 weights = weights.masked_fill(mask == 0, 0) # Zero out masked positions
 
 output = torch.matmul(weights, V)
 
 return output, weights

# Test: causal mask (autoregressive)
seq_len = 10
causal_mask = torch.tril(torch.ones(seq_len, seq_len)).unsqueeze(0) # [1, 10, 10]

Q = torch.randn(2, seq_len, 64)
K = torch.randn(2, seq_len, 64)
V = torch.randn(2, seq_len, 64)

output, weights = masked_attention(Q, K, V, mask=causal_mask)

print(f"Masked attention output shape: {output.shape}")
assert output.shape == V.shape, "Should preserve value shape"
print("✓ Masked attention working")

if __name__ == "__main__":
 print("Lab 3: Masked Attention - PASSED")

### Lab 4: Attention Visualization

import torch
import torch.nn.functional as F
import numpy as np

def analyze_attention_heads(attention_weights):
 """Analyze attention head patterns"""
 # attention_weights: [batch, num_heads, seq_len, seq_len]
 
 batch_size, num_heads, seq_len, _ = attention_weights.shape
 
 # Entropy per head (measure of focus)
 entropy = -(attention_weights * torch.log(attention_weights + 1e-8)).sum(dim=-1).mean()
 
 # Diagonal focus (attending to self)
 diag_focus = torch.diagonal(attention_weights, dim1=-2, dim2=-1).mean()
 
 return entropy.item(), diag_focus.item()

# Test
weights = torch.softmax(torch.randn(2, 8, 10, 10), dim=-1)

entropy, diag = analyze_attention_heads(weights)

print(f"Attention entropy: {entropy:.4f}, Diagonal focus: {diag:.4f}")
assert 0 <= entropy, "Entropy should be non-negative"
assert 0 <= diag <= 1, "Diagonal focus should be in [0,1]"
print("✓ Attention analysis working")

if __name__ == "__main__":
 print("Lab 4: Analysis - PASSED")