Attention Mechanisms and Transformer Applications

# Attention Mechanisms and Transformer Applications

## Introduction & Motivation

Attention mechanisms enable models to focus on relevant information, critical for processing sequential and structured data in NLP, time series, and engineering applications. Transformers based on attention have revolutionized machine learning, enabling efficient parallel processing and long-range dependencies crucial for scientific applications.

Motivation: Implement attention for improved learning on sequential and structured data.

Applications: Time series analysis, text processing, document understanding, sequential prediction.

---

## Core Concepts & Theory

### Self-Attention

Query-key-value mechanism.

### Multi-Head Attention

Parallel attention subspaces.

### Positional Encoding

Sequence position information.

### Attention Weights

Learned focus distribution.

---

## Mathematical Formulation

Scaled Dot-Product Attention:
$$ ext{Attention}(Q, K, V) = ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}} ight)V$$

Multi-Head:
$$ ext{MultiHead}(Q, K, V) = ext{Concat}( ext{head}_1, \ldots, ext{head}_h)W^O$$

Positional Encoding:
$$PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d})$$

---

## Advanced Theory & Extensions

### Cross-Attention

Attending to external sequences.

### Hierarchical Attention

Multi-level focus.

### Sparse Attention

Efficient computation.

---

## Computational Considerations

Self-Attention: O(n²·d) for n tokens, d dimensions.

Multi-Head: O(h·n²·d/h) = O(n²·d).

Transformer Block: O(n²·d + n·d²).

---

## Practical Implementation Strategies

### Attention Initialization

Weight initialization schemes.

### Mask Application

Causal and padding masks.

### Attention Dropout

Regularization techniques.

---

## Benchmark Datasets & Evaluation

GLUE: Language understanding.

SQuAD: Question answering.

Time Series: Sequential forecasting.

---

## Key Challenges & Limitations

### Quadratic Complexity

Scales poorly with sequence length.

### Interpretability

Understanding attention patterns.

### Training Stability

Large attention weights.

---

## Hyperparameter Tuning

Number of heads: 4-16.

Head dimension: 32-128.

Dropout: 0.0-0.3.

---

## Real-World Applications & Case Studies

Document Processing: Long document understanding.

Time Series: Sequential forecasting.

Scientific Texts: Literature mining.

---

## Integration with Other Methods

Attention + RNNs; + CNNs; + graph neural networks.

---

## Summary & Key Takeaways

Attention mechanisms enable focus on relevant information.

Principles:
1. Query-Key-Value: Define attention computation.
2. Scaling: Stabilize gradient flow.
3. Multi-Head: Capture diverse patterns.
4. Positional: Encode sequence position.
5. Application: Solve domain problems.

---

## Appendix: Practical Labs

### Lab 1: Scaled Dot-Product Attention

import numpy as np

def scaled_dot_product_attention(Q, K, V, mask=None):
 """Compute scaled dot-product attention"""
 d_k = Q.shape[-1]
 
 # Compute attention scores
 scores = Q @ K.T / np.sqrt(d_k)
 
 # Apply mask
 if mask is not None:
 scores = scores + mask * -1e9
 
 # Softmax
 attention_weights = np.softmax(scores, axis=-1)
 
 # Weighted sum of values
 output = attention_weights @ V
 
 return output, attention_weights

# Test
Q = np.random.randn(4, 8) # 4 queries, 8 dimensions
K = np.random.randn(6, 8) # 6 keys
V = np.random.randn(6, 10) # 6 values, 10 dimensions

output, weights = scaled_dot_product_attention(Q, K, V)

print(f"✓ Scaled dot-product attention:")
print(f" Output shape: {output.shape}")
print(f" Attention weights shape: {weights.shape}")
print(f" Attention sum: {weights.sum(axis=1)}") # Should be all 1s

### Lab 2: Multi-Head Attention

import numpy as np

class MultiHeadAttention:
 def __init__(self, d_model=64, num_heads=4):
 self.d_model = d_model
 self.num_heads = num_heads
 self.d_k = d_model // num_heads
 
 # Weights
 self.W_q = np.random.randn(d_model, d_model) * 0.1
 self.W_k = np.random.randn(d_model, d_model) * 0.1
 self.W_v = np.random.randn(d_model, d_model) * 0.1
 self.W_o = np.random.randn(d_model, d_model) * 0.1
 
 def forward(self, Q, K, V):
 """Forward pass"""
 batch_size = Q.shape[0]
 
 # Linear projections
 Q = Q @ self.W_q
 K = K @ self.W_k
 V = V @ self.W_v
 
 # Split into heads
 Q = Q.reshape(batch_size, -1, self.num_heads, self.d_k).transpose(0, 2, 1, 3)
 K = K.reshape(batch_size, -1, self.num_heads, self.d_k).transpose(0, 2, 1, 3)
 V = V.reshape(batch_size, -1, self.num_heads, self.d_k).transpose(0, 2, 1, 3)
 
 # Attention
 scores = Q @ K.transpose(0, 1, 3, 2) / np.sqrt(self.d_k)
 attention = np.softmax(scores, axis=-1)
 
 # Combine
 context = attention @ V
 context = context.transpose(0, 2, 1, 3).reshape(batch_size, -1, self.d_model)
 
 output = context @ self.W_o
 
 return output, attention

# Test
mha = MultiHeadAttention(d_model=64, num_heads=4)

Q = np.random.randn(2, 10, 64) # batch=2, seq_len=10, d_model=64
K = np.random.randn(2, 10, 64)
V = np.random.randn(2, 10, 64)

output, attention = mha.forward(Q, K, V)

print(f"✓ Multi-head attention:")
print(f" Output shape: {output.shape}")
print(f" Attention shape: {attention.shape}")
print(f" Number of heads: 4")

### Lab 3: Positional Encoding

import numpy as np

def positional_encoding(seq_length, d_model):
 """Create positional encoding"""
 position = np.arange(seq_length)[:, np.newaxis]
 dim_indices = np.arange(d_model)[np.newaxis, :]
 
 # Compute angles
 angle_rates = 1 / np.power(10000, (2 * (dim_indices // 2)) / d_model)
 angles = position * angle_rates
 
 # Apply sin to even indices and cos to odd indices
 angles[:, 0::2] = np.sin(angles[:, 0::2])
 angles[:, 1::2] = np.cos(angles[:, 1::2])
 
 return angles

# Test
seq_len = 10
d_model = 64

pos_enc = positional_encoding(seq_len, d_model)

print(f"✓ Positional encoding:")
print(f" Shape: {pos_enc.shape}")
print(f" First position (first 8 dims): {pos_enc[0, :8]}")
print(f" Second position (first 8 dims): {pos_enc[1, :8]}")
print(f" Range: [{pos_enc.min():.2f}, {pos_enc.max():.2f}]")

### Lab 4: Transformer Block

import numpy as np

class TransformerBlock:
 def __init__(self, d_model=64, num_heads=4, d_ff=256):
 self.d_model = d_model
 self.num_heads = num_heads
 
 # Attention
 self.W_q = np.random.randn(d_model, d_model) * 0.1
 self.W_k = np.random.randn(d_model, d_model) * 0.1
 self.W_v = np.random.randn(d_model, d_model) * 0.1
 self.W_o = np.random.randn(d_model, d_model) * 0.1
 
 # Feed-forward
 self.W_1 = np.random.randn(d_model, d_ff) * 0.1
 self.W_2 = np.random.randn(d_ff, d_model) * 0.1
 
 def forward(self, X):
 """Forward pass"""
 batch_size, seq_len, d_model = X.shape
 
 # Self-attention
 Q = X @ self.W_q
 K = X @ self.W_k
 V = X @ self.W_v
 
 # Simplified attention
 scores = Q @ K.transpose(0, 2, 1) / np.sqrt(d_model)
 attention = np.softmax(scores, axis=-1)
 context = attention @ V
 
 # Output projection
 attn_output = context @ self.W_o
 attn_output = X + attn_output # Residual
 
 # Feed-forward
 ff_hidden = np.maximum(0, attn_output @ self.W_1) # ReLU
 ff_output = ff_hidden @ self.W_2
 output = attn_output + ff_output # Residual
 
 return output

# Test
block = TransformerBlock(d_model=64, num_heads=4)

X = np.random.randn(2, 10, 64) # batch=2, seq_len=10
output = block.forward(X)

print(f"✓ Transformer block:")
print(f" Input shape: {X.shape}")
print(f" Output shape: {output.shape}")

---