Attention Mechanisms Transformers Deep Dive

# Attention Mechanisms & Transformers Deep Dive

## Introduction & Motivation

Attention Mechanisms: focus computation on relevant elements. Scaled dot-product attention, multi-head attention. Applications: sequence modeling, vision, multimodal tasks.

Motivation: Enable models to selectively focus on important information.

Applications: Machine translation, image captioning, visual reasoning.

---

## Core Concepts & Theory

### Query-Key-Value Framework

Compute relevance and aggregate.

### Scaled Dot-Product Attention

Normalize attention by dimension.

### Multi-Head Attention

Parallel attention subspaces.

### Positional Encoding

Inject position information.

---

## Mathematical Formulation

Scaled Dot-Product Attention:
$$ ext{Attention}(Q, K, V) = ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}} ight)V$$

Multi-Head Attention:
$$ ext{MultiHead}(Q,K,V) = ext{Concat}( ext{head}_1,..., ext{head}_h)W^O$$

Positional Encoding:
$$PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d}), \quad PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d})$$

---

## Advanced Theory & Extensions

### Cross-Attention

Query from one sequence, key/value from another.

### Self-Attention

Query, key, value from same sequence.

### Relative Position Bias

Learnable relative position embeddings.

---

## Computational Considerations

Attention: O(T²·d).

Multi-head: O(h·T²·d).

Positional encoding: O(T·d).

---

## Practical Implementation Strategies

### Efficient Attention

Linear attention, sparse patterns.

### Attention Visualization

Interpret learned patterns.

### Dropout

Regularize attention weights.

---

## Benchmark Datasets & Evaluation

WMT14: Machine translation benchmark.

ImageNet: Vision transformer evaluation.

GLUE: Language understanding tasks.

---

## Key Challenges & Limitations

### Quadratic Complexity

Scales poorly with sequence length.

### Memory Usage

Attention matrices consume memory.

### Inductive Bias

Limited positional inductive bias.

---

## Hyperparameter Tuning

Attention heads: 8-16.

Hidden dimension: 512-2048.

Dropout rate: 0.1-0.2.

---

## Real-World Applications & Case Studies

Neural Machine Translation: Seq2seq translation.

Vision Transformers: Image classification.

Multimodal Learning: Vision-language models.

---

## Integration with Other Methods

Attention + CNN for hybrid architectures; + RNN for recurrent attention.

---

## Summary & Key Takeaways

Attention mechanisms enable selective focus via learned weights.

Principles:
1. Query-key-value: Relevance computation.
2. Softmax: Normalization.
3. Multi-head: Diverse representations.
4. Positional encoding: Position awareness.
5. Efficiency: Trade-offs in scaling.

---

## Appendix: Practical Labs

### Lab 1: Scaled Dot-Product Attention

import numpy as np

def scaled_dot_product_attention(query, key, value, d_k):
 """Compute scaled dot-product attention"""
 scores = query @ key.T / np.sqrt(d_k)
 attn_weights = np.exp(scores) / np.sum(np.exp(scores), axis=1, keepdims=True)
 output = attn_weights @ value
 return output, attn_weights

np.random.seed(42)
q = np.random.randn(5, 64)
k = np.random.randn(5, 64)
v = np.random.randn(5, 64)
output, weights = scaled_dot_product_attention(q, k, v, d_k=64)
assert output.shape == v.shape, "Correct output shape"
print("✓ Scaled dot-product attention working")

### Lab 2: Multi-Head Attention

import numpy as np

def multi_head_attention(query, key, value, num_heads, d_k):
 """Multi-head attention computation"""
 batch_size, seq_len, d = query.shape
 head_outputs = []
 
 for h in range(num_heads):
 q_h = query[:, :, h*d_k:(h+1)*d_k]
 k_h = key[:, :, h*d_k:(h+1)*d_k]
 v_h = value[:, :, h*d_k:(h+1)*d_k]
 
 scores = q_h @ k_h.transpose(0, 2, 1) / np.sqrt(d_k)
 attn = np.exp(scores) / np.sum(np.exp(scores), axis=2, keepdims=True)
 head_outputs.append(attn @ v_h)
 
 return np.concatenate(head_outputs, axis=2)

np.random.seed(42)
q = np.random.randn(2, 5, 512)
k = np.random.randn(2, 5, 512)
v = np.random.randn(2, 5, 512)
output = multi_head_attention(q, k, v, num_heads=8, d_k=64)
assert output.shape == v.shape, "Correct output shape"
print("✓ Multi-head attention working")

### Lab 3: Positional Encoding

import numpy as np

def positional_encoding(seq_len, d_model):
 """Generate positional encodings"""
 pe = np.zeros((seq_len, d_model))
 position = np.arange(seq_len)[:, np.newaxis]
 div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000) / d_model))
 
 pe[:, 0::2] = np.sin(position * div_term)
 pe[:, 1::2] = np.cos(position * div_term)
 
 return pe

pe = positional_encoding(seq_len=100, d_model=512)
assert pe.shape == (100, 512), "Correct PE shape"
print("✓ Positional encoding working")

### Lab 4: Attention Masking

import numpy as np

def apply_attention_mask(attention_scores, mask):
 """Apply mask to attention scores"""
 masked = attention_scores.copy()
 masked[mask == 0] = -np.inf
 attn_weights = np.exp(masked) / np.sum(np.exp(masked), axis=1, keepdims=True)
 return np.nan_to_num(attn_weights)

np.random.seed(42)
scores = np.random.randn(5, 5)
mask = np.triu(np.ones((5, 5)), k=1)
weights = apply_attention_mask(scores, mask)
assert weights.shape == scores.shape, "Correct weights shape"
print("✓ Attention masking working")

---