Self-Attention - Scaled Dot-Product

# Self-Attention - Scaled Dot-Product

## Introduction & Motivation

Self-attention: relate sequence elements to each other. Scaled dot-product attention mechanism. Applications: foundational to transformers.

Motivation: Enable parallel sequence processing with global dependencies.

Applications: Transformers, sequence models, universal architecture.

---

## Core Concepts & Theory

### Query, Key, Value

Different projections for attention computation.

### Scaled Dot-Product

Normalize attention scores.

### Softmax

Attention weight normalization.

### Sequence Length Scaling

Handle variable sequence lengths.

---

## Mathematical Formulation

Scaled Dot-Product Attention:
$$ ext{Attention}(Q, K, V) = ext{softmax}(\frac{QK^T}{\sqrt{d_k}}) V$$

Query-Key Similarity:
$$ ext{similarity} = Q \cdot K^T$$

Scaling Factor:
$$ ext{scale} = \frac{1}{\sqrt{d_k}}$$

---

## Advanced Theory & Extensions

### Causal Masking

Prevent attending to future tokens.

### Relative Position Bias

Position-aware attention.

### Attention Dropout

Regularize attention patterns.

---

## Computational Considerations

Time complexity: O(T²·D).

Space complexity: O(T²).

Quadratic scaling: Bottleneck for long sequences.

---

## Practical Implementation Strategies

### Numerical Stability

Log-softmax for stability.

### Attention Masking

Apply mask before softmax.

### Gradient Flow

Careful gradient computation.

---

## Benchmark Datasets & Evaluation

Machine Translation: BLEU scores.

Language Modeling: Perplexity.

Question Answering: Accuracy metrics.

---

## Key Challenges & Limitations

### Computational Cost

Quadratic memory and time.

### Attention Sparsity

Most weight on few positions.

### Long Sequences

Difficult for very long inputs.

---

## Hyperparameter Tuning

Scaling factor: 1/sqrt(d_k).

Dropout: 0.1-0.3.

Masking threshold: Task-dependent.

---

## Real-World Applications & Case Studies

Machine Translation: Seq2Seq architecture.

Language Understanding: BERT foundation.

Question Answering: Retrieve relevant passages.

---

## Integration with Other Methods

Self-attention + positional encoding; + feed-forward network.

---

## Summary & Key Takeaways

Scaled dot-product attention enables efficient sequence processing.

Principles:
1. Query-key similarity: Dot product.
2. Scaling: Numerical stability.
3. Softmax: Attention weights.
4. Value weighting: Output generation.
5. Parallelism: Efficient computation.

---

## Appendix: Practical Labs

### Lab 1: Scaled Dot-Product Attention

import numpy as np

def scaled_dot_product_attention(query, key, value, mask=None):
 """Compute scaled dot-product attention"""
 d_k = query.shape[-1]
 
 # Compute similarity
 scores = query @ key.T / np.sqrt(d_k)
 
 # Apply mask
 if mask is not None:
 scores = scores + mask * -1e9
 
 # Softmax
 attention = np.exp(scores) / np.sum(np.exp(scores), axis=-1, keepdims=True)
 
 # Apply to values
 output = attention @ value
 
 return output, attention

np.random.seed(42)
q = np.random.randn(10, 64)
k = np.random.randn(10, 64)
v = np.random.randn(10, 64)
out, attn = scaled_dot_product_attention(q, k, v)
assert out.shape == (10, 64)
print("✓ Scaled dot-product attention working")

### Lab 2: Causal Masking

import numpy as np

def create_causal_mask(seq_len):
 """Create triangular causal mask"""
 mask = np.tril(np.ones((seq_len, seq_len)))
 mask = (1 - mask) * -1 # Invert: 0 for attend, -1 for mask
 return mask

mask = create_causal_mask(10)
assert mask.shape == (10, 10)
assert np.allclose(mask[0, 1:], -1)
print("✓ Causal mask working")

### Lab 3: Attention Entropy

import numpy as np

def attention_entropy(attention_weights):
 """Compute entropy of attention distribution"""
 epsilon = 1e-10
 entropy = -np.sum(attention_weights * np.log(attention_weights + epsilon), axis=-1)
 return entropy

np.random.seed(42)
attn = np.random.rand(10, 20)
attn = attn / attn.sum(axis=1, keepdims=True)
entropy = attention_entropy(attn)
assert entropy.shape == (10,)
assert np.all(entropy >= 0)
print("✓ Attention entropy computed")

### Lab 4: Query-Key Similarity

import numpy as np

def analyze_attention_similarity(query, key):
 """Analyze query-key similarity patterns"""
 d_k = query.shape[-1]
 
 # Compute raw scores
 scores = query @ key.T / np.sqrt(d_k)
 
 # Statistics
 mean_sim = np.mean(scores)
 max_sim = np.max(scores)
 min_sim = np.min(scores)
 
 return mean_sim, max_sim, min_sim

np.random.seed(42)
q = np.random.randn(10, 64)
k = np.random.randn(10, 64)
mean, max_s, min_s = analyze_attention_similarity(q, k)
assert mean > min_s and mean < max_s
print(f"✓ Similarity stats: mean={mean:.2f}, max={max_s:.2f}, min={min_s:.2f}")

---