Positional Encoding Position Variants

# Positional Encoding & Position Variants

## Introduction & Motivation

Positional encoding: add position information to embeddings. Critical for transformers. Applications: sequence models, position awareness.

Motivation: Enable models to understand token positions.

Applications: Transformers, RNNs, sequence modeling.

---

## Core Concepts & Theory

### Absolute Positions

Fixed position embeddings.

### Relative Positions

Position-relative attention.

### Rotary Embeddings

Encode rotation in complex space.

### ALiBi

Attention with Linear Biases.

---

## Mathematical Formulation

Sinusoidal Encoding:
$$PE(pos, 2i) = \sin(pos/10000^{2i/d})$$
$$PE(pos, 2i+1) = \cos(pos/10000^{2i/d})$$

Rotary:
$$ ext{RoPE}(x, m) = x \cdot e^{im heta}$$

ALiBi Bias:
$$ ext{bias}(i,j) = -\alpha |i - j|$$

---

## Advanced Theory & Extensions

### Extrapolation

Handle longer sequences.

### Multi-Scale Positions

Hierarchical positioning.

### Length Extrapolation

Generalize to longer sequences.

---

## Computational Considerations

Sinusoidal: O(1) per position.

Learnable: O(L·D).

Rotary: O(1) per dimension.

---

## Practical Implementation Strategies

### Position Frequency

Choose base for frequencies.

### Embedding Dimension

Allocate to position info.

### Interpolation

Extend to longer lengths.

---

## Benchmark Datasets & Evaluation

Length Extrapolation: Test generalization.

Position Sensitivity: Analyze effectiveness.

Downstream Tasks: Evaluate on applications.

---

## Key Challenges & Limitations

### Extrapolation

Limited to trained lengths.

### Efficiency

May increase computation.

### Interactions

Complex position-content interplay.

---

## Hyperparameter Tuning

Base: 10000 typical.

Dimension: 50%-100% of d_model.

Learnable: Vs fixed tradeoff.

---

## Real-World Applications & Case Studies

Language Models: GPT, BERT.

Vision: ViT positional embeddings.

Long Sequences: Document understanding.

---

## Integration with Other Methods

Positional encoding + self-attention; + rotary for efficiency.

---

## Summary & Key Takeaways

Positional encoding enables position awareness.

Principles:
1. Sinusoidal: Fixed, extrapolatable.
2. Learnable: Flexible, task-specific.
3. Rotary: Efficient rotation-based.
4. ALiBi: Simple linear bias.
5. Extrapolation: Generalize to longer sequences.

---

## Appendix: Practical Labs

### Lab 1: Sinusoidal Encoding

import numpy as np

def sinusoidal_encoding(seq_len, d_model, base=10000):
 """Create sinusoidal positional encoding"""
 pos = np.arange(seq_len)[:, np.newaxis]
 div = np.exp(np.arange(0, d_model, 2) * -(np.log(base) / d_model))
 
 pos_enc = np.zeros((seq_len, d_model))
 pos_enc[:, 0::2] = np.sin(pos * div)
 pos_enc[:, 1::2] = np.cos(pos * div)
 
 return pos_enc

enc = sinusoidal_encoding(100, 512)
assert enc.shape == (100, 512)
print("✓ Sinusoidal encoding working")

### Lab 2: Rotary Embeddings

import numpy as np

def rotary_embedding(x, m, base=10000):
 """Apply rotary positional embedding"""
 d = x.shape[-1]
 theta = base ** (-2 * np.arange(0, d, 2) / d)
 
 # Rotation angle per position
 angle = m * theta
 
 # Apply rotation (simplified)
 x_rot = x.copy()
 x_rot[0::2] = x[0::2] * np.cos(angle) - x[1::2] * np.sin(angle)
 x_rot[1::2] = x[0::2] * np.sin(angle) + x[1::2] * np.cos(angle)
 
 return x_rot

np.random.seed(42)
x = np.random.randn(512)
rot = rotary_embedding(x, 10)
assert rot.shape == x.shape
print("✓ Rotary embedding working")

### Lab 3: ALiBi Bias

import numpy as np

def alibi_bias(seq_len, num_heads, alpha=1.0):
 """Attention with Linear Biases"""
 # Distance matrix
 distances = np.abs(np.arange(seq_len)[:, np.newaxis] - 
 np.arange(seq_len)[np.newaxis, :])
 
 # Linear bias per head
 biases = []
 for h in range(num_heads):
 head_alpha = alpha * (2 ** (-8 * h / num_heads))
 bias = -head_alpha * distances
 biases.append(bias)
 
 return np.array(biases)

biases = alibi_bias(100, 8)
assert biases.shape == (8, 100, 100)
print("✓ ALiBi bias working")

### Lab 4: Position Interpolation

import numpy as np

def interpolate_positions(old_max_len, new_max_len, pos_emb):
 """Interpolate positional embeddings to new length"""
 # Linear interpolation
 old_pos = np.linspace(0, 1, old_max_len)
 new_pos = np.linspace(0, 1, new_max_len)
 
 # Interpolate each dimension
 interp = np.interp(new_pos, old_pos, pos_emb[:, 0])
 
 return interp

np.random.seed(42)
pos_emb = np.random.randn(512, 768)
interp = interpolate_positions(512, 1024, pos_emb)
assert len(interp) == 1024
print("✓ Position interpolation working")

---