learnable position embedding

**Learnable Position Embedding** is a **position encoding method where position vectors are treated as trainable parameters** — each position in the sequence has its own learned embedding vector that is added to the token embedding, allowing the model to discover optimal position representations. **How Does It Work?** - **Parameters**: $P in mathbb{R}^{N_{max} imes d}$ — one $d$-dimensional vector per position. - **Application**: $x_i' = x_i + P_i$ (add position embedding to token embedding). - **Training**: Position vectors are optimized via backpropagation alongside all other parameters. - **Used In**: BERT, GPT-2, ViT, most modern transformers. **Why It Matters** - **Simplicity**: The simplest position encoding — just add learned vectors. - **Flexibility**: The model discovers whatever positional patterns are useful for the task. - **Limitation**: Fixed maximum sequence length. Cannot generalize to longer sequences than training. **Learnable Position Embedding** is **the model teaching itself about position** — letting optimization discover the best way to encode sequential or spatial position.