sinusoidal position encoding

**Sinusoidal Position Encoding** is the **original position encoding from the Transformer paper** — using fixed sine and cosine functions at different frequencies to encode absolute position, based on the idea that relative positions can be represented as linear transformations. **How Does It Work?** - **Formula**: $PE_{(pos, 2i)} = sin(pos / 10000^{2i/d})$, $PE_{(pos, 2i+1)} = cos(pos / 10000^{2i/d})$ - **Frequencies**: Each dimension uses a different frequency, from high (position 0) to low (position $d-1$). - **Relative Position**: $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$ for any fixed $k$. - **Paper**: Vaswani et al. (2017). **Why It Matters** - **No Parameters**: Completely deterministic — no learnable parameters for position encoding. - **Extrapolation**: Can theoretically encode positions beyond the training length. - **Foundation**: Inspired RoPE, ALiBi, and other modern position encodings. **Sinusoidal Position Encoding** is **the mathematical clock of the original Transformer** — encoding position through harmonic frequencies at no parameter cost.