insertion-based generation

**Insertion-Based Generation** is a **text generation approach where the model builds the output sequence by inserting tokens into an initially empty (or seed) sequence** — at each step, the model decides WHERE to insert and WHAT token to insert, growing the sequence from the inside out rather than left-to-right. **Insertion Generation Methods** - **Balanced Binary Tree**: Insert at the midpoint of gaps — $O(log N)$ steps for a sequence of length $N$. - **Arbitrary Order**: Learn to insert at any position — the model predicts both the position and the token simultaneously. - **Multiple Insertions**: Insert multiple tokens per step — parallel insertion for faster generation. - **Stern-Brocot Tree**: A specific insertion ordering that efficiently covers all positions. **Why It Matters** - **Speed**: $O(log N)$ insertion steps vs. $O(N)$ for autoregressive — exponentially faster for long sequences. - **Bidirectional Context**: Each inserted token can attend to BOTH left and right context — unlike left-to-right AR models. - **Flexibility**: The generation order naturally adapts to the content — important words can be generated first. **Insertion-Based Generation** is **building text from the inside out** — generating sequences by inserting tokens at chosen positions rather than strict left-to-right order.