prefix language modeling

**Prefix Language Modeling** combines **bidirectional encoding of a prefix with autoregressive generation of continuation** — creating a unified architecture where prefix tokens attend bidirectionally (like BERT) while generation tokens attend autoregressively (like GPT), enabling better context understanding for conditional generation tasks like summarization, translation, and dialogue. **What Is Prefix Language Modeling?** - **Definition**: Hybrid architecture with bidirectional prefix encoding + autoregressive generation. - **Prefix**: Initial tokens attend to each other bidirectionally. - **Generation**: Subsequent tokens attend to prefix + previous generation tokens autoregressively. - **Unified Model**: Single model handles both encoding and generation. **Why Prefix Language Modeling?** - **Better Prefix Understanding**: Bidirectional attention captures full prefix context. - **Fluent Generation**: Autoregressive generation maintains coherence. - **Natural for Conditional Tasks**: Many tasks have input (prefix) + output (generation). - **Unified Architecture**: One model for many tasks, no separate encoder-decoder. - **Flexible**: Can adjust prefix/generation boundary per task. **Architecture** **Attention Masks**: - **Prefix Tokens**: Can attend to all other prefix tokens (bidirectional). - **Generation Tokens**: Can attend to all prefix tokens + previous generation tokens (causal). - **Implementation**: Position-dependent attention masks. **Example Attention Pattern**: ``` Prefix: [A, B, C] Generation: [X, Y, Z] Attention Matrix: A B C X Y Z A [ 1 1 1 0 0 0 ] (bidirectional prefix) B [ 1 1 1 0 0 0 ] C [ 1 1 1 0 0 0 ] X [ 1 1 1 1 0 0 ] (autoregressive generation) Y [ 1 1 1 1 1 0 ] Z [ 1 1 1 1 1 1 ] ``` **Model Components**: - **Shared Transformer**: Same transformer layers for prefix and generation. - **Position Embeddings**: Distinguish prefix from generation positions. - **Attention Masks**: Control bidirectional vs. causal attention. **Comparison with Other Architectures** **vs. Pure Autoregressive (GPT)**: - **GPT**: All tokens attend causally (left-to-right only). - **Prefix LM**: Prefix tokens attend bidirectionally. - **Advantage**: Better prefix understanding for conditional tasks. - **Trade-Off**: Slightly more complex attention masking. **vs. Encoder-Decoder (T5, BART)**: - **Encoder-Decoder**: Separate encoder (bidirectional) and decoder (autoregressive). - **Prefix LM**: Unified model with position-dependent attention. - **Advantage**: Simpler architecture, shared parameters. - **Trade-Off**: Less architectural separation between encoding and generation. **vs. Pure Bidirectional (BERT)**: - **BERT**: All tokens attend bidirectionally, no generation. - **Prefix LM**: Adds autoregressive generation capability. - **Advantage**: Can generate fluent text, not just representations. **Training** **Objective**: - **Prefix**: No loss on prefix tokens (or optional MLM loss). - **Generation**: Standard autoregressive language modeling loss. - **Formula**: L = -Σ log P(x_i | x_