ctrl (conditional transformer language)

**CTRL (Conditional Transformer Language model)** is a **1.63 billion parameter** language model developed by **Salesforce Research** (2019) that introduced the concept of **control codes** — special tokens prepended to the input that steer the style, content, domain, and format of generated text. **How Control Codes Work** - **Training**: CTRL was trained on a large, diverse corpus where each text segment was prefixed with a **control code** indicating its source or domain (e.g., "Reviews," "Wikipedia," "Reddit," "Links," "Questions"). - **Generation**: At inference time, users prepend a control code to their prompt to guide the model's output style and content. For example: - `Reviews` prefix → generates product review-style text - `Wikipedia` prefix → generates encyclopedia-style factual text - `Reddit` prefix → generates conversational, informal text - `Horror` prefix → generates horror fiction **Key Innovations** - **Controllable Generation**: Unlike standard language models that generate text in an uncontrolled manner, CTRL gives users explicit knobs to adjust output characteristics. - **Source Attribution**: The model can predict which control code is most likely for a given text, essentially performing **source attribution** — identifying the style, domain, or register of unknown text. - **No Fine-Tuning Required**: Different output styles are achieved through control codes rather than separate fine-tuned models. **Limitations** - **Fixed Control Codes**: The set of control codes is determined at training time — you can't add new ones without retraining. - **Coarse Control**: Control codes influence general style but don't provide fine-grained attribute control. - **Model Size**: At 1.63B parameters, CTRL was large for 2019 but small by modern standards. **Legacy** CTRL pioneered the idea that language models could be **explicitly steered** through conditioning signals. This concept influenced later work on **prompt engineering**, **instruction tuning**, and **controllable generation** systems that are central to modern LLM usage.