scratchpad

**Scratchpad prompting** is the technique of providing the language model with a designated **workspace area** where it can show **intermediate calculations, working notes, and step-by-step reasoning** before producing a final answer — mimicking how humans use scratch paper to work through complex problems. **Why Scratchpads Help** - Without a scratchpad, the model must compute everything "in its head" — maintaining intermediate results in its hidden state, which is prone to errors for multi-step problems. - A scratchpad **externalizes working memory** — the model writes down intermediate results as text tokens, which then become part of the visible context for subsequent reasoning. - This is especially important for **arithmetic, symbolic manipulation, and multi-step logic** where tracking intermediate values is critical. **Scratchpad Format** ``` Question: What is 47 × 83? Scratchpad: 47 × 83 = 47 × 80 + 47 × 3 = 3760 + 141 = 3901 Answer: 3901 ``` **Scratchpad vs. Chain-of-Thought** - **Chain-of-Thought**: Natural language reasoning narrative — "First, I note that... then I consider... therefore..." - **Scratchpad**: More structured, often using notation, symbols, and compact working — closer to how you'd write on actual scratch paper. - **Overlap**: Both externalize reasoning. Scratchpad tends to be more compact and calculation-focused. CoT tends to be more narrative and explanation-focused. - In practice, they're often combined — natural language reasoning with interspersed calculations. **When Scratchpads Are Most Effective** - **Arithmetic**: Multi-digit multiplication, division, compound calculations — the model writes out partial products and carries. - **Symbolic Manipulation**: Algebra, equation solving, simplification — each transformation step written explicitly. - **Code Tracing**: Stepping through code execution — tracking variable values at each line. - **Logic Problems**: Truth tables, constraint tracking, elimination — writing out what's known and what's ruled out. - **State Tracking**: Problems involving changing state (puzzles, simulations) — recording state after each action. **Scratchpad Training** - **Few-Shot**: Include scratchpad demonstrations in the prompt — the model learns to use the scratchpad format from examples. - **Fine-Tuning**: Models fine-tuned on data with scratchpad traces learn to produce scratchpads without explicit prompting. - **Verifier Training**: A separate model can be trained to check the scratchpad work — identifying errors in intermediate steps. **Benefits** - **Accuracy**: Scratchpads can improve math accuracy by **20–50%** on complex calculations compared to direct answering. - **Debuggability**: When the answer is wrong, you can inspect the scratchpad to find exactly where the error occurred. - **Reproducibility**: The explicit working makes the reasoning transparent and reproducible. **Practical Tips** - Explicitly instruct: "Use a scratchpad to show your work before giving the final answer." - For few-shot prompting, include examples with scratchpad work shown. - Keep the scratchpad focused — too much extraneous work can distract the model from the core calculation. Scratchpad prompting is a **simple but powerful technique** — by giving the model space to show its work, it transforms error-prone mental computation into reliable, step-by-step written reasoning.