scan
**SCAN (Simplified Compositional Generalization)** is the **sequence-to-sequence compositional generalization benchmark that translates natural language navigation commands into action sequences** — with carefully designed train/test splits that hold out specific command combinations to test whether models learn compositional rules or memorize input-output pairs, revealing a fundamental failure mode of standard neural architectures.
**What Is SCAN?**
- **Origin**: Lake & Baroni (2018) — positioned as a challenge for the compositionality of neural sequence models.
- **Task**: Map natural language commands to sequences of primitive actions.
- "jump" → `JUMP`
- "jump twice" → `JUMP JUMP`
- "run around left" → `LTURN RUN LTURN RUN LTURN RUN LTURN RUN`
- "jump and walk opposite right" → `JUMP RTURN RTURN WALK`
- **Grammar**: Commands compose verbs (jump, walk, run, look), adverbs (twice, thrice, around, opposite), and directions (left, right).
- **Scale**: ~16,728 training examples; several test splits of ~4,182 examples each.
**The Critical Splits**
**Simple/Random Split**: Standard random train/test. Most models achieve >99% accuracy — SCAN is easy for standard seq2seq if splits are random.
**"Add Jump" Split**: "Jump" appears only in primitive form in training ("jump" → `JUMP`). Test contains composition with "jump" ("jump around left," "jump twice and run"). Standard LSTM/Transformer: <2% accuracy. Human: ~100%.
**"Around Right" Split**: "Around" and "right" appear separately in training, but "around right" is held out. Tests right-directional compositional application.
**Length Split**: Training contains commands generating short action sequences (≤22 actions). Test requires long sequences (24-48 actions). Standard models: ~14% accuracy — reveals length generalization failure.
**Why SCAN Failure Is Significant**
The "Add Jump" split failure is one of the most striking results in compositionality research. A standard LSTM trained on all other commands perfectly should immediately generalize "jump twice" → `JUMP JUMP` by applying the "twice" rule it learned from "walk twice" and "run twice." The model fails completely because:
- **Lexical Memorization**: The model learned "walk twice → WALK WALK" as a holistic mapping, not as "WALK" + "apply_twice_rule." It cannot apply the rule to a new verb.
- **Distributional Dependence**: "Jump" never appeared in compositional contexts in training, so its embedding has no compositional signal.
- **Absence of Symbolic Abstraction**: True compositional understanding requires an abstract "verb" category that can bind to "twice" — neural models lack this explicit abstraction.
**Approaches That Solve SCAN**
- **Compositional Data Augmentation (GECA, SEAR)**: Augment training data by recombining elements — partially helps.
- **Hierarchical Seq2Seq**: Explicitly model the grammar hierarchy — achieves ~85%+ on Add Jump.
- **Meta-Learning (Meta-Seq2Seq)**: Train on many tasks where compositional generalization is required — dramatically improves.
- **Neuro-Symbolic Parsers**: Parse the command into a formal grammar tree, then execute — achieves 100% on all splits.
- **LLM In-Context**: GPT-4 with detailed instructions achieves ~90%+ on Add Jump via implicit meta-learning from pretraining.
**Performance Comparison**
| Model | Random | Add Jump | Length |
|-------|--------|---------|--------|
| LSTM seq2seq | 99.7% | 1.9% | 13.8% |
| Transformer | 99.8% | 3.1% | 11.0% |
| GECA augmentation | 99.7% | 81.0% | — |
| Meta-Seq2Seq | 99.9% | 99.7% | 78.2% |
| GPT-4 (few-shot) | ~99% | ~88% | ~70% |
**Why SCAN Matters**
- **Compositionality Debate**: SCAN reignited the debate about whether deep learning can achieve "human-like" compositional generalization or fundamentally relies on memorization — a central question in cognitive AI.
- **Data Efficiency**: Humans learn compositional rules from very few examples (one or a handful). SCAN quantifies how many training samples each architecture requires to generalize.
- **Systematic Evaluation**: SCAN's algorithmic splits enable perfectly controlled experiments impossible in real NLP data.
- **Architecture Design**: Results drive the development of structured, modular, and neuro-symbolic architectures.
SCAN is **learning the syntax of action** — a clean, controlled test of whether neural networks can truly abstract and compose the rules they observe, or whether they merely store and interpolate between training examples, with profound implications for the compositional intelligence required by real-world language-guided robotics and AI agents.