ar-lsat
**AR-LSAT (Analytical Reasoning from the Law School Admission Test)** is the **constraint satisfaction benchmark derived from the "Logic Games" section of the LSAT** — presenting models with problems where entities must be arranged under a set of rules, testing whether AI can perform systematic constraint propagation and state-space search that symbolic reasoners handle naturally but neural networks struggle with.
**What Is AR-LSAT?**
- **Scale**: 2,046 questions from 230 logic game scenarios (LSAT exams 1991-2016).
- **Format**: Scenario description + constraint rules + 5 multiple-choice questions per scenario.
- **Difficulty**: Among the hardest standardized test sections for humans — average performance ~57-60% for LSAT takers; perfect scores are extremely rare.
- **Reasoning Type**: Constraint satisfaction and logic spatial reasoning — the same class of problems as CSP (Constraint Satisfaction Problems) in classical AI.
**The Logic Game Structure**
A typical AR-LSAT problem:
**Scenario**: "Six students — A, B, C, D, E, F — are assigned to study groups 1, 2, and 3. Each group has exactly 2 students."
**Constraints**:
- "A and B cannot be in the same group."
- "C must be in group 1 or 2."
- "If D is in group 3, then E must be in group 1."
- "F cannot be in the same group as both C and D."
**Questions** (5 per scenario):
1. "Which of the following is a possible assignment?" — Direct constraint checking.
2. "If A is in group 2, which must also be in group 2?" — Conditional inference.
3. "Which entity could be placed in any group?" — Universal flexibility question.
4. "Which pair cannot be in the same group?" — Mutual exclusion derivation.
5. "What is the maximum number of students that could be in group 1?" — Optimization under constraints.
**Why AR-LSAT Is Hard for Transformers**
- **State Space Management**: Solver must maintain a graph of possible assignments and propagate implications across all constraints simultaneously — transformers' lack of persistent working memory makes this difficult without explicit scratchpad use.
- **Chain Reasoning**: A single constraint implication can cascade: "D in group 3 → E in group 1 → F not in group 1 → F in group 2 or 3 → but F cannot be with C (who is in group 1 or 2) → if C in group 2, F must be in group 3..." Each step is individually simple; 5-6 chained steps overwhelm standard attention.
- **Distractors Under Uncertainty**: Wrong answers are carefully constructed to correspond to invalid arrangements that violate exactly one constraint — models without exhaustive constraint checking will be fooled.
- **High Stakes Decisions**: One wrong constraint inference invalidates the entire solution, unlike NLI tasks where partial understanding suffices.
**Performance Results**
| Model | AR-LSAT Accuracy |
|-------|-----------------|
| Random baseline | 20% |
| LSAT human average | ~57% |
| RoBERTa-large (fine-tuned) | ~30% |
| GPT-3.5 (few-shot) | ~39% |
| GPT-4 (few-shot) | ~58% |
| GPT-4 + scratchpad + CoT | ~70% |
| GPT-4 + code (constraint solver) | ~85%+ |
**The Code Execution Solution**
The most effective approach routes AR-LSAT to a Python constraint solver:
1. Parse scenario → Python variables and constraint functions.
2. Use `itertools` or `python-constraint` to enumerate valid assignments.
3. Answer questions by querying the solved assignment graph.
This approach achieves ~85%+ accuracy but requires robust NL-to-code translation of constraint specifications.
**Why AR-LSAT Matters**
- **Neuro-Symbolic Boundary**: AR-LSAT sits exactly at the boundary where symbolic AI (CSP solvers) is provably superior to neural methods for pure constraint satisfaction — the benchmark clarifies what hybrid architectures need to deliver.
- **Legal and Regulatory AI**: Real-world regulatory compliance ("Can entity X do action Y given these contractual constraints?") is structurally identical to AR-LSAT logic games.
- **Planning and Scheduling**: Scheduling AI must satisfy mutually exclusive resource constraints — the same problem class.
- **Cognitive AI**: LSAT logic games are used by psychologists as measures of working memory capacity and fluid intelligence in humans.
- **Tool Use Motivation**: AR-LSAT is a primary motivating example for giving LLMs access to external constraint solvers and improving NL-to-formal-specification translation.
AR-LSAT is **logic puzzles at the gates of law school** — constraint satisfaction problems that test whether AI can maintain a mental model of multiple interacting rules and infer valid arrangements, revealing the boundary where trained neural pattern matching must give way to systematic symbolic search.