Previous token heads is the attention heads that strongly attend to the immediately preceding token position - they provide local context routing that supports many higher-level circuits.

What Is Previous token heads?

• Definition: Attention pattern is concentrated on token index minus one relative position.
• Functional Use: Creates short-range context features used by downstream heads.
• Circuit Role: Often upstream of induction and local-grammar processing mechanisms.
• Detection: Identified through average attention maps and positional preference metrics.

Why Previous token heads Matters

• Foundational Routing: Local token transfer is a building block for many model computations.
• Interpretability Baseline: Simple positional behavior provides clear mechanistic anchors.
• Composition Insight: Helps explain how later heads build complex behavior from local signals.
• Error Analysis: Weak or noisy local routing can degrade syntax and continuation quality.
• Comparative Study: Useful for scaling analyses across model sizes and architectures.

How It Is Used in Practice

• Positional Probes: Measure head attention by relative position across diverse prompts.
• Circuit Mapping: Trace which later components consume previous-token features.
• Intervention: Ablate candidate heads and monitor local dependency performance drops.

Previous token heads is a basic but important positional mechanism in transformer attention - previous token heads are critical primitives for constructing higher-order sequence-processing circuits.