leaderboard climbing

Leaderboard climbing refers to optimizing specifically for benchmark performance, sometimes at the expense of genuine capability. **The problem**: Models or training pipelines tuned specifically to benchmark performance may not generalize to real-world tasks. **Manifestations**: Training on benchmark-similar data, prompt engineering for specific benchmarks, architectural choices that help benchmarks but not deployment. **Goodharts Law**: When a measure becomes a target, it ceases to be a good measure. Optimizing for metric rather than underlying capability. **Examples**: Models scoring high on GLUE but poor at real tasks, code models passing HumanEval but struggling with production code. **Community concerns**: Suspicious score jumps, undisclosed training data, specialized evaluation code. **Mitigations**: Held-out test sets, multiple diverse benchmarks, human evaluation, real-world deployment testing, contamination checking. **Healthy perspective**: Benchmarks are proxies for capability, not the goal itself. Celebrate real-world performance. **Current landscape**: Growing skepticism of benchmark claims, emphasis on contamination detection, move toward harder benchmarks. Important to validate claims with independent testing.