capability testing

**Capability Testing** is a **model evaluation approach that tests specific, named capabilities rather than overall accuracy** — evaluating whether the model can handle negation, handle rare classes, maintain consistency, or perform other specific skills required for its application. **Capability Categories** - **Core Skills**: Basic classification/regression accuracy on standard inputs. - **Edge Cases**: Performance on boundary cases, rare events, and extreme values. - **Compositional**: Ability to handle combinations of features or conditions not seen during training. - **Temporal**: Stability over time as data distribution drifts. **Why It Matters** - **Targeted Improvement**: When a capability test fails, you know exactly what to improve. - **Requirements Alignment**: Map model capabilities directly to application requirements. - **Regression Detection**: Track capabilities across model versions to catch regressions in specific skills. **Capability Testing** is **testing skills, not scores** — evaluating whether the model possesses specific abilities required for its real-world role.