data labeling
**Data Labeling and Annotation**
**What is Data Labeling?**
Data labeling is the process of adding informative tags or annotations to raw data, creating the ground truth that supervised machine learning models learn from.
**Types of Annotations**
**Text Annotation**
| Type | Use Case | Example |
|------|----------|---------|
| Classification | Sentiment analysis | Positive/Negative/Neutral |
| NER | Information extraction | [PERSON: John] works at [ORG: Google] |
| Sequence labeling | POS tagging | The/DT cat/NN sat/VBD |
| Pairwise | Preference learning | Response A > Response B |
**Image Annotation**
- **Bounding boxes**: Object detection
- **Segmentation masks**: Pixel-level labeling
- **Keypoints**: Pose estimation
- **Polygons**: Instance segmentation
**Annotation Quality Metrics**
**Inter-Annotator Agreement**
| Metric | Formula | Good Threshold |
|--------|---------|----------------|
| Cohen's Kappa | Agreement beyond chance | >0.8 |
| Krippendorff's Alpha | Multi-rater reliability | >0.8 |
| Fleiss' Kappa | Multiple annotators | >0.7 |
**Quality Control Strategies**
1. **Gold standard questions**: Test annotators against known answers
2. **Overlap**: Have multiple annotators label same item
3. **Auditing**: Regular review of annotation samples
4. **Training**: Calibration sessions for new annotators
**Annotation Platforms**
| Platform | Type | Highlights |
|----------|------|------------|
| Scale AI | Commercial | High quality, expensive |
| Labelbox | SaaS | Good UI, collaborative |
| Label Studio | Open source | Self-hosted, flexible |
| Prodigy | Commercial | Active learning, efficient |
| Amazon SageMaker Ground Truth | AWS | Integrated with AWS ML |
**Best Practices for LLM Data**
- Create detailed annotation guidelines with examples
- Include edge cases and ambiguous scenarios
- Measure and report annotator agreement
- Version control your annotation guidelines
- Use synthetic data generation to augment limited labels