meta-dataset
**Meta-Dataset** is a **large-scale benchmark** for evaluating few-shot learning algorithms, consisting of a diverse collection of datasets spanning **different visual domains**. Introduced by Triantafillou et al. (2020), it addressed critical limitations of earlier single-domain evaluations.
**Why Meta-Dataset Was Needed**
- **Single-Domain Limitation**: Earlier benchmarks (miniImageNet, Omniglot) evaluated few-shot learning within a **single visual domain**. Models could achieve high accuracy by learning domain-specific features rather than general few-shot learning strategies.
- **Fixed Episode Structure**: Standard benchmarks used fixed 5-way 5-shot or 5-way 1-shot episodes, which doesn't reflect real-world variability.
- **Overfit to Benchmark**: Many methods were optimized specifically for miniImageNet, achieving high scores without truly general few-shot capabilities.
**Component Datasets (10 Domains)**
| Domain | Dataset | Classes | Description |
|--------|---------|---------|-------------|
| Natural Images | ImageNet | 1,000 | General object recognition |
| Handwriting | Omniglot | 1,623 | Handwritten characters from 50 alphabets |
| Aircraft | FGVC-Aircraft | 100 | Fine-grained aircraft model recognition |
| Birds | CUB-200 | 200 | Fine-grained bird species |
| Textures | DTD | 47 | Describable texture patterns |
| Drawings | Quick Draw | 345 | Hand-drawn sketches |
| Fungi | FGVCx Fungi | 1,394 | Mushroom species identification |
| Flowers | VGG Flower | 102 | Flower species recognition |
| Signs | Traffic Signs | 43 | Traffic sign classification |
| Objects | MSCOCO | 80 | Object categories in context |
**Key Design Innovations**
- **Variable-Way Variable-Shot**: Episodes have **variable numbers of classes and examples per class** — reflecting realistic scenarios where you might have 3 examples of one class and 10 of another.
- **Realistic Distributions**: Class and sample counts follow realistic distributions rather than fixed configurations.
- **Cross-Domain Evaluation**: Train on a subset of datasets, test on **held-out datasets** to measure generalization to entirely new visual domains.
- **Within-Domain Testing**: Also evaluate on unseen classes from training datasets to measure both cross-domain and within-domain generalization.
**Evaluation Protocol**
- **Training Sources**: Typically train on ImageNet, Omniglot, Aircraft, CUB-200, DTD, Quick Draw, Fungi, VGG Flower.
- **Test Sources**: Evaluate on held-out test classes from training datasets PLUS entirely unseen datasets (Traffic Signs, MSCOCO).
- **Metric**: Average accuracy across many sampled episodes, reported per dataset.
**Key Findings**
- Many methods optimized for miniImageNet **performed poorly** across diverse domains — exposing the limitation of single-domain benchmarks.
- Large pre-trained feature extractors significantly outperformed meta-learning methods trained from scratch.
- **Universal representations** (features that work across all domains) are more effective than domain-specific adaptation for most target domains.
Meta-Dataset established the **gold standard for few-shot learning evaluation** — any new few-shot method must demonstrate effectiveness across its diverse domains to be considered truly general.