open vocabulary detection
**Open-Vocabulary Object Detection** is the **capability to detect and localize objects in images using arbitrary text descriptions rather than a fixed set of predefined categories** — leveraging vision-language models (CLIP, ALIGN) to match image regions with text embeddings, enabling detectors that can find objects for any query ("red fire hydrant partially covered by snow") without retraining, fundamentally changing object detection from a closed-set classification problem to an open-ended visual search.
**Closed-Set vs. Open-Vocabulary**
```
Closed-set (traditional YOLO/Faster R-CNN):
Train on {car, person, dog, cat, ...} → can only detect these exact classes
New class detected: IMPOSSIBLE without retraining
Open-vocabulary:
Input: Image + text query "red fire hydrant"
Output: Bounding boxes for matching objects
New class detected: Just change the text query! Zero retraining.
```
**Key Systems**
| Model | Developer | Year | Approach |
|-------|----------|------|----------|
| ViLD | Google | 2022 | Distill CLIP into detector |
| OWL-ViT / OWLv2 | Google | 2022/2023 | End-to-end vision-language detection |
| Grounding DINO | IDEA | 2023 | DINO detector + language grounding |
| GLIP | Microsoft | 2022 | Grounded language-image pretraining |
| Florence-2 | Microsoft | 2024 | Unified vision foundation model |
| Grounding SAM | Community | 2023 | Grounding DINO + SAM segmentation |
**How Open-Vocabulary Detection Works**
```
[Image] → [Visual Encoder (ViT)] → [Region features / proposals]
↓
[Text query] → [Text Encoder (CLIP/BERT)] → [Text embedding]
↓
[Cross-modal matching: cosine similarity between regions and text]
↓
[Bounding boxes + confidence scores for matching regions]
```
**Grounding DINO Architecture**
```
[Image] [Text: "cat sitting on a red chair"]
↓ ↓
[Image backbone (Swin)] [Text backbone (BERT)]
↓ ↓
[Feature enhancer with cross-modality fusion]
↓
[Language-guided query selection]
↓
[Cross-modality decoder]
↓
[Bounding boxes + phrase grounding]
→ Box 1: "cat" at [x1,y1,x2,y2]
→ Box 2: "red chair" at [x3,y3,x4,y4]
```
**Performance**
| Model | COCO Novel AP50 (zero-shot) | LVIS AP (rare) |
|-------|-----------------------------|----------------|
| Faster R-CNN (supervised baseline) | 0 (can't detect novel) | 12.3 |
| ViLD | 27.6 | 16.7 |
| OWLv2 | 36.2 | 31.4 |
| Grounding DINO (L) | 52.5 | 33.1 |
**Grounding SAM Pipeline**
```
Step 1: Grounding DINO → detect and localize objects from text
Step 2: SAM (Segment Anything) → produce precise masks for detected boxes
Result: Open-vocabulary detection + segmentation from any text prompt
Example:
Query: "all the coffee cups on the desk"
→ Grounding DINO finds 3 boxes for coffee cups
→ SAM produces pixel-precise masks for each cup
```
**Applications**
| Application | How It's Used |
|------------|---------------|
| Robotics | "Pick up the blue screwdriver" → detect + grasp |
| Autonomous driving | Detect rare objects without training on them |
| Visual search | Find specific items in image/video databases |
| Content moderation | Detect any described content without per-class training |
| Medical imaging | Describe anomaly in text → locate in scan |
Open-vocabulary detection is **the paradigm shift from training detectors to querying them** — by replacing fixed class vocabularies with open-ended text queries, these systems make object detection a natural language interface to visual understanding, enabling applications that were previously impossible without expensive per-class training data collection and model retraining.