image-text matching

**Image-Text Matching (ITM)** is a **classic pre-training objective** — where the model predicts whether a given image and text pair correspond to each other (positive pair) or are mismatched (negative pair), forcing the model to learn fine-grained alignment. **What Is Image-Text Matching?** - **Definition**: Binary classification task. $f(Image, Text) ightarrow [0, 1]$. - **Usage**: Used in models like ALBEF, BLIP, ViLT. - **Hard Negatives**: Crucial strategy where the model is shown text that is *almost* correct but wrong (e.g., "A dog on a blue rug" vs "A dog on a red rug") to force detail attention. **Why It Matters** - **Verification**: Acts as a re-ranker. First retrieve top-100 candidates with fast dot-product (CLIP), then verify best match with slow ITM. - **Fine-Grained Alignment**: Unlike CLIP (unimodal encoders), ITM usually uses a fusion encoder to compare specific words to specific regions. **Image-Text Matching** is **the quality control of multimodal learning** — teaching the model to distinguish between "close enough" and "exactly right".