Image-Text Matching (ITM) is a classic pre-training objective — where the model predicts whether a given image and text pair correspond to each other (positive pair) or are mismatched (negative pair), forcing the model to learn fine-grained alignment.

What Is Image-Text Matching?

• Definition: Binary classification task. $f(Image, Text) ightarrow [0, 1]$.
• Usage: Used in models like ALBEF, BLIP, ViLT.
• Hard Negatives: Crucial strategy where the model is shown text that is almost correct but wrong (e.g., "A dog on a blue rug" vs "A dog on a red rug") to force detail attention.

Why It Matters

• Verification: Acts as a re-ranker. First retrieve top-100 candidates with fast dot-product (CLIP), then verify best match with slow ITM.
• Fine-Grained Alignment: Unlike CLIP (unimodal encoders), ITM usually uses a fusion encoder to compare specific words to specific regions.

Image-Text Matching is the quality control of multimodal learning — teaching the model to distinguish between "close enough" and "exactly right".