Pretext Tasks in self-supervised learning are artificially constructed proxy objectives that train neural networks to solve defined problems on unlabeled data — where solving the pretext task forces the network to learn representations that capture genuine semantic and structural features of the data, which then transfer usefully to downstream supervised tasks — the original paradigm of self-supervised learning that predated contrastive methods, building the conceptual foundation through decades of work on colorization, rotation prediction, jigsaw puzzles, masked prediction, and temporal ordering before contrastive learning unified and superseded many handcrafted designs.

What Are Pretext Tasks?

• Core Concept: A pretext task generates its own supervision signal from the data structure — no human labels needed. The task is "pretext" because it is not the actual downstream objective, but is designed so that solving it requires learning useful features.
• Self-Generated Labels: The "label" for a pretext task is derived automatically from the data — a rotated image's rotation angle, an image's original color from its grayscale version, the correct order of shuffled patches.
• Representation Learning Goal: The representation in the penultimate layer is what we care about — not the task head's output. After pretraining, the task head is discarded and the backbone is fine-tuned on labeled data.
• Design Challenge: A good pretext task requires understanding semantically meaningful structure — not just low-level statistics. A bad pretext task (e.g., predict the hash of an image) teaches nothing transferable.

Classic Pretext Tasks by Domain

Visual Pretext Tasks:

• Rotation Prediction (Gidaris et al., 2018): Rotate an image by 0°, 90°, 180°, or 270°; classify the rotation angle. Forces the model to understand object orientation and visual semantics.
• Jigsaw Puzzles (Noroozi & Favaro, 2016): Shuffle image patches; predict the correct permutation. Forces learning of spatial relationships between parts.
• Colorization (Zhang et al., 2016): Predict the full-color image from its grayscale version. Forces learning of semantic content (grass is green, sky is blue).
• Inpainting: Predict masked regions of an image from surrounding context.
• Relative Patch Position: Predict the relative spatial position of two randomly sampled image patches.

Language Pretext Tasks:

• Masked Language Modeling (BERT): Predict randomly masked tokens from bidirectional context — the dominant NLP pretraining objective.
• Next Sentence Prediction: Classify whether two sentences are consecutive or random (original BERT, since partly superseded).
• Next Token Prediction (GPT): Predict the next word given all previous words — the generative pretraining objective.

Video / Temporal Pretext Tasks:

• Temporal Order Verification: Classify whether a sequence of video frames is in correct temporal order.
• Arrow of Time: Predict whether a video clip is playing forward or backward.
• Frame Prediction: Predict the next frame given previous frames.

Evolution Toward Contrastive and Masked Approaches

Modern contrastive methods (SimCLR, DINO) and masked autoencoders (MAE) are conceptually still pretext tasks — but with learned augmentation policies and task-agnostic objectives that generalize better than handcrafted designs.

Pretext Tasks are the intellectual origin of self-supervised learning — the insight that supervision can be manufactured from the structure of data itself, eliminating the label bottleneck and enabling neural networks to learn from the vast ocean of unlabeled images, text, audio, and video that constitutes the majority of human-generated information.