perceiver
**Perceiver** is a **general-purpose transformer architecture that uses cross-attention to project arbitrary-size inputs into a fixed-size latent array** — decoupling the computational cost from input size so that a 100K-pixel image, a 50K-token audio clip, and a 10K-point cloud all get processed through the same small latent bottleneck (e.g., 512 latent vectors), enabling a single architecture to handle any modality without modality-specific design choices.
**What Is Perceiver?**
- **Definition**: A transformer architecture (Jaegle et al., 2021, DeepMind) where the input (of any size) is processed through cross-attention with a small learned latent array (typically 256-1024 vectors), and all subsequent self-attention operates on this compact latent space rather than the high-dimensional input space.
- **The Problem**: Standard transformers apply O(n²) self-attention directly on the input. For a 224×224 image (50K pixels), that's 2.5 billion attention computations per layer — impossible. CNNs and ViTs work around this with patches, but each modality needs custom architecture.
- **The Solution**: Project ANY input into a fixed-size latent array via cross-attention (cost: O(n × M) where M is latent size << n), then apply self-attention only on the small latent array (cost: O(M²), independent of input size).
**Architecture**
| Step | Operation | Input | Output | Complexity |
|------|----------|-------|--------|-----------|
| 1. **Cross-Attention** | Latent queries attend to input | Latent: M × d, Input: N × d_in | M × d (latent updated) | O(M × N) |
| 2. **Self-Attention** | Latent self-attention (multiple blocks) | M × d | M × d (refined) | O(M²) per block |
| 3. **Repeat** (optional) | Additional cross-attention + self-attention | Updated latent + original input | M × d (further refined) | O(M × N + M²) |
| 4. **Decode** | Task-specific output (class token, etc.) | M × d | Task output | O(M) |
**Key Insight: The Latent Bottleneck**
| Property | Standard Transformer | Perceiver |
|----------|---------------------|-----------|
| **Self-attention cost** | O(N²) — depends on input size | O(M²) — depends on latent size (fixed) |
| **Input flexibility** | Fixed tokenization per modality | Any byte array, any modality |
| **Scalability** | Cost grows quadratically with input | Cost fixed regardless of input size |
| **Architecture per modality** | Different: ViT for images, BERT for text | Same architecture for everything |
**Example**: M=512 latents, N=50,000 input elements:
- Standard: Self-attention = 50,000² = 2.5B operations per layer
- Perceiver: Cross-attn = 512 × 50,000 = 25.6M; Self-attn = 512² = 262K per block
**Modality Flexibility**
| Modality | Input Representation | Same Perceiver Architecture |
|----------|---------------------|---------------------------|
| **Images** | Pixel array (H×W×C) with positional encoding | ✓ |
| **Audio** | Raw waveform or spectrogram | ✓ |
| **Point Clouds** | 3D coordinates (N×3) | ✓ |
| **Video** | Pixel frames (T×H×W×C) | ✓ |
| **Text** | Token embeddings | ✓ |
| **Multimodal** | Concatenate all modalities as one input array | ✓ |
**Perceiver is the universal perception architecture** — using cross-attention to a fixed-size latent array to decouple computational cost from input size and modality, enabling a single unchanged architecture to process images, audio, video, point clouds, and multimodal inputs with O(M²) self-attention cost regardless of whether the input has 1,000 or 1,000,000 elements, pioneering the movement toward truly modality-agnostic deep learning.