RWKV is the recurrent architecture that blends RNN-like recurrence with transformer-style gating, enabling efficient vision modeling by unfolding states in a single backward-ordered pass — it processes tokens with a linear recurrence that resembles attention but uses fixed recurrence weights, making it fast for long sequences while retaining competitive accuracy.

What Is RWKV?

• Definition: A model whose name stands for Receptance-Weighted Key Value; it mixes RNN recurrence with gating mechanisms reminiscent of transformers.
• Key Feature 1: The recurrence updates a hidden state based on previous activations, modulated by learned receptance gates.
• Key Feature 2: During training, the recurrence is computed in parallel by reversing the sequence, so it remains efficient.
• Key Feature 3: For vision tasks, RWKV treats flattened patches as sequences, letting the recurrence capture spatial patterns without attention matrices.
• Key Feature 4: The model has fixed (non-adaptive) recurrence weights, simplifying inference and lowering memory usage.

Why RWKV Matters

• Streaming: Works with online inference because it maintains a small hidden state per head.
• Low Memory: No need to store all past keys and values, just the recurrent state.
• Performance: Competes with ViTs on some benchmarks while using less compute.
• Simplicity: Its deterministic recurrence makes it easy to deploy on CPUs and edge accelerators.
• Compatibility: Can replace attention blocks in hybrid architectures to reduce overhead.

Recurrence Components

Receptance Gate:

• Controls how much new information enters the hidden state.
• Similar to the gating in GRUs and LSTMs.

Value Update:

• Computes linear combinations of keys and previous hidden states.
• Allows the model to accumulate context without storing entire histories.

Backward Training:

• Flips the sequence so the recurrence can be parallelized with standard matrix operations.
• Maintains high throughput during training.

How It Works / Technical Details

Step 1: Flatten the image into a sequence, project to hidden dimension, and apply the gated recurrent update that blends current input with past state according to receptance weights.

Step 2: For inference, process patches sequentially and maintain only the current hidden state; use any residual connections if necessary to align with transformer blocks.

Comparison / Alternatives

Tools & Platforms

• RWKV-LM repo: Includes PyTorch and CUDA implementations that can be adapted to vision.
• Hugging Face: Hosts RWKV models for text, with vision variants emerging.
• TensorRT / ONNX: Support streaming inference by folding the recurrent update.
• Evaluation Suites: Compare RWKV to ViT on long sequence tasks to verify efficiency gains.

RWKV is the recurrent comeback that gives vision transformers a streaming sibling without dropping transformer-style gating — it keeps state small and updates fast while modeling spatial dependencies.