Two-Stream Networks are a foundational deep learning architecture for video understanding that explicitly decomposes the video recognition problem into two parallel, independently processed information channels — a Spatial Stream analyzing individual RGB frames for appearance and object identity, and a Temporal Stream analyzing pre-computed Optical Flow fields for motion patterns — before fusing their predictions at the decision level.

The Fundamental Decomposition

• The Human Visual Analogy: Human visual cortex processes "what" (ventral stream — color, texture, object identity) and "where/how" (dorsal stream — motion, spatial relationships) through anatomically distinct neural pathways. Two-Stream Networks directly mirror this biological architecture in silicon.
• The Spatial Stream ("What does it look like?"): A standard 2D CNN (e.g., VGGNet, ResNet) processes a single RGB video frame. It extracts appearance features: the texture of a football, the shape of a person, the color of a jersey. It has no concept of motion.
• The Temporal Stream ("How is it moving?"): A separate 2D CNN processes a stack of pre-computed Optical Flow frames. Optical Flow explicitly encodes the pixel-level displacement vectors between consecutive frames — it is a dense motion field showing exactly which pixels moved, in which direction, and by how much. This stream recognizes motion patterns: "swinging," "running," "throwing."

The Optical Flow Computation

Before training, Optical Flow is pre-computed offline using classical algorithms (TV-L1, Farneback). For each pair of consecutive frames, a 2D displacement field ($u, v$) is generated. Stacking 10-20 consecutive flow frames creates a dense temporal volume capturing the dynamic motion signature of the action.

The Fusion

Both streams produce independent class probability vectors (e.g., "70% Kicking" from spatial, "90% Kicking" from temporal). Late Fusion combines these predictions via simple weighted averaging or an SVM to produce the final video-level classification. This late-fusion architecture means each stream can be pre-trained and fine-tuned independently on different data.

The Legacy and Limitations

Two-Stream Networks dominated video recognition benchmarks for years but suffer from the massive computational overhead of pre-computing Optical Flow offline. Modern architectures (SlowFast, Video Swin, TimeSformer) learn temporal dynamics implicitly from raw frames without requiring explicit flow computation.

Two-Stream Networks are the rods and cones of artificial video perception — parallel biological processing of color and motion through independently specialized neural pathways, fused only at the final moment of conscious recognition.