temporal segment networks

**Temporal Segment Networks (TSN)** are the **sparse sampling video framework that divides a video into segments and aggregates predictions from sampled snippets** - this design captures long-range context efficiently without processing every frame densely. **What Is TSN?** - **Definition**: Segment-based action recognition where one snippet is sampled from each temporal segment and fused for final prediction. - **Sampling Logic**: Sparse coverage of full timeline preserves global semantics at low cost. - **Backbone Type**: Often uses 2D CNN per snippet with consensus aggregation. - **Fusion Rule**: Average or weighted consensus over segment-level scores. **Why TSN Matters** - **Efficiency**: Reduces computation compared with dense frame-by-frame processing. - **Long Video Coverage**: Sees broad timeline despite limited snippet count. - **Practical Baseline**: Easy to train and deploy in resource-constrained environments. - **Strong Legacy**: Influential architecture in large-scale action recognition benchmarks. - **Extension Friendly**: Can integrate optical flow, transformers, and temporal modules. **TSN Pipeline** **Segment Sampling**: - Partition video into K equal temporal segments. - Sample one snippet from each segment during training and inference. **Per-Snippet Encoding**: - Extract features with shared visual backbone. - Optional multimodal streams capture complementary signals. **Consensus Aggregation**: - Combine snippet predictions into video-level score. - Train end-to-end with segment-consensus loss. **How It Works** **Step 1**: - Perform sparse temporal sampling over full video and encode each snippet. **Step 2**: - Aggregate snippet logits with consensus function and optimize action classification objective. Temporal Segment Networks are **a high-efficiency approach for long-video recognition that balances temporal coverage with manageable compute** - they remain a strong reference for sparse temporal modeling.