temporal shift module
**Temporal Shift Module (TSM)** is the **parameter-free operation that shifts a fraction of feature channels across neighboring timesteps to inject temporal context into 2D backbones** - it adds motion awareness with almost zero additional FLOPs.
**What Is TSM?**
- **Definition**: Channel shift operator where some channels move forward in time and some move backward, while remaining channels stay unchanged.
- **Design Goal**: Provide temporal interaction without expensive 3D convolutions.
- **Placement**: Inserted into residual blocks of standard 2D CNNs.
- **Cost Profile**: No learned parameters, minimal arithmetic overhead.
**Why TSM Matters**
- **Efficiency Breakthrough**: Near-free temporal modeling for edge and real-time systems.
- **Backbone Reuse**: Existing image models can become video models with light modification.
- **Strong Accuracy-Speed Tradeoff**: Good performance under tight latency budgets.
- **Deployment Simplicity**: Uses basic tensor shift operations supported by common runtimes.
- **Scalable Integration**: Can be combined with segment sampling and transformer heads.
**TSM Mechanics**
**Channel Partitioning**:
- Split channels into forward-shift, backward-shift, and static groups.
- Typical ratio keeps most channels static to preserve spatial signal.
**Temporal Mixing**:
- Forward-shift channels import previous-step context.
- Backward-shift channels import next-step context.
**Residual Compatibility**:
- Shift operation wraps around convolution blocks and maintains shape consistency.
- Easy insertion into existing ResNet-like pipelines.
**How It Works**
**Step 1**:
- Reshape features by time dimension and apply deterministic channel shifts between adjacent timesteps.
**Step 2**:
- Process shifted features with standard 2D convolutions and aggregate predictions across clips.
Temporal Shift Module is **a lightweight temporal context mechanism that upgrades image backbones for video with minimal compute overhead** - it is a practical option when efficiency is a primary deployment constraint.