dueling dqn

**Dueling DQN** is a **DQN architecture that separates the Q-function into a state value function and an advantage function** — $Q(s,a) = V(s) + A(s,a) - ext{mean}(A(s,cdot))$, allowing the network to independently learn the value of being in a state and the relative advantage of each action. **Dueling Architecture** - **Shared Backbone**: Convolutional layers shared for feature extraction. - **Value Stream**: Fully connected layers outputting $V(s)$ — scalar value of the state. - **Advantage Stream**: Fully connected layers outputting $A(s,a)$ — advantage of each action relative to average. - **Combination**: $Q(s,a) = V(s) + A(s,a) - frac{1}{|A|}sum_{a'} A(s,a')$ — centering for identifiability. **Why It Matters** - **State vs. Action**: Many states have similar value regardless of action — dueling architecture captures this. - **Sample Efficiency**: The value stream updates for every action — more efficient learning of state values. - **Complements**: Combines well with Double DQN and Prioritized Experience Replay. **Dueling DQN** is **separating what matters from what to do** — independently learning state value and action advantages for more efficient Q-learning.