naf

**NAF** (Normalized Advantage Functions) is a **continuous control RL algorithm that represents the Q-function as a quadratic function of actions** — $Q(s,a) = V(s) + A(s,a)$ where the advantage is a negative-definite quadratic: $A(s,a) = -frac{1}{2}(a-mu(s))^T P(s)(a-mu(s))$. **NAF Architecture** - **Value**: Neural network outputs $V(s)$ — state value. - **Action**: Neural network outputs $mu(s)$ — optimal action (the quadratic peak). - **Advantage Matrix**: Neural network outputs lower-triangular $L(s)$ — $P(s) = L(s)L(s)^T$ ensures positive definiteness. - **Closed-Form Max**: $argmax_a Q(s,a) = mu(s)$ — no separate actor network needed. **Why It Matters** - **No Actor**: The optimal action is computed analytically — no separate actor network or actor optimization. - **Simple**: Single network outputs value, action, and advantage matrix — cleaner than DDPG. - **Limitation**: The quadratic assumption limits expressiveness — can't represent complex, multi-modal Q-functions. **NAF** is **Q-learning with a quadratic shortcut** — using a quadratic advantage function for closed-form continuous action optimization.