in-place operations

**In-place operations** is the **tensor updates that modify existing memory buffers instead of allocating new outputs** - they can reduce memory pressure and allocation overhead, but must be used carefully with autograd dependencies. **What Is In-place operations?** - **Definition**: Operation variants that overwrite input tensor storage with result values. - **Memory Benefit**: Avoids creating extra temporary tensors and lowers peak allocation footprint. - **Autograd Risk**: Overwriting values needed for backward pass can break gradient computation. - **Safety Condition**: Valid when overwritten tensor is not required by later gradient or reuse paths. **Why In-place operations Matters** - **Memory Efficiency**: In-place updates can increase feasible batch size under tight VRAM budgets. - **Allocation Reduction**: Lower allocator churn can improve runtime stability and reduce fragmentation. - **Performance**: Avoiding extra copies may speed elementwise-heavy workloads. - **Tradeoff Awareness**: Unsafe in-place use causes subtle correctness bugs and training instability. - **Optimization Scope**: Useful selective tool when applied with explicit gradient-safety analysis. **How It Is Used in Practice** - **Dependency Audit**: Confirm tensor is not required by future backward graph nodes before overwriting. - **Controlled Usage**: Apply in-place ops in memory-critical paths with targeted tests. - **Numerical Validation**: Compare gradients and final metrics against non-in-place baseline. In-place operations are **a memory optimization tool with strict correctness constraints** - deliberate use can save memory, but unsafe overwrites can invalidate training.