Pytorch

PyTorch is an open-source deep learning framework for Python that provides GPU-accelerated tensors, automatic differentiation, and a library of neural network building blocks. It was created at Facebook AI Research, with an early public release in 2016 and its open-source debut in January 2017, by a team including Adam Paszke, Sam Gross, Soumith Chintala and Gregory Chanan. It descends from the Lua-based Torch library, but its defining idea is that a model is ordinary Python code executed line by line, so ordinary debugging tools, control flow and data structures work directly inside a network definition.

Eager execution, often called define-by-run, is PyTorch's signature design choice. In a static-graph framework the programmer first builds a symbolic computation graph and then runs it in a separate session. In PyTorch each operation runs immediately when the line executes, returns a real tensor, and can be printed or inspected, so an if statement, a loop over variable-length sequences or a breakpoint behaves exactly as in any Python program. The approach drew on earlier dynamic frameworks such as Chainer and on the autograd research libraries, and its stated goals appear in the 2019 NeurIPS paper "PyTorch: An Imperative Style, High-Performance Deep Learning Library" by Paszke and coauthors. The tradeoff is that the system sees only one operation at a time, which limits whole-program optimization unless extra machinery is added.

Autograd is the engine that makes training possible without hand-derived gradients. When a tensor is created with requires_grad enabled, PyTorch records each operation applied to it on a dynamically built graph, and calling backward on a scalar loss walks that graph in reverse, applying the chain rule to fill each parameter's grad field. This is reverse-mode automatic differentiation, the same mathematics as backpropagation, but generalized to arbitrary code. Because the graph is rebuilt on every forward pass, a model may change structure from one input to the next. A frequent beginner pitfall is that gradients accumulate by default, so training loops must clear them with zero_grad before each step, and another is forgetting torch.no_grad or inference mode during evaluation, which wastes memory by tracking history.

Tensors with requires_grad = True

model parameters live on CPU or GPU

Forward pass runs eagerly

each operation executes at once and is recorded on the autograd graph

loss.backward()

reverse-mode autodiff walks the graph and fills each parameter's grad

One training step

optimizer.step()

update parameters from gradients

optimizer.zero_grad()

clear accumulated gradients

Repeat per batch

the graph is rebuilt every iteration

Optional: torch.compile

captures graphs and generates fused kernels for speed

Performance work has gradually moved PyTorch toward compilation without abandoning eager mode. Under the hood the tensor operations are implemented in C++ and CUDA, and the Python layer is a thin interface over them. TorchScript, introduced with PyTorch 1.0 in 2018 alongside a merge with parts of Caffe2, let models be serialized and run without Python, which suited production. PyTorch 2.0, released in March 2023, added torch.compile, which uses TorchDynamo to capture graphs from Python bytecode and TorchInductor to generate fused kernels, so a one-line wrapper can speed up a model while the programmer keeps writing ordinary code. Distributed training is supported by DistributedDataParallel and Fully Sharded Data Parallel, and the ecosystem includes torchvision, torchaudio, and many libraries built on top, such as Hugging Face Transformers and PyTorch Lightning.

Governance changed in 2022, when PyTorch moved from Meta to a neutral foundation. In September 2022 the PyTorch Foundation was announced under the Linux Foundation, with founding members including AMD, Amazon Web Services, Google Cloud, Meta, Microsoft Azure and Nvidia. The goal was a vendor-neutral home for trademarks, infrastructure and roadmap decisions, in recognition that the project had become shared industry infrastructure. The reason it matters is practical: hardware vendors can contribute backends for their accelerators, and users gain some assurance that a single company's priorities will not decide its direction.

Its adoption in research is among the clearest shifts in machine learning tooling, though exact shares depend on how they are counted. Around 2018 to 2020, papers at major conferences increasingly released code in PyTorch, and large language model work such as the Llama and many open-weight releases uses it. Misconceptions persist: PyTorch is not merely a Python wrapper around another library, eager mode is not inherently slow once operations are large enough to keep a GPU busy, and the framework is not only for research, since it powers many production systems. Other pitfalls include silent device mismatches between CPU and GPU tensors, in-place operations that break autograd, and nondeterminism from GPU kernels that makes bit-exact reproducibility require explicit settings.

Read PyTorch as the framework that made neural networks feel like ordinary Python: eager execution and a tape-based autograd make models easy to write, inspect and change, while compilation, distributed training and a foundation-governed ecosystem supply the performance and neutrality that large-scale work demands.

Pytorch

Take PyTorch further