GPT

GPT (Generative Pre-trained Transformer) is a family of large language models from OpenAI built on a decoder-only transformer that is trained to predict the next token in a sequence, one token at a time, over huge amounts of text. Generating text then simply means sampling a token, appending it to the input, and predicting again, which is why the approach is called autoregressive. The name describes a recipe as much as a product: generative pretraining on unlabeled text first, adaptation (by fine-tuning, prompting or later preference training) second. The same recipe was adopted far beyond OpenAI, so "GPT-style" now commonly means any decoder-only, next-token-predicting transformer.

The first model, GPT-1, appeared in June 2018 in "Improving Language Understanding by Generative Pre-Training" by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, and it showed that unsupervised pretraining followed by supervised fine-tuning could beat task-specific architectures. It was a 12-layer transformer decoder with roughly 117 million parameters, pretrained on the BooksCorpus of unpublished books and then fine-tuned separately on tasks such as entailment, question answering and classification, with structured inputs flattened into a single token sequence. A few months later BERT used a bidirectional encoder with a fill-in-the-blank objective and overtook GPT-1 on many understanding benchmarks, but the two lines chose different bets: the encoder route optimized for reading, while the decoder route kept the objective that also lets a model write.

GPT-2 (February 2019, "Language Models are Unsupervised Multitask Learners") scaled the same design to 1.5 billion parameters and trained it on WebText, about 40 gigabytes of pages linked from highly upvoted Reddit posts, and it made the claim that a sufficiently large language model can perform tasks without any task-specific training. Prompted with the right framing, it could attempt translation, summarization and question answering zero-shot, though far below dedicated systems. OpenAI initially withheld the largest model citing misuse concerns and released it in stages through the year, an early and much-debated example of staged release. GPT-3 (2020, "Language Models are Few-Shot Learners" by Brown and colleagues) then reached 175 billion parameters, with 96 layers and a 2,048-token context window, and made in-context learning the headline result: giving the model a few examples in its prompt, with no weight updates, often sufficed to perform a new task.

Decoder-only means the model keeps just the decoder half of the original 2017 transformer and applies a causal mask, so that during training every position is graded on predicting the token after it using only the tokens before it. That is an unusually efficient objective, because one forward pass over a sequence of length n yields n training signals, and it requires no human labeling since the text supplies its own answers. The scaling-law papers (Kaplan and colleagues at OpenAI, 2020; Hoffmann and colleagues' Chinchilla study at DeepMind, 2022) showed that loss falls predictably as parameters, data and compute grow, which justified ever larger runs. At inference, key and value tensors from earlier tokens are cached (the KV cache) so each new token costs one incremental step, not a recomputation of the whole prompt.

A raw pretrained GPT is a text continuer, not an assistant, and the step that turned it into a conversational product was alignment training. InstructGPT (Ouyang and colleagues, 2022) fine-tuned GPT-3 with supervised demonstrations and reinforcement learning from human feedback, and human raters preferred a 1.3-billion-parameter InstructGPT's outputs to those of the 175-billion-parameter GPT-3, showing that behavior shaping mattered as much as raw size. ChatGPT launched in November 2022 on that lineage, and later GPT releases added chat formatting, tool use, longer contexts and image input. OpenAI has not published parameter counts or training details for its later models, so any figure quoted for them is speculation and should not be treated as fact; what can be said reliably is only what the papers and system cards state.

Common misconceptions are that GPT means ChatGPT, that the model looks facts up, and that it understands in the way people do. "GPT" is the model family and architecture while ChatGPT is a product built on top of such models; a pure next-token predictor produces what is statistically plausible given its context, so it can state falsehoods fluently (often called hallucination), and its knowledge is frozen at the training cutoff unless retrieval or tools are attached. Practical pitfalls include sensitivity to prompt wording, context-window limits, non-deterministic sampling at nonzero temperature, and the fact that training-data memorization can leak text. Neighbors worth distinguishing: BERT (encoder-only, bidirectional), T5 (encoder-decoder), and open decoder-only families such as LLaMA, which share the GPT-style design without being OpenAI products.

Read GPT as the claim that predicting the next token, done at sufficient scale on enough text with a decoder-only transformer, becomes a general-purpose engine: GPT-1 proved pretraining pays, GPT-2 hinted at zero-shot ability, GPT-3 made few-shot prompting real at 175 billion parameters, and alignment training turned that engine into an assistant, while its fluency should never be mistaken for guaranteed truth.