Natural Language Processing

Natural language processing (NLP) is the field of AI concerned with getting a machine to work with human language — parsing it, understanding it, generating it — and its entire modern era can be summarized as one architectural lineage replacing another: hand-built grammar rules gave way to statistical models, which gave way to recurrent neural networks, which were almost entirely replaced by the transformer after 2017. Nearly every NLP task today (translation, summarization, question answering, chatbots) is now solved by the same underlying architecture pointed at different training objectives, which is a historically unusual level of convergence for a field that used to need a different specialized technique per task. At a high level, that lineage looks like this:

Tokenization is the unglamorous first step that decides how everything downstream even sees the text, and the choice has real compute consequences. Early NLP split text into whole words, which breaks on misspellings, rare words, and languages without clear word boundaries; modern LLMs use subword tokenization (byte-pair encoding or similar) that splits text into frequency-based chunks — common words stay whole, rare words get split into pieces, and any input can be represented without an "unknown word" failure case. This matters economically because a model's context window and compute cost are both measured in tokens, not words or characters — a language that tokenizes less efficiently in a given model's vocabulary literally costs more to process for the same amount of actual content.

The transformer's self-attention mechanism solved the specific bottleneck that limited how far NLP could scale. RNNs and LSTMs process a sequence one token at a time, which means a GPU's thousands of parallel cores sit mostly idle waiting for each sequential step — you cannot meaningfully speed up an RNN by throwing more GPUs at a single sequence. Self-attention instead computes relationships between all tokens in a sequence simultaneously as one large matrix multiply, which is exactly the operation a GPU is built to do in parallel; this hardware-fit, more than any pure accuracy argument, is why the transformer displaced recurrent architectures for virtually all large-scale NLP within a few years of the 2017 "Attention Is All You Need" paper.

Most NLP capability improvements since 2018 have come from scaling the same architecture with more data and parameters, not from new architectural ideas. BERT (2018) demonstrated that pretraining a transformer on massive unlabeled text, then fine-tuning on a small labeled set, beat task-specific architectures across nearly every standard NLP benchmark at the time. GPT-style models pushed the same pretraining idea further by scaling parameter count and training-token count together, eventually reaching the point where a single pretrained model, prompted rather than fine-tuned, could handle translation, summarization, and question answering without any task-specific architecture at all — collapsing what used to be a dozen separate NLP subfields into one model family.

Read modern NLP as a story about one architecture winning because it fit the available hardware, not because it was conceptually smarter than its predecessors: self-attention's fully parallel matrix multiply is what a GPU wants to compute, and the resulting ability to scale that one operation with more data and more silicon is what collapsed translation, summarization, and dialogue into variations on the same pretrained transformer.