Optical Character Recognition
Optical character recognition (OCR) is the technology that converts images of text, such as scanned pages, photographs of signs, receipts or PDFs without a text layer, into machine-readable characters that can be searched, copied, indexed and processed. It sits at the boundary of computer vision and language: the system must first find where text is in an image, then decide which characters or words the marks represent, and often also recover the structure of the page (columns, tables, reading order). The output is not a picture of text but real text, which is why OCR is the front door for digitizing archives, processing invoices and passports, reading license plates and feeding documents to language models.
The history runs from template matching to learned vision. Early machines in the first half of the twentieth century and commercial systems of the 1950s and 1960s matched characters against stored templates in fixed fonts, and Ray Kurzweil's 1976 reading machine for the blind brought omni-font recognition to a broader public. A decisive shift came from neural networks: Yann LeCun and colleagues at Bell Labs trained convolutional networks on handwritten ZIP-code digits in 1989, and LeNet-5 (LeCun, Bottou, Bengio and Haffner, 1998) was deployed to read a large share of bank cheques in the United States in the 1990s and gave rise to the MNIST digit dataset, still the world's best-known machine learning exercise. Meanwhile, Tesseract, an engine begun at Hewlett-Packard in the mid-1980s, was open-sourced in 2005 and sponsored by Google from 2006; its version 4 (2018) added an LSTM-based recognizer, and it remains a standard free option.
A modern OCR system is usually two stages, detection and recognition, because finding text and reading it are different problems. Detectors such as EAST (2017) and CRAFT (2019) draw boxes or polygons around words and lines, including curved text in natural scenes; recognizers then read each crop. The CRNN design (Shi, Bai and Yao, 2015) combines a convolutional feature extractor, a recurrent layer and CTC loss, the same alignment trick used in speech recognition, so a whole line is read without segmenting individual letters, which matters for cursive handwriting and connected scripts such as Arabic where letters cannot be cleanly separated. Recent transformer systems such as TrOCR (Li and colleagues at Microsoft, 2021) pair a vision-transformer encoder with a text decoder pretrained as a language model, and open toolkits such as PaddleOCR and EasyOCR package these stages for dozens of languages.
Layout understanding is increasingly the hard part, because real documents are not plain paragraphs. An invoice has tables, a form has labeled boxes, a newspaper has columns and captions, and reading words in the wrong order scrambles the meaning even if every character is correct. LayoutLM (Xu and colleagues, 2019-2020) added each word's position on the page as input to a BERT-style model so it could classify fields and extract key-value pairs, and OCR-free models such as Donut (Kim and colleagues, 2021) skip the explicit text-reading stage and generate structured output directly from the image. Multimodal large language models can now answer questions about a page image directly, but dedicated OCR still wins on exact transcription, bounding boxes for auditing, speed and cost, and many production pipelines combine both.
OCR errors are systematic, and the most common pitfall is trusting the output without checking it. Visually similar characters (0 and O, 1 and l and I, rn and m, 5 and S) are classic confusions, low-resolution scans under roughly 200 to 300 dots per inch degrade accuracy, and handwriting, stylized fonts, stamps, faded thermal-paper receipts and background texture lower it further. A subtle danger is that language-model-assisted correction can fix text into a plausible but wrong word, and a transposed digit in an amount or an account number is silent unless validated, so serious pipelines use per-word confidence scores, format checks (checksums, date ranges), human review for low-confidence fields and, for archives, preservation of the original image alongside the text. Accuracy is typically measured as character error rate or word error rate against a reference transcription, the same edit-distance idea used in speech recognition. Another misconception is that OCR and handwriting recognition (sometimes called HTR or ICR) are equally mature: printed Latin text is close to solved on clean scans, while historical manuscripts and unconstrained handwriting remain research problems.
Read OCR as the bridge from pixels to text: detection finds where the writing is, a recognizer converts each region into characters, layout models restore the page's structure, and the lasting engineering lesson is that every output deserves a confidence score and a check.