Unicode
Unicode is the universal standard that assigns a unique number, called a code point, to every character in the world's writing systems, together with the properties and rules needed to process text: which characters are letters or digits, how they change case, how they sort, and which direction they run. Code points are written in the form U+ followed by hexadecimal digits, so the Latin capital A is U+0041 and the euro sign is U+20AC. The standard defines up to 1,114,112 code points (U+0000 to U+10FFFF), of which well over 150,000 are assigned in recent versions, covering modern and historic scripts, mathematical symbols and emoji.
Unicode began as an attempt to end the patchwork of national code pages. In 1987 and 1988 engineers at Xerox and Apple, including Joe Becker, Lee Collins and Mark Davis, began a design for one fixed-width, 16-bit code for all scripts; the Unicode Consortium was incorporated in 1991, and version 1.0 appeared that year. Unicode was closely aligned with the parallel ISO/IEC 10646 project, and the two have been kept synchronised since the early 1990s. The assumption that 65,536 slots would suffice proved wrong once historic scripts and the large set of Chinese, Japanese and Korean ideographs were counted, so Unicode 2.0 (1996) extended the range to 17 planes using surrogate pairs. Emoji entered the standard in Unicode 6.0 in 2010 and have been a driver of new additions since.
Unicode is a character set, not an encoding; UTF-8, UTF-16 and UTF-32 are the encodings that turn code points into bytes. UTF-32 uses four bytes for every code point, simple but wasteful; UTF-16 uses two bytes for the first plane and surrogate pairs for the rest, and underlies the strings of Windows, Java and JavaScript; UTF-8 uses one to four bytes, with ASCII characters kept as the same single bytes. UTF-8 was devised by Ken Thompson and Rob Pike in September 1992, famously sketched on a placemat in a New Jersey diner, for the Plan 9 operating system at Bell Labs. Its self-synchronising design, no embedded zero bytes, and backward compatibility with ASCII made it the dominant encoding of the web and of most modern software.
Because a user-perceived character is not the same as a code point, Unicode text handling has pitfalls that ASCII never had. The letter e with an acute accent can be one code point (U+00E9) or two (the letter e followed by the combining acute accent U+0301), and they look identical but compare as different strings until both are put in a common normalization form (NFC or NFD). Flags, family emoji and many Indic syllables are sequences of several code points glued together, called grapheme clusters, so counting "length" by code points or by UTF-16 units gives surprising answers, and slicing a string in the middle of a cluster can split a symbol in half. Case conversion is also language-dependent, the German sharp s capitalises to two letters, and Turkish treats dotted and dotless i specially.
Practical Unicode work follows a few firm habits. Use UTF-8 for files, APIs and databases and declare it; decode input as soon as it arrives and encode only on output; normalize before comparing or hashing user-entered names; and use libraries that implement the Unicode algorithms for collation, segmentation and bidirectional text instead of writing byte-level code. Security issues exist too: look-alike characters from different scripts (a Cyrillic letter that looks like a Latin one) enable homograph spoofing in domain names and filenames, and invisible control characters can hide in source code. In machine learning, tokenizers for language models build vocabularies over Unicode text or its UTF-8 bytes, and mis-normalized training text shows up as duplicate tokens.
A common misconception is that Unicode "is 16-bit" or "is two bytes per character"; that was true of its first design and has not been true since 1996.
Read Unicode as the single address book of human writing: every character gets one permanent number, and UTF-8 and its siblings are merely the different ways of writing those numbers down.