Character Encoding

A character encoding is an agreed rule for turning text into numbers and those numbers into bytes, and back again, so that a computer which only stores bytes can store, transmit and display letters, digits and symbols. Two separate decisions are bundled inside the idea: first which number stands for which character (the character set, or code points), and second how those numbers are laid out as actual bytes in memory, a file or a network packet (the encoding proper). A byte sequence has no intrinsic meaning as text; it becomes the word "café" only when a reader applies the same encoding that the writer used.

The history of character encoding is a history of incompatible local agreements. Telegraph codes such as Baudot (late 19th century) used five bits per character; IBM's EBCDIC, used on System/360 mainframes from 1964, and the 7-bit ASCII standard of 1963 were rivals that arranged the alphabet differently, so text moved between them needed translation tables. When 8-bit bytes became standard, the spare eighth bit was used by dozens of incompatible "code pages": ISO 8859-1 (Latin-1, 1987) for Western European languages, ISO 8859-5 for Cyrillic, Microsoft's Windows-1252, plus multi-byte systems such as Shift JIS for Japanese and GB 2312 for Chinese. The same byte could mean a different character on each, so a document was only readable if the reader guessed or was told which code page applied. Unicode, begun in the late 1980s, ended this by assigning one number to every character, while UTF-8 became the dominant way of storing it.

Mojibake, garbled text such as an accented letter turning into two odd symbols, is the visible signature of an encoding mismatch. The word comes from Japanese (moji-bake, "character transformation"), where many competing encodings made the failure routine. The usual modern cause is UTF-8 bytes being read as Latin-1 or Windows-1252, so that the two bytes C3 A9 become two characters instead of one. It is not data corruption: the bytes are intact, and decoding them with the correct encoding recovers the text exactly. Fixes happen at the point of interpretation, not by editing the damaged-looking text, although repeated wrong decode-and-save cycles can make the damage permanent.

Encoding must always be declared or agreed, because the bytes alone do not carry it. Web servers state it in an HTTP header such as Content-Type: text/html; charset=utf-8; HTML pages repeat it with a meta charset tag; XML defaults to UTF-8 and may declare otherwise; databases have a per-column or per-database collation and charset; email uses MIME headers. Files can optionally begin with a byte order mark (the bytes EF BB BF for UTF-8, or FE FF and FF FE to show UTF-16 byte order), a hint that is helpful for UTF-16 but often unwanted at the start of UTF-8 files, where it can break scripts and shell shebang lines. Where nothing is declared, software guesses with heuristics, which works often and fails unpredictably, so explicit declarations are the only robust option.

Programming languages make the distinction between text and bytes explicit, and ignoring it is a common source of bugs. Python 3 separates str (text) from bytes and requires calls such as text.encode("utf-8") and data.decode("utf-8"), and open() on text files accepts an encoding argument that should be given instead of relying on the platform default. Java and JavaScript strings are internally UTF-16, whereas Go and Rust strings are UTF-8 by convention. In data science, reading a CSV exported from a legacy Windows tool without specifying encoding="cp1252" or "latin-1" commonly raises a UnicodeDecodeError or silently corrupts names, and a model trained on mis-decoded text learns the mojibake as if it were language. Tokenizers for large language models often operate on bytes precisely to avoid this class of problem.

Several misconceptions persist. There is no such thing as "plain text" without an encoding; "Unicode" is not itself an encoding (UTF-8, UTF-16 and UTF-32 are); a character is not always one byte, and a byte is not always one character; and the number of characters a user sees need not equal the length reported by code. The practical rules are short: store and exchange text as UTF-8, declare it everywhere, decode at the boundary of the program and encode on the way out, and treat any unexplained accented-letter garbage as an encoding question first.

Read character encoding as the shared codebook between writer and reader: the text is only as readable as the agreement about which numbers mean which letters and how those numbers were packed into bytes.