Algorithm

An algorithm is a finite, unambiguous sequence of steps that transforms an input into an output — not code itself, but the abstract procedure code implements, which is why the same algorithm can be written in Python, run on an NVIDIA GPU, or burned directly into a chip's logic gates and still be "the same algorithm." What makes a procedure count as an algorithm rather than just a set of instructions is that it must terminate, must be precise enough that a machine (or a person with no judgment calls) can execute it, and must produce a correct, repeatable result. At a high level, how a new algorithm moves from idea to running hardware looks like this:

Complexity, not raw speed, is what separates a good algorithm from a bad one. Big-O notation describes how an algorithm's running time or memory use grows as input size n grows — O(log n) binary search versus O(n) linear search versus O(n^2) bubble sort versus O(n log n) merge sort are not stylistic choices, they are the difference between a search finishing in microseconds or hours once n reaches billions. Sorting a billion-row database with an O(n^2) algorithm instead of an O(n log n) one is the difference between seconds and literal years of compute time — the same hardware, radically different outcomes, purely from algorithm choice.

The hardware-aware version of algorithm design is what separates a CPU-era CS curriculum from how modern AI infrastructure actually runs. Flash Attention (used across essentially every modern transformer-based LLM, including models served by OpenAI, Anthropic, and Meta) isn't a new mathematical operation — it's the same attention computation restructured to minimize how often data moves between a GPU's slow HBM memory and its fast on-chip SRAM, because on an NVIDIA H100 that memory-movement cost, not the arithmetic itself, is the real bottleneck. Ring-allreduce and tree-allreduce, used to synchronize gradients across thousands of GPUs during large model training, are algorithms chosen specifically for network topology, not raw operation count — the "best" algorithm on paper is often wrong once real interconnect bandwidth is accounted for.

Famous algorithms earn their fame by solving a problem naive brute force cannot touch at scale. Google's PageRank turned the entire web's link graph into a single ranking number using iterative eigenvector approximation instead of exhaustively comparing every page pair; Dijkstra's and A* algorithms route GPS navigation across millions of road segments in milliseconds instead of checking every possible path; RSA and elliptic-curve algorithms secure essentially all internet traffic by relying on problems (large-integer factoring, discrete logarithms) that are efficient to verify but computationally infeasible to reverse with classical hardware — which is exactly why Shor's algorithm, a quantum algorithm that breaks that asymmetry, is taken seriously as a long-term threat to current encryption.

Read any piece of software not as code but as an algorithm wearing a particular language and running on particular silicon: the same binary-search logic is O(log n) whether it's written in C or Python, but whether that logic lands in a CPU cache, a GPU's SRAM, or dedicated fixed-function hardware is what turns a correct algorithm into a fast one — the same yield-and-layers discipline that decides whether a chip design is manufacturable also decides, one level up the stack, whether an algorithm is actually usable at the scale it was designed for.