Search Engine AI
AI in search engines covers the use of machine learning, and now generative language models, to interpret queries, rank documents and sometimes write the answer directly. A classic search engine returns a ranked list of links; an AI-driven one may also read the retrieved pages and compose a synthesised response with sources. The shift changes what is being optimised, from pointing to documents toward producing statements whose accuracy is the system's own responsibility.
The baseline was link analysis and hand-tuned signals. Larry Page and Sergey Brin described PageRank and the Google prototype in their 1998 paper at Stanford, "The Anatomy of a Large-Scale Hypertextual Web Search Engine". PageRank treated a link as a vote and weighted votes by the importance of the voter, computed as the stationary distribution of a random surfer on the web graph. Ranking then combined it with hundreds of other signals, such as term matches, anchor text, freshness and location, assembled and tuned by engineers. Query understanding relied on synonyms, spelling correction and click data, not on reading comprehension.
Neural ranking brought language understanding into the pipeline. Google confirmed in 2015 that RankBrain, a machine-learning system, helped interpret rare and never-seen queries. Dense retrieval, in which queries and documents are mapped into vectors and matched by similarity, such as Karpukhin and colleagues' Dense Passage Retrieval (2020), complemented keyword matching. The most visible step was BERT, the transformer model from Devlin and colleagues at Google (2018), which Google announced in October 2019 was being applied to search in English in the United States, affecting about one in ten queries. Its value was in small words that carry meaning, such as "to" and "for" in a query about travel directions, which older bag-of-words methods ignored. Google followed with MUM in 2021, and Microsoft's Bing used similar models.
AI answer engines changed the interface in 2023 and 2024. Microsoft launched a chat-based Bing in February 2023, and Google's Bard announcement the same month became notable when a demonstration answer about the James Webb Space Telescope contained a factual error. Perplexity built a product around cited answers, and Google introduced AI Overviews to US search users in May 2024, followed by publicised mistakes in odd queries that led to adjustments. These systems rely on retrieval-augmented generation, described by Lewis and colleagues in 2020: fetch relevant text, then have a model answer from it, which is intended to reduce unsupported claims compared with answering from memory.
Grounding and citation are the weak points. A citation signals that a claim is supported, but a model can attach a real link to a sentence that the page does not actually say, summarise a source too confidently, or merge conflicting pages into one smooth answer. Research on generative search engines, including a 2023 study of verifiability by Liu, Zhang and Liang, found that a notable share of generated statements lacked full support from the cited pages. Retrieval itself can surface low-quality or manipulated pages, and content written to be picked up by models is an emerging form of search spam. Quoting a source faithfully, ranking its reliability and showing its provenance remain unsolved in general.
Economics and misconceptions. A common belief is that the model simply knows the answer; in a grounded system its quality depends heavily on what was retrieved. Another is that fewer clicks always means a better result. Direct answers can serve users for simple facts while reducing traffic to the publishers whose pages supplied the information, a dispute that has prompted licensing deals and litigation. Neighbouring entries cover retrieval-augmented generation components and ranking models; this entry concerns the search product that combines them.
Read AI search as ranking plus synthesis: older link-and-signal ranking still decides what is retrieved, language models decide how it is read and phrased, and trust depends on whether the answer can be traced back to evidence that actually says it.