RAG-Token is a variant of Retrieval-Augmented Generation (RAG) that retrieves a set of relevant documents and allows the language model to attend over different retrieved documents independently for each generated token, marginalizing over the retrieval distribution at the token level rather than the sequence level. This fine-grained integration enables the model to synthesize information from multiple retrieved passages within a single generated response.

Why RAG-Token Matters in AI/ML: RAG-Token provides per-token document selection that enables the model to compose answers drawing from different source documents for different parts of the response, producing more comprehensive and accurate outputs for complex queries requiring multi-source synthesis.

• Token-level marginalization — For each generated token y_t, the model computes: p(y_t|x, y_{Multi-document synthesis — Unlike RAG-Sequence (which generates the entire response conditioned on a single document and then marginalizes), RAG-Token can seamlessly combine facts from document A in one sentence and document B in the next, producing more comprehensive answers • Retrieval distribution sharing — The retrieval probabilities p(z|x) are computed once from the input query and shared across all generation steps, amortizing the retrieval cost while enabling flexible per-token document selection • Generator architecture — A BART-based sequence-to-sequence model generates text with cross-attention over the concatenation of multiple retrieved passages, with the marginalization over documents enabling soft selection at each decoding step • Factual grounding — By conditioning each token on retrieved evidence and marginalizing over multiple sources, RAG-Token reduces hallucination compared to pure generative models while maintaining fluent, coherent text generation

RAG-Token enables fine-grained retrieval-augmented generation by marginalizing over retrieved documents at every generation step, allowing the model to seamlessly synthesize information from multiple source documents within a single response and producing more comprehensive, factually grounded outputs for complex knowledge-intensive generation tasks.