Sequence to Sequence

Sequence-to-sequence, usually shortened to seq2seq, is a neural network framework for mapping an input sequence of one length to an output sequence of a different, independently determined length, such as a French sentence to an English one, a question to an answer, or an audio clip to a transcript. It works with two cooperating networks: an encoder that reads the entire input and summarizes it, and a decoder that generates the output one token at a time, each token conditioned on that summary and on the tokens already produced. The approach made it possible to treat translation and many similar tasks as a single end-to-end trainable model instead of a pipeline of hand-built components.

The framework arrived almost simultaneously from two groups in 2014. Sutskever, Vinyals, and Le's "Sequence to Sequence Learning with Neural Networks" (NIPS 2014) used a deep LSTM to encode a source sentence into one fixed-length vector and a second LSTM to decode the translation from it, and reported a BLEU score of 34.8 on the WMT 2014 English-to-French task, ahead of the phrase-based statistical baseline of the time; they also found that reversing the order of the source words made optimization markedly easier by shortening the distance between early source words and early target words. Cho and colleagues' "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation" (EMNLP 2014) proposed the same encoder-decoder structure and introduced the gated recurrent unit as its cell, using the network at first to score phrase pairs inside a statistical system.

The fixed-length summary was the design's bottleneck, and attention was the fix that changed the field. Cho, van Merriënboer, Bahdanau, and Bengio's 2014 study "On the Properties of Neural Machine Translation: Encoder-Decoder Approaches" showed that translation quality fell off as sentences grew longer, since a single vector had to carry everything. Bahdanau, Cho, and Bengio's "Neural Machine Translation by Jointly Learning to Align and Translate" (posted 2014, ICLR 2015) let the decoder compute, at every output step, a weighted combination of all the encoder's states, effectively learning an alignment between output and input words. Luong and colleagues' 2015 variants simplified the scoring, and Google's neural translation system (Wu and colleagues, 2016) combined deep stacked LSTMs, attention, and residual connections to power Google Translate in production.

Training and inference use the model differently, and the gap between them is a classic pitfall. In training, the decoder is usually fed the true previous target token at each step, a practice called teacher forcing (from Williams and Zipser's 1989 work on recurrent networks), which makes learning fast and stable. At inference there is no ground truth, so the decoder consumes its own earlier outputs, and a single early mistake can push it into territory it never saw in training, a mismatch called exposure bias. Decoding itself is a search problem: greedy choice is fast but short-sighted, and beam search keeps several candidate sequences alive, which is why beam width and length penalties are tuning parameters in translation systems.

Seq2seq is a problem formulation, not a synonym for recurrent networks, and its architecture has been largely replaced while the framing survived. The transformer (Vaswani and colleagues, 2017) was itself introduced as an encoder-decoder for translation, with attention replacing recurrence, and it removed the step-by-step bottleneck that made training slow. T5 (Raffel and colleagues, 2019) recast many language tasks, including classification, as text-to-text problems for an encoder-decoder, BART (Lewis and colleagues, 2019) used the design for denoising pretraining, and Whisper (Radford and colleagues, 2022) uses an audio encoder with a text decoder for speech recognition. Decoder-only language models such as the GPT family treat the prompt and the answer as one continuous sequence, which handles most seq2seq tasks without a separate encoder, but encoder-decoder models remain common in translation, summarization, and speech.

Several misconceptions and limits are worth keeping in mind. The input and output do not need to share a vocabulary, a language, or a modality, and need not be the same length; the output is not guaranteed to be faithful to the input, since a fluent decoder can invent content, which shows up as hallucination in summarization and as dropped or repeated phrases in translation. Quality metrics such as BLEU (Papineni and colleagues, 2002) score overlap with reference text rather than correctness, so they can miss exactly these failures. And because output is generated token by token, errors compound and generation time grows with output length, a cost that caching of attention keys and values reduces without removing.

Read sequence-to-sequence as the encoder-decoder bargain: compress or represent the input, then generate the output step by step with access to it, and the 2014 papers' lasting contribution is less the recurrent networks they used than the idea that variable-length inputs and outputs can be handled by one trained model, with attention as the mechanism that made it work.