Encoder-Decoder

An encoder-decoder is a neural network architecture with two parts: an encoder that reads an input, such as a sentence, and turns it into an internal representation, and a decoder that generates an output, such as its translation, from that representation. When input and output are both sequences, such models are also called sequence-to-sequence (seq2seq) models.

History

In 2014 Ilya Sutskever, Oriol Vinyals and Quoc Le at Google showed that a pair of recurrent neural networks could learn to translate, while Kyunghyun Cho and colleagues in Yoshua Bengio’s group in Montreal, who coined the term “RNN Encoder-Decoder”, used such a pair to improve a statistical translation system. Squeezing a whole sentence into one fixed-size vector was a bottleneck, so Dzmitry Bahdanau, Cho and Bengio added attention, letting the decoder look back at every input word. Google Translate switched to a neural encoder-decoder system in 2016 (see Machine Translation), and the transformer of 2017 was itself designed as an encoder-decoder.

Three families

Transformers later split into three branches. Encoder-only models such as BERT suit understanding and classifying text. Decoder-only models such as GPT generate text one token at a time and dominate today’s chatbots (see Autoregressive Model). Encoder-decoder models such as Google’s T5 (2019) are still used for translation, summarization and speech recognition – OpenAI’s Whisper, for example. Autoencoders, which learn to rebuild their own input, share the same two-part structure.

Why it matters

The encoder-decoder idea – encode first, then generate – shaped machine translation and the transformer, and its split into separate branches explains the architecture of today’s chatbots (see Large Language Model).

Written with Claude, made by Anthropic, whose Claude models generate text one token at a time, as do the decoder-only models of competitors OpenAI, Google (also an Anthropic investor) and Meta; Anthropic does not publish its models’ architecture. Ilya Sutskever, a co-author of the 2014 paper, later co-founded OpenAI and, in 2024, Safe Superintelligence.

Related

Transformer · Recurrent Neural Network · Attention · Machine Translation · Autoregressive Model · BERT

Sources: Ilya Sutskever, Oriol Vinyals and Quoc V. Le: Sequence to Sequence Learning with Neural Networks (arXiv) (2014) · Kyunghyun Cho et al.: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (arXiv) (2014) · Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio: Neural Machine Translation by Jointly Learning to Align and Translate (arXiv) (2014) · Colin Raffel et al.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (arXiv) (2019)

Leave a Reply

Your email address will not be published. Required fields are marked *