Seq2Seq: the encoder–decoder
One LSTM reads the whole sentence into a vector; another LSTM generates the translation word by word.
Why it matters
Established the encoder-reads, decoder-generates framework. The Transformer kept this overall structure and replaced the recurrent networks inside with attention.
Costs and controversy
The whole sentence is squeezed into one fixed-length vector, so long sentences lose a lot of information.
Led directly to
Source: arxiv.org ↗