BERT
A bidirectional Transformer encoder that learns language through fill-in-the-blank pre-training.
340Mparameters (Large)
Why it matters
Set new records on 11 NLP tasks, set off a pre-training boom across the field, and soon went into Google Search.
Costs and controversy
It understands but does not generate; generative models later converged on GPT-style decoders.
Built on
Led to
Source: arxiv.org ↗