GPT-1: Generative Pre-Training
Pre-train on huge amounts of text by predicting the next word, then fine-tune for specific tasks.
117Mparameters
Why it matters
Established the pre-train-then-fine-tune paradigm, and gave GPT its name: Generative Pre-trained Transformer.
Costs and controversy
Small in scale, and outperformed by BERT later that year.
Built on
Led to
Source: cdn.openai.com ↗