Chinchilla: data matters too
For the same compute, a smaller model trained on more data beats a bigger one.
70Bparameters, beating the 280B Gopher
Why it matters
Corrected the scaling laws. "About 20 tokens per parameter" became an industry rule of thumb and pointed the way for efficient open models.
Built on
Led to
Source: arxiv.org ↗