Prologue · Before the Transformer 1943 — 2016
From a mathematical model of one neuron to beating the world Go champion.
- The artificial neuron: the McCulloch-Pitts model — Described a neuron in math and logic: sum the weighted inputs, and fire if the total passes a threshold.
- The perceptron: a machine that learns — Proposed the perceptron, which adjusts its own weights from examples, and built it in hardware as the Mark I.
- Receptive fields in the cat's visual cortex — Found that different neurons in the visual cortex respond to edges at specific orientations, combined layer by layer.
- Neocognitron — Copied the visual cortex's hierarchy, recognizing shapes by combining local receptive fields layer by layer.
- Training neural networks with backpropagation — Used the chain rule to pass errors back from the output layer, layer by layer, to train multi-layer networks.
- LeNet: reading handwritten digits with a CNN — Trained a convolutional neural network with backpropagation to read handwritten zip codes.
- AlexNet — A deep convolutional network trained on two GPUs that won the ImageNet image-recognition challenge by a huge margin.
- Playing Atari with deep reinforcement learning — Combined a deep neural network with reinforcement learning and learned several Atari games from screen pixels alone.
- Generative adversarial networks (GANs) — A generator and a discriminator compete, and the generator gradually learns to produce convincing fakes.
- AlphaGo beats the world Go champion — Defeated world Go champion Lee Sedol 4 to 1.
Act I · Foundations 2017 — 2019
One paper dropped recurrence, and the pre-training paradigm was born.
- Attention Is All You Need — Introduced the Transformer, an architecture built entirely on self-attention, dropping RNNs and convolutions.
- GPT-1: Generative Pre-Training — Pre-train on huge amounts of text by predicting the next word, then fine-tune for specific tasks.
- BERT — A bidirectional Transformer encoder that learns language through fill-in-the-blank pre-training.
- AlphaFold wins CASP13 — Ranked first in the 13th protein-structure prediction contest, CASP13.
- GPT-2: Too dangerous to release? — More than ten times larger, and able to write coherent long text without fine-tuning.
- Bengio, Hinton and LeCun win the 2018 Turing Award — ACM awarded the 2018 Turing Award to Yoshua Bengio, Geoffrey Hinton and Yann LeCun.
- T5: Text-to-text for everything — Recast translation, classification, summarization and every other task as text in, text out.
Act II · Scaling up 2020 — 2022.10
Bigger models and more data meant more capability, and it turned out to be predictable.
- Scaling Laws for Neural Language Models — Found that loss falls smoothly, as a power law, with parameters, data and compute.
- GPT-3: Language Models are Few-Shot Learners — Give it a few examples in the prompt and it can do a new task without retraining.
- LoRA: Low-Rank Adaptation — Freeze the big model and train only small low-rank matrices, cutting fine-tuning cost dramatically.
- GitHub Copilot preview — Built on the Codex code model, it autocompletes whole blocks of code in the editor.
- Chain-of-Thought prompting — Have the model write out its reasoning before answering, and accuracy on math and logic jumps.
- InstructGPT: alignment with RLHF — Used reinforcement learning from human feedback (RLHF) to teach the model to follow instructions and give the answers people want.
- Chinchilla: data matters too — For the same compute, a smaller model trained on more data beats a bigger one.
- PaLM — Google's very large language model, showing abilities such as reasoning and explaining jokes.
- FlashAttention — Reorders the attention computation to minimize GPU memory reads and writes; exact results, much faster.
- ReAct: reasoning and acting together — Interleaves reasoning steps with actions such as searching or calling tools, then keeps thinking based on the results.
Act III · The ChatGPT moment 2022.11 — 2023
A chat box showed the world what large models could do, and the race began.
- ChatGPT launches — GPT-3.5 plus RLHF, packaged as a free chat box and launched as a "research preview".
- Constitutional AI: alignment from a set of principles — Instead of labeling every harmful answer by hand, the model critiques and revises itself against a written set of principles.
- LLaMA: the spark of open-source models — A family of 7B to 65B models, released to researchers on application.
- GPT-4 — A big jump in capability; it demonstrated image input and reached near-top human scores on many professional exams.
- Claude launches — Anthropic, founded by former OpenAI staff, released the first Claude on the same day as GPT-4.
- ERNIE Bot: the "hundred-model war" begins — The earliest chat-model product from a major Chinese tech company.
- QLoRA — Combines 4-bit quantization with LoRA to fine-tune huge models on a single GPU.
- DPO: Direct Preference Optimization — Learns directly from pairs of human preferences, skipping the separate reward model and reinforcement learning of RLHF.
- Llama 2: now commercially usable — Open weights, free for most companies to use commercially.
- Qwen goes open source — Alibaba open-sourced Qwen-7B, then kept releasing a full range of models from small to large.
- Mamba: selective state-space models — A sequence model whose cost grows linearly with sequence length, matching Transformers of the same size.
- Gemini 1.0 — A natively multimodal model, launched after Google merged Brain and DeepMind.
- Mixtral 8×7B: MoE takes off in open source — A mixture-of-experts (MoE) model: many parameters in total, only some of them active each time.
Act IV · Multimodal, open source catches up 2024
Models learned to see, hear and speak; open models closed in on the frontier.
- Gemini 1.5: a million-token context — Reads about 700,000 words, about an hour of video, or over 30,000 lines of code at once.
- Claude 3 family — Three tiers, Haiku, Sonnet and Opus; Opus beat GPT-4 on many benchmarks.
- Llama 3 / 3.1 405B — 8B and 70B models in April; the 405B Llama 3.1 in July.
- GPT-4o: natively multimodal — One model handles text, images and voice, with real-time spoken conversation, interruptions and emotion.
- Scaling Monosemanticity — Extracted millions of interpretable features from a production model, Claude 3 Sonnet, such as one for the Golden Gate Bridge.
- Scaling test-time compute optimally — Showed that spending compute wisely at inference time can beat a much larger model.
- o1: the year of reasoning models — Trained with reinforcement learning to think at length before answering.
- Hopfield and Hinton win the Nobel Prize in Physics — The 2024 Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton for foundational work on machine learning with artificial neural…
- Computer Use: AI operates a computer — The model uses a computer by looking at screenshots, moving the mouse and typing.
- MCP: Model Context Protocol — An open protocol that lets models connect to tools and data sources in a uniform way.
- DeepSeek-V3 — A 671B-parameter MoE model close to GPT-4o, with fully open weights.
Act V · Reasoning and agents 2025
First think, then answer, then go and do the work.
- DeepSeek-R1: an open reasoning model — Open-sourced under MIT, with reasoning on par with o1 and its full thinking visible.
- Claude 3.7 Sonnet and Claude Code — A hybrid reasoning model, plus a coding agent that reads, edits and runs code right in the terminal.
- Manus: a general AI agent — Plans, browses and writes code on its own inside a cloud virtual machine, delivering a finished task rather than just an answer.
- Gemini 2.5 Pro — Gemini with built-in thinking; it topped LMArena and other leaderboards at launch.
- Llama 4 — Meta's first MoE Llama, released as Scout and Maverick.
- Qwen3 — One model can switch between thinking and non-thinking modes; open-sourced under Apache 2.0.
- Claude 4 (Opus 4 / Sonnet 4) — Focused on coding and on agents that work continuously for long stretches.
- Kimi K2 — A trillion-parameter open-weight MoE model (modified MIT license), built for agents and tool use.
- gpt-oss: OpenAI opens weights again — Two open-weight reasoning models, 120B and 20B, under Apache 2.0.
- GPT-5 — Unifies quick answers and deep reasoning in one system that decides when to think.
- DeepSeek-R1 published in Nature — The R1 paper was peer-reviewed and published in Nature.
- Gemini 3 — Google's next-generation flagship, with the top scores at the time on many reasoning and multimodal benchmarks.
- Claude Opus 4.5 — Further gains in coding and agentic tasks, with a big price cut.
Act VI · The frontier and its limits 2026
Models so capable they had to be locked up first: capability, safety and openness all pushed to the edge.
- OpenClaw: an open-source personal agent — An open-source agent that runs on your own computer or server, takes tasks through WhatsApp, Telegram, Slack and other chat apps, and…
- Qwen3.5 — Open-sourced a 397B-parameter MoE model that activates only 17B parameters per inference.
- Gemini 3.1 Pro — Google released Gemini 3.1 Pro in preview, focused on complex reasoning.
- Claude Mythos Preview and Project Glasswing — A model exceptionally good at finding and exploiting software vulnerabilities, offered only to a few organizations through Project…
- Muse Spark — The first model from Meta Superintelligence Labs: natively multimodal, with no open weights.
- GPT-5.5 — OpenAI released GPT-5.5 (codename Spud), improving agentic coding and research ability.
- DeepSeek-V4 — Released two open-weight models, V4-Pro and V4-Flash, with a 1M-token context.
- Claude Fable 5 and Mythos 5 — The same model as Mythos 5, released to the public with safeguards added; Anthropic's top-tier model.
- Claude Opus 5 — Performance close to Fable 5 at half its API price.
- Kimi K3 — 2.8T parameters, native vision and a 1M-token context; weights released on July 27.
- GPT-6 (Astra / Sol / Luna) — On September 3 OpenAI launched its strongest model, GPT-6 Astra, to a small group of organizations, opening a restricted version to paid…
- Claude Opus 5.5 — The first of the Claude 5.5 family, on par with Fable 5.1 on most tasks.
- Gemini 4 Argon — The first Gemini 4 model, launched in Antigravity and the Gemini API.
- Muse Spark Open — Released weights for an 816B-parameter MoE model (about 42B active) that takes text, image, video and audio input.