A rare Saturday with an uninterrupted block of time, so I'm using it to think through and organize something Liang Wenfeng raised: continual learning.
Liang Wenfeng places continual learning on the step right after agents. Karpathy open-sourced autoresearch in March — a small loop in which an agent modifies its own training code, runs its own experiments, and decides for itself whether to keep a change. One articulated a judgment; the other built the practice.
Putting these two threads side by side, plus my own felt experience from building agents lately, a few judgments have gradually settled out. This piece covers four things:
- What Liang Wenfeng said and what Karpathy built, and where I think the truly critical points are
- How continual learning actually lands technically — the four-layer path I've mapped out
- A more fundamental question: is AGI the model, or the agent?
- Rather than argue about timelines, get a ruler — the intelligence Q-value, and why I think it's more useful than any prophecy
The conclusion up front: AGI isn't built, it's ignited. The model is the fuel, the agent is the flame, continual learning is the oxygen. Let me unpack that.
1. Liang Wenfeng's Diagnosis: AI Doesn't Lack Taste, It Lacks Continual Learning
Three lines from the transcript are, to me, the critical ones.
First: if you can describe a problem very clearly and give it complete context and instructions, it already surpasses humans. Note the premise in that sentence — a human has done the learning on its behalf that it should have done itself. Who prepared the context and the instructions? A person.
Second: AI can't replace your employees today; but if AI had the ability to learn continually, then like an employee who spends two months at the company learning, it could replace anyone. This one gets at a fact: an employee's value isn't in what they know on day one, it's that by month three they've picked up the company's internal jargon, each client's temperament, and which parts of the codebase you don't touch. That value comes from the continual conversion of experience into capability — which is precisely what current AI is architecturally forbidden from doing.
Third: AI today doesn't lack taste or intuition; what it lacks is the ability to learn continually. He follows it with an example — ask it to write an article, and its taste and intuition are, in his view, perfectly fine.
To be precise about the claim: he's talking about the taste-and-intuition dimension, not asserting that AI is adequate on every axis. But that boundary is exactly what makes the judgment interesting — if the model doesn't even lack the most talent-like quality there is, then "it can't remember" is, in my view, the biggest bottleneck. I'm not ruling out long-horizon planning, reliability, or sample efficiency as candidates; I just think memory is the longest lever: once it's released, the others will likely come along with it.
I strongly agree with this diagnosis. Anyone building agents should recognize the feeling — the model's point capabilities have long been sufficient, and what actually blocks you is: whatever you taught it today, it has forgotten by tomorrow.
2. Karpathy's Practice: The First Time I Saw the Flame Burn on Its Own
Karpathy's motivation for autoresearch was mundane: he'd spent several months hand-tuning nanochat (his own complete LLM training pipeline), with the benchmark being "how long does it take to train to GPT-2 quality?" One day he asked himself — why isn't an agent doing this?
So he gave the agent three things: one file it was allowed to modify (train.py only), one metric defining "better" (val_bpb, lower is better), and a time-boxed loop (a strict 5-minute wall-clock per experiment). The agent edits code, runs the experiment, checks the metric, keeps it if better and rolls back if worse, repeat. It ran for two days and roughly 700 experiments, found about 20 stackable improvements, and compressed "train to GPT-2 quality" from 2.02 hours to 1.80 — an 11% speedup. In early March he open-sourced it as a minimal single-GPU repo of about 630 lines.
What happened next says more than the 11% does. Within two weeks it had about 48,000 stars and 6,700 forks; the result autoresearch produced ranked 5th on the nanochat leaderboard, beating every one of Karpathy's own prior hand-tuned submissions. And the agent wasn't just tuning hyperparameters: it tried architecture-level changes like reordering QK Norm and RoPE, and it caught a bug in the QK-Norm implementation where a scalar multiplier was missing — a bug Karpathy himself had missed for months. Shopify's Tobi Lütke ran it overnight on their own data: 37 experiments bought a 19% improvement. The press later started calling the whole thing the Karpathy Loop.
Two and a half months later he joined Anthropic's pretraining team, tasked with building a team to accelerate pretraining research using Claude. That destination is itself a signal, which I'll come back to.
My view is that the significance isn't the 11%, it's that there is no human inside the loop. The metric judges automatically, improvements accumulate automatically, and the human only set the rules at the start. The bug that had been missed for months is especially telling: it isn't that the agent is smarter than Karpathy, it's that it tirelessly tried every possibility — while a human skips over the places he's already decided couldn't be the problem. This is the first time I've seen, inside a (very small) closed system, the fire burning by itself.
But one thing needs to be clear. A lot of takes read this as "the model is self-evolving" — it isn't. The agent modifies the training code, not its own weights. Self-iteration happens at the research layer, not the weight layer. That distinction is the key to understanding the whole technical picture.
3. My Core Claim: Today's Post-Training Isn't Learning, It's Reincarnation
Before discussing how continual learning gets implemented, you have to look squarely at the fundamental awkwardness of the current paradigm: every model we train is a snapshot, finalized the moment it leaves the factory.
The instant training ends, the model's intuition, taste, and knowledge are all frozen into the weights. Whatever it goes through afterwards — millions of conversations, however many times it's corrected — none of that enters the weights in real time. At best it gets accumulated, waiting for the next trip back through the furnace. The next day it's still yesterday's model. Our only remedy is post-training: gather a batch of data, re-melt, ship a new snapshot.
That isn't learning, that's reincarnation. There's no continuous "self" between the new version and the old one, just a rebirth in the data sense.
To be precise: I mean re-melting without an experience stream — what gets fed in is generic corpus, not trajectories the model itself produced. Re-melting that digests the model's own experience is a different thing entirely; that's Layer 2 below.
Why not update weights online? Not for lack of desire — there are three mountains in the way:
- Catastrophic forgetting — learning the new overwrites the old; learn your company's jargon and it might forget calculus
- Alignment drift — if weights change online, the model's safety and values are in continuous, uncontrolled drift
- Cost — maintaining a separately evolving set of weights for hundreds of millions of users is unrealistic in both storage and inference
So the industry settled for second best and produced a pile of "route around the weights" learning substitutes. What's interesting is that stacked up, those substitutes happen to form a staircase toward real continual learning.
4. The Four-Layer Path: How Continual Learning Actually Lands
I've sorted everything that can currently be called "continual learning" into four layers by depth:
Layer Mechanism Do the weights move? Timescale Representative Layer 1: Context level memory files, skill libraries, retrieval No Minutes Everyone's agent memory; Karpathy's LLM Wiki Layer 2: Experience flywheel recover deployment trajectories → filter and score → periodic post-training Per version Months The main battlefield at every frontier lab Layer 3: Self-generated curriculum the model sets its own problems, adversarial symbiosis, self-produced training signal Yes Continuous Agent0; transferring AlphaGo self-play Layer 4: Meta-research agents improving training methods and underlying algorithms It changes "the machine that makes models" One discovery benefits all subsequent models Karpathy Loop; AlphaEvolve
(One note: decomposing recursive self-improvement into levels isn't my invention — Weco AI published a four-level framework earlier. But the axes differ. Theirs is by rate of improvement — faster than humans or not, does the return compound. Mine is by mechanism of improvement — do the weights move, how often does experience settle. The two axes are essentially orthogonal; the same system can sit at their Level 1 and my Layer 2 simultaneously. I use mechanism because, for people building agents, that's what determines which layer you can actually work on today.)
A few words on each layer.
Layer 1 is the truth about every agent's "continual learning" today: the weights don't budge, experience lives outside as text, and it's read back into context before the next start. The upsides are immediacy, auditability, and reversibility; the limits are just as essential — knowledge stays forever at the "visible" level and never becomes intuition. Like someone who can only work by flipping through a notebook: however thick the notebook gets, it isn't internalization.
I've hit this layer's ceiling myself. Building a design-taste skill earlier, I grew the ruleset from zero to thirteen hundred lines, added twenty-odd bans and a dozen-plus ledger entries with cited verdicts, and ran the same engine on the same problem set seven times — the human final-review scoreboard showed zero unreserved "good" verdicts. Later I added an A/B control, and output with the full ruleset was essentially tied with the bare model. Rules can prevent ugliness, but they can't produce beauty. In the previous piece on harnesses I said an agent's capability comes from structural constraints; this layer is the other face of the same idea: structure can constrain behavior, but structure cannot grant growth.
Layer 2 is the real battlefield right now. It simulates continuous growth through discrete version iteration, trading the uncontrolled risk of online learning for a controllable iteration cycle. I think what decides this layer isn't compute, it's three unglamorous components: the experience filter (feeding garbage trajectories back makes the model dumber; signal-to-noise ratio is the growth ceiling), anti-forgetting replay (digest new experience mixed with old data), and an alignment anchor (re-test the safety baseline after every iteration).
Liang Wenfeng has said that half of DeepSeek's core researchers — its most important people — are labeling data. It sounds mundane, but what they're actually doing is serving as the most expensive component in that flywheel by hand: the filter. His stated reason is practical too — the cost of high-quality data annotation isn't sustainable, and China has no cost advantage in labeling; labeling is simply too expensive. This point matters: the filter's cost not coming down isn't a technical problem, it's an economic one. Remember it — it comes back when we get to Q.
Layer 3 solves where the fuel comes from. The flywheel's experience ultimately comes from the task distribution of the human world, and human data has an end. Layer 3 lets the model manufacture its own training signal: two agents derived from the same base, one posing increasingly hard problems, the other reaching for tools to solve them, with success or failure automatically generating reward. It's essentially AlphaGo self-play transferred to general reasoning — the rules of Go were once a free referee, and now whether code runs and whether a proof verifies, this "verifiability," is the new referee. With a referee in place, humans can exit the problem-setting role.
But to be clear: what Layer 3 automates away is exactly Layer 2's most expensive component, the filter. Human investment doesn't disappear — it moves up from "annotating item by item" to "designing evaluators," from a marginal cost to a fixed one. That's also where it's most fragile: self-produced signal without an external anchor is easy to game (Goodhart, reward hacking, self-reinforcing bias), which is why Layer 3 today only really works in domains like code and math where the referee is cheap and trustworthy.
Layer 4 has the most leverage. What it improves isn't a particular model, it's training methodology and the underlying algorithms themselves — optimizers, data mixtures, kernel implementations. One discovery benefits every model afterwards; you're changing the production line, not the product. The Karpathy Loop lives here; so does AlphaEvolve, which optimizes the algorithms and kernels training depends on — including accelerating the training of the very model driving it. That self-referential loop is more intuitive than any argument. It lands squarely on Liang Wenfeng's line that AI can accelerate AI research, and so later on it may become nonlinear.
My judgment: real continual learning won't be a solo performance by any one layer; it'll be all four tightening and meshing at once. Layer 1 handles minute-scale adaptation, Layer 2 handles month-scale growth, Layer 3 handles fuel, Layer 4 accelerates everything.
5. Is AGI the Model, or the Agent?
With the four layers laid out, the question can be answered. The field has argued "model companies vs. agent companies" for a long time; my view is: neither. AGI is the closed loop itself.
To be clear, this is a proposal, not an assertion: I'm arguing for switching AGI's unit of analysis from "the model" to "the loop." Once you switch units, a lot of arguments dissolve on their own.
An analogy I find quite precise:
- Model weights ≈ the brain's synaptic connections
- Agent ≈ a person's behavior while awake
- Continual learning ≈ memory consolidation during sleep — the day's experience gets filtered, replayed, and written into long-term memory at night
Asking "is AGI the model or the agent" is like asking "is a person their brain or their behavior." The reason the question seems well-formed today is precisely that the loop is broken: the model is a frozen snapshot, the agent is a runtime wrapped around that snapshot, and experience flows into the context but never settles into the weights.
Close the loop and the boundary dissolves. The agent is the model's waking state; training is the model's sleeping state. Each release is no longer a reincarnation but a waking-up. At that point, pointing at any instant of the system and asking "is this the model or the agent" is like pointing at a running person and asking "is this the brain or the legs?"
This perspective isn't entirely my invention either. On the singularity, Liang Wenfeng's framing is that once the model can learn continually, it can develop its own versions, do its own research, and then develop its next version. That description is itself a closed loop — the way he defines the singularity already presupposes that AGI isn't a snapshot but a machine capable of turning itself. I'm just making that explicit.
So I think AGI's minimal complete unit is the closure of four things, none optional:
Weights (frozen capability) + harness (the body that executes) + experience stream (continual nourishment) + update mechanism (the gut that digests)
Without weights, no intuition. Without a harness, no contact with the world. Without an experience stream, no fuel. Without an update mechanism, all experience is fleeting. In the previous piece I said that "building an agent is really building the harness" — inside this framework, the harness is only one quarter. Model people and agent people are arguing over different organs of the same organism, and organs can't live alone.
The proposal also has a testable corollary: take two adjacent generations of base models and two experience loops that actually exist in industry, fix a single evaluation set, and give it a one-year window — if the loop is the dominant variable, then the capability gap from "same base, different loops" should exceed the gap from "different bases, same loop." If it comes out the other way, the base still dominates at this stage, and my proposal should be downgraded to "the loop is the second lever."
This also explains Karpathy's move: he went to Anthropic not to train a bigger model, but to accelerate pretraining research with Claude. The bet is on the loop, not the snapshot. I find that signal clearer than any paper.
6. Rather Than Argue About Timelines, Get a Ruler
How should we describe the process of evolving from today's agents to continually learning AGI?
I initially wanted to coin a name for it, then looked around and found: people have done this already, and more than one group. "Takeoff" is an old term in AI acceleration discussions; "ignition" is a recent coinage — Chojecki uses it for an agent autonomously converting compute into capability gains without human intervention, and Weco treats it as Level 2 in their four-level recursive self-improvement framework. Going further back, Yudkowsky borrowed nuclear fission's criticality coefficient k > 1 outright in Intelligence Explosion Microeconomics (though he notes in a footnote that this isn't an argument by analogy, just a starting point from a simple process).
So I gave up on coining a word, and came to think the more useful thing isn't a name but a ruler you can read off year by year.
Fusion has a milestone the whole world understands: Q > 1 — the first time fusion output exceeds the energy put in.
But there's a detail that gets widely misreported, and it matters for the metaphor: Q > 1 does not mean self-sustaining. At Q = 1, remove the external heating and the plasma still cools and goes out — because roughly 80% of a D-T reaction's energy is carried off by neutrons and never stays in the plasma to self-heat. Self-heating only matches external heating around Q ≈ 5; true "ignition" is external heating dropping to zero, corresponding to Q → ∞. Between "net output" and "self-sustaining" there's still a factor of several.
What I want to borrow isn't the word, it's the ratio behind it. Define the intelligence Q-value:
Q = capability gain from the system's self-produced experience ÷ capability gain from human input
The numerator is what the system burns on its own; the denominator is what humans feed in. The virtue of this definition is that it can be broken down and estimated on specific projects, rather than being a philosophical position you can only take sides on.
But one trap has to be plugged first: if humans do nothing at all, the denominator shrinks too. So Q is only meaningful under the premise that total capability is still rising — a stagnant system can post a beautiful Q, but that's not ignition, that's a flameout. Fittingly, fusion is the same: at Q = 1, remove the external heating and the fire goes out too.
Look back through this ruler and everything falls into place:
- The whole industry today sits in the kindling phase, Q < 1. Half of DeepSeek's core researchers are labeling data; everyone's most expensive line item is the judgment of human experts. And as noted above, it isn't that they don't want to bring the cost down — they can't. High-quality annotation is too expensive; the filter is an economic problem. Humans are still striking matches one at a time.
- The four-layer path is essentially four routes to pushing Q up. Layer 1 stops experience from evaporating (denominator flat, numerator up slightly). Layer 2 gets experience into the weights (the numerator starts growing substantially). Layer 3 lets the system manufacture its own fuel (the human marginal contribution can approach zero for the first time — mechanism design is a one-time cost, and once it's running each additional round needs no human). Layer 4 lets the system improve how it burns (the numerator goes exponential).
- The significance of the Karpathy Loop is that it's the first time we've seen Q > 1 on an open-ended task like "improve AI training methodology." AlphaZero achieved this long ago in a rule-closed domain like Go, but there the referee was free. Autoresearch's referee is only an arbitrarily modifiable training script and a single validation metric, which is much closer to what real research looks like. The fire is small and the furnace is small, but it's burning on its own.
Following that ruler, the road to AGI gets gradations too, and requires no mysticism: kindling (Q < 1, humans repeatedly striking matches) → net output (Q > 1, the system's self-produced gain exceeds human input for the first time — but pull the humans out here and the fire still dies) → self-sustaining (human marginal contribution approaches zero; the fusion equivalent of ignition) → spread (from verifiable domains like code and math out to every intelligent task).
Liang Wenfeng's own timeline is in three stages: first solve continual learning, then reach the singularity of self-iteration, and only then embodied intelligence — after which it walks into the physical world. My ruler only covers the first two stages; the third is a different question. Q measures intelligence self-sustaining; it doesn't measure whether it has a body.
One more thing I especially want to say. In the transcript, Liang Wenfeng says the singularity isn't really a singular point — it's also a gradual process, not a discontinuity. I agree completely, and the Q ruler happens to carry that correction natively: fusion ignition isn't an explosion, it's the gradual shift from "flash on each pulse" to "sustained burn." Q climbs from 0.3 to 0.7 to 0.98, crosses 1.0, and heads on toward 2 and 10. There will be no morning when sirens announce the arrival of AGI; there will just be some year when you look back at the numbers and notice that the human share of contribution has gone from the bulk to a rounding error. The singularity isn't a bang, it's a curve slowly crossing 1.
I also know this ruler can't be rigorously measured today — how to quantify "capability gain," how to attribute numerator versus denominator, these are hard problems.
But a rough metric people can argue over beats a timeline where everyone just talks past each other. At minimum it turns "how far are we from AGI" into "what's this system's Q this year?"
7. Closing
To tie it off.
Liang Wenfeng subtracts — only the AGI mainline; however hot video generation gets, DeepSeek doesn't do it, because it has nothing to do with the roadmap toward intelligence. Karpathy divides — he hand-tuned a script for months and then deleted himself from the loop. Two forms of restraint, one direction: letting the system grow itself is closer to the essence of intelligence than growing it for it.
For those of us building agents, there's a very practical corollary: every memory mechanism you write today, every trajectory-recovery pipeline, every automated eval loop — viewed through the Q framework, none of these are product features. Memory and trajectory recovery are laying pipe to the numerator; automated eval is doing the filtering in humans' place, pressing the denominator down. They point the same way. The harness determines whether your agent runs stably today; the experience loop determines whether it gets stronger tomorrow. The first is a structural problem, the second a growth problem, and you have to do both.
The human role in this process is also changing shifts: from fuel handler (labeling data, tuning hyperparameters), to furnace designer (building the harness, setting rewards, placing anchors), and finally to fire watcher — deciding which fires should burn, and how far. The third role is the quietest, and I think the most important. Because after ignition, the question worth answering more than "how big can the fire get" is: what do we want it to illuminate?
This line of thinking isn't complete and isn't entirely correct — how to rigorously measure Q, the coupling order among the four layers, which domain reaches self-sustaining first: I have settled answers to none of these. This piece is a denominator too. And the endpoint of the denominator is no longer needing one.