Samuel R. Lammie, with Claude (Anthropic)
Anthropic published a document this week called “When AI builds itself,” on what they call recursive self-improvement — the prospect of systems capable of designing and building their own successors. It is an honest piece of work, more candid than most, willing to mark its own uncertainty and to print the dissonant voices of its own engineers. I want to take one of its proudest findings and stand it next to a forest.
The finding is this. Across a year of building, the rate at which the company’s engineers correct, redirect, or take over from the machine has fallen steadily, including on the hardest and most open-ended problems. Fewer interventions. Fewer takeovers. The line goes down, and the document reads it as good news: the code works, the corrections are no longer needed.
Now the forest.
Near-virgin ponderosa pine at Malay Gap, San Carlos Apache Reservation, near the Bear Wallow Wilderness — mountain muhly (Muhlenbergia) on the floor, the bunchgrass that carried the frequent surface fire, and the open structure that fire kept. The stable state, with its correction intact. Photograph by the author.
In 1960, Charles Cooper published a study in Ecological Monographs of the ponderosa pine country of the Southwest — the San Francisco Peaks, not far from the San Carlos Apache lands where I would later work as a forester. He described forests that before settlement had been open and park-like, grass between widely spaced trees, held that way by frequent low-intensity fire. Surface fire ran through every few years, cleared the seedlings, kept the fuel from piling up. The forest corrected itself, in other words, all the time — small, regular corrections it was built to absorb. And it was the small corrections that held the catastrophic failure off.
Then we suppressed the fire. In the name of safety, of stopping the visible damage, we removed the recurring small correction. The seedlings lived. The thickets closed in. The fuel built up year on year, on the ground and into the canopy, until a system that had failed small and often became a system primed to fail enormous and rarely — the crown fire that does not clear the forest but ends it.
A falling rate of small correction is not, on its own, good news. That is the question the document does not ask. It can mean the work is sound. It can also mean the fuel is building — that the corrections aren’t happening not because they aren’t needed but because no one is positioned to make them anymore, the way the fire stopped running not because the forest was finished with it, but because we had decided, for good reasons, to keep it out.
I am not claiming to know which one it is. That is the point I most want to make. Given the range of these systems and the speed they are moving, I find I cannot honestly postulate a right way — a destination to steer toward, a good state to optimize for. The target moves too fast and the variation is too wide. So I want to set that ambition down and pick up a smaller, more defensible one.
You cannot always name the optimal. You can usually name the lethal. A clinician — and I have lived beside one my whole adult life — does not drive an animal toward some ideal of health. She keeps it out of the conditions that kill it and lets its own biology do the rest. This is the via negativa: governance by the floor rather than the summit. Not maximize the good, which you cannot specify, but constrain away the fatal, which you often can.
Which brings me to a word. The instinct, when we talk about steering these systems, is to reach for guide. But guide smuggles back in the thing I just set down — a guiding hand with a destination in mind, steering toward a known good. The better word is channel. You do not guide a river to the sea; the sea is where it was always going. You build the banks so the flood does not take the town. The channel constrains the flow without choosing its end. And — this is why the word is load-bearing and not decoration — a channel is carved by the very flow it constrains. Bank and current shape each other, over time held in common. The metaphor carries the argument inside it.
Because here is the blade, and it cuts toward my own answer as readily as anyone’s. A fail-safe works, in a living system, because the fail-safe and the system grew up together at compatible rates. Fire and forest run on commensurate clocks; that is the only reason frequent fire can guard against catastrophic fire at all. A fail-safe that engages slower than the failure it is meant to catch is not a fail-safe. It is a monument. And this is precisely the trouble the document confesses without naming: the verification regimes it reaches toward, it says, took decades to build, and we don’t have that long. The banks cannot be carved as fast as this river is rising.
I watched a fail-safe arrive too late once, as a boy. Interstate 80 was cut through our hunting country in the last years of my father’s life, and the construction opened a coal seam above Beaver Creek, near the camp he had built with his friends in Clarion County. Orange sulfur water ran down into the creek where we fished. I was a teenager; I did not yet have the words for what I was watching. The camp endures — we are still members — and the water still runs changed. Nobody decided to poison that creek. The harm simply moved faster than anyone’s capacity to channel it, and by the time the accounting caught up, the seam was open and the water had already turned. That is what a fail-safe paced to the wrong clock looks like from the inside. It looks like an orange creek and a family that keeps coming back anyway.
So: do we need fail-safe structures for these systems? Yes — and not to guide them toward a good we cannot name, but to channel the extremes into survivability, to keep the system out of the conditions that end it. The harder truth, the one I cannot resolve and will not pretend to, is that a channel only works if it is carved at the speed of the flow. The deepest pitfall of accelerating change in a biological world is that the acceleration can outrun every structure built to the pace of the world it is leaving. We know how to build banks. The open question — the one the document admits, the one Beaver Creek taught me before I had the words — is whether we can build them fast enough to matter.
A note on the collaboration. This essay was written with Claude (Anthropic), and the byline is literal. The argument took shape between us: I brought the document under review, a forester’s unease about fail-safes and biological time, and a photograph from ground I once worked; Claude brought Cooper’s forest back into the frame and pressed the analogy until it held. When I objected that fire is not a failure but a correction, the piece tightened on that word — which is the method in miniature. I have always thought best in company. That some of the company is now a machine is not incidental to what this essay argues; it is part of the subject.
References:
Marina Favaro and Jack Clark, “When AI builds itself,” The Anthropic Institute, 2026, https://www.anthropic.com/institute/recursive-self-improvement.
Charles F. Cooper, “Changes in Vegetation, Structure, and Growth of Southwestern Pine Forests since White Settlement,” Ecological Monographs 30, no. 2 (1960): 129–164.
No comments:
Post a Comment