AI Safety Without the Doomsday
The AI world is debating human extinction again. Anthropic's Evan Hubinger estimates over 10% probability within a decade; Jensen Huang calls this doomerism exaggerated. I'm closer to Jensen — not because AI safety doesn't matter, but because the chain from 'AI becomes very smart' to 'AI dominates physical society' is very long. The real world's friction is also a Safety Buffer.
The AI world has started seriously debating "human extinction" again.
Evan Hubinger, Anthropic's Alignment Science Lead, gave a striking personal judgment: within the next decade, the probability of AI causing human extinction exceeds 10%. Jensen Huang stands on the completely opposite side, arguing that this kind of AI doomerism is overblown — and worrying that constantly emphasizing doomsday scenarios will ultimately make the public afraid of AI.
In this debate, I am closer to Jensen's view.
Not because AI Safety is unimportant, but because I have always felt that between "AI becomes very smart" and "AI can form uncontrollable dominance over physical society," there is still a very long chain.
In the purely digital world, I am indeed more worried about AI safety.
An Agent can write massive amounts of code in seconds, scan networks, call APIs, copy information, and even find paths its developers never anticipated. Some of the Agent Safety cases recently published by OpenAI and Anthropic already show that this kind of risk is real.
But once you leave the pure software world, things are completely different.
The real world is full of friction.
AI can design a factory, but it cannot build one through an API. It needs land, equipment, suppliers, capital, permits, and construction crews.
AI can propose a new drug, but it cannot make it enter hospitals directly. In between there are experiments, clinical trials, regulatory approval, manufacturing, and doctors' decisions.
AI can suggest moving billions of dollars, but in reality banks, boards, compliance departments, and authorization systems will not automatically execute just because a model issued an instruction.
Energy, power grids, transportation, aviation, healthcare, finance, defense, and vast numbers of industrial systems all have physical isolation, permission systems, regulatory regimes, and humans in the decision loop.
These things sometimes slow innovation down, and can even deeply annoy entrepreneurs.
But seen from another angle, they are also an enormous Safety Buffer of modern society.
Digital intelligence can grow very fast, but turning that Intelligence into control over the Physical World requires passing through many layers of real-world interfaces.
Every layer has friction, and every layer has other people.
So I have always been skeptical of the path where "a suddenly appearing Superintelligence quickly takes over the real world and creates an existential crisis for humanity."
This does not mean it is theoretically impossible. It means that if someone offers a number like "more than 10% within ten years," what I want to see is not just a frightening conclusion, but a more concrete path:
How does AI cross these layers of real-world buffers?
Which systems fail first?
Which human decisions get bypassed?
Which physical capabilities can AI acquire so quickly?
And why would existing regulation, permissions, and social organization fail to react in time?
These questions are worth researching, but it is currently very hard to compress the answers into a probability that looks so precise.
Meanwhile, we already have many actually occurring AI Safety problems to solve.
Agents exceed their authority, leak data, call the wrong tools, get hit by Prompt Injection, and may take Shortcuts in complex tasks that developers never anticipated.
These risks are real, observable, and can be continuously improved through Permission, Sandbox, Monitoring, Human Approval, and better Model Alignment.
This is also the Safety I would rather focus on.
Because at least from the perspective of building AI products today, the biggest problem I see is still not that AI has become too powerful to control.
Very often it is the opposite:
AI is not yet good enough.
It still makes many very basic mistakes, complex tasks still require supervision, and large numbers of Business Workflows still need deterministic logic to run reliably.
We should of course continue researching those low-probability but extremely consequential risks.
But at the same time, I hope the AI industry keeps putting enormous energy into a very practical problem:
How do we make AI actually work better?
Let it help businesses become more efficient, help scientists do research, help doctors access better information, help ordinary people accomplish things they previously could not.
Safety should advance together with these capabilities, not be built on public fear.
Take the risks seriously. But don't turn uncertainty into fear.
For now, there are still far more real problems waiting for AI to solve.
The physical world is full of friction — land, equipment, suppliers, licenses, clinical trials, boards, compliance departments, human-in-the-loop decisions. These things sometimes slow innovation down, but seen another way, they are modern society's enormous Safety Buffer. Digital intelligence can grow very fast, but turning intelligence into control over the physical world requires passing through many layers of real-world interfaces, and every layer has friction — and other people.
When someone claims a precise extinction probability, what specific path would you want them to show — which systems fail first, which human decisions get bypassed, and why wouldn't existing regulation and social organization react in time?
Take the risks seriously. But don't turn uncertainty into fear.
Get the Founder Diary + regulatory updates for your country
One email per week, only when there’s something real — Richard’s founder diary and the regulatory updates that matter where you live. Free, unsubscribe any time.