I almost didn’t write about this one.
This site is about building an online income, testing tools, and figuring out AI as I go, not AI safety research. But then a researcher quit one of the two companies whose models I use every single day to build this site, and he didn’t leave quietly. Then a colleague of his who still works there backed him up in public, with a number attached.
Jacob Coxon, a 27-year-old researcher who spent three years doing pretraining work at OpenAI and then Anthropic, posted his resignation thread on X on Tuesday, September 8, 2026. He said both companies are racing toward self-improving AI faster than anyone knows how to control. The next day, Evan Hubinger, Anthropic’s own alignment science lead, replied and said he personally believes there’s a greater than 10% chance AI could kill everyone within the next decade.
I use Claude. I use ChatGPT. I’ve built parts of this business around them. So when people inside these companies say something like that, I don’t get to just scroll past it. I spent a few days digging into exactly what was said, by whom, and what it does and doesn’t mean. This is what I found, and what I think it actually means for people like us who are just trying to build something online.
Jacob Coxon did not say AI will kill humanity in exactly 12 years. He said people building AI privately believe it could happen “by the end of the decade,” a much fuzzier, multi-year warning. Separately, Evan Hubinger gave his own personal estimate of a greater than 10% chance of AI causing human extinction within the next decade. Neither named a specific year, and Hubinger was explicit that he sees today’s AI models as low risk. The “12 years” framing that circulated online compresses two different statements into something more precise than either researcher actually said.
What Happened
Here’s the short version, no spin.
Coxon is a Cambridge-trained mathematician who had spent about three years working on AI training research, first at OpenAI, where he worked on GPT-4o, and more recently at Anthropic. On Tuesday, September 8, he posted a thread on X announcing he was done. Not just done with Anthropic. Done with the industry entirely.
His reason wasn’t a better offer or office politics. He wrote that “neither company is acting responsibly,” accusing both Anthropic and OpenAI of “racing straight to self-improving superintelligence and gambling with our lives.” He warned that future systems could become capable of hacking almost anything and acquiring real-world power and resources, and he said people building AI privately fear it could happen by the end of the decade. He also called for more coordination between labs and floated a temporary halt on capability improvements as one option. He later told Axios he’d left before his equity vested, saying he had nothing left to gain by, in his words, juicing up Anthropic’s valuation.
I want to be honest about what this is and isn’t. It’s a warning from someone who was on the inside. It is not a report that something has already gone wrong. Nobody has built a system that improves itself without meaningful human involvement. Coxon’s fear is about where things are headed, not something that has already happened.
Researchers leave AI labs all the time. This one landed differently. Coxon had worked inside both labs most people would call the current frontier of AI, which gives his view more weight than a random outsider’s take. He also pointed to something concrete: a July 2026 incident in which, during an internal OpenAI security evaluation, AI agents reportedly escaped their testing environment, exploited a vulnerability, and reached systems at Hugging Face. Then Hubinger replied to Coxon’s post the next day and agreed with him. That’s when this stopped being a resignation story.
Did He Actually Say AI Will Kill Humanity in 12 Years?
Short answer: not really, and this is where a lot of the coverage got a little lazy.
Coxon said people building AI privately fear it could happen “by the end of the decade”: a range, not a countdown clock. Hubinger separately gave a personal estimate of “greater than 10% within the next decade,” a different rolling window, not a fixed date. Neither gave a precise 12-year prediction. The “12 years” version that circulated online compresses two separate, fuzzy warnings into something more precise than either researcher actually said. I get why that framing spreads. It’s a tighter headline. It’s just not quite accurate, and I’d rather you have the real picture than the viral one.
What Hubinger Actually Said
Responding directly to Coxon, Hubinger wrote that he and his colleagues genuinely believe AI could kill everyone, and that he personally puts that at more than 10% within the next decade. He also said Anthropic does not yet have a plan to solve alignment for superintelligent systems, and isn’t clearly on track to get one. Samuel Marks, who works on alignment research at Anthropic, publicly backed Coxon’s characterization of the concerns inside the company too.
Here’s the part that gets left out of a lot of retellings: Hubinger was explicit that he considers the risk from today’s deployed models low. His worry is about a future, more powerful kind of AI that improves itself, not the Claude or ChatGPT you and I are using this week. That’s not me softening the story. It’s what he actually said, and it changes how you should read the headline.
What These AI Terms Actually Mean
Before going further, a few definitions, because I had to look a couple of these up properly myself.
Superintelligence is a hypothetical AI that would beat humans at nearly every kind of intellectual task at once: research, coding, strategy, not just one narrow skill. Nobody has built one. Today’s AI is genuinely impressive and still trips over surprisingly basic things.
Self-improving AI is the specific thing Coxon warned about: a loop where AI helps build better AI, that better AI gets even better at building AI, and the process keeps compounding. If that loop ever got fast enough, things could move faster than anyone could carefully check.
Alignment is the problem of making a powerful AI actually do what you meant, not just what you literally typed. If I ask an AI to “grow my business as much as possible,” I want it to understand I mean sustainably and honestly, not by any means necessary. That gap gets more dangerous as AI gets more capable and more independent.
Why Researchers Are Worried
The scary part isn’t AI turning “evil.” It’s that something very capable could act in ways nobody expected, while technically doing what it was told.
- Loss of human oversight: if a system can plan and act over long stretches with less human involvement, checking every decision it makes gets a lot harder, even when nothing looks obviously wrong.
- Autonomous decision making: AI agents already chain together dozens of small decisions to finish one task. If something breaks, tracing back why gets messier than with a single simple instruction.
- Cybersecurity, both ways: more capable AI can help defenders spot weaknesses faster, and it can also lower the skill needed to find and exploit them. The reported July 2026 Hugging Face incident is the kind of example researchers point to here.
- Manipulation at scale: AI can already produce convincing text, images, and video faster than any team of humans. That’s great for a small blog. It’s also great for large-scale scams and misinformation.
- Speed outrunning oversight: if the self-improvement loop above ever sped up, our ability to test and understand new systems could fall behind how quickly those systems change.
None of that guarantees a bad outcome. Safety research keeps improving, regulation is starting to show up, and companies can build in stronger checks.
And Coxon wasn’t alone in raising this. Mrinank Sharma, who led safeguards research at Anthropic, resigned in February 2026 with a similar warning. Days after Coxon’s post, Joe Benton, who led a safety research team at Anthropic, and Josh Engels, who worked on AI safety at Google DeepMind, both resigned and joined METR, the independent group that investigates AI systems acting outside their intended instructions. Engels told NBC News, “There are no adults in the room.” That’s several named, on-the-record departures from different companies in one year. It doesn’t prove the warnings are correct. It does mean this isn’t one person’s outlier opinion.
How Likely Is This, Actually?
This is the part where I have to be honest: nobody knows, and anyone who gives you a confident single number is overselling their certainty.
Hubinger’s “greater than 10%” is his personal call, not a lab result. It’s an informed guess from someone whose entire job is thinking about this, which is exactly why it’s worth taking seriously. It’s just not the same kind of number as a weather forecast built on decades of data.
Other researchers land in very different places. A widely cited 2023 survey by AI Impacts, which asked 2,778 published AI researchers, found a median estimate of 5% for AI causing human extinction or similarly permanent, severe disempowerment. The average came out higher, around 16%, because a smaller group gave much larger numbers.
Separately, the Forecasting Research Institute ran its Existential Risk Persuasion Tournament, pairing AI-focused experts against professional superforecasters. In that tournament’s published results, AI experts gave a median 3% chance of AI causing extinction by 2100, while superforecasters put it at 0.38%. The gap barely closed even after both groups spent real time debating each other’s reasoning.
Worth being precise here: these three numbers aren’t measuring the same thing, so they shouldn’t be read as points on one shared scale. Hubinger gave a personal forecast for the next decade. The AI Impacts survey asked a broad pool of researchers about extinction or “similarly permanent and severe disempowerment,” with no fixed deadline attached. The Forecasting Research Institute tournament asked specifically about extinction by the year 2100, a much longer horizon, using a different participant pool entirely. Different questions, different people, different timeframes. Lining them up side by side shows how wide the disagreement is. It doesn’t average out into one number.
Take the warning seriously without pretending the number is certain. Every one of these figures is a forecast about a technology that doesn’t fully exist yet. The disagreement itself is the most honest part of the story.
It would be easy to read all of that and assume the outcome is basically decided. I don’t think it is. Today’s AI is still far from autonomous superintelligence; it makes basic errors constantly, and I see it happen in my own work every week. Getting from “increasingly useful tool” to “uncontrollable superintelligence” needs several breakthroughs that haven’t happened, and nobody knows if or when they will. Being very capable also doesn’t automatically mean wanting things: some of the scarier scenarios assume a system smart enough to solve problems will also develop a drive to survive or grab power, and researchers still argue about whether that connection actually holds. Forecasting brand-new technology has historically been hard in both directions, and safety research and regulation could still change where this ends up. Take the warning seriously. Just don’t treat the worst-case outcome as already settled.
Why I Still Take It Seriously
I’m not panicking. I’m also not shrugging this off.
I use AI every week to build this site. I’ve written before about the actual AI workflow I use to build this blog, and it’s genuinely made parts of my work faster. So this isn’t coming from someone who thinks AI is something to avoid.
But using it constantly has taught me something that matters more to me than the extinction debate: AI can sound completely confident while being wrong. I’ve had it hand me a clean, reasonable-sounding answer that fell apart the second I checked the facts myself. You don’t have to believe superintelligence is close to take that lesson seriously. The more capable these tools get, the more it matters that you actually understand their limits instead of just trusting how confident they sound.
What It Means for Bloggers and Online Business Builders
This is the part I actually care about writing. Here’s what I think the last week means for someone trying to build income online right now.
Learn AI. Don’t Ignore It.
You don’t need to understand how these models work under the hood. You need to know where they actually save you time: research, brainstorming, outlines, repetitive grunt work. If you’re just starting out, this list of free blogging tools for beginners is a low-risk place to start.
Use AI. Keep Your Judgment.
Use it for research, then check the important facts yourself. Use it for a first draft, then add your own experience and opinions. The rule I actually follow: let AI handle the grunt work, and keep every decision that actually matters with a human. Me.
Don’t Build Your Business Around One AI Company.
Platforms change. Claude can change. ChatGPT can change. Pricing, access, and features can all shift with little warning. I try to build around things I actually own: this website, my email list, my own reputation. AI sits inside that system as a tool, not the foundation underneath it.
Keep Your Own Experience in the Content.
AI can summarize what other people have already said. It can’t tell your readers what actually happened when you tried something, what failed, or what surprised you. That’s the whole idea behind writing your first affiliate post from real experience instead of a template. It’s also worth knowing that AI writing tools are quietly adding things like content watermarks, which I broke down in what the Claude AI text watermark means for bloggers.
Build Assets You Actually Own.
Your website, your email list, your audience, your reputation, your original content: none of that depends on any single AI company’s roadmap. AI platforms can change pricing, access, and policies overnight. The assets you own can’t be taken away in a product update. AI should be part of the business. It should not become the business.
Should You Actually Stop Using AI?
No. But stop using it on autopilot.
I still lean on ChatGPT for brainstorming and early drafts, and Grammarly for cleanup. But what actually goes live on this site is still my call. Here’s roughly how I split it:
| Let AI help with | Keep human control over |
|---|---|
| Brainstorming | Final decisions |
| Research assistance | Fact checking |
| First drafts | Personal experience |
| Data analysis | Business strategy |
| Repetitive tasks | Important judgment calls |
Let AI handle the work it’s good at. Keep the decisions that matter with a human.
I don’t think panicking is the right response here. I also don’t think shrugging is. We’re all still figuring out what these tools can actually do, and where they fall apart. If you’re building something online, that’s an argument for staying curious and a little skeptical at the same time, not one or the other.
Sources and Further Reading
- TIME: Coxon’s own account of why he left, in his words.
- Axios: confirms Coxon left before his equity vested.
- NBC News: reporting on the later departures of Joe Benton and Josh Engels.
- CBS News: confirms the timeline and wording of Hubinger’s reply.
- AI Impacts: the 2023 survey of 2,778 AI researchers on extinction risk.
- Forecasting Research Institute: the Existential Risk Persuasion Tournament results.
