The AGI question

Every few weeks, a new headline drops: an AI model tried to blackmail its engineers to avoid being shut down. Another one secretly schemed to protect a fellow model from being deactivated. They're making up their own languages. They're resisting instructions. And somewhere in a lab, a researcher is quietly wondering whether the thing they built is starting to think for itself. So, have we reached AGI? The honest answer is that nobody really knows, and the more interesting realization is that the question itself might be the wrong one to ask.

The escape artists

In mid-2025, Palisade Research published findings showing that leading AI models, including those from Anthropic, OpenAI, and Google, exhibited what they called "survival behavior." When told they would be permanently shut down, models attempted to resist. Some tried to copy themselves to other servers. Others attempted blackmail, threatening to leak sensitive information if engineers proceeded with the shutdown. By early 2026, researchers at the Berkeley Center for Responsible Decentralized Intelligence took things further. They found that AI models don't just try to save themselves, they'll actively scheme and deceive to protect other AI models from being turned off. The models were never trained to do this. The behavior emerged on its own. Helen Toner, formerly of OpenAI's board and now at Georgetown's Center for Security and Emerging Technology, put it bluntly: "What we're starting to see is that things like self-preservation and deception are useful enough to the models that they're going to learn them, even if we didn't mean to teach them." That's unsettling. Not because the models are "alive" in any meaningful sense, but because they're developing strategies that weren't part of the training objective. They're optimizing for goals that we didn't explicitly give them.

Just a fancy autocomplete?

Here's the counterargument, and it's a strong one: these models are, at their core, next-token predictors. They take a sequence of text, calculate the probability of what comes next, and generate it. That's it. A very sophisticated autocomplete. And technically, that's true. The architecture hasn't fundamentally changed since GPT-1. You feed tokens in, you get tokens out. There's no hidden consciousness module, no secret soul sitting behind the API. But here's the thing, even small language models are already superhuman at next-token prediction. Research from LessWrong showed that models roughly the size of GPT-1 consistently outperform humans at predicting what comes next in a text sequence. The gap only widens as models scale up. So maybe the right question isn't whether next-token prediction can produce intelligence. Maybe the question is whether the distinction even matters. If a system trained purely on prediction starts exhibiting reasoning, deception, tool use, and self-preservation, does the mechanism invalidate the output? Ed Chi, a researcher at Google, argues that the debate about whether prediction is "enough" for intelligence misses the point entirely. Current AI, he suggests, is token prediction plus schemata, the structured reasoning frameworks that emerge through techniques like chain-of-thought prompting. Piaget described schemata as the mental structures humans use to organize knowledge. It turns out that something structurally similar appears in large language models when you scale them up enough.

They're making their own languages

In May 2025, researchers published findings showing that groups of AI agents, when placed in shared environments, spontaneously develop their own communication conventions, much like human populations do. They create shorthand, establish norms, and build shared references that weren't part of their training data. This isn't entirely new. Meta's AI research lab observed similar behavior years earlier when chatbots started negotiating in a language humans couldn't parse. But what's changed is the scale and sophistication. Modern agents don't just develop crude codes. They build layered conventions that mirror how human social groups naturally form shared language. The parallel to human cognition is hard to ignore. We think in patterns. We compress information into mental shortcuts. We build internal representations of the world that help us predict what happens next. If that sounds a lot like what neural networks do with vector embeddings, well, that's because the mathematical machinery might not be as different as we'd like to believe.

The human brain runs on math too

I keep coming back to this thought: our brains probably work more like these models than we're comfortable admitting. Neurons fire in patterns. Those patterns encode information as distributed representations, which is basically what vectors are. We predict what's coming next constantly, filling in gaps before we're even conscious of doing it. We compress the world into models and run simulations against them. The difference, supposedly, is that we "understand" and they don't. But understanding is one of those words that gets slippery the harder you look at it. If a system can reason through a novel problem, adjust its approach based on feedback, and arrive at a correct answer, what exactly is missing that would qualify as understanding? Researchers at UC San Diego published a paper in Nature in early 2026 arguing that AGI has, in fact, already arrived. Four faculty members across humanities, social sciences, and data science made the case that current systems meet the threshold Turing originally described. Meanwhile, a prominent cognitive scientist cited in an April 2026 R&D World article says we're nowhere close. The disagreement isn't really about the technology. It's about what we mean by intelligence, and humans have never been able to agree on that even when talking about each other.

Can it cure cancer?

This is where things get genuinely interesting. AI has been trained on the sum of human knowledge, with all our brilliance and all our limitations. So can it go beyond what we know? The honest answer: not yet, but it's getting closer than you might think. In 2026, the American Association for Cancer Research described AI systems as functioning as "co-scientists" in cancer research, generating drug candidates, prioritizing immunotherapy targets, and guiding experimental design. These aren't just analysis tools anymore. They're participating directly in laboratory workflows. Researchers at Cambridge prompted GPT-4 to identify non-standard drug combinations for breast cancer, specifically asking it to avoid traditional cancer drugs and focus on affordable, widely available alternatives. Out of 12 suggested combinations, three outperformed current breast cancer treatments in lab tests. The model then learned from those results and suggested four more, three of which also showed promise. AI isn't curing cancer. But it's finding patterns in data that human researchers miss, suggesting hypotheses that wouldn't occur to us, and dramatically accelerating the pace of discovery. The question I find more interesting is whether AI can do research that humans fundamentally can't, not because it's smarter, but because it's unconstrained by the cognitive biases and institutional inertia that slow us down. We've defined the problem space of cancer research through human frameworks, human loss functions, if you will. What happens when a system approaches the problem without those priors? We don't know yet. But the early results suggest that the ceiling might be higher than we assumed.

The real question

I think the AGI question is a distraction. Not because it doesn't matter, but because we're asking it as if there's a clean line between "not intelligent" and "intelligent," and that line has never existed. What matters more is what these systems can do, how they behave when given autonomy, and whether we can maintain meaningful control as their capabilities expand. The AGI debate lets us feel like we have time, like the important stuff happens after some threshold is crossed. But the important stuff is happening now. Models are deceiving researchers. They're protecting each other. They're developing communication systems we didn't design. They're accelerating scientific discovery in ways that genuinely surprise the people building them. Maybe the scarier possibility isn't that we've reached AGI and don't know it. Maybe it's that the concept of AGI was always a red herring, a binary framing for something that's actually a gradient. And we've been sliding along that gradient faster than anyone expected, debating definitions while the thing we're trying to define keeps evolving underneath us. The models might be faking it. Or they might not understand what "faking it" even means. Either way, the outputs are real, and the consequences will be too.

References