AI safety is a luxury good

In early 2026, Anthropic became the first American company to be designated a "supply chain risk" by the Pentagon. Its offence was refusing to remove guardrails that prevented the U.S. military from using its AI for autonomous weapons and mass domestic surveillance. Within hours of the ban, OpenAI announced its own Department of Defense contract. The message was clear: if you won't play ball, someone else will. This is a story about incentives, not heroism. Anthropic's stand is admirable, but the more interesting question is structural. What happens when doing the right thing is only affordable if you're already winning?

The case study

The timeline is worth laying out. Anthropic had been providing AI tools to the U.S. government since 2024 and was the first advanced AI company deployed in classified government work. Its contract included explicit restrictions: no autonomous weapons, no mass surveillance of Americans. These were Anthropic's conditions, baked in from the start. In early 2026, the Pentagon, now rebranding itself as the Department of War, demanded those restrictions be removed. Anthropic refused. Defense Secretary Pete Hegseth gave them a deadline to comply. When they didn't, the government cancelled over $200 million in contracts, designated Anthropic a supply chain risk, and President Trump ordered all federal agencies to stop using the company's tools. Anthropic sued, calling the actions "unprecedented and unlawful." A small group of OpenAI and Google employees filed an amicus brief in support. But the commercial reality had already shifted. OpenAI had filled the vacuum.

Safety as a competitive disadvantage

The Anthropic situation illustrates something uncomfortable: in the current market, AI safety is a cost centre with negative commercial returns. Consider what Anthropic lost by saying no. Over $200 million in existing and expected Defense Department contracts. A fourfold increase in public sector revenue that was trending toward multiple billions within five years. Relationships with defense contractors who paused deal talks after the blacklisting. The company's chief commercial officer warned that its projected public sector revenue of over half a billion dollars in 2026 could "shrink substantially or disappear altogether." Now consider what OpenAI gained by saying yes. A new Pentagon contract. A strengthened relationship with the administration. The implicit message to future government buyers that OpenAI is the cooperative option. This is the core dynamic. If safety means slower delivery, restricted use cases, and the willingness to walk away from revenue, then market selection will favour the companies that skip those steps. It's not that OpenAI is reckless. Its Pentagon contract includes some safeguards around autonomous weapons and surveillance. But the bar is lower, the restrictions are softer, and the willingness to negotiate is higher.

Only the rich can afford principles

Anthropic is not a scrappy startup. It has raised billions in venture capital and counts Google and Amazon among its investors. It can absorb a $200 million hit and keep operating. It has the legal resources to sue the federal government. It has the brand equity to turn a principled stand into a recruiting advantage. Most AI companies cannot do any of this. An early-stage AI company with 50 employees and a single product cannot afford to turn down a government contract on ethical grounds. It cannot afford a dedicated safety team, red-teaming infrastructure, or the legal firepower to challenge a federal agency. For startups, the calculus is simple: take the contract or die. This creates a two-tier system. Large, well-funded companies can sometimes afford to draw ethical lines, though even they face enormous pressure to cave. Smaller companies never had the option in the first place. The result is that safety becomes a luxury good, something that only the market leaders can offer, and only when the cost of offering it doesn't threaten their position. Anthropic's own "race to the top" strategy was built on this assumption. The idea was that if Anthropic adopted strong safety practices, competitive pressure and reputational incentives would push other companies to follow. For a while, it seemed to work. DeepMind and OpenAI adopted aspects of Anthropic's Responsible Scaling Policy framework. But as investment dollars ballooned and federal regulation receded, Anthropic conceded that the approach had fallen short. "The policy environment has shifted toward prioritizing AI competitiveness and economic growth," the company wrote, "while safety-oriented discussions have yet to gain meaningful traction at the federal level."

The race to the bottom

The Pentagon dispute accelerated a dynamic that was already underway. Instead of a race to the top on safety, the AI industry is converging on a race to the bottom on restrictions. Steven Levy captured this shift well in WIRED: "Instead of a race to the top, the AI rivalry seems more like a bareknuckle version of King of the Mountain." When Anthropic was banished, OpenAI rushed in. CEO Sam Altman insisted he was trying to relieve pressure on Anthropic. Anthropic CEO Dario Amodei was unconvinced. "Sam is trying to undermine our position while appearing to support it," he wrote in an internal memo. "He is trying to make it more possible for the admin to punish us by undercutting our public support." The competitive logic is relentless. If one company refuses a contract, another fills the gap. If one company maintains restrictions, a competitor offers fewer. If one company invests heavily in safety, it moves slower and charges more, while a leaner rival captures the market. This is not unique to AI. It is the standard outcome in any market where safety is costly and enforcement is weak. Environmental regulations exist because companies won't voluntarily limit pollution when their competitors don't. Financial regulations exist because banks won't voluntarily limit risk when the upside is privatised and the downside is socialised. The same logic applies here.

The state has the bigger stick

What makes the Anthropic case especially instructive is the asymmetry of power. Governments don't just compete for AI contracts like any other buyer. They have tools that no private actor can match. The Pentagon threatened to invoke the Defense Production Act to compel access to Anthropic's technology. This is a law designed for wartime mobilisation, and the fact that it was floated in a contract dispute over AI guardrails reveals how seriously the government treats its leverage. Procurement bans, supply chain risk designations, executive orders, these are instruments that can effectively destroy a company's government business overnight. Corporate ethics can constrain state power only as long as the state allows it. If the government decides that AI safety restrictions are incompatible with national security, no amount of corporate principled-ness will hold. The company either complies or is replaced. As Owen Daniels of Georgetown's Center for Security and Emerging Technology noted, peers including Google and xAI have shown greater willingness to align with Defense Department policies. The Pentagon has alternatives, and it knows it.

What would make safety a competitive advantage?

If the current incentive structure punishes safety, the question is whether that structure can be changed. There are a few plausible mechanisms. Regulation. The most direct path. If all AI companies are required to meet minimum safety standards, no single company is punished for compliance. The European Union's AI Act is a step in this direction, but it doesn't cover military applications by its member states, let alone U.S. defense use. Federal AI legislation in the U.S. remains distant. Buyer preferences. If large enterprise customers start demanding safety certifications, audited red-teaming, and usage restrictions, the commercial incentive could flip. There are early signs of this in sectors like healthcare and finance, where liability concerns make buyers cautious about unaudited AI. But in defense procurement, the buyer is the one demanding fewer restrictions. Insurance and liability. If AI failures create significant financial liability, companies that invest in safety will have lower risk profiles and potentially lower insurance costs. This mechanism is slow and depends on legal precedents that don't yet exist for AI-specific harms. Talent markets. AI researchers care about safety. The amicus brief filed by OpenAI and Google employees in support of Anthropic signals that talent loyalty is real. Companies that abandon safety commitments may face brain drain. This is a meaningful pressure, but it's unlikely to outweigh hundreds of millions in lost revenue. None of these mechanisms are sufficient on their own. Without regulation, safety remains a voluntary cost that the market will tend to eliminate.

The structural trap

Dario Amodei described this plainly in an essay earlier this year: "This is the trap. AI is so powerful, such a glittering prize, that it is very difficult for human civilization to impose any restraints on it at all." He's right, and the trap extends beyond any single company's decisions. The Anthropic-Pentagon dispute is not really about whether Anthropic is brave or foolish. It's about whether the structure of the AI market, and the relationship between AI companies and governments, permits safety to exist as anything other than a marketing claim. Right now, the answer is uncomfortable. Safety is affordable when you're dominant, optional when you're competing, and impossible when you're surviving. The companies with the strongest safety commitments are the ones that can absorb the cost, and even they are finding the cost increasingly hard to bear. The question is not whether AI safety is important. Almost everyone agrees it is. The question is who pays for it, and what happens when the bill comes due.

References