The irony of testing AI

Every time we send a prompt to an LLM, we're rolling the dice. The same input, the same model, the same temperature setting, and yet the output can be different each time. That's the nature of non-deterministic systems. And somehow, the way we've decided to test these systems is by expecting them to behave deterministically. I find this genuinely funny. We've built the most unpredictable text generators in history, and our primary quality assurance strategy boils down to: "Did it say the right thing this time?"

The evaluation paradox

Traditional software testing works because software is deterministic. You give a function an input, you get a predictable output, and you write a test that checks for it. If the test passes today, it passes tomorrow. Simple. LLMs broke that contract. You can ask GPT the same question twice and get two different answers, both perfectly reasonable, both technically correct, both phrased completely differently. So how do you write a test for that? You can't just assert that the output equals some expected string. You need something fuzzier, something that understands whether the response is "good enough." And here's where it gets truly absurd.

Using a coin flip to judge a coin flip

The industry's most popular solution to the evaluation problem is called "LLM-as-a-judge." The idea is straightforward: you use one LLM to evaluate the output of another LLM. You prompt a model, typically something powerful like GPT-4, to act as a grader. It reads the output, considers some criteria, and assigns a score or a verdict. Think about that for a second. We're using a non-deterministic system to judge the output of another non-deterministic system, and then we treat the result as ground truth. It's like asking one lottery player to verify whether another lottery player's ticket is a winner, without checking the actual numbers. Researchers have found that LLMs are actually worse at evaluating answers than they are at generating them. A 2024 paper from Seoul National University called this "The Generative AI Paradox," showing that models exhibit significantly lower performance on evaluation tasks compared to generation tasks. The models can produce great answers but can't reliably tell you whether an answer is great.

The biases run deep

It gets worse. LLM judges don't just produce inconsistent scores, they produce systematically biased ones. Research from Weights & Biases and others has documented several recurring problems:

These aren't edge cases. They're consistent patterns that distort evaluation results at scale. A study on expert knowledge tasks found that subject matter experts only agreed with LLM judges about 64 to 68 percent of the time. That means roughly a third of the time, the LLM judge is just wrong, at least by human expert standards.

The feedback loop problem

Here's the part that keeps me up at night. If we use LLM-as-a-judge to evaluate our models, and then use those evaluation scores to fine-tune our models, we're creating a closed feedback loop. The models learn to produce outputs that score well with the judge, not outputs that are actually good. Over time, the models optimize for pleasing another model rather than being useful to humans. This is essentially Goodhart's Law applied to AI: once the metric becomes the target, it stops being a good metric. When your judge is an LLM, the models learn the judge's preferences and biases. They learn to be verbose because the judge likes verbosity. They learn to hedge and qualify because the judge rewards hedging. The system converges on outputs that game the evaluation, not outputs that serve the user.

So what actually works?

I'm not saying we should throw out automated evaluation entirely. The scale problem is real. You can't have humans review every single output from a production system processing millions of requests. But we need to be honest about what LLM-as-a-judge actually gives us: a rough, biased, non-deterministic approximation of quality. Some approaches that seem more promising:

Accepting the uncertainty

I think the deeper issue is philosophical. We want AI systems to be testable the way software is testable. We want clean pass/fail results. We want CI/CD pipelines with green checkmarks. But LLMs aren't software in the traditional sense. They're closer to creative collaborators, and evaluating a creative collaborator's work has always been subjective, contextual, and messy. The irony isn't just that we're testing non-deterministic systems with non-deterministic judges. It's that we're so uncomfortable with uncertainty that we'd rather have a confident but unreliable automated score than sit with the ambiguity. We'd rather pretend the problem is solved than do the harder work of defining what "good" actually means for each specific use case. Maybe the first step is admitting that LLM evaluation is genuinely hard, that there's no clean solution, and that anyone selling you a fully automated eval pipeline is probably oversimplifying the problem. The best evaluation strategy might just be accepting that you're gambling, and learning to manage the odds instead of pretending they don't exist.

References