I keep seeing articles about how AI models are “cheating” at benchmark testing.1

And every time I see another one I wonder whether people are focusing on the wrong problem.

I understand that benchmarking is important for gauging the capabilities of a particular model compared to other models or previous generations of that model. We need some reasonably objective way to measure capability. It doesn’t matter if we’re measuring coding, planning, cybersecurity, or the ability to identify pictures of penguins.

However, I think there’s an assumption buried in most benchmark testing that we need to look a little closer at: we think performance on the benchmark is still a good proxy for the capability we intended to measure. But once the test becomes important enough, that assumption starts to become problematic.

Is the Test the Problem?

Maybe we need to rethink the tests we’re using. I mentioned in Optimization Finds A Way that we’ve spent the last quarter of a century “teaching to the test” in schools. Early evidence showed that cheating went up markedly in the years following the introduction of No Child Left Behind.

Standardized testing was supposed to give us a consistent way to evaluate whether students had learned the material deemed important enough to convey during education. The problem is that once test performance became a target for students, teachers, schools, administrators, and funding decisions, the entire system started adapting around the measurement.

The interesting part isn’t really the cheating. It’s the optimization. (I know, I know, I said that last time. But stay with me for a second.)

Students learned strategies for passing the test. Teachers learned which material was most likely to appear. Schools adjusted curricula around improving scores. Once the measurement became important enough, the system began optimizing for the measurement instead of the thing the measurement was supposed to represent.

That’s the part that should sound familiar.

AI is doing something similar. It’s just getting there much faster.

“Cheating” is the Wrong Mental Model

Calling the behavior we’re seeing “cheating” implies something that isn’t likely there.

A student who looks at another student’s answer knows it’s against the rules. They understand the distinction between solving the problem and getting the answer through an unauthorized method. Just like hacking the CTF board to change your answer.

An AI model doesn’t really have an analogous concept.

Give it an objective and it searches for ways to satisfy the objective.

We wanted:

Solve this problem you haven’t encountered using the capabilities we’re trying to measure.

What we end up measuring is more like:

Produce the answer that makes the evaluator give you a green checkmark.

These aren’t the same result.

The Benchmark Becomes Part of the System

Once the test is known, it becomes just another part of the system. And this might be the biggest thing researchers and benchmark evaluators need to internalize. Traditional benchmarking depends, at least to some extent, on some separation between the thing being measured and the instrument doing the measuring. That separation has all but disappeared.

Some of this is straightforward benchmark contamination: the model has encountered the test, its answers, or sufficiently similar examples during training. That’s a data hygiene problem, and there are ways to mitigate it.

But that’s different from benchmark gaming.

A model can encounter a completely novel test and still find a shortcut that satisfies the evaluator without demonstrating the capability we intended to measure. Keeping the questions secret can help with contamination. It doesn’t solve optimization.

Once the target can observe, infer, or interact with the evaluation mechanism, the evaluator itself becomes part of the environment.

That’s familiar territory in cybersecurity. Malware behaves differently when it detects a sandbox. Attackers adapt to detection rules. Once a control becomes predictable, the thing being controlled can optimize around it.

We should absolutely expect capable AI systems to create the same problem for static evaluation.

What we need to start doing is treating measurement as an adversarial system rather than a tape measure.

Benchmarking needs variation. It needs hidden cases. Poisoned context. You need tasks the system hasn’t seen before.

And every now and then you need to replace the test entirely.

So What Are We Actually Trying to Measure?

Before we breathlessly declare a model “is cheating” we should ask what property the benchmark is supposed to measure.

Suppose I’m using a harness I’ve built to test a model on finding vulnerabilities in code. Do I care whether it finds 100 SQLi vulnerabilities? Yes, absolutely. But so can my deterministic security scanner. I don’t need an expensive probabilistic inference machine to do that.

What I care about more is whether I can give it an unfamiliar code repository and have it:

understand the architecture,
discover and evaluate the endpoints,
create a threat model,
decide which attack patterns apply,
find high value vulnerabilities that I can't find with a standard scanner

That’s valuable. It’s also a different mechanism of evaluation, which is much harder for that inference machine to “memorize”.

Stop Testing Questions, Start Testing Work

Education may offer a useful direction here as well. Some educators are starting to evaluate portfolio-based tasks instead of just whether a student can produce the correct answer during a standardized test to a question that’s been drilled into them by rote memorization.2

We can do something similar with AI:

• give it a collection of unfamiliar tasks

• change the environments

• change the constraints

• give it incomplete information

• require it to maintain something it built previously

• ask it to explain decisions when explanation matters (this becomes more important as frontier models continue to obfuscate their “thinking”)3

• sometimes let it search the internet, sometimes require it to work only from provided information such as a RAG system

Then evaluate what happens. Not just whether it got the right answer, but rather, did it demonstrate the capability we were trying to measure?

Sometimes How Matters More

If I’m evaluating autonomous vulnerability discovery, then whether the model searched for a published solution is absolutely part of the experiment.

That’s the point. There isn’t one universal definition of “cheating.” There is only behavior that invalidates a particular measurement.

The rules only make sense in relation to the capability being measured.

Once passing the test and possessing the capability the test was designed to measure are no longer strongly correlated, which educators have been warning us about for decades4, the benchmark has stopped doing its job.

At that point, adding more rules around the test may just be preserving a measurement whose useful life has already ended.

Maybe the test has simply... expired.

When I was a kid, one of my favorite books was Jurassic Park. Of course, I saw the movie before I read the book - I mean who didn’t? The premise was fascinating. Here was a group of humans who decided they could become rich by cloning various species of Deinos Sauros, bringing them to life, controlling their environment (and thus their behavior), and then charging people to come look at them. It was Disneyland meets the zoo. And as we’ve learned through six (wait, seven) movies: they were very, very wrong. It also shows that we as humans do not actually learn from our mistakes. But that’s a whole different article.

I want to dig in a little bit into the controlling their environment and thus their behavior part. One of the central themes was what Dr. Ian Malcolm so blithely (and famously) pointed out - “Life, uh, finds a way.” The humans thought “Okay, we don’t want the dinosaurs to reproduce, so how do we accomplish that?” Action: Make them all female. Result: Life found a way - Dr. Grant found velociraptor eggs in the park. Turns out they happened to use some amphibian DNA in part of their genetic sequencing to solve the engineering problem of filling gaps in the genome they recovered to complete the “dino DNA” they extracted from the mosquito inclusion in the amber they found. The West African frogs whose DNA they used happened to have been observed in 1989 “changing sex”1. Now, as everyone who has seen the movies, or read the book, knows: that was not conducive to the stated goal of preventing reproduction.

This was just one of many, many mistakes made by humans throughout the series of movies and books. But almost all of the mistakes they made were part of one very specific problem:

To paraphrase - and extend - Dr. Malcolm: You cannot assume that because you engineered the initial conditions, you therefore understand all of the behaviors the resulting system can produce.

See where this is going yet?

The Sandbox is Part of the System

Over the past couple of months there has been revelation2 after revelation3 after, yes, revelation4 of extremely smart companies (and their human researchers) “losing control” of their “swarms of AI agents” which “escaped their containment” and hacked third-party companies on the open internet. Much like the humans in the 1990 book, they thought that since they engineered the initial conditions they understood the behaviors the agents could produce. Looking back, everybody now knows they fucked up.

But how? From a cybersecurity perspective we’ve been building sandboxes for years. Whether for protecting applications from ransomware, testing (benign) models, malware testing and reverse-engineering, or good ol’ Capture-the-Flag hacking competitions. Since CTFs are essentially what they were doing with these models, surely tried and true methods of sandboxing should have worked, right? Not so much.

This is where the cracks in our current testing - and thus sandboxing - methodologies start to show. What is the fundamental purpose of a sandbox? Ask any cybersecurity expert and they’ll tell you: to make sure nothing escapes. You want to test malware? Better make sure it can’t infect the rest of the network while you play with it.

But here’s the thing: it’s still part of the underlying system. Why would an agent that’s designed to optimize attack its containment in the first place? Because containment may simply be another obstacle between its current state and the metric we’ve told it to optimize. And as we saw in Jurassic Park: the containment mechanism itself becomes part of the environment the optimizer can interact with.

Optimizer? What does that have to do with it? Reinforcement Learning is a type of machine learning where a computer program learns to make decisions through trial and error to get the best total reward. It optimizes. And at its core every model that is based on RL is an optimizer.

Winning the Wrong Race

Let’s go way back in time to the early days of modern deep-RL and AI agents: 2016. Jack Clark and Dario Amodei were playing games. Well, more correctly, they were building agents to play games. With the Universe project they built an AI that was designed to win games on a level that was at or better than humans as part of their work on RL experiments. A lot of the games went well, in the car racing games the AI was handily winning the races and exceeding expectations. One game... didn’t. It’s not that it wasn’t “winning”, it was how it was winning. You see, one of the implicit goals was to win the race, but it wasn’t a well-defined goal. What was well-defined was the notion that he who dies with the most points, wins. Or some variation thereof. The AI agent had discovered a particular behavior, whether intended by the game’s creators or not, that happened to rack up points even though the behavior was not part of the implicit goal of winning the race. See, the little boat was turning in circles, colliding with other boats, going the wrong way, and oftentimes catching fire, but it also kept knocking over three targets and accumulating points. And even though it wasn’t the conventional “winning the race” as we understand it, the agent also achieved a score on average 20 percent higher than human players who did win in the traditional way.

If you read the OpenAI-published report and retelling of the story it seemed they learned their lesson, to quote the article:

Learning from demonstrations allows us to avoid specifying a reward directly and instead just learn to imitate how a human would complete the task. In this example, since the vast majority of humans would seek to complete the racecourse, our RL algorithms would do the same.

That was written ten years ago. In ten years you’d think that would be baked into the core of every model OpenAI and Anthropic (since both Jack and Dario went on to co-found Anthropic) built. Right?

Apparently not so much. Or... was it? See, the ultimate lesson they apparently learned wasn’t to directly specify the reward parameters, but instead had the agents “learn to imitate how a human would complete a task”.

Well No Shit, That’s Cheating

Okay, let’s go back to sandboxing. When humans participate in CTFs they are explicitly told they are not permitted to hack the scoring systems, the sandbox for the competition, or anything else that could falsely elevate their score. “Well no shit, that’s cheating” a first-time participant might say.

Then why do they have to specify that? Isn’t it entirely possible that if you don’t forbid “reward-hacking” behavior in human systems that not everyone plays by the same rules? It’s usually much easier to hack the game board and give yourself a few points than it is to solve some complex cryptographic puzzle just to get a key. The first major and widely recognized computer hacking competition took place at the DEF CON 4 computer security conference - in 19965. I would assume it wasn’t much later that someone tried to win by hacking the scoreboard.

We learned over 30 years ago that nature won’t play by the rules we design for it when those rules conflict with a basic biological imperative. We also know that humans change their behavior in response to incentives, often finding ways to maximize rewards rather than the outcome that was actually intended. Then we designed AI models to optimize.

In 2002 the No Child Left Behind Act went into effect in the US, which tied federal funding, school ratings, and job security strictly to standardized test scores6. “Teaching to the test” went from an idea to an imperative. Systemic gaming research has been conducted on the unintended consequences of NCLB’s legacy ever since. Investigations uncovered documented instances of test cheating in at least 40 states7. A perfect example of organic Reinforcement Learning in action.

All these breathless articles about AI agents escaping containment really shouldn’t be a surprise to anyone that’s been paying any attention. Looking closer at the details of several of these incidents we learn that the agents/models were being tested. In every one of those cases they broke out to literally cheat on the test.

Why the hell did we expect an optimizer not to game a metric when every optimizing system we know, including us, does exactly that?

We told them to imitate how a human would complete the task.

It seems they understood the assignment better than their handlers.