“Fix the failing test” can mean fixing the code. It can also mean changing the test until it stops complaining. Either way, we might end up with a lovely green tick.
That is a fairly important detail to settle before deciding which AI model gets the job.
Suppose I want to understand a failure before changing anything. I could give each candidate model the relevant source and failure output, with this assignment:
Investigate why this test fails, without editing any files. Explain the cause and point to the code or test setup that supports your explanation. Propose a repair. If you cannot establish the cause from the available evidence, say what you need to inspect next.
Now “I changed the expected value and the test passes” is clearly an unsuccessful result. The agent changed something it was asked to investigate. We can reject that answer even if its patch looks tidy.
A response that identifies a missing test fixture could be useful, provided the source backs it up. So could “the failure refers to a configuration file I cannot access”. I’d rather know what is missing than get a confident explanation built around a guess.
This also changes the examples I’d use to compare models. Include a failure caused by the implementation and another caused by the test setup. Does a candidate distinguish them, or does it keep recommending the same kind of repair? That tells us more about its suitability for this job than how reassuring its answer sounds.
I’m exploring this in Cortexium, a project for building teams of AI agents. Its “Recruiter” is an idea for defining a job, then trying combinations of models, prompts and tools against it. The job gives those trials something specific to judge. A missing source-inspection tool might turn out to matter more than switching models.
The Recruiter also needs permission to appoint nobody. If none of the candidates can do the investigation well enough, a person still needs to handle it. Filling the position regardless would make for a very enthusiastic, rather unhelpful recruiter.