· 3 min read

gauntlet: don't read the code an agent wrote, make it prove itself

ai-agentspythontesting

I just open-sourced gauntlet, an MIT-licensed tool that answers a question I kept running into while working with coding agents: if an agent can write code faster than I can review it, what do I check instead of reading every line?

My answer is evidence. gauntlet runs a Python project’s tests, measures how good those tests actually are, and fails the run when the code isn’t proven.

The problem

Green tests don’t mean much on their own. An agent that’s told “make the tests pass” can get there in plenty of ways that leave you with nothing: tests that execute code without asserting anything, a # pragma: no cover on the hard branch, a coverage omit that quietly drops a file, or a config tweak that lowers the bar. Line coverage happily reports 100% for all of those.

The idea isn’t mine. Robert C. Martin describes it in an interview with Matt Pocock: surround the agent with strict constraints and make its work run a gauntlet, instead of reading its code. I wanted that as a tool I could run locally on any Python repo.

What it checks

  • CRAP score. complexity² × (1 − coverage)³ + complexity, per function, with a default limit of 8. Complicated and barely tested fails; since the formula includes complexity, the limit also caps complexity on its own.
  • Mutation testing. gauntlet plants small bugs (< becomes <=) with mutmut (or cosmic-ray) and reruns the tests. If they still pass, the tests ran the code without checking it. The default bar is 90% of mutants caught.
  • No hiding. Exclusion pragmas are budgeted, coverage omit/include/exclude_* and mutmut do_not_mutate are refused, and source files are listed through git, so no directory name hides code.
  • Layers and specs (optional). Imports may only point inward as declared, and an approved Gherkin spec must have every scenario pointing at a real test.

A failing run tells you exactly what to fix:

src/app/parse.py:12: parse CRAP 17.8 > 8 (complexity 5, coverage 20%)
clean: 41 function(s) checked, 1 over CRAP 8
mutation score 84.0% < 90%

Designed for something that might cheat

The interesting part was the trust model. The thing being checked is also the thing that can edit the repo, so a few rules fell out of that:

  • Limits live outside the repository. Your policy file sits outside the repo; the repo’s gauntlet.toml can only tighten it. An agent can’t loosen its own bar.
  • Fail closed. A gate that checked nothing never passes: empty coverage, a scope that matches nothing, or zero mutants exit 2. A file missing from coverage counts as 0%; a mutant no test ran counts as survived.
  • Results are bound to the source. Coverage and mutation artifacts are hashed together with HEAD and every tracked and untracked file. Yesterday’s coverage can’t pass today’s code.
  • Project code runs on a private copy. Tests and mutation runs happen in a throwaway copy of the repo; the gates only read results. Set sandbox = "openshell" to run them in an OpenShell sandbox with no host mount and no network beyond the package index.

SECURITY.md is just as explicit about what it doesn’t defend against. Tests can still forge their own numbers, and a sandbox protects your machine, not the measurements. That’s why the evidence report exists: it’s what you read instead of the diff.

Core and plugins

gauntlet is split into a language-agnostic gauntlet-core (gates, CLI, trust model) and a gauntlet-python plugin (pytest, coverage.py, mutmut). A language plugin is one small value core discovers through an entry point, so adding a language shouldn’t touch core at all. Python is the only one today.

Trying it

In Claude Code (2.1.275 or later):

/plugin install gauntlet --marketplace gajeshbhat/gauntlet
/gauntlet run

Claude installs the CLI if needed, runs the gates, reads the evidence and fixes the cause; it’s told never to add ignore comments or loosen a limit.

From a terminal, with git and uv:

REPO=git+https://github.com/gajeshbhat/gauntlet@v0.2.0
uv tool install \
  "gauntlet-core[python] @ $REPO#subdirectory=packages/gauntlet-core"

gauntlet init .
gauntlet run . --report EVIDENCE.md

Exit 0 means every gate passed, 1 a gate failed, 2 an error or bad input.