Write unit tests with AI that catch real bugs, not tautologies

You can write unit tests with AI quickly, but by default many of them restate the implementation and pass whether the code is right or wrong. This guide shows how to get tests that catch bugs: test-first workflows with a coding agent, prompts that pin down behavior, a review checklist, and mutation testing to measure what your suite actually detects.

Can you write unit tests with AI that actually catch bugs?

Yes, if the tests encode what the code should do rather than what it currently does. When you write unit tests with AI by pointing a model at a function and asking for tests, it reads the implementation and writes assertions that match it. If the implementation has a bug, the test faithfully expects the bug.

The fix is to give the model a source of truth that is independent of the code: a spec, a ticket with acceptance criteria, real examples of input and expected output, or a bug report. With that, AI is good at the tedious parts of testing: enumerating edge cases, writing table-driven cases, building fixtures and keeping test names readable.

A coding agent adds one more advantage over a chat window: it can run the tests, read the failures and iterate. Anthropic's Claude Code best practices make the same point: without a check it can run, "looks done" is the only signal available.

What makes an AI-generated test useless?

Most weak AI tests fall into a few recognizable patterns. Learn to spot them in review and you will reject the worst output in seconds.

Here is a tautology next to a useful test for the same function, a price calculator that applies a discount rate and rounds to cents:

def test_apply_discount():    price = 100    rate = 0.2    # Recomputes the implementation. If the code is wrong, so is this.    assert apply_discount(price, rate) == round(price - price * rate, 2)

import pytest@pytest.mark.parametrize("price, rate, expected", [    (100.00, 0.20, 80.00),    (19.99, 0.10, 17.99),   # rounds to cents    (50.00, 0.00, 50.00),   # no discount    (50.00, 1.00, 0.00),    # full discount])def test_apply_discount_examples(price, rate, expected):    assert apply_discount(price, rate) == expected@pytest.mark.parametrize("rate", [-0.1, 1.5])def test_apply_discount_rejects_invalid_rates(rate):    with pytest.raises(ValueError):        apply_discount(100.00, rate)

The second version uses literal expected values that a person can check by hand, covers boundaries and rejects invalid input. If someone changes the rounding or forgets the validation, it fails.

How do you do test-driven development with a coding agent?

Test-driven development works well with agents because it splits the job into two steps you can check separately: first agree on the tests, then let the agent make them pass. The tests become the contract, and you review a small test file instead of a large implementation.

1. Describe the behavior. Write a short spec or paste the ticket. Include concrete examples and the error cases you care about.
2. Ask for tests only. Tell the agent not to write implementation code yet.
3. Review and confirm they fail. Read every assertion. Run the suite and check that each new test fails for the right reason, such as a missing function, not a typo.
4. Commit the tests. A commit gives you a clear diff later if the agent touches them.
5. Implement with the tests locked. Tell the agent the test files are read-only and to iterate until the suite passes.
6. Review the implementation and the evidence. Ask for the final test output, then read the diff.

The prompt for step 2:

We're adding apply_discount(price, rate) to src/pricing.py.Spec: returns the price after discount, rounded to 2 decimals.rate must be between 0 and 1 inclusive; anything else raises ValueError.Examples: (100, 0.2) -> 80.00, (19.99, 0.1) -> 17.99, (50, 1) -> 0.00.Write pytest tests in tests/test_pricing.py only. Do not write theimplementation. Use literal expected values, never recompute them.Cover the examples, both boundaries, and invalid rates. Then run thetests and confirm they fail because apply_discount doesn't exist.

In Claude Code you can enforce step 5 with a temporary deny rule in .claude/settings.local.json. The permissions docs use Edit(path/**) rules for file edits:

{  "permissions": {    "deny": ["Edit(tests/**)"]  }}

The best practices guide also suggests a writer and tester split across two sessions: one writes the tests, a fresh session writes code to pass them. A fresh context has no attachment to an implementation it never saw.

How do you add tests to code that has none?

For existing code without tests, start with characterization tests: tests that record what the code does today, so you notice when a change alters behavior. This is the one case where testing against current behavior is the point, as long as the test names say so.

Ask the agent to find the riskiest functions first, then pin down their behavior:

Read src/billing/ and list the 5 functions with the most branching logicand the most callers. For each, write characterization tests that recordcurrent behavior for typical inputs, boundaries and error paths.Name them test_current_behavior_<what>. Avoid mocks except for thepayment gateway client. Where current behavior looks like a bug, keep thetest but add a comment "SUSPECTED BUG:" with your reasoning.

The SUSPECTED BUG comments turn the exercise into a free code review. Read them; some will be real. Our guide to refactoring legacy code with AI builds on this step.

Which prompts produce meaningful unit tests?

Good test prompts name the source of truth, the cases to cover and the style rules. Compare:

Rules worth adding to almost every test prompt, or once to CLAUDE.md:

• Use literal expected values; never recompute the expectation with the code's own logic.
• Each test checks one behavior and its name says which.
• Mock only what is slow, non-deterministic or external (network, clock, payment provider).
• Cover boundaries, empty inputs, invalid inputs and the error message or type.
• Add a regression test for every bug you fix, named after the bug.

A prompt that hardens an existing suite:

For src/pricing.py, imagine 5 realistic bugs a developer could introduce(off-by-one, wrong rounding, missing validation, swapped arguments,wrong default). For each, check whether tests/test_pricing.py would fail.Write a new test for every bug the current suite would miss.

This is mutation testing done by hand, and it leads to the next section.

How do you know AI-written tests are good?

Coverage tells you which lines ran during the tests; mutation testing tells you whether the tests would notice if those lines were wrong. Since AI makes high coverage cheap, mutation score is the better signal for important code.

A mutation testing tool makes small changes to your code, such as flipping > to >= or replacing a return value, and reruns the tests. If a test fails, the mutant is "killed". Surviving mutants show behavior your tests don't check. Researchers have started combining the two, using surviving mutants to guide a model toward better tests, as in this mutation-guided test generation paper.

The StrykerJS setup generates a stryker.config.mjs you can trim to the folders you care about. mutmut needs fork support, so on Windows run it inside WSL.

Mutation runs are slow on large codebases, so point them at critical modules (pricing, permissions, data transformations) rather than the whole repository. Then feed the survivors back to the agent:

Here is the list of surviving mutants from mutmut for src/pricing.py: [paste].For each one, write a test that kills it, or explain why the mutant isequivalent to the original code and can't be killed.

How should you review AI-written tests?

Review tests with the same care as production code, because a wrong test is worse than no test: it gives false confidence. A quick checklist:

• Every expected value is a literal or comes from a fixture, not from the code under test.
• Each test would fail if the behavior it names broke. Pick one and break the code to check.
• Mocks stand in only for external or slow dependencies.
• Edge cases and error paths are present, not three variations of the happy path.
• Test names describe behavior (rejects_rate_above_one), not methods (test_apply_discount_2).
• No test was weakened, skipped or deleted to get a green run.
• Snapshots are small and were read, not accepted blindly.

The last two points matter most with agents. When a test fails, an agent may "fix" it by loosening the assertion. Make it a rule in CLAUDE.md that changing an existing assertion requires an explanation in the final summary. For broader review habits, see our guide to AI code review.

How do you make the agent run tests every time?

Write the exact test commands into CLAUDE.md and require evidence. An agent that knows pnpm test path/to/file runs one file will run focused tests while it works and the full suite at the end.

## Testing- Run one file: `pnpm vitest run path/to/file.test.ts`- Before saying a task is done: `pnpm test && pnpm typecheck`, and paste the summary- Never edit an existing assertion without explaining why in your summary- New behavior needs a test; every bug fix needs a regression test

Instructions in CLAUDE.md are advisory. For a guarantee, a Claude Code hook can run your test command when Claude tries to finish and block the stop until it passes, which the best practices guide describes as the deterministic option. In CI, the same suite runs on every pull request, so nothing merges on the agent's word alone.

Test-backed tasks are also the ones you can safely leave running: the agent works until the suite is green, and you review the evidence later. If those runs take longer than your laptop stays awake, a hosted box such as Orkio keeps Claude Code working and opens the pull request when it's done.

Start with one module this week

Pick one module where a bug would cost real money or trust, and work through this:

• Write a five-line spec with three concrete examples and two error cases.
• Have the agent write tests only, confirm they fail, and commit them.
• Let it implement or refactor with test files locked.
• Run a mutation tool on that module and have the agent kill the survivors.
• Add the testing rules to CLAUDE.md so the next task starts from them.

Once unit tests are in shape, the same ideas apply one level up: our guide to AI test automation with Playwright covers end-to-end tests, and debugging with AI shows how a failing test becomes the fastest way to fix a bug.

Frequently asked questions