Testing AI-Written Code: Contracts & Playwright

Sharing is caring!

Ask an AI agent to add tests and you will usually get a green suite in under a minute. That is the problem. The tests were written after the code, by the same system that wrote the code, asserting the behaviour that the code happens to have — including the bug. Green means the two artifacts agree with each other. It does not mean either one is right.

Testing AI-written code is not about writing more tests. It is about writing tests the model cannot fake.

The short answer

1. Contract before code — define the behaviour in a form that can fail before the implementation exists.
2. A test that never failed is not a test — see it red, then green.
3. Unit tests for logic, browser tests for behaviour — Playwright for what a user does, not for what a function returns.
4. Control the seams — network, time, randomness, auth, storage.
5. Measure what is verified, not what is executed — mutation score and changed-lines coverage beat a global percentage.

Rule 1: Contract before code

The cheapest way to stop a model from inventing behaviour is to write the acceptance criteria first. Not prose in a ticket — executable expectations.

For a data boundary, that means a runtime schema. TypeScript disappears at runtime; a Zod schema does not:

import { z } from "zod";

export const OrderSchema = z.object({
  id: z.string().uuid(),
  total: z.number().int().nonnegative(),
  currency: z.enum(["USD", "EUR", "GBP"]),
  status: z.enum(["pending", "paid", "refunded"]),
});

export type Order = z.infer<typeof OrderSchema>;

The test asserts the contract, not the implementation:

import { describe, expect, it } from "vitest";
import { OrderSchema } from "./order";

describe("order contract", () => {
  it("rejects a negative total", () => {
    const result = OrderSchema.safeParse({
      id: "6f1c2f34-3a6e-4f5b-9d0e-6a7b8c9d0e1f",
      total: -1,
      currency: "USD",
      status: "paid",
    });
    expect(result.success).toBe(false);
  });
});

Now the agent can implement whatever it wants, as long as the boundary holds. That is the point: constrain the contract, free the implementation.

Rule 2: A test that never failed is not a test

Before accepting any new test, break the code on purpose. Change a return value. Remove a guard. Flip a condition. The test must fail for the right reason — not because of a missing mock, not because of a timeout, but because the behaviour changed.

This one habit catches most of the “tests” AI writes. Three common failure modes:

• The mirror test: it asserts the same expression the function returns, so it can never disagree.
• The mock test: it verifies that the mocks were called, not that the feature works.
• The snapshot test: it locks in the current output, bugs included, and nobody reads the diff again.

For a stronger signal, run mutation testing on the changed files. Stryker mutates operators and conditions, then checks whether the suite notices. A file with 95% line coverage and a 40% mutation score is a file with decorative tests.

Rule 3: Unit tests for logic, Playwright for behaviour

Agents are good at unit tests for pure functions. They are shaky at integration behaviour: focus order, loading states, navigation, and the browser doing something unexpected. That is exactly where Playwright earns its place.

import { expect, test } from "@playwright/test";

test("a customer can apply a valid coupon", async ({ page }) => {
  await page.goto("/checkout");

  await page.getByLabel("Coupon code").fill("SAVE10");
  await page.getByRole("button", { name: "Apply" }).click();

  await expect(page.getByRole("status")).toHaveText("10% applied");
  await expect(page.getByTestId("total")).toHaveText("$81.00");
});

Three details make this test worth keeping:

• Role and label selectors, not CSS classes. The test fails when accessibility breaks, which is a feature.
• Web-first assertions (await expect(...)) instead of manual sleeps. Playwright retries until the assertion passes or the timeout expires, which removes most flakiness.
• One behaviour per test. “Apply a coupon” is a behaviour. “Checkout works” is a wish.

When a test fails in CI, open the trace, not the video. The trace viewer shows the DOM snapshot, network log, and console at every step. It is the fastest way to tell whether the product broke or the test did.

If you are using an agent to write browser tests, give it a real browser loop. Playwright’s MCP server lets an agent inspect the page through the accessibility tree and verify selectors against the live DOM, which is far more reliable than guessing from source. The rule stays the same: the agent may drive the browser, but you decide what counts as correct.

Rule 4: Control the seams

Most flaky tests are not random. They are tests that touch the real world without a boundary. There are five seams to control:

Playwright gives you all five without extra libraries: page.route() for network, page.clock for time, storageState for auth, and a fresh browser context per test by default. Use them before you add a retry.

A concrete example: a checkout test fails once a day because the confirmation toast disappears before the assertion runs. The fix is not a longer timeout. Assert on the order status or the order row that persists, not on an animation. The same rule applies to loading spinners — assert the final state, not the intermediate one.

Rule 5: Kill coverage theater

Coverage measures execution, not verification. A file can be 100% covered by tests that assert nothing. Three numbers are more honest:

• Changed-lines coverage: did the diff bring untested branches? Enforce it on the PR, not the whole repo.
• Mutation score: when the logic changes, does the suite notice? Run it on critical modules, not everywhere.
• Flake rate: how often does the same commit pass and fail? A flaky suite trains everyone to ignore red.

Set a budget and keep it boring. For most product teams: changed-lines coverage above 80%, mutation score above 60% on pricing/auth/permissions, flake rate under 1%. Everything else is a conversation, not a gate.

Where AI-written tests break first

After reviewing a few hundred of these suites, the failures cluster. Watch for three patterns:

• The assertion that restates the implementation. The test imports the same helper the function uses and asserts the same expression. It can never disagree, so it can never protect you.
• The mock that replaces the thing under test. The database is mocked, the repository is mocked, and the service is mocked — what remains is a test of the mocks. Mock the boundary you don’t own, not the logic you are verifying.
• The sleep instead of a wait. await new Promise(r => setTimeout(r, 500)) is a race condition with a delay attached. It passes on a fast laptop and fails in CI. Use web-first assertions or fake timers.

The pyramid still applies

AI changes the economics of tests, not the shape of a healthy suite. You still want many fast unit tests, a meaningful layer of integration tests at the seams, and a small number of browser tests for the flows that must not break.

What changes is who writes what. Let the agent produce the volume — table-driven cases, boundary values, error branches — and keep the judgement calls for yourself: which flows are critical, what “correct” means for the business, and which tests are worth the maintenance. An agent can generate two hundred browser tests in an afternoon; deleting them will take a week.

Debugging a test the agent wrote

When a new test fails, resist the urge to ask the model to “fix the test”. That is how a real bug gets papered over. Check, in order: is the expectation wrong, is the fixture stale, is the selector ambiguous, and is the code actually broken? Only the first two are test problems.

If the failure disappears when you delete an assertion, the assertion was the only thing checking the behaviour. That is a finding, not an inconvenience.

A workflow that works with agents

1. Write the spec as examples. One sentence per behaviour, with input and expected output.
2. Write the failing test first — or have the agent write it, then review it before any implementation exists.
3. Run it. Watch it fail. If it passes, the test is wrong or the feature already exists.
4. Let the agent implement against the test, with the AGENTS.md commands and boundaries in context.
5. Break the code once after it goes green, to confirm the test still has teeth.
6. Add one browser test for the user-visible path, then stop. Not every branch needs an end-to-end test.

This is slower for the first hour and much faster for the next month. The agent gets a target it cannot redefine, and you get a review that is about behaviour instead of taste.

FAQ

Should AI write the tests at all?

Yes, with supervision. It is good at boilerplate, edge-case enumeration, and converting a spec into a table-driven test. It is bad at deciding what matters. You own the spec and the review.

What coverage should we require?

Changed-lines coverage is more useful than a global number. 80% on the diff is a reasonable default; critical modules deserve mutation testing instead of a higher percentage.

Playwright or Cypress in 2026?

Both are viable. Playwright’s cross-browser support, auto-waiting, trace viewer, and agent tooling make it my default for new projects. The important decision is what you test in the browser, not which runner you pick.

How many end-to-end tests do we need?

Enough to cover the revenue paths and the flows that break silently: sign-in, checkout, permissions, and the one workflow your business cannot lose. Usually a dozen, not a hundred.

How do we stop tests from becoming flaky as the suite grows?

Treat flakiness as a bug with an owner. Quarantine the test, fix the seam, and delete the retry. Retries hide the problem and teach the team to ignore red builds.

Should AI generate the fixtures too?

Small, readable fixtures are fine; large generated ones are a liability. If nobody can explain why a fixture has fourteen fields, the test will drift away from reality and nobody will notice. Keep fixtures minimal and name them after the scenario.

The bottom line: the test is the specification, not the souvenir. Write the expectation first, watch it fail, control the seams, and measure whether the suite notices when the code changes. That is the part AI cannot fake for you.

Previously: Reviewing AI-Written Code and Context Engineering for AI Coding. Next: the security checklist.

Images: Lachlan Hardy (CC BY), *n3wjack (CC BY-SA), ajmexico (CC BY) via Openverse.