AI Wrote My Unit Tests. They All Passed. The Bug Still Shipped.

By Kamlesh Bhor - DotNet Wisdom

When I opened the PR, every test was green. Six of them. The names looked senior. Moq verified the calculator was called. I was one click from merge.

Then I typed one External customer + EMP-SAVE into a tiny console and watched the total print ₹93.50 instead of ₹110.00.

Green does not mean correct. It means the asserts matched whatever the code (or mock) did today. If asserts were copied from buggy output, the suite protects the bug.

TL;DR: AI-style unit tests against a buggy price calculator all passed while an External customer still got an employee discount. The fix was characterization tests for the real rules, then mutation testing.

Honesty first

This is a condensed teaching demo on .NET 10. BuggyPriceCalculator and the weak AiStyle suite are intentional fixtures. Same failure modes: happy paths only, mock call-counts, copied totals, no External+EMP-, and no EU midpoint edge.

The rules that matter

Bugs: EMP- applies to everyone; EU tax uses AwayFromZero. Both calculators ship in src/OrdersPricing/.

Step 1 - AiStyle tests: all green

export PATH="$HOME/.dotnet:$PATH"
dotnet test tests/OrdersPricing.AiStyle.Tests
# Passed: 6

Tip: Install the .NET 10 SDK. If dotnet is not on your PATH, add the exported PATH above.

What those AI-style tests got wrong

1. Happy path only

// Internal + EMP- only
var result = _sut.Calculate(new PriceRequest(
    Subtotal: 100m, DiscountCode: "EMP-SAVE",
    CustomerType: CustomerType.Internal, Region: "IN"));
Assert.Equal(15.00m, result.DiscountAmount);
Assert.Equal(93.50m, result.Total);

Never asks the scary question: what about External?

2. Assert the mock was called

mock.Verify(c => c.Calculate(It.IsAny<PriceRequest>()), Times.Once);
Assert.Equal(93.50m, result.Total); // total came from the stub

That proves wiring, not the Internal-only business rule.

3. Expected values copied from buggy output

var result = _sut.Calculate(new PriceRequest(
    100m, "EMP-SAVE", CustomerType.External, "IN"));
Assert.Equal(15.00m, result.DiscountAmount); // wrong rule
Assert.Equal(93.50m, result.Total);

The suite notarizes the bug.

4. No edges for EU midpoint

var result = _sut.Calculate(new PriceRequest(
    12.125m, null, CustomerType.External, "EU"));
Assert.Equal(2.43m, result.TaxAmount); // buggy; correct ToEven is 2.42

Happy path + mock theater + copied wrong oracle + missing edges = merge confidence with no business confidence.

Step 2 - The wrong money still ships

dotnet run --project demo/WrongTotalDemo

Green tests. Wrong money.

Step 3 - Solid specs fail on the buggy code

Write the asserts from the business rules, then point them at BuggyPriceCalculator:

dotnet test tests/OrdersPricing.Solid.AgainstBuggy.Tests
# Failed: 3

What goes red:

• External EMP- on 100 - expect discount 0 (buggy gives 15)
• EU midpoint 12.125 - expect tax 2.42 (buggy gives 2.43)
• External EMP- total - expect 110.00 (buggy gives 93.50)

Characterization move: red against a known-bad implementation is success. Your specs have teeth.

Step 4 - Correct calculator + Solid green

dotnet test tests/OrdersPricing.Solid.Tests
# Passed: 13

Solid coverage includes External EMP- means 0, Internal EMP- means 15 on 100, EU ToEven edges (12.125 means tax 2.42), Non-EU AwayFromZero control, and boundaries including zero total, negative throws, and case-insensitive codes and regions.

Ship CorrectPriceCalculator in production. Keep BuggyPriceCalculator in the sample so readers can replay the red/green arc.

Step 5 - Mutation testing with Stryker

Happy green bars can lie. Mutants are less polite.

dotnet tool restore
cd tests/OrdersPricing.AiStyle.Weak.Tests
dotnet tool run dotnet-stryker --project OrdersPricing.csproj   --mutate "**/BuggyPriceCalculator.cs" --break-at 0
# mutation score ~15.79%

cd ../OrdersPricing.Solid.Tests
dotnet tool run dotnet-stryker --project OrdersPricing.csproj   --mutate "**/CorrectPriceCalculator.cs" --break-at 0
# mutation score ~90.91%

A wrong oracle can still look covered because asserts defend the bug. The weak slice scored ~15.79%. Solid characterization on the correct calculator scored ~90.91%. Reports ship under reports/stryker-aistyle/ and reports/stryker-solid/.

Checklist before you trust AI tests

1. Spec the quirks first (Internal-only EMP-, EU ToEven).
2. Assert money and state - not only Times.Once.
3. Include the scary negative case (External + EMP-).
4. Include the midpoint separating ToEven from AwayFromZero.
5. Run characterization against a known-bad build - expect red.
6. Run Stryker and read the survivors.
7. Spot-check one path in a console before you merge.

Core lesson

Green tests mean the asserts matched. They do not mean the business is safe.

When I treat AI-authored tests like a junior PR - useful, fast, and never merge-ready without characterization - I sleep better. When I don't, External customers get employee pricing and finance starts asking about cents.

FAQ

Q: Are you saying ban AI from writing tests?
A: No. I still use it for scaffolding. I do not let it be the source of truth for money or auth quirks.

Q: Why ship both calculators?
A: So you can reproduce green-wrong, red-correct, and the swap without digging through git history.

Q: Why does AgainstBuggy fail on purpose?
A: That failure is the lesson. Prefer filtered commands instead of expecting the whole solution to stay green.

Q: Do I need Stryker in CI on day one?
A: Start local. Add a threshold once the suite is honest. A high score on a wrong oracle is worse than a medium score on a true one.

Q: .NET version?
A: net10.0 / SDK 10.0.x (global.json).

Reproduce this demo

export PATH="$HOME/.dotnet:$PATH"
cd dnw-ai-unit-tests-article
dotnet build OrdersPricing.sln
dotnet test tests/OrdersPricing.AiStyle.Tests
dotnet run --project demo/WrongTotalDemo
dotnet test tests/OrdersPricing.Solid.AgainstBuggy.Tests   # expect fail
dotnet test tests/OrdersPricing.Solid.Tests
dotnet tool restore
# Stryker steps: toolkit/Reproduce.md

More detail: toolkit/Reproduce.md. Open the calculator source and test projects first.

Your turn

Which AI-authored test smell have you already merged - happy-path only, mock call-counts, copied expected totals, or missing midpoint edges? Drop it in the comments.