Circular Validation: The Hidden Risk in AI-Generated Tests

Software testing is undergoing a fundamental transformation with AI-generated test suites. But this advancement introduces a critical risk I call circular validation - a pattern that could undermine the entire purpose of testing.

The Circular Validation Problem Link to heading

When an AI assistant analyzes implementation code to generate tests, it creates a closed feedback loop. These tests aren’t validating that software meets business needs - they’re simply confirming the code does exactly what it was written to do, including any bugs or misinterpretations.

It’s like having a student both write and grade their own exam. The tests pass brilliantly while potentially reinforcing critical flaws.

Business Intent as the Only Valid Reference Point Link to heading

Effective tests must validate the business intent - not just implementation details. Quality tests answer questions that matter:

• Does the feature deliver intended value to users?
• Does it satisfy all business requirements?
• Are edge cases handled properly?
• Does performance meet expectations?
• How does it behave when integrated?

Tests derived solely from code can’t address these fundamental questions. At best, they confirm internal consistency; at worst, they conceal fundamental misalignments with actual requirements.

Real-World Example: Discount Calculation Bug Link to heading

Consider this user story:

As an online shopper
I want to filter products by discount percentage
So that I can quickly find the best deals

Acceptance Criteria:

1. Filter products by minimum discount percentage (10%, 25%, 50%, 75%)
2. Show original price, discounted price, and discount percentage
3. Hide products below selected discount threshold
4. Calculate discount as ((original - current) / original) * 100
5. Work with existing category filters
6. Allow sorting by highest discount first

The Implementation Bug Link to heading

public class ProductService
{
    public IReadOnlyList<Product> GetDiscountedProducts(IEnumerable<Product> products,
                                                        int? minDiscountPercentage = null,
                                                        string category = null,
                                                        bool sortByHighestDiscount = false)
    {
        // The BUG: Incorrect discount calculation formula
        // Correct: ((originalPrice - currentPrice) / originalPrice) * 100
        // Buggy: ((originalPrice - currentPrice) / currentPrice) * 100
        var filtered = products.Where(p =>
        {
            decimal discountPercentage = ((p.OriginalPrice - p.CurrentPrice) / p.CurrentPrice) * 100;
            p.DiscountPercentage = discountPercentage;

            bool meetsDiscountFilter = minDiscountPercentage == null || discountPercentage >= minDiscountPercentage;
            bool meetsCategoryFilter = string.IsNullOrEmpty(category) || p.Category == category;

            return meetsDiscountFilter && meetsCategoryFilter;
        });

        if (sortByHighestDiscount)
        {
            return filtered.OrderByDescending(p => p.DiscountPercentage);
        }

        return filtered.OrderBy(p => p.Name)
                 .ToList();
    }
}

This bug causes incorrect discount percentages, wrong filtering, and improper sorting.

For a $100 product now $80:

• Correct: ((100 - 80) / 100) * 100 = 20% discount
• Buggy: ((100 - 80) / 80) * 100 = 25% discount

AI-Generated Tests (Circular Validation) Link to heading

[Test]
public void GetDiscountedProducts_WithMinimumDiscount_ReturnsMatchingProducts()
{
    // Act
    var result = _service.GetDiscountedProducts(_testProducts, 30);

    // Assert - This passes with the buggy implementation!
    Assert.AreEqual(2, result.Count);
    CollectionAssert.Contains(result, _testProducts[1]); // Premium Phone: ((800-600)/600)*100 = 33.33%
    CollectionAssert.Contains(result, _testProducts[2]); // Designer Bag: ((300-150)/150)*100 = 100%
}

These tests pass despite the bug because they’re based on implementation logic, not business requirements.

Intent-Based Tests Link to heading

[Test]
public void DiscountPercentage_CalculatedCorrectly()
{
    // Arrange
    var product = new Product { OriginalPrice = 100, CurrentPrice = 75 };

    // Act - Using the correct formula directly in the test
    decimal expectedDiscountPercentage = ((product.OriginalPrice - product.CurrentPrice) / product.OriginalPrice) * 100;
    var result = _service.GetDiscountedProducts(new List<Product> { product });

    // Assert
    Assert.AreEqual(expectedDiscountPercentage, result[0].DiscountPercentage);
    Assert.AreEqual(25, result[0].DiscountPercentage); // Explicitly check the expected percentage
}

This test would fail, correctly identifying the implementation bug.

Breaking the Circular Validation Cycle Link to heading

To create tests that truly validate business intent:

1. Start with Requirements, Not Implementation Link to heading

• Derive test cases from user stories and acceptance criteria
• Document the “why” behind each test

2. Challenge Implementation Assumptions Link to heading

• Probe edge cases and boundary conditions
• Include negative testing scenarios
• Simulate realistic failure modes

3. Adopt a Stakeholder Perspective Link to heading

• Collaborate with product owners and end-users
• Incorporate real-world usage patterns
• Validate business value delivery

4. Use AI as a Complement, Not a Replacement Link to heading

• Review AI-generated tests against business requirements
• Use AI to suggest additional test cases
• Maintain human oversight

Conclusion Link to heading

AI can accelerate testing, but introduces circular validation risks. The key distinction is whether tests validate what software should do versus what it does do.