Evaluate Pester Test Suite Quality

When to Use

• User asks to evaluate, audit, or assess test quality
• User asks about coverage gaps or uncovered code
• User asks to check mock completeness or test isolation
• User mentions flaky tests, test reliability, or test performance
• User says "evaluate", "audit", "check coverage", "find gaps"

Evaluation Dimensions

Step 1: Run Tests with PassThru

$config = New-PesterConfiguration
$config.Run.Path = './tests'
$config.Run.PassThru = $true
$config.CodeCoverage.Enabled = $true
$config.CodeCoverage.Path = './src'
$config.CodeCoverage.OutputFormat = 'JaCoCo'
$config.CodeCoverage.OutputPath = './reports/coverage.xml'
$config.TestResult.Enabled = $true
$config.TestResult.OutputFormat = 'NUnitXml'
$config.TestResult.OutputPath = './reports/test-results.xml'
$config.Output.Verbosity = 'Detailed'
$result = Invoke-Pester -Configuration $config

Step 2: Coverage Gap Analysis

# Identify uncovered lines
$missed = $result.CodeCoverage.CommandsMissed
$grouped = $missed | Group-Object File | Sort-Object Count -Descending

foreach ($file in $grouped) {
    Write-Host "`nUNCOVERED: $($file.Name) — $($file.Count) lines"
    $file.Group | ForEach-Object {
        Write-Host "  Line $($_.Line): $($_.Command)"
    }
}

# Coverage summary
$hit = $result.CodeCoverage.CommandsExecutedCount
$total = $hit + $result.CodeCoverage.CommandsMissedCount
$pct = [math]::Round(($hit / $total) * 100, 1)
Write-Host "`nOverall coverage: $pct% ($hit/$total lines)"

Common Gap Patterns

Step 3: Mock Completeness Check

Scan test files for mock quality:

# Find all Mocks
$mocks = Select-String -Path ./tests/*.Tests.ps1 -Pattern 'Mock\s+(\S+)' |
    ForEach-Object { $_.Matches.Groups[1].Value } | Sort-Object -Unique

# Find all Should -Invoke verifications
$verified = Select-String -Path ./tests/*.Tests.ps1 -Pattern 'Should\s+-Invoke\s+(\S+)' |
    ForEach-Object { $_.Matches.Groups[1].Value } | Sort-Object -Unique

# Unverified mocks (Mock exists but no Should -Invoke)
$unverified = $mocks | Where-Object { $_ -notin $verified }
if ($unverified) {
    Write-Warning "Mocks without Should -Invoke verification:"
    $unverified | ForEach-Object { Write-Warning "  $_" }
}

Mock Quality Checklist

• [ ] Every Mock has a matching Should -Invoke -Times N -Exactly
• [ ] Connect-AzAccount and Get-AzContext mocked in every Azure test
• [ ] Mock return types match actual cmdlet output shapes
• [ ] ParameterFilter used for conditional behavior (not blanket mocks)
• [ ] Mocks scoped correctly: Describe-level (shared) vs Context-level (scenario)
• [ ] No mock of the function under test — only its dependencies

Step 4: Flaky Test Detection

# Run tests 3x and compare results
$results = 1..3 | ForEach-Object {
    $config = New-PesterConfiguration
    $config.Run.Path = './tests'
    $config.Run.PassThru = $true
    $config.Output.Verbosity = 'None'
    Invoke-Pester -Configuration $config
}

# Compare pass/fail status for each test
$testNames = $results[0].Tests.Name
foreach ($name in $testNames) {
    $statuses = $results | ForEach-Object {
        ($_.Tests | Where-Object { $_.Name -eq $name }).Result
    }
    if (($statuses | Sort-Object -Unique).Count -gt 1) {
        Write-Warning "FLAKY: $name — results: $($statuses -join ', ')"
    }
}

Common Flaky Causes

Step 5: Test Isolation Verification

Check that tests are truly independent:

• [ ] No $script: or $global: variables shared between Describe blocks
• [ ] Each Describe has its own BeforeAll setup (not inherited)
• [ ] TestDrive used for file operations (auto-cleaned per Describe)
• [ ] No real network, file system, or registry operations
• [ ] Tests pass when run individually: Invoke-Pester -Path ./tests/X.Tests.ps1
• [ ] Tests pass in random order (shuffle)

Step 6: Performance Profiling

$result = Invoke-Pester -Path './tests' -PassThru
$result.Containers | ForEach-Object {
    $_.Blocks | ForEach-Object {
        Write-Host "$($_.Name): $([math]::Round($_.Duration.TotalSeconds, 2))s"
    }
} | Sort-Object -Descending

Flag any Describe block > 5 seconds for optimization.

Output: Test Quality Report

Generate a structured quality report:

## Test Quality Evaluation Report
| Metric             | Result  | Target  | Status |
|--------------------|---------|---------|--------|
| Line coverage      | 93.2%   | ≥90%    | PASS   |
| Branch coverage    | 85.1%   | ≥80%    | PASS   |
| Function coverage  | 100%    | 100%    | PASS   |
| Mock completeness  | 100%    | 100%    | PASS   |
| Flaky tests        | 0       | 0       | PASS   |
| Avg test duration  | 1.8s    | <5s     | PASS   |
| Test isolation     | OK      | Independent | PASS |