toMatchScreenshot skips createDiff:true when the first capture is stable against the reference #11178

Describe the bug

toMatchScreenshot never calls the comparator with createDiff: true when the first capture is already “stable” against the stored reference.

Stability detection always calls the comparator with createDiff: false. If that first comparison passes, getStableScreenshot returns retries === 0, and determineOutcome short-circuits:

// first capture matched reference (used as baseline) - no further comparison needed
if (retries === 0) {
  return { type: 'matched-immediately' }
}

The docs say a final comparison with createDiff: true happens after stability. In this path it never does.

That is correct for the built-in pixelmatch comparator, where createDiff only controls whether a diff buffer is allocated. It is wrong for a custom comparator that uses the two flags as two phases:

• createDiff: false — cheap stability (consecutive frames / vs reference)
• createDiff: true — the actual match (SSIM, empty-reference guards, messages, …)

We hit this in production: a golden comparator uses pixel tolerance for stability and SSIM plus an empty-image guard for the final match. An empty reference plus an empty capture passes the 0.1% pixel stability check on the first try, retries === 0 skips createDiff: true, and the empty-image guard never runs. False green.

The pixelmatch PNG-byte fast path is already gated on comparatorName === 'pixelmatch' (see #10278). This retries === 0 short-circuit is not, so it still bypasses custom comparators.

I intend to send a PR that removes the short-circuit (keep the pixelmatch byte fast path) and adds a regression test.

Reproduction

Register a comparator that passes only on the stability call:

declare module 'vitest/browser' {
  interface ScreenshotComparatorRegistry {
    'stability-only': Record<string, never>
  }
}

export default defineConfig({
  test: {
    browser: {
      expect: {
        toMatchScreenshot: {
          comparators: {
            'stability-only': (_reference, _actual, { createDiff }) => ({
              pass: !createDiff,
              diff: null,
              message: createDiff ? 'createDiff:true rejected' : null,
            }),
          },
        },
      },
    },
  },
})

test('first-capture stability still runs createDiff:true', async () => {
  const locator = page.getByTestId('el')
  await locator.screenshot({ save: true, path: '…reference.png' })
  await expect(locator).toMatchScreenshot('name', {
    comparatorName: 'stability-only',
  })
})

Expected: assertion fails (Screenshot does not match the stored reference. / createDiff:true rejected).

Actual on 5.0.0 and current main: assertion passes (matched-immediately).

The same comparator fails correctly once stability needs a retry (retries > 0), because that path does call createDiff: true.

System Info

System:
  OS: macOS 26.3.1
Binaries:
  Node: 26.5.0
  pnpm: 11.20.0
npmPackages:
  vitest: 5.0.0
  @vitest/browser: 5.0.0
  @vitest/browser-playwright: 5.0.0
  vite: 8.2.2
  playwright: 1.63.0

Also confirmed the same retries === 0 branch is still in main (packages/browser/src/node/commands/screenshotMatcher/index.ts).

Used Package Manager

pnpm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

Drafted with assistance from Grok (xAI); I verified the retries === 0 branch against 5.0.0 and main, and against a custom comparator that treats createDiff as two phases.