Visual testing

[Newbie012]

A third-party jest matcher called toMatchImageSnapshot performs image comparison. It's primarily useful when you do visual tests. It would be pretty cool if vitest could support this kind of testing as one of its built-in features.

The lack of visual testing prevents me from migrating from jest to vitest. I believe many other developers who perform visual testing will be interested in this feature.

[sheremet-va]

This library should work with Vitest, no?

[Newbie012]

I haven't really tried. After inspecting the source code, it looks like it really should. I'll keep you posted :)

[hcampos-professional]

Agreed 👍. It would be great to support something similar to what Playwright has built in: https://playwright.dev/docs/test-snapshots

With Vitest we can take the screenshots but don't have a built in mechanism for comparing and asserting them. And even if we implement our own comparison (e.g. with pixelmatch or phash) there is no way (that I could find, at least) to display results in the Vitest UI. At the moment we're forced to build a custom reporter that produces a separate report that developers need to check separately.

It would be awesome if we could pass some metadata along with errors to let Vitest display Before, After and Diff, like Playwright does:

Vitest already has something similar, in the form of meta.failScreenshotPath:

vitest/packages/ui/client/components/views/ViewReport.vue

Lines 117 to 123 in 68e6c63

	const currentScreenshotUrl = computed(() => {
	const file = currentTask.value?.meta.failScreenshotPath
	// force refresh
	const t = timestamp.value
	// browser plugin using /, change this if base can be modified
	return file ? `/__screenshot-error?file=${encodeURIComponent(file)}&t=${t}` : undefined
	})

Alternatively (or in addition), Vitest could add support for rendering HTML error messages. That would let apps render before/after/diff directly in the UI. Probably a way of opting out of escaping HTML here:

vitest/packages/ui/client/components/views/ViewReport.vue

Lines 33 to 61 in 68e6c63

	function createHtmlError(filter: Convert, error: ErrorWithDiff) {
	let htmlError = ''
	if (error.message?.includes('\x1B')) {
	htmlError = `<b>${error.nameStr || error.name}</b>: ${filter.toHtml(
	escapeHtml(error.message),
	)}`
	}
	const startStrWithX1B = error.stackStr?.includes('\x1B')
	if (startStrWithX1B || error.stack?.includes('\x1B')) {
	if (htmlError.length > 0) {
	htmlError += filter.toHtml(
	escapeHtml((startStrWithX1B ? error.stackStr : error.stack) as string),
	)
	}
	else {
	htmlError = `<b>${error.nameStr || error.name}</b>: ${
	error.message
	}${filter.toHtml(
	escapeHtml((startStrWithX1B ? error.stackStr : error.stack) as string),
	)}`
	}
	}
	if (htmlError.length > 0) {
	return htmlError
	}
	return null
	}

[Shiwa]

I agree it would be pretty cool to do visual regression testing just using vitest built-ins.

I’m working on a project with lots of home made components and design that can be technically usable but visually broken (popovers, sticky toolbars, responsive layouts, etc.).
We already use chromatic for our design system and it’s awesome, and I’d really like to have the same kind of tooling in the app, without setting up extra libs.

[Shiwa]

FYI I implemented visual regression testing with vitest browser mode / playwright. It was painful and the current solution is not perfect.

Here are the pain points I encountered:

• the jest-image-snapshot does not work in browser mode, because it tries to use Buffer. I tried to use it in a server-side command but in this context expect does not work
• the expect().toHaveScreenshot() utility from playwright does not work because we don’t have access to the playwright version of expect
• so I used page.screenshot() and toMatchSnapshot() on the base64. This works, but the match must be pixel perfect.
• since the base64 snap is not very practical, I tried to use readFile and toMatchFileSnapshot(ctx.task.name + '.png'), but even with encoding: 'binary' the resulting snapshot is not a valid png file. I ended up doing a readFile/writeFile to save the png.

Here’s the current version of our visual regression testing utility, if that can help:

export async function renderAndScreenshot(
  element: JSX.Element,
  ctx: TaskContext,
) {
  renderInBrowser(element, {
    wrapper: ({ children }) => <BrowserTestContext>{children}</BrowserTestContext>, // BrowserTestContext imports CSS and stuff
  });

  const screenshotPath = `./__screenshots__/${ctx.task.name}.png`;
  const screenshot = await page.screenshot({ base64: true, path: screenshotPath });

  await expect(screenshot.base64).toMatchFileSnapshot(`./__snapshots__/${ctx.task.name}.snap`); // match base64

  // test ok, save png in __snapshots__ for reference
  const data = await server.commands.readFile(screenshot.path, { encoding: 'binary' });
  await server.commands.writeFile(`./__snapshots__/${ctx.task.name}.png`, data, { encoding: 'binary' });
  await server.commands.removeFile(screenshot.path);
}

[JoaoCnh]

Yeah I believe if we get something like this working within Vitest a lot of migration can happen.

We're currently already using Vitest but we would really love to have some regression testing completely integrated into vitest but we've not been successful with jest-image-snapshot or any tries with playwright custom commands.

[JoaoCnh]

I've tried writing my own custom matcher but a problem I can't seem to get around is having to use fs and path for file related stuff to handle screenshots and snapshots. I will always get hit with the Module "path" has been externalized for browser compatibility. Cannot access "path.join" in client code. error.

Is there a way to get around this?

[JoaoCnh]

I'm using Vitest browser testing btw

[Shiwa]

You should do file system actions through server commands, either using the provided readFile, etc. (like in my prototype above) or creating a custom one.
Custom server commands receive a context object with a testPath prop that helps to find your screenshots.

[JoaoCnh]

I was able to save the screenshot with success but pngjs then cannot read the contents from the readFile server command so I guess I'm hitting some sort of limitation of data, buffers and stuff that I'm not fully knowledgeable of. But maybe I can create a custom command to run this stuff on the server for me. I'll try that approach next

[JoaoCnh]

I was able to get it working. Now I need to fine-tune the logic of file organisation and all of that. Essentially the test page takes the screenshot and saves it -> passes the path to expect that was extended with a custom matcher called toMatchImageSnapshot -> that in turn calls a custom command that receives the test name and the screenshot path and does file reading/writing/deleting.