PDF visual regression testing: compare every page in CI

PDF visual regression testing compares images of generated PDF pages with approved references. It catches document defects that a successful render response misses: a missing logo, a substituted font, a clipped total, or a table pushed onto an extra page. Our recommended CI check combines page-image comparisons with assertions about the document's content.

This is especially useful after a Chromium update or an engine migration such as wkhtmltopdf to Playwright. The test target should be the PDF your application delivers, including every printed page.

What should a PDF regression test verify?

Check document content, structure, and appearance separately. A pixel comparison can reveal a shifted amount without determining whether the amount is correct. A text assertion can verify a total while missing that the total is clipped at the page edge.

Playwright's visual comparison documentation describes screenshot baselines and the importance of consistent environments. A browser screenshot is useful for the editor, but it is a separate test from the PDF artifact. The PDF API uses print media by default, so an apparently correct screen preview does not establish print-layout correctness.

Keep a small set of fixtures that target real failure modes. Our suggested starting set is a short invoice, a multipage table, a long customer name, an optional-field case, and a document containing the non-Latin scripts you support.

How do you make PDF comparisons repeatable?

Freeze the sample data and the rendering environment before choosing an image threshold. Our recommended fixture record includes the template revision, input JSON, fonts, browser build, locale, timezone, PDF options, and rasterizer version.

Replace live timestamps, random identifiers, and changing external charts with fixed values. Prefer versioned local assets in your test fixture. If the production document uses remote assets, test that integration separately and keep the visual baseline's assets stable.

Wait for application-specific document readiness before printing. That might mean chart rendering has completed or all required data has arrived. A fixed sleep can pass on a fast machine and capture an unfinished document on a slower worker. The migration guide includes a static-template render script with explicit font and image readiness.

Choose the comparison environment deliberately. Playwright notes that rendering can vary with operating system, browser version, hardware, and other settings. An unexpected font difference is easier to diagnose when the CI run records exactly which image and browser produced it.

How do you rasterize every PDF page?

Use the same PDF rasterizer and resolution for reference and candidate documents. Poppler's pdftoppm writes one image per page and supports PNG output at an explicit DPI.

With Poppler installed, run the following in a fresh working directory containing reference.pdf and candidate.pdf:

mkdir baseline actual
pdftoppm -png -r 144 reference.pdf baseline/page
pdftoppm -png -r 144 candidate.pdf actual/page

The resolution of 144 DPI is a starting choice for this example, not a compliance standard. Keep it constant and review whether it exposes the defects your templates have experienced. Avoid -singlefile, which would test only the first page.

Use clean output directories on every run. Leftover PNGs from an older, longer document can conceal a page-count change. Keep the PDF files too: reviewers often need to zoom into the original or inspect text after seeing the image diff.

The PDF-to-PNG tool is useful for a manual page review. For CI, a pinned rasterizer gives the test a recorded, repeatable conversion step.

How can a Node.js script compare the page images?

Begin with exact RGB image comparison and fail on a changed page inventory or image dimensions. The following script uses sharp, installed in the Node.js project, and writes a diff image for each pixel mismatch. It assumes the clean baseline and actual directories from the preceding step.

Save this as compare-pages.mjs:

import { mkdir, readdir } from 'node:fs/promises';
import sharp from 'sharp';
 
const pages = async (dir) => (await readdir(dir))
  .filter((name) => /^page-\d+\.png$/.test(name)).sort();
const baseline = await pages('baseline');
const actual = await pages('actual');
if (!baseline.length || baseline.join('\n') !== actual.join('\n')) {
  throw new Error('PDF page inventory changed or is empty');
}
 
await mkdir('diffs', { recursive: true });
const failures = [];
const pixels = (file) => sharp(file)
  .flatten({ background: '#fff' }).toColourspace('srgb').removeAlpha()
  .raw().toBuffer({ resolveWithObject: true });
 
for (const name of baseline) {
  const [left, right] = await Promise.all([
    pixels(`baseline/${name}`), pixels(`actual/${name}`),
  ]);
  if (left.info.width !== right.info.width ||
      left.info.height !== right.info.height) {
    failures.push(`${name}: dimensions changed`);
    continue;
  }
  if (left.data.equals(right.data)) continue;
  const diff = Buffer.alloc(left.data.length);
  for (let i = 0; i < diff.length; i++) {
    diff[i] = Math.abs(left.data[i] - right.data[i]);
  }
  await sharp(diff, { raw: {
    width: left.info.width, height: left.info.height, channels: 3,
  } }).png().toFile(`diffs/${name}`);
  failures.push(`${name}: pixels changed`);
}
 
if (failures.length) throw new Error(failures.join('\n'));
console.log(`Matched ${baseline.length} PDF page image(s)`);

Run node compare-pages.mjs. Treat exact comparison as a diagnostic starting point. If your environment has acceptable variation, document it and add a reviewed tolerance. A global tolerance chosen simply to make CI green can hide a small but important defect, such as a missing decimal point.

How should the CI review handle changed baselines?

Require review of intentional document changes and retain the failing artifacts. Our recommendation is to upload the candidate PDF, reference pages, candidate pages, diff images, and content-check results with each run.

Update baselines in the same reviewed change that modifies a template or renderer. Do not regenerate all references automatically on a failing build. A passing comparison against an accidentally approved broken invoice preserves the defect.

Add text assertions alongside the image check. Our PDF text extraction guide covers that layer, and the PDF-to-text tool helps inspect a sample manually. The print CSS guide explains common layout changes that the visual check will expose.

The same acceptance workflow works with hosted rendering. Save the PDF returned by your PDF4.dev integration as candidate.pdf, then run the page and content checks. PDF4.dev handles rendering; your tests define what a correct customer document must contain and how it should look.