Running Playwright tests in CI with GitHub Actions

8 min read · TypeScript · 5 linked exercises

Tests that only run on your laptop protect nobody. Running them on every push and pull request is what turns a test suite into a safety net.

This guide sets up Playwright in GitHub Actions: the workflow file, the settings that should differ in CI, how to read a failure you didn't see happen, and how to keep the run fast. The same ideas apply to GitLab CI, Jenkins and other systems.

The workflow file

GitHub Actions runs every YAML file in .github/workflows. This one installs the project and the browsers, runs the tests and keeps the report:

# .github/workflows/playwright.yml
name: Playwright tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14

You don't have to type it. npm init playwright@latest offers to add a workflow like this one when it creates a project, with the current versions of each action.

npm ci installs exactly the versions in package-lock.json, so CI runs the same Playwright version as you. --with-deps also installs the system libraries the browsers need on a bare Linux machine.

Settings that change in CI

GitHub Actions sets the environment variable CI. The config reads it to behave differently on the two machines:

export default defineConfig({
  // Fail the run if someone committed test.only.
  forbidOnly: !!process.env.CI,

  // Retry in CI only. Locally you want to see a failure at once.
  retries: process.env.CI ? 2 : 0,

  // Annotations in the pull request, plus the HTML report.
  reporter: process.env.CI ? [['github'], ['html', { open: 'never' }]] : 'html',

  use: {
    baseURL: 'https://playwright-playground.com',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },
});

forbidOnly matters more than it looks. A forgotten test.only makes the suite run one test and pass. Nothing looks wrong, and the other tests stop protecting you.

The github reporter adds each failure as an annotation on the line that failed, where reviewers of the pull request see it. The HTML report holds the details.

Read a failure you never saw

In CI there is no browser window to watch. The upload step is what you have instead. if: ${{ !cancelled() }} makes it run when the tests failed, which is exactly when you need it. Without that line, a failed test step skips every step after it.

Download the playwright-report artifact from the run's summary page, unzip it and open it:

npx playwright show-report path/to/playwright-report

With trace: 'on-first-retry', every test that failed and was retried has a trace in the report: the page before and after each action, the network and the console. The Trace Viewer guide shows how to read one.

Practise it
  • Trace ViewerIntermediateFind a bug from a trace alone, as you would after a CI failure.
  • Reading FailuresBeginnerRead five error messages and fix each test from the text.

Retries and flaky tests

A test that fails and then passes on a retry is reported as flaky, and the run is green. Retries keep the pipeline moving, but they also hide problems. A flaky test is either a test with a timing mistake or an app with a real bug that appears some of the time.

Check the flaky list in the report after every run, and fix what's on it. To be strict, fail the run when any test needed a retry:

npx playwright test --fail-on-flaky-tests

CI machines are slower than your laptop, so timing mistakes show up there first. Find them before you push by repeating new tests:

npx playwright test cart.spec.ts --repeat-each=20
Practise it
  • Flaky LabAdvancedFix tests that pass most of the time, and prove it with twenty runs.

Speed: browsers, workers and shards

Install only the browsers you test. Downloading all three takes a noticeable part of a short run. If your config has one project:

npx playwright install --with-deps chromium

Use the workers you have. Playwright runs test files in parallel, by default on half of the machine's CPU cores. Standard CI runners have few cores, so the gain is smaller than on your laptop. Parallel runs only work when tests don't share data, so check that locally first with --workers=4.

Split a large suite across machines. Sharding gives each machine a part of the tests:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      # …checkout, install…
      - run: npx playwright test --shard=${{ matrix.shard }}/4

Each shard writes its own report. To get one report for the whole run, use the blob reporter in the shards and combine the results in a final job with npx playwright merge-reports. Add sharding when the run takes longer than you're willing to wait, not before.

Practise it

What the tests run against

Tests need a running app. There are two common setups.

A deployed environment. The tests open a URL that already exists, such as a staging site. Pass the address in, so the same tests can run against any environment:

use: {
  baseURL: process.env.BASE_URL ?? 'http://localhost:5173',
},

The app started in the job. Playwright starts it, waits until the URL answers, runs the tests and stops it:

webServer: {
  command: 'npm run dev',
  url: 'http://localhost:5173',
  reuseExistingServer: !process.env.CI,
},

reuseExistingServer lets you keep your own dev server running locally, while CI always starts a clean one.

The exercises on this site are the first kind. Point baseURL at https://playwright-playground.com and there's nothing to start, so a repository with your solutions and the workflow above is a complete project you can show.

Secrets and sign-in

Never commit a real password. Store it in the repository's settings under "Secrets and variables", pass it to the test step as an environment variable, and read it in the test:

- run: npx playwright test
        env:
          TEST_USER: ${{ secrets.TEST_USER }}
          TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
await page.getByLabel('Password').fill(process.env.TEST_PASSWORD!);

GitHub hides secret values in the logs. They can still end up in a trace or a screenshot, so use an account made for testing, with access to test data only.

A setup project that signs in once works in CI without changes: it runs first and writes the session file on the runner. Keep that file out of the repository. The authentication guide covers the setup.

Practise it

A checklist before you rely on it

  • The workflow runs on pull requests, not only on the main branch.
  • forbidOnly is on in CI.
  • The report is uploaded when tests fail, and you have opened one.
  • Retries are on, and somebody looks at the flaky tests.
  • The job has a timeout, so a hung run doesn't use an hour of minutes.
  • No password, token or session file is in the repository.

Exercises in this guide

More guides