Back to Sanity

Test Reports & Artifacts

.agents/skills/playwright-best-practices/infrastructure-ci-cd/reporting.md

5.24.011.6 KB
Original Source

Test Reports & Artifacts

Table of Contents

  1. CLI Commands
  2. Reporter Configuration
  3. Custom Reporter
  4. Trace Configuration
  5. Screenshot & Video Settings
  6. Artifact Directory Structure
  7. CI Artifact Upload
  8. Decision Guide
  9. Anti-Patterns
  10. Troubleshooting

When to use: Configuring test output for debugging, CI dashboards, and team visibility.

CLI Commands

bash
# Display last HTML report
npx playwright show-report

# Specify reporter
npx playwright test --reporter=html
npx playwright test --reporter=dot           # minimal CI output
npx playwright test --reporter=line          # one line per test
npx playwright test --reporter=json          # machine-readable
npx playwright test --reporter=junit         # CI integration

# Combine reporters
npx playwright test --reporter=dot,html

# Merge sharded reports
npx playwright merge-reports --reporter=html ./blob-report

Reporter Configuration

Environment-Based Setup

typescript
// playwright.config.ts
import {defineConfig} from '@playwright/test'

export default defineConfig({
  reporter: process.env.CI
    ? [['dot'], ['html', {open: 'never'}], ['junit', {outputFile: 'results/junit.xml'}], ['github']]
    : [['list'], ['html', {open: 'on-failure'}]],
})

Reporter Types

ReporterOutputUse Case
listOne line per testLocal development
lineSingle updating lineLocal, less verbose
dot. pass, F failCI logs
htmlInteractive HTML pagePost-run analysis
jsonMachine-readable JSONCustom tooling
junitJUnit XMLCI platforms
githubPR annotationsGitHub Actions
blobBinary archiveShard merging

JSON Output to File

typescript
import {defineConfig} from '@playwright/test'

export default defineConfig({
  reporter: [['json', {outputFile: 'results/output.json'}]],
})

JUnit Customization

typescript
import {defineConfig} from '@playwright/test'

export default defineConfig({
  reporter: [
    [
      'junit',
      {
        outputFile: 'results/junit.xml',
        stripANSIControlSequences: true,
        includeProjectInTestName: true,
      },
    ],
  ],
})

Custom Reporter

Build custom reporters for Slack notifications, database logging, or dashboards.

typescript
// reporters/notification-reporter.ts
import type {FullResult, Reporter, TestCase, TestResult} from '@playwright/test/reporter'

class NotificationReporter implements Reporter {
  private passed = 0
  private failed = 0
  private skipped = 0
  private failures: string[] = []

  onTestEnd(test: TestCase, result: TestResult) {
    switch (result.status) {
      case 'passed':
        this.passed++
        break
      case 'failed':
      case 'timedOut':
        this.failed++
        this.failures.push(`${test.title}: ${result.error?.message?.split('\n')[0]}`)
        break
      case 'skipped':
        this.skipped++
        break
    }
  }

  async onEnd(result: FullResult) {
    const total = this.passed + this.failed + this.skipped
    const status = this.failed > 0 ? 'FAILED' : 'PASSED'
    const message = [
      `Tests ${status}`,
      `Passed: ${this.passed} | Failed: ${this.failed} | Skipped: ${this.skipped}`,
      `Duration: ${(result.duration / 1000).toFixed(1)}s`,
    ]

    if (this.failures.length > 0) {
      message.push('', 'Failures:')
      this.failures.slice(0, 5).forEach((f) => message.push(`  - ${f}`))
      if (this.failures.length > 5) {
        message.push(`  ...and ${this.failures.length - 5} more`)
      }
    }

    const webhookUrl = process.env.NOTIFICATION_WEBHOOK
    if (webhookUrl) {
      const controller = new AbortController()
      const timeout = setTimeout(() => controller.abort(), 5000)
      try {
        await fetch(webhookUrl, {
          method: 'POST',
          headers: {'Content-Type': 'application/json'},
          body: JSON.stringify({text: message.join('\n')}),
          signal: controller.signal,
        })
      } catch (error) {
        // Intentionally swallow notifier failures to avoid blocking test completion
        console.warn('Webhook notification failed:', error.message)
      } finally {
        clearTimeout(timeout)
      }
    }
  }
}

export default NotificationReporter

Register custom reporter:

typescript
import {defineConfig} from '@playwright/test'

export default defineConfig({
  reporter: [['dot'], ['html', {open: 'never'}], ['./reporters/notification-reporter.ts']],
})

Trace Configuration

Traces capture actions, network requests, DOM snapshots, and console logs.

typescript
import {defineConfig} from '@playwright/test'

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    trace: 'on-first-retry',
  },
})

Trace Options

ValueBehaviorOverhead
'off'Never recordsNone
'on'Every testHigh
'on-first-retry'On first retry after failureMinimal
'retain-on-failure'Records all, keeps failuresMedium
'retain-on-first-failure'Records all, keeps first failureMedium

Viewing Traces

bash
# Local trace viewer
npx playwright show-trace results/my-test/trace.zip

# From HTML report (click Traces tab)
npx playwright show-report

# Online viewer: https://trace.playwright.dev

Screenshot & Video Settings

typescript
import {defineConfig} from '@playwright/test'

export default defineConfig({
  use: {
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
})

Video with Custom Size

typescript
use: {
  video: {
    mode: 'retain-on-failure',
    size: { width: 1280, height: 720 },
  },
},

Screenshot Options

ValueCapturesDisk Cost
'off'NeverNone
'on'Every testHigh
'only-on-failure'Failed testsLow

Video Options

ValueRecordsKeepsDisk Cost
'off'NeverNone
'on'Every testAllVery high
'on-first-retry'On retryRetriedLow
'retain-on-failure'Every testFailedMedium

Artifact Directory Structure

text
test-results/
├── checkout-test-chromium/
│   ├── trace.zip
│   ├── test-failed-1.png
│   └── video.webm
├── login-test-firefox/
│   ├── trace.zip
│   └── test-failed-1.png
└── junit.xml

playwright-report/
├── index.html
└── data/

blob-report/
└── report-1.zip

CI Artifact Upload

GitHub Actions

yaml
- uses: actions/upload-artifact@v4
  if: ${{ !cancelled() }}
  with:
    name: playwright-report
    path: playwright-report/
    retention-days: 14

- uses: actions/upload-artifact@v4
  if: failure()
  with:
    name: test-traces
    path: |
      test-results/**/trace.zip
      test-results/**/*.png
      test-results/**/*.webm
    retention-days: 7

Decision Guide

ScenarioReporter Configuration
Local development[['list'], ['html', { open: 'on-failure' }]]
GitHub Actions[['dot'], ['html'], ['github']]
GitLab CI[['dot'], ['html'], ['junit']]
Azure DevOps / Jenkins[['dot'], ['html'], ['junit']]
Sharded CI[['blob'], ['github']]
Custom dashboard[['json', { outputFile: '...' }]] + custom reporter
ArtifactWhen to CollectRetentionUpload Condition
HTML reportAlways14 daysif: ${{ !cancelled() }}
TracesOn failure7 daysif: failure()
ScreenshotsOn failure7 daysif: failure()
VideosOn failure7 daysif: failure()
JUnit XMLAlways14 daysif: ${{ !cancelled() }}
Blob reportAlways (sharded)1 dayif: ${{ !cancelled() }}

Anti-Patterns

Anti-PatternProblemSolution
No reporter configuredDefault list only; no persistent reportConfigure html + CI reporter
trace: 'on' in CIMassive artifacts, slow uploadsUse trace: 'on-first-retry'
video: 'on' in CIEnormous storage, slower testsUse video: 'retain-on-failure'
Upload artifacts only on failureNo report when tests passUpload with if: ${{ !cancelled() }}
No retention limitsCI storage fills quicklySet retention-days: 7-14
Only dot reporterCannot drill into failuresPair dot with html
JUnit to stdoutInterferes with console outputWrite to file
Blocking onEnd in custom reporterSlow HTTP calls delay pipelineUse Promise.race with timeout

Troubleshooting

Empty HTML Report

Check reporter config. HTML report defaults to playwright-report/:

typescript
import {defineConfig} from '@playwright/test'

export default defineConfig({
  reporter: [['html', {outputFolder: 'playwright-report', open: 'never'}]],
})

Traces Too Large

Switch from trace: 'on' to 'on-first-retry' with retries enabled:

typescript
import {defineConfig} from '@playwright/test'

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    trace: 'on-first-retry',
  },
})

JUnit XML Not Recognized

Ensure path matches CI configuration:

typescript
reporter: [['junit', { outputFile: 'results/junit.xml' }]],
yaml
# GitHub Actions
- uses: dorny/test-reporter@latest
  with:
    path: results/junit.xml
    reporter: java-junit

# Azure DevOps
- task: PublishTestResults@latest
  inputs:
    testResultsFiles: 'results/junit.xml'

# Jenkins
junit 'results/junit.xml'

Empty Merged Report

Use blob reporter for sharded runs (not html):

typescript
import {defineConfig} from '@playwright/test'

export default defineConfig({
  reporter: process.env.CI ? [['blob'], ['dot']] : [['html', {open: 'on-failure'}]],
})

Missing Screenshots in Report

Enable screenshots and keep both directories:

typescript
use: {
  screenshot: 'only-on-failure',
},

The HTML report embeds screenshots from test-results/. Deleting that directory removes screenshots from the report.