Playwright AI Automation Framework — Saucedemo

An end-to-end test framework where a Jira ticket becomes a reviewed, TCMS-mirrored Playwright pull request — authored by AI agents, gated by deterministic CI. Built on saucedemo as both a reusable template and a portfolio piece.

In four bullets:

• A ticket goes in; a reviewed pull request comes out — the tests, the Page Objects they need, and the acceptance criterion each test covers.
• The AI writes once. Every run after that is plain Playwright reading committed files. Clone this and run npm test with no Claude Code installed and you still get the Jira links and the criteria.
• A run never opens a red PR. It diagnoses and retries up to three times; if the _application_ is what contradicts the ticket, it opens nothing and says so.
• 12 of 16 generated PRs landed exactly as generated — measured, with its caveats, below.

overview

Quick start

git clone https://github.com/Diegocortes15/playwright-ai-framework.git
cd playwright-ai-framework
npm install
npx playwright install chromium
cp .env.example .env          # saucedemo defaults work out of the box
npm test

112 tests, ~34 seconds. The tests need nothing else — no Claude Code, no Jira, no Qase account. The AI-authoring layer is additive and entirely optional.

Node 22.x is enforced rather than suggested: .nvmrc, an engines field and engine-strict=true mean npm install refuses another major instead of warning.

Another engine, when you want one. Cross-browser is implemented and opt-in (ADR-0027) — npm test never pays for it:

npx playwright install firefox webkit    # ~175 MB, once
npm run test:firefox                     # the standard user on Gecko
npm run test:cross -- --grep "@smoke"    # both engines, smoke only — 17 tests, ~13s

The full standard suite passes on both: 91 tests on Firefox (~46s), 91 on WebKit (~33s), against 112 on chromium in ~34s. Only the standard user goes cross-browser — engine differences live in the framework's interaction code, not in saucedemo's per-user bugs.

Its first real use paid for itself: WebKit exposed a race in InventoryPage.goto() that chromium had always won, and the fix landed with it (ADR-0027).

See it work — a five-minute demo

Everything here runs locally against the public app. Good for showing someone what the framework does without reading any code.

1. A run where failing is the correct outcome

npx playwright test --project=chromium-problem --grep "@problem"

The --grep narrows to the tests pinned to this user. Without it the project also runs the seven footer tests that @all-users routes to every authenticated context, which are not the point here.

✓ problem_user sees one identical broken image for every product
✘ problem_user sorts products by name descending
✘ problem_user sorts products by price ascending

4 passed (7.6s)

Two tests failed and the suite is green. They are locked to a real, open defect with test.fail(), so they assert the _correct_ behaviour while the bug is open. The day it is fixed, the run reports "Expected to fail, but passed." and someone has to come remove the marker — the suite notifies you instead of relying on memory (ADR-0024).

2. Why it failed, in a sentence a non-engineer can read

npm run report

Click either failing test → Annotations. Every generated test carries its Jira links, the acceptance criterion it covers, and — when it is locked to a defect — a plain-language explanation:

known defect · SW-14 — the inventory sort dropdown discards the selection for problem_user: the <select>'s own value reverts to az, so neither the active label nor the product order ever changes. This test asserts the correct behaviour and is expected to fail while the defect is open, so the suite stays green. The day it is fixed the run reports "Expected to fail, but passed." — remove the test.fail() marker in that same pull request.

3. What the app did that no test asserted on

npm run observations

Every run records console errors, uncaught exceptions, failed requests and native dialogs, then renders them as prose — network and console without opening devtools:

**0 not yet reviewed · 14 reviewed · 2 not seen when last exercised.**

#### ⚠️ failed request (third party) (reviewed)
The page asked a third-party service for `POST https://events.backtrace.io/…` and got
back **401** — the request was rejected as unauthorised.

Observations never fail a test — the fixture records, it never judges (ADR-0021). Triage is human and durable: mark an entry ignored with a note and it stays quiet across runs.

4. The full picture — DOM, network, console, step by step

trace: 'on' means every test has a trace, not just failures. In the HTML report, click a test → Traces for an inline timeline with DOM snapshots at every action — including for tests that passed, which is the point.

The trace also carries the full network log, so /report-bug extracts it as a network.har — openable in any browser's DevTools under Network → Import HAR, with no checkout and no npx. It is an extraction, not a second recording: turning on recordHar would write the same bytes twice.

5. Turn a failure into a bug report draft

With Claude Code: `/report-bug`. It assembles repro steps, the acceptance criterion the test traces to, expected vs. actual, correlated observations, and copies the screenshot, video, trace and a network.har extracted from that trace into one attachable folder. It files nothing — you read the draft and decide.

How it works — the pipeline

flowchart LR
    J["Jira ticket<br/>(project SW)"] -->|"/refine-ticket"| R["Hardened<br/>acceptance criteria"]
    R -->|"/from-issue"| G["Tests + Page Objects<br/>+ TCMS record"]
    G --> PR["GitHub Pull Request"]
    PR -->|"CI: smoke + changed specs<br/>+ typecheck + strict lint"| Rev["Human review"]
    Rev -->|merge| M["main"]
    M -->|"CI: full suite<br/>+ catalog sync"| Q["Qase TCMS<br/>(human-readable cases)"]

1. `/refine-ticket` hardens a ticket's acceptance criteria against a rubric and writes them back to Jira — so the next step has nothing to guess.
2. `/from-issue` reads the ticket through the Atlassian MCP, generates tests + Page Objects, runs them, writes a committed TCMS record, and opens a PR. It never opens a red PR: on failure it diagnoses and retries up to three times, and if the _app_ is what contradicts the criterion, it reports and opens nothing (ADR-0020).
3. CI on the PR runs the smoke set plus the specs that PR changed, behind a typecheck + strict-lint gate (ADR-0032). Smoke is unconditional, which is what keeps a change that touches no spec from running nothing.
4. A human reviews and merges — the PR is the review gate.
5. CI on merge runs the full suite, syncs the catalog to Qase, and commits the refreshed id map back.

The skills

Five Claude Code skills — four written here, one vendored from @playwright/cli.

/from-issue is the conductor: it calls /scaffold-page-object when a Page Object is missing, uses playwright-cli to confirm selectors against the live DOM, and grows the auth matrix when a ticket needs an unwired user.

Why skills, and why exactly one MCP server

The interesting part is not that this uses skills — it is that it uses one MCP server and deliberately refuses another, on a rule rather than a preference:

• A CLI, when a good one already exists. GitHub goes through gh (ADR-0007) and the browser through playwright-cli (ADR-0006) — both already installed and authenticated, more token-efficient than a server, and neither needs a PAT in the repo or an MCP registration.
• An MCP server, when there is no such CLI and OAuth beats a secret. Jira reads go through the Atlassian MCP (ADR-0011) for two named reasons: no Jira CLI is as ubiquitous as gh, and its OAuth means no API token lives in this repository.
• A skill for the workflow itself. Skills are the _procedure_ — read the ticket, scaffold, generate, run, open the PR. The CLI and the MCP are just how it reaches out.

ADR-0011 says outright that it scopes ADR-0007 rather than reversing it, which is the honest shape: gh still owns GitHub, the MCP owns Jira only. Only /refine-ticket ever writes to Jira (ADR-0013); Qase is written one-way at merge, and Slack receives scheduled-run outcomes.

What it costs: an MCP server is a connection that can drop, and it did — Jira access was lost to inactivity and needed a fresh session to reconnect. A CLI does not have that failure mode. The trade bought no secrets in the repo, and that was judged worth it.

How well it works, measured: of 16 merged /from-issue pull requests, 12 landed exactly as generated. The other four needed one follow-up commit each — extracting a component, removing a conditional from a test body, tightening a locator to an exact match, and correcting a comment that explained a correct decision with the wrong mechanism. Three caveats that keep the number honest: there is a single reviewer; a change amended into the original commit would be invisible to this count; and the fourth follow-up changed only a comment, which is a weaker miss than the other three — counted anyway, because a metric that forgives the small ones stops meaning anything.

refined Jira ticket

📋 A real example → PR #25 — _automate SW-11 burger menu scenarios_ is an actual /from-issue pull request. Its description carries the auto-generated "What I understood" summary, the AC-coverage table, and the ⚠️ Assumptions the agent flagged for review.

generated-pr

Architecture

Strict one-directional composition: tests know Pages; Pages compose Components; Components hold Locators. Tests never touch a raw Locator.

flowchart TD
    Spec["Spec — one per feature<br/>tests/.../*.spec.ts"] -->|"imports test/expect"| Fix["Fixture — @fixtures/test<br/>(injects ready Page Objects)"]
    Fix --> Pages
    subgraph Pages["Page Objects — src/pages"]
        LP[LoginPage]
        IP[InventoryPage]
        CP[CartPage]
        CK["checkout/ — Info · Overview · Complete"]
    end
    Pages -->|compose| Comps
    subgraph Comps["Components — src/components"]
        H[Header]
        F[Footer]
        CB[CartBadge]
        BM[BurgerMenu]
    end
    Comps -->|hold| Loc["Locators — data-test / getByRole"]
    Pages -.->|"page-direct locators"| Loc

• The rules are enforced, not suggested — Pages never return Pages, queries return data (never a Locator), nesting depth ≤ 2, no XPath, no waitForTimeout. ESLint fails the build on the checkable ones. Full list in CLAUDE.md and docs/architecture.md.
• Data-driven projects — playwright.config.ts derives every project from tests/users.ts AUTH_USERS (['standard', 'problem', 'error'] today), giving seven projects. Each user yields setup-<user> + chromium-<user>, plus chromium-no-auth. New users wire in on demand (ADR-0014); cross-browser adds four more, opt-in behind CROSS_BROWSER=1 (ADR-0027).
• Tags route tests to projects — @no-auth, @standard, @problem, @error, @all-users, @smoke. They live in the { tag } option, never in the title (ADR-0015) — a lint rule enforces it.

Coverage: login · inventory (content + sort) · product detail · footer · cart · checkout (information → overview → complete) · logout · burger menu.

Continuous integration

"Smoke plus changed specs" is a deliberate choice over Playwright's --only-changed, which follows the import graph and re-runs the world when a shared fixture changes; the full suite on merge is the real integration gate. A resolver that mapped changed source files back to specs was built, measured at 77 of 88 tests for one Page Object change, and deleted (ADR-0032). Each scheduled run posts pass/fail counts, duration, environment, and links to the Qase run and the report artifact.

What's captured when:

trace: 'on' buys always-on debuggability at a measured price. Measured on the standard project when it held 58 tests — it holds 65 now, and the percentage is what carries over, not the seconds — with n=10 per setting, interleaved and then repeated in reverse order to rule out drift:

+1.17s, or +8.7%. An earlier version of this section claimed recording was free; that measurement took three samples in block order and the drift between blocks inverted the result. Interleave, or do not bother.

Disk is the cheap part: 66 MB across 112 traces locally, a ~50 MB artifact in CI, 4 seconds to upload it, 7-day retention.

8.7% of a 30-second suite is a deliberate trade — reading the steps of a test that _passed_ is a real use, and no failure-only setting can serve it. If the suite ever outgrows that, the replacement is 'retain-on-failure' and not 'on-first-retry': retries are 0 locally, so 'on-first-retry' would keep no trace at all on a developer's machine.

From CI, download the `playwright-report` artifact and run npx playwright show-report ./playwright-report; the traces are bundled inside.

TCMS mirror (Qase)

Opt-in and one-way, so non-technical reviewers can browse human-readable cases. Off unless QASE_* is configured.

At merge, CI creates/updates/archives Qase cases (suite tree feature › context › bucket, steps from test.step, expected result = the ticket's AC text) and commits the refreshed id map. No run is created — merges stay noise-free. Runs are explicit: npm run qase:smoke / qase:regression. Every HTTP call to Qase lives in src/tcms/qase-client.ts and nowhere else, so Xray/Zephyr is a sibling client — though the vendor's _name_ leaks further than that, and docs/failure-modes.md measures exactly how far. Design in docs/tcms.md.

qase board

Reference

<details> <summary><strong>npm scripts</strong></summary>

</details>

<details> <summary><strong>Project structure</strong></summary>

.
├── src/
│   ├── pages/          # Page Objects (LoginPage, InventoryPage, CartPage, checkout/*)
│   ├── components/     # Reusable components (Header, Footer, CartBadge, BurgerMenu)
│   ├── fixtures/       # Playwright fixture — injects Page Objects into tests
│   ├── observations/   # Runtime observation capture, dedup index and digest
│   ├── tcms/           # Qase TCMS mirror (seam, case-mapper, sync, client)
│   └── utils/          # env config (single process.env read point), network helpers, run metadata
├── data/               # Reference data + typed loaders (@data/*)
├── tests/              # Specs (one folder per feature) + auth.setup.ts + users.ts
├── auth/               # Generated storageState (git-ignored)
├── scripts/            # Repo-level checks (documented commands, ADR invariants)
├── .claude/skills/     # AI authoring skills
├── docs/               # architecture, app/, adr/, skill guides, tcms
└── .github/workflows/  # GitHub Actions CI

</details>

<details> <summary><strong>Configuration</strong></summary>

All environment config flows through src/utils/env.ts — the single process.env read point. Copy .env.example → .env:

In CI these are GitHub Actions secrets.

</details>

<details> <summary><strong>Editing a skill</strong></summary>

ADR-0019 rests on one invariant — no markdown link inside a skill resolves outside it — and this is the whole check:

grep -rn "](\.\./\|](/\|](docs/\|](src/\|](tests/\|](data/" .claude/skills/

No output means clean. It catches both failure modes (a link escaping the repo, and a skill pointing at a sibling skill's file) and returns nothing on the current tree. Deliberately manual rather than a CI gate; ADR-0019 records why, and names the scale that would change it.

For what each skill costs in context, Claude Code ships /skill-doctor — observed usage, a different question from whether a skill is well-formed.

</details>

Documentation