evaluating-as-user — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluating-as-user (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Teach the methodology and judgement needed to evaluate a running product as a real user would. Defines how to derive what to test from the PRD, how to structure the evaluation, what quality dimensions to assess, and how to format findings.
This skill covers what to evaluate and how to think about quality. Browser interaction mechanics are in playwright-cli (preloaded). Agent role, inputs, status codes, and handoff are in the agent definition.
Applies to: web applications, CLI tools, APIs.
Does not cover: source code review (reviewing-code), automated test generation, accessibility audits (WCAG), or performance benchmarking.
Read the PRD. For each P0 requirement, derive one user journey using this structure:
| Field | Description |
|---|---|
| Starting point | Where the user begins (URL, command, state) |
| Actions | Step-by-step sequence the user would take |
| Observable outcome | What confirms the requirement is satisfied |
How to derive actions: Treat the requirement as a goal, not a task. Ask: what would a person with this goal actually do? Start from the entry point a new user would reach, not the most direct technical path.
Priority: P0 requirements first. Add P1 journeys only if time permits after all P0 journeys and universal checks are complete.
Non-web projects: For CLI tools, the starting point is the terminal and the actions are command sequences. For APIs, the starting point is an endpoint and the actions are the sequence of requests a user of the API would make. The journey structure is the same.
Attempt each derived journey in sequence. As you go:
walkthrough.md (what you did, what you expected, what you saw)When a journey succeeds, note which quality dimensions (Step 4) were satisfied or strained along the way. When a journey fails, record at what step and what the failure was.
If a severity-4 issue is encountered mid-journey: record it, then attempt to continue or skip to the next journey. Do not abandon the evaluation.
After PRD journeys, run these checks on every product regardless of what the PRD specifies:
| Check | What to do | What to look for |
|---|---|---|
| First landing | Load the primary screen fresh | Does the product communicate its purpose within 5 seconds? Is there a clear visual hierarchy? Does the user know what to do next? |
| First-time empty state | Access the product as a new user with no data | Are empty views explained and actionable, or blank and confusing? |
| Error paths | Submit invalid input, trigger validation, attempt impossible actions | Are errors in plain language with a clear fix suggestion? Do they appear where the user is looking? |
| Zero-results | Search or filter for something that will not exist | Is there a clear message that the search worked but found nothing? Is there an actionable next step? |
| Visual consistency | Navigate to three or more different views | Do colour, spacing, typography, and interaction patterns hold consistent throughout? |
| Feedback absence | Trigger any async action (form submission, data save, deletion) | Does the product visibly confirm that the action happened? |
Non-web CLI tools: Replace first landing with "first run of the tool with no arguments or --help". Replace visual consistency with "consistency of output format, flag naming, and error message style". Skip screenshot-dependent checks.
APIs: Replace first landing with "calling the root endpoint or documentation endpoint". Replace visual checks with "consistency of response envelope structure, error format, and status code usage".
Note: These dimensions are a lens you apply throughout Steps 2, 3, and 5 — not a separate sequential activity. Return here as a reference when assessing findings, not as a distinct pass after universal checks.
For each journey and each universal check, assess the product across these five dimensions. You do not need a score per check — use the dimensions as a lens to ensure findings cover the full experience, not just functional failures.
| Dimension | The question | What failure looks like |
|---|---|---|
| Communicativeness | Does the product tell me what's happening? | Silent submissions, missing loading states, absent feedback, cryptic error codes |
| Learnability | Can I figure out how to use it without help? | Jargon, non-standard navigation patterns, no contextual guidance, forcing users to memorise |
| Reliability feel | Does it feel trustworthy and consistent? | Visual inconsistencies, unpredictable behaviour, poor error recovery, performance anxiety |
| Efficiency | Does it respect my time? | Unnecessary steps, buried primary actions, no defaults, repetitive data entry |
| Emotional tone | Does it feel crafted or careless? | Unstyled edge states, ignored empty states, missing microinteraction feedback, placeholder text in production |
Treat feedback absence as severity 3 by default. If a user action produces no visible response, that is a major finding unless there is a clear reason it does not need one.
After journeys and universal checks, use the product freely as a real person would. Navigate, go back, try things the PRD did not specify, use it at an unexpected pace. This layer surfaces issues no checklist captures.
Focus exploratory time on:
Record what you try and what you notice. Not all observations become findings — use judgement to decide what matters.
If a design system file was provided, compare the rendered product against it:
Report deviations as findings. Use severity 2 for cosmetic drift, severity 3 for systematic token violations or broken interactive states.
Write each finding using this format:
**[SEVERITY-N]** Location (screen, view, component, or command)
_Finding:_ What the issue is, described as a user would experience it.
_Impact:_ Why it matters — what the user loses or cannot do.
_Suggestion:_ A specific, actionable fix the developer can implement.Group findings in findings.md by severity, highest first. Include references to walkthrough steps and screenshots.
| Level | Meaning | Example |
|---|---|---|
| 4 Catastrophic | Prevents task completion | Core feature does not work; user cannot achieve their goal |
| 3 Major | Significant user impact — important to fix | Confusing flow leads users to a dead end; form submits silently; primary action is hidden |
| 2 Minor | Causes friction but does not block | Missing loading indicator; inconsistent button labels; empty state with no guidance |
| 1 Cosmetic | Fix if time permits | Slightly inconsistent spacing; minor colour drift from design system |
Honesty rule: Do not inflate severity. A cosmetic issue is not catastrophic because it is frequent. A major issue is not cosmetic because it is hard to fix.
A finding is actionable when a developer reading it knows exactly what to change. Test your findings against these criteria:
Vague finding (not actionable): "The onboarding experience is confusing."
Actionable finding: **[SEVERITY-3]** Onboarding — Step 2 (account setup) — After clicking "Create account", no confirmation or loading indicator appears for ~3 seconds. _Impact:_ Users assume the click did not register and click again, creating duplicate submissions. _Suggestion:_ Show a spinner or disabled state on the button immediately on click; display a "Creating your account..." message.
PRD has three P0 requirements. The app is running at localhost.
Expected: Agent derives three user journeys (one per P0), runs each in the browser using playwright-cli, applies all six universal checks, assesses all five quality dimensions during the walk, runs exploratory evaluation, and produces a walkthrough.md and findings.md with severity-rated, actionable findings.
Example walkthrough entry (Step 2):
Journey 1, Step 3 — Submit registration form
Did: Filled in name, email, and password fields; clicked "Create account"
Expected: Loading state on button, then redirect to dashboard or confirmation screen
Saw: Button remained active with no visual change for ~4 seconds, then page refreshed to the same form with no error message
Quality lens: Communicativeness failure (no feedback); Reliability feel strained (unpredictable outcome)PRD describes a CLI utility with two core commands. No system.md.
Expected: Agent derives journeys as command sequences, runs them in the terminal, applies universal checks adapted for CLI (first run with no args, error message quality, output consistency, feedback on completion). Skips design compliance step. Produces findings with terminal output evidence in walkthrough.md instead of screenshots.
Example finding (CLI-adapted format):
**[SEVERITY-3]** CLI — first run with no arguments
_Finding:_ Running `mytool` with no arguments exits silently with code 1. No usage information, no hint that `--help` exists.
_Impact:_ A new user has no way to discover how to use the tool without consulting external documentation. Standard CLI convention is to print usage on invocation with no args.
_Suggestion:_ When invoked with no arguments, print a short usage summary (same output as `--help`) before exiting. Exit code 0 for informational output; exit code 1 only for genuine errors.
_Evidence:_ walkthrough.md step 1 — terminal output:
$ mytool
$ echo $?
1During journey 1, the primary feature crashes with an unhandled exception.
Expected: Agent records the severity-4 finding, notes the step at which evaluation was blocked, attempts any remaining steps or alternative paths, continues with remaining journeys and universal checks (rather than abandoning entirely), and returns DONE_WITH_CONCERNS with the catastrophic finding highlighted.
Evaluating the code, not the product. If you find yourself reasoning about implementation, stop. You are the user — you observe what happens, not why.
Judging intent, not experience. "The developer clearly intended X" is not relevant. "As a user, I encountered Y" is.
Over-inflating severity. Reserve 4 for genuine task blockers. Reserve 3 for issues that meaningfully harm the experience or cause user error. Not everything is major.
Skipping empty states because you filled them with test data. Explicitly test the zero-data view: create a new account, clear your test data, or access the product as a fresh user.
Only reporting failures. The walkthrough should document what works too — both as context for findings and as confirmation that the evaluation was thorough.
Dismissing visual inconsistency as cosmetic. Research shows visual quality is a proxy for product quality — users who see inconsistent spacing or mismatched colours assume the underlying product is unreliable too. Rate visual coherence issues at severity 2 minimum.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.