Harness Engineering

Design the system around AI agents for reliable, safe production use.

Harness Engineering protects the quality ceiling of Agentic Engineering. It turns fast agent output into controlled execution through permissions, observability, recovery, and adversarial validation.

Agent Runtime Model

Treat the agent harness as the kernel around an LLM OS:

Productized Agent Harness

Google I/O '26 added a practical pressure test for harness design: the same runtime pattern now appears in developer tools, personal agents, search, commerce, generative media, and smart glasses.

If a harness cannot produce an audit trail for what the agent saw, decided, called, changed, and verified, the agent is not ready for delegated action.

Managed Agent Runtime Model

For production-like agents, separate the runtime into three resources:

The session event log is the backbone of reliability. It should capture user messages, tool calls, tool results, agent responses, verification evidence, errors, and recovery actions. A response without an inspectable event trail is not enough for delegated production work.

Permission Bike Method

Escalate autonomy by proven reliability, not by confidence in a prompt:

Do not give a tool key and rely on text instructions to prevent misuse. The real boundary is the key, endpoint, account, filesystem, network, budget, and approval path.

Usage Template

Prompt

Use harness-engineering for this agent workflow. Design permissions, tools, feedback loops, observability, and failure handling.

Use Case

• Moving an agent workflow from ad hoc prompting toward a reliable runtime architecture.

Expected Result

• The agent produces a harness design with permission tiers, tool boundaries, logs, evals, and recovery paths.

Output Example

• A runtime spec with permission matrix, tool allowlist, approval gates, logs, evals, and incident response.

Verification Case

• The design names what the agent can do automatically, what needs approval, and what is denied.

Verified Effect

• An ad hoc agent workflow becomes a controlled runtime with explicit permissions, observability, and failure handling.

Success Metrics

• Design specifies permissions, tool contracts, observability, failure handling, and recovery path.
• High-risk actions have approval or sandbox boundaries.
• Verification evidence is defined before deployment or automation.

When to Use

• Deploying agents to production
• Setting up permissions/guardrails/approval workflows
• Designing multi-agent systems
• Agent behaved unpredictably → needs better constraints
• Configuring auto-mode or permission tiers

Three Domains

Six Components

1. Context & Knowledge Layer

• Curated access to code, docs, schemas, logs
• Use CLAUDE.md for project-level context
• Use context-manager for token budgeting
• Never inject raw 10K+ token files
• Persist reusable outputs to wiki, docs, logs, or state files; chat history is not durable memory

2. Tooling & API Surface

Three-Tier Permission Model:

Tier 3 heuristic:

1. Can destroy data irreversibly? → BLOCK
2. Accesses credentials? → BLOCK
3. Affects shared infrastructure? → BLOCK
4. Target inferred (not explicit)? → BLOCK

Delegated action gate:

Before any tool call that buys, publishes, schedules, messages, edits shared state, or acts through a user's account, require:

Mandate: what the user explicitly authorized
Scope: allowed accounts, surfaces, vendors, files, or domains
Limit: budget, time, rate, data, or blast radius cap
Preview: what the user can inspect before execution
Receipt: durable audit record after execution
Rollback: how to undo or compensate if wrong

3. Architectural Constraints

• Linters, structural tests
• File/directory access boundaries
• Token budget: never exceed 80% context window

4. Feedback & Validation Loops

• Write-Test-Fix cycle
• Generator + Evaluator pattern (GAN-inspired)
• Deny-and-continue: try safer alternative on block
• Escalate to human after 3 consecutive blocks
• For high-risk work: Builder -> Evaluator -> Red Team -> Fixer -> final proof

5. Observability & Governance

• Log every tool call + result
• Metrics: success rate, revert rate, token usage
• Cost tracking per session
• Evidence ledger: verification command, source link, screenshot, test result, or diff for each completion claim
• Provenance ledger for generated media, search summaries, and commerce decisions: source, prompt/edit trail, model/tool used, and user confirmation point

6. Maintenance

• Periodic agents: dead code, outdated docs, architectural drift
• Wiki lint for broken links/missing frontmatter
• Context garbage collection: summarize completed work, close idle processes, and record residual risk

GAN-Inspired Multi-Agent Pattern

"Tuning a standalone evaluator to be skeptical is far more tractable than making a generator critical of its own work."

Generator ←──sprint contract──→ Evaluator
    │                              │
    ▼                              ▼
 Produces output              Playwright/Test

Implementation:

1. Planner → 1-sentence prompt → full spec
2. Generator → builds in sprints
3. Evaluator → tests, files bugs
4. Feedback loop → 5-15 iterations

Physical → Cognitive Analogies

Multi-Agent Patterns

Permissioned Tool Design

Define every tool like a system call:

Prefer narrow tools with explicit inputs over broad shell/API access. If broad access is unavoidable, wrap it with approval gates and post-action verification.

Security-Aware Integration

Before shipping high-risk agent output, run an adversarial review:

Use simulated hostile agents, security tests, or a skeptical evaluator before release. Do not let the same builder be the only judge of safety.

Closed-Loop System (Aladdin)

Data Lake → Risk Engine → Optimizer → Stress Test → OMS → Compliance → Feedback Loop
     ↑                                                              ↓
     └──────────────────── 反馈闭环 ────────────────────────────────┘

Key: Closed loop is the moat — not any single component. MECE principle applies.

Quality Gates

• [ ] Each agent has defined permission scope
• [ ] Agent, Environment, and Session responsibilities are separated
• [ ] Three-tier permission model documented
• [ ] Permission escalation follows observe -> co-drive -> scoped action -> monitored autonomy
• [ ] Write-Test-Fix feedback loop in place
• [ ] Tool calls logged and observable
• [ ] Token costs tracked per session
• [ ] Delegated actions have mandate, scope, limit, preview, receipt, and rollback
• [ ] Generated media and public claims have provenance or disclosure path
• [ ] High-risk actions require approval
• [ ] Multi-agent isolation prevents interference
• [ ] Generator and Evaluator separate for complex tasks
• [ ] High-risk outputs receive adversarial or red-team review
• [ ] Reusable outputs have durable write-back outside chat history
• [ ] Tool contracts include bounds, failure path, evidence, and audit trail