zuowo-s-record (做我所录)

Record once, replay forever. Windows desktop automation toolkit for AI agents.

Overview

A skill for AI agents that need to automate Windows GUI workflows. The agent reads a recording file (JSON + screenshots) and emits a workflow.json that the bundled smart_replay engine can execute deterministically.

┌─────────┐    ┌─────────┐    ┌─────────┐
│  Record │ →  │ Design  │ →  │ Replay  │
│  (exe)  │    │ (agent) │    │  (CLI)  │
└─────────┘    └─────────┘    └─────────┘
   JSON + PNG     workflow.json    run actions

When to use this skill

Trigger this skill when the user wants to:

• Automate a repetitive Windows GUI task
• Convert a manual click-by-click workflow into a script
• Reproduce a recorded sequence on demand

If the task is one-off or already covered by a CLI / API, this skill is overkill.

How it works

1. Record (human + recorder)

The user launches app/ActionRecorder_v2.exe and performs the task once:

• F9 start
• F10 stop
• F8 pause

The recorder writes to app/recordings/<timestamp>/:

• recording_<timestamp>.json — event stream (clicks, drags, keys, types, scrolls)
• full_NNNN.png — full screenshots
• zoom_NNNN.png — 200x200 center crops (used for visual re-anchoring during replay)

2. Design (AI agent)

The agent reads the recording JSON, examines the screenshots, and writes a workflow.json next to the recording:

• workflow.json — machine-readable plan: intent + steps + acceptance
• tools.md — tool inventory (human-readable)
• dev_log.md — design notes (human-readable)

The schema is defined in references/schema.md. Agent 速成路径: 先看 references/schema.md §0 v3.1 范式速成 (5 分钟上手), 再查字段细节 (§1+). A minimal example:

{
  "meta": {
    "workflow_name": "open_notepad",
    "version": "1.0",
    "created_at": "2026-06-05T23:00:00",
    "author": "<agent-name>",
    "description": "Open Notepad and type a greeting",
    "recording": "recording_20260605_225500.json"
  },
  "intent": {
    "goal": "Open Notepad and type 'Hello world'",
    "success_looks_like": "Notepad window is open with the text typed in"
  },
  "steps": {
    "reproduce": [
      {
        "step": 1,
        "action": "launch",
        "params": {"target": "notepad.exe"},
        "description": "Launch Notepad"
      },
      {
        "step": 2,
        "action": "wait",
        "params": {"duration": 1.0},
        "description": "Wait for Notepad to open"
      },
      {
        "step": 3,
        "action": "type",
        "params": {"text": "Hello world"},
        "description": "Type the greeting"
      }
    ]
  }
}

3. Replay (CLI or library)

Two ways to execute a workflow.json:

CLI (no Python required):

python tools/replay.py run path/to/workflow.json
python tools/replay.py validate path/to/workflow.json   # check required fields
python tools/replay.py info path/to/workflow.json       # show plan summary
python tools/replay.py tools                            # list available actions

Python library (programmatic):

import sys
sys.path.insert(0, "tools")
from smart_replay import ActionExecutor, Logger, snap

logger = Logger("my_run")
executor = ActionExecutor(logger=logger, snap_func=snap)

for step in workflow["steps"]["reproduce"]:
    executor.execute_action(step)

Available actions

The smart_replay engine supports 15 GUI actions. 5 mirror recorder events 1:1; 10 are added by the AI agent during workflow design (launches, window focus, waits, hotkeys, etc.) — see references/event_mapping.md for the full mapping.

Use python tools/replay.py tools to print the live list.

Failure handling

The engine supports per-step retry with these strategies:

• simple — re-execute the step up to max_attempts
• scan_nearby — re-anchor coordinates via template matching against img_zoom (within tolerance_px)
• reboot — re-activate the target window, then re-execute

Failures write a snapshot to disk and a structured log line so the agent (or the user) can diagnose from outside.

File layout

zuowo-s-record/
├── SKILL.md                          ← this file (AI agent entry point)
├── README.md                         ← project overview (for humans)
├── CHANGELOG.md                      ← version history
├── CONTRIBUTING.md                   ← how to contribute
├── LICENSE                           ← MIT license
├── requirements.txt                  ← Python dependencies
├── .github/
│   └── workflows/ci.yml              ← CI: runs the 63 tests on push / PR
├── app/
│   ├── ActionRecorder_v2.exe         ← Windows recorder (PyInstaller bundle, ~66 MB)
│   ├── ActionRecorder_v2.exe.bak     ← previous EXE (kept for emergency rollback; not in repo)
│   ├── ActionRecorder_v2.spec        ← PyInstaller spec
│   ├── recorder_settings.json        ← user's recorder config (gitignored, contains local paths)
│   ├── recorder_settings.json.example← recorder config template
│   └── recordings/                   ← generated recordings
│       └── <timestamp>/              ← one folder per recording session
│           ├── recording_<ts>.json   ← raw pynput event stream (gitignored, regenerated)
│           ├── full_NNNN.png         ← full-window screenshots (gitignored, large)
│           ├── zoom_NNNN.png         ← 200x200 center crops (gitignored)
│           ├── workflow.json         ← AI-designed replay plan (in repo, v3.1 layout)
│           ├── tools.md              ← AI tool inventory (in repo)
│           └── dev_log.md            ← AI design notes (in repo)
├── src/
│   ├── recorder_v2.py                ← recorder source
│   └── settings_dialog.py            ← recorder settings dialog
├── tools/
│   ├── smart_replay.py               ← Python replay engine (15 actions, ~1000 lines)
│   ├── app_launcher.py               ← open_app 5-layer fallback (Win10/11 UWP-safe)
│   ├── app_index.json                ← generated Start Menu index (gitignored)
│   └── replay.py                     ← CLI entry point (`run` / `validate` / `info` / `tools`)
├── tests/
│   ├── test_smart_replay.py          ← engine unit tests (37 tests)
│   ├── test_app_launcher.py          ← open_app fallback tests (11 tests)
│   └── test_replay_cli.py            ← CLI end-to-end tests (15 tests)
└── references/
    ├── schema.md                     ← workflow.json schema (post-v3.1.1)
    └── event_mapping.md              ← recorder event → smart_replay action mapping

Limitations

• Windows only — recorder and replay both use Win32 APIs (pyautogui + ctypes)
• Coordinate-based — replay uses recorded mouse coordinates; dynamic UIs

need scan_nearby retry or human supervision

• No scheduler — this skill does not run workflows on a cron; trigger it

explicitly when needed

• Requires an AI agent to design workflows — the recorder captures events,

but turning them into a workflow.json requires an LLM to read the JSON + screenshots and emit the plan. run itself does not need an LLM.

License

MIT