CUE — Computer Use Enforcer

English | 한국어

The missing safety layer between AI agents and your desktop.

CUE is safety middleware for AI computer use agents. It monitors, enforces policies, and provides guardrails so that AI agents can interact with your desktop — safely and compliantly.

Problem

AI computer use is accelerating fast, but safety infrastructure hasn't kept up:

• Rapid adoption — 40% of enterprises plan to deploy AI computer use agents by end of 2026
• No guardrails — only 50% of organizations have any safety controls for autonomous agents
• Regulation is coming — the EU AI Act (Aug 2026) mandates human oversight for high-risk AI systems

There is no open-source framework that sits between AI agents and the desktop to enforce safety policies. CUE fills that gap.

What CUE Does

┌─────────────────────────────────┐
│  Any AI Agent                   │  Claude, GPT, Agent-S, ...
│  (natural language in/out)      │
└──────────┬──────────────────────┘
           │ MCP protocol (stdio)
┌──────────▼──────────────────────┐
│  CUE — Computer Use Enforcer   │  ← this project
│  ┌────────────────────────────┐ │
│  │ Policy Engine              │ │  risk classification, action filtering
│  │ Guardrails                 │ │  app blocklist, key blocking, rate limits
│  │ Audit Logger               │ │  JSONL compliance trail
│  │ Monitor (coming soon)      │ │  real-time dashboard & event streaming
│  └────────────────────────────┘ │
└──────────┬──────────────────────┘
           │ pyautogui / pygetwindow / pywin32
┌──────────▼──────────────────────┐
│  Desktop OS                     │
└─────────────────────────────────┘

Core Value Proposition

Quick Start

1. Install

git clone https://github.com/yonghwan1106/computer-use-engine.git
cd computer-use-engine
pip install -e .

2. Register with Claude

python scripts/register.py

This automatically adds CUE to both Claude Desktop and Claude Code configurations.

3. Restart Claude and go

Restart Claude Desktop or Claude Code. Then just ask:

"Take a screenshot of my screen"

"Open Notepad and type 'Hello, CUE!'"

"Show me all open windows"

Current Features

MCP Tools (12)

#### Screenshot & Screen

#### Mouse

#### Keyboard

#### Window Management

Safety Features

Safety Policy

All safety settings are configurable in config/default.yaml:

safety:
  max_actions_per_session: 100
  action_delay: 0.05
  failsafe: true
  allowed_apps: []
  blocked_apps:
    - "Windows Security"
    - "Registry Editor"
    - "Task Manager"
  blocked_keys:
    - "win+r"
    - "alt+f4"
    - "ctrl+alt+del"

Manual Registration

If you prefer to configure manually instead of using register.py:

Claude Desktop — edit %APPDATA%/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "cue": {
      "command": "python",
      "args": ["-m", "cue"]
    }
  }
}

Claude Code — edit ~/.claude/settings.json:

{
  "mcpServers": {
    "cue": {
      "command": "python",
      "args": ["-m", "cue"]
    }
  }
}

Project Structure

computer-use-engine/
├── cue/
│   ├── __init__.py            # Package version
│   ├── __main__.py            # python -m cue entry point
│   ├── server.py              # FastMCP server initialization
│   ├── tools/
│   │   ├── screenshot.py      # Screen capture tools (3)
│   │   ├── mouse.py           # Mouse control tools (4)
│   │   ├── keyboard.py        # Keyboard input tools (2)
│   │   └── window.py          # Window management tools (3)
│   ├── safety/
│   │   ├── guardrails.py      # Action limits, app/key blocking
│   │   └── logger.py          # JSONL audit logger
│   ├── core/                  # Policy engine, risk scoring (Phase 1)
│   ├── monitor/               # Real-time dashboard (Phase 2)
│   ├── adapters/              # Agent-agnostic backends (Phase 4)
│   └── utils/
│       ├── screen.py          # DPI awareness, image processing
│       └── keymap.py          # xdotool → pyautogui key mapping
├── config/
│   └── default.yaml           # Safety configuration
├── scripts/
│   └── register.py            # Auto-registration for Claude
├── tests/                     # Unit tests
├── pyproject.toml
├── LICENSE                    # Apache 2.0
└── README.md

Development

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Run the server directly (stdio mode)
python -m cue

Roadmap

Requirements

• Python 3.11+
• Windows 10/11
• Claude Desktop or Claude Code with MCP support

Contributing

Contributions are welcome! Please see LICENSE for details.

License

Apache 2.0