Alphaavatar— agent skill

<div align="center"> <a name="readme-top"></a>

<picture> <source media="(prefers-color-scheme: dark)" srcset="./.github/banner_dark.svg"> <source media="(prefers-color-scheme: light)" srcset="./.github/banner_light.svg"> <img width="100%" alt="AlphaAvatar logo and banner" src="./.github/banner_dark.svg"> </picture>

<br />

<h3 align="center"> Learnable, configurable, and pluggable Omni Personal Assistant for everyone </h3>

<p align="center"> <a href="https://www.alphaavatar.ai"> <img src="https://img.shields.io/badge/Website-alphaavatar.ai-6366F1?style=flat-square&logo=googlechrome&logoColor=white" alt="Website"> </a> <a href="https://docs.alphaavatar.io"> <img src="https://img.shields.io/badge/Docs-Documentation-2563EB?style=flat-square&logo=readthedocs&logoColor=white" alt="Docs"> </a> <a href="https://www.alphaavatar.ai/demo"> <img src="https://img.shields.io/badge/Demo-Live-7C3AED?style=flat-square&logo=vercel&logoColor=white" alt="Demo"> </a> <a href="https://github.com/AlphaAvatar/AlphaAvatar/blob/main/ROADMAP.md"> <img src="https://img.shields.io/badge/Roadmap-Project%20Plan-F97316?style=flat-square&logo=github&logoColor=white" alt="Roadmap"> </a> </p>

<p align="center"> <a href="https://discord.gg/fAEUtSzRyK"> <img src="https://img.shields.io/badge/dynamic/json?style=flat-square&logo=discord&logoColor=white&label=Discord&color=5865F2&query=%24.approximate_member_count&suffix=%20members&url=https%3A%2F%2Fdiscord.com%2Fapi%2Fv10%2Finvites%2FfAEUtSzRyK%3Fwith_counts%3Dtrue" alt="Discord Members"> </a> <a href="https://github.com/AlphaAvatar/AlphaAvatar/discussions"> <img src="https://img.shields.io/badge/Discussions-GitHub%20Community-181717?style=flat-square&logo=github&logoColor=white" alt="GitHub Discussions"> </a> </p>

</div>

<h2>AlphaAvatar Introduction</h2>

AlphaAvatar is a self-hostable Omni Personal Assistant framework designed to evolve into an intelligent personal butler — a continuous, personalized, and proactive assistant that can remember, understand, plan, and act on behalf of the user.

It is built around a plugin-based real-time Agent architecture, combining:

• 🧠 Memory for long-term user, assistant, and tool interaction history
• 🧬 Persona for user understanding, identity continuity, and personalization
• 💡 Reflection for self-improvement and long-term behavioral adaptation
• 📅 Planning for task decomposition, reminders, and future-oriented actions
• ⚙️ Behavior for response style, workflow policy, and proactive assistance
• 🧰 Tools through MCP, RAG, DeepResearch, and external integrations
• 😊 Virtual Character for real-time voice/avatar interaction

✨ Self-hostable and privacy-first — AlphaAvatar can run on your own infrastructure, giving you control over runtime configuration, memory and tools. Selected cloud model, embedding and remote-tool providers still require their own network access and credentials; self-hosting the framework does not make those providers local.

<h2>Runtime Architecture 🧠</h2>

<p align="center"> <img src=".github/assets/alphaavatar_architecture.png" alt="AlphaAvatar Runtime Architecture" width="100%" /> </p>

AlphaAvatar follows a layered realtime multimodal architecture:

• 🎙️ Native Client Inputs: realtime text, voice, camera, and screen observations from first-party clients.
• 🏠 Application Host: assembles Agent execution and RTC adapters, coordinates their lifecycle, and hosts the application entrypoint.
• 🔌 RTC Adapter: provides LiveKit transport while keeping the application boundary replaceable; other backends are not yet implemented.
• 👁️ Core Perception: normalizes typed multimodal observations, source and segment lineage, ordered perception events, annotations, timelines, and historical snapshots.
• ⚙️ Agent & Runtime: owns Turn admission, native Loop execution, shared state, capability invocation and precise interruption. Router separately owns addressing and turn-taking decisions.
• 🧩 Plugin Ecosystem: adds Memory, Persona, RAG, MCP, Character, and other capabilities.
• 🧠 Foundation & Infrastructure: exposes ContextManager, Loop, ProviderService and VoiceBundle; structured extraction and native foreground streaming retain separate backend capabilities.
• 💾 Storage & Data: stores identity, memory, vectors, traces, artifacts, and media.
• 📤 Assistant Outputs: publishes validated EnvOutput content through OutputRuntime, with separate canonical-record and execution-observation streams. Router coordinates synthesis/transcripts; RTC owns delivery and playback feedback.

<h3>What AlphaAvatar Is Designed For</h3>

<table> <tr> <td width="50%">

<h4>1️⃣ Personal Data & Life Metrics Management</h4>

• 📊 Track and analyze personal metrics such as health, fitness, sleep, and study progress
• 📈 Provide long-term insights and trend analysis
• 🎯 Suggest improvements based on historical patterns

</td> <td width="50%">

<h4>2️⃣ Knowledge & Notes Management</h4>

• 📖 Organize personal notes, documents, and knowledge
• 🔍 Retrieve relevant information through RAG
• 🧠 Build a personal knowledge base over time

</td> </tr>

<tr> <td width="50%">

<h4>3️⃣ Task & Event Management</h4>

• 📅 Schedule tasks and reminders
• ⏰ Proactively notify based on context and priority
• 🔄 Break down long-term goals into actionable steps

</td> <td width="50%">

<h4>4️⃣ Autonomous Planning & Execution</h4>

• 🧠 Plan multi-step workflows such as learning plans, projects, and research
• 🔧 Call tools automatically to complete tasks
• 📌 Maintain consistency across long time horizons

</td> </tr>

<tr> <td width="50%">

<h4>5️⃣ Personalized Companion & Context Awareness</h4>

• 🧬 Understand user preferences, habits, and personality
• 💬 Provide highly personalized responses
• 🤝 Maintain continuity across conversations and modalities

</td> <td width="50%">

<h4>6️⃣ External World Interaction</h4>

• 🌐 Search, research, and summarize real-world information
• 🧰 Integrate with tools such as email, databases, APIs, and messaging apps
• 🔗 Act as a bridge between user intent and external systems

</td> </tr> </table>

💡 AlphaAvatar is not just a chatbot. It is a foundation for building stateful, proactive, multimodal, and self-evolving personal AI assistants.

<h2>AlphaAvatar Plugins</h2>

AlphaAvatar organizes plugins into three layers: shared foundation services, perception and interaction capabilities, and tools for external tasks.

Available means implemented in this source tree, not necessarily enabled by default or API-stable. Planned entries are not yet available as standalone plugins.

⚙️ Foundation

Shared components that other plugins use through FoundationRuntime.

FoundationRuntime owns ContextManager, Loop, ProviderService and VoiceBundle. Memory and Persona borrow structured-task gateways while retaining processor-owned configuration and business prompts. The foreground Loop explicitly selects a streaming backend; the LangChain task backend is not made stream-capable by sharing the service.

🧠 Perception

Capabilities for understanding users, maintaining memory, and coordinating interaction.

🧰 Tools

Integrations for research, document access, and external actions.

See Plugin Layout and Boundaries for development details and the Roadmap for planned work.

[plugin-voice]: avatar-plugins/foundation/avatar-plugins-voice/README.md [plugin-provider]: avatar-plugins/foundation/avatar-plugins-provider/README.md [plugin-context]: avatar-plugins/foundation/avatar-plugins-context/README.md [plugin-loop]: avatar-plugins/foundation/avatar-plugins-loop/README.md [plugin-router]: avatar-plugins/perception/avatar-plugins-router/README.md [plugin-memory]: avatar-plugins/perception/avatar-plugins-memory/README.md [plugin-persona]: avatar-plugins/perception/avatar-plugins-persona/README.md [plugin-character]: avatar-plugins/perception/avatar-plugins-character/README.md [plugin-status]: avatar-plugins/perception/avatar-plugins-status/README.md [plugin-mcp]: avatar-plugins/tools/avatar-plugins-mcp/README.md [plugin-deepresearch]: avatar-plugins/tools/avatar-plugins-deepresearch/README.md [plugin-rag]: avatar-plugins/tools/avatar-plugins-rag/README.md [status-available]: https://img.shields.io/badge/Available-28a745?style=flat [status-planned]: https://img.shields.io/badge/Planned-6c757d?style=flat [status-pending]: https://img.shields.io/badge/Pending-d29922?style=flat

<h2>Latest News 🔥</h2>

• [2026/10] Released AlphaAvatar [version 0.6.8](docs/releases/v0.6.8.md): Completed the AlphaAvatar-owned native Host and streaming Agent runtime, removed the LiveKit Agents SDK, separated RTC access from execution ownership, introduced processor-owned Persona and Memory, runtime capabilities and committed-turn consumers, synchronized PCM speech and captions, structured diagnostics, and the Python 3.13 toolchain.
• [2026/09] Released AlphaAvatar version 0.6.7: Added AlphaAvatar-owned multimodal turn taking, local Semantic Addressing with per-speaker conversation focus, typed addressing evidence and fusion, exact annotation-driven turn snapshots, full-duplex interruption, and removed LiveKit text turn detection and the legacy acoustic invocation path.
• [2026/08] Released AlphaAvatar version 0.6.6: Added a unified time-aligned multimodal runtime, adaptive audiovisual ENV Memory, provider-neutral model input, runtime capability awareness, time-based perception retention, participant-scoped timezone context, and date-grouped session storage.
• [2026/07] Released AlphaAvatar version 0.6.4: Added a transport-agnostic perception runtime with typed multimodal streams, shared timelines, annotated payload views, and online ENV memory extraction from live visual observations.
• Released AlphaAvatar version 0.6.5: Added shared realtime audio perception, the Interaction Router, AlphaAvatar-native VAD and STT, isolated per-runner inference processes, and migrated all AlphaAvatar VDB workloads away from LiveKit’s shared inference executor.

• [2026/06] Released AlphaAvatar version 0.6.0: Added the Status plugin, sampled visual input support, and status-aware DeepResearch / RAG / MCP tool feedback.
• Released AlphaAvatar version 0.6.1: Added visual identity support for Persona, including face detection, face vector matching, speaker-face identity fusion, and several bug fixes.
• Released AlphaAvatar version 0.6.2: introduced the unified provider layer, nested configuration, provider tracing, and a cleaner session/runtime foundation for future multi-user multimodal memory.
• Released AlphaAvatar version 0.6.3: Added the first graph-aware Memory foundation, including multi-object memory items, session-scoped graph node mentions, LanceDB graph-node retrieval, alias-ready graph lookup, and cleaner session-content memory extraction prompts.

• [2026/05] Released AlphaAvatar version 0.5.4:
• Added LanceDB-backed MCP tool retrieval, enabling AlphaAvatar to semantically search relevant MCP tools from Agent queries.
• Refactored system prompt and runtime prompt composition, improved Persona runtime state tracking, added temporary-user to real-user identity merging, and improved RAG runtime behavior.
• Released AlphaAvatar version 0.5.5: Fixed the inference runner registration lifecycle for production start mode, ensuring plugins runners are registered after config parsing and before LiveKit creates the inference executor.

• [2026/04] Released AlphaAvatar version 0.5.3:
• Added localized Markdown backup for the Memory plugin.
• Added LanceDB as the default local VDB option when Qdrant credentials are not provided.

• [2026/03] Released AlphaAvatar version 0.5.0:
• Added the MCP plugin, enabling retrieval and concurrent invocation of MCP tools.
• Released AlphaAvatar version 0.5.1: Added the original WhatsApp bridge via Baileys. This is a historical milestone; external messaging bridges are removed from the v0.6.8 development line.
• Released AlphaAvatar version 0.5.2: Added the AlphaAvatar Voice plugin with Voice.ai TTS support.

• [2026/02] Released AlphaAvatar version 0.4.0:
• Added RAG support through RAG-Anything.
• Optimized the Memory and DeepResearch modules.
• Released AlphaAvatar version 0.4.1: Fixed Persona plugin bugs and added a new MCP plugin.

• [2026/01] Released AlphaAvatar version 0.3.0:
• Added DeepResearch support through the Tavily API.
• Released AlphaAvatar version 0.3.1: Added tool-call memory extraction during user–assistant interactions.

<details> <summary>2025 Release History</summary>

• [2025/12] Released AlphaAvatar version 0.2.0:
• Added AIRI Live2D-based virtual character display.

• [2025/11] Released AlphaAvatar version 0.1.0:
• Added automatic memory extraction.
• Added automatic user persona extraction and matching.

</details>

<br/>

<h2>Installation ⚙️</h2>

AlphaAvatar v0.6.8 uses an AlphaAvatar-owned native Host and targets standard CPython 3.13. Install from the matching release tag and its lockfile; older released packages may have different APIs.

git clone --branch v0.6.8 --recurse-submodules https://github.com/AlphaAvatar/AlphaAvatar.git
cd AlphaAvatar

uv python install 3.13
uv sync --all-packages --locked
source .venv/bin/activate

The alphaavatar command is provided by alpha-avatar-host. No LiveKit Agents installation or separate model-download command is required. Enabled model runners resolve their files during initialization. Native model backends may require additional platform-specific build tools.

When upgrading a previous Python 3.11 checkout, create a new environment and follow the development environment migration guidance.

<h2>Quick Start ⚡️</h2>

Start the native Host in dev mode against your configured LiveKit deployment. Application and execution-process hosting belong to the native AvatarHost; foreground generation belongs to AvatarEngine and Loop.

🧩 Step 1. Configure Environment Variables

cd AlphaAvatar

# Copy template
cp .env.template .env.dev

Edit .env.dev and provide credentials for the selected RTC, model, embedding and tool services. Example model/voice identifiers are deployment choices, not guarantees of account access.

✅ Step 2. Run the Agent

Enabled inference runners resolve their model files during initialization. Missing files are downloaded automatically; no separate download command is required.

Set ALPHAAVATAR_MODEL_CACHE to choose the model cache directory. Set ALPHAAVATAR_MODEL_OFFLINE=1 to require locally cached, validated files. See Model cache for downloads and offline use, and Avatar workspace for personal data, backups and cleanup.

ENV_FILE=.env.dev alphaavatar dev examples/agent_configs/voice/pipeline_openai_tools_minimal.yaml
# or
ENV_FILE=.env.dev alphaavatar dev examples/agent_configs/mm/pipeline_openai_tools.yaml

See the Host README for startup/lifecycle boundaries and Examples README for complete configurations and their service requirements.

<h2>Usage 🚀</h2>

AlphaAvatar focuses on first-party realtime interaction. The browser demo and Developer Playground provide client surfaces; dedicated App and OS experiences remain planned.

External messaging bridges are not included in the current development scope. This does not remove native text chat or tool-based integrations with external services.

<h3>🌐 Web Access</h3>

<img src="https://img.shields.io/badge/Available-28a745?style=flat" />

AlphaAvatar now provides a browser-based realtime demo interface built on LiveKit.

👉 Try the Web Demo: https://www.alphaavatar.ai/demo

The browser client surfaces include (availability depends on the deployed version and configuration):

• 🎙️ Real-time voice interaction
• 💬 Text chat with the Avatar
• 📷 Camera preview and video-ready interaction
• 🔊 Agent audio playback
• 😊 Optional native AIRI presentation with a separately hosted frontend; renderer/RTC integration must be validated for the deployment
• 🧠 Configurable Memory, Persona, RAG, MCP and DeepResearch, subject to enabled services and execution policy
• 🌍 Browser timezone metadata, enabling AlphaAvatar to understand local login time

<p align="center"> <img src=".github/assets/web-demo-screenshot.png" alt="AlphaAvatar Web Demo Screenshot" width="100%" /> </p>

Screenshots and hosted deployments may reflect another revision. In the current native examples, AIRI-named configurations retain a voice persona, not animated Character transport.

<h3>📲 Native Mobile App</h3>

<img src="https://img.shields.io/badge/Planned-6c757d?style=flat" />

A dedicated AlphaAvatar mobile application providing:

• 🎙️ Real-time voice communication
• 😊 Live2D / Virtual character visualization
• 🧠 Persistent memory & persona

<h3>🧪 Developer Playground</h3>

<img src="https://img.shields.io/badge/Available-28a745?style=flat" />

Developers can immediately access AlphaAvatar via the LiveKit Playground.

👉 https://agents-playground.livekit.io/

After starting your AlphaAvatar server:

1. Connect to your LiveKit instance
2. Configure the Agent name to match avatar.name in the selected YAML (the supplied examples use AlphaAvatar) for Explicit Dispatch.
3. Connect to the agent room
4. Start testing real-time interaction

Supported capabilities:

• 🎙️ Voice interaction
• 🧠 Memory extraction
• 🔍 RAG retrieval
• 🧰 MCP tool invocation
• 😊 Character presentation is opt-in and requires the separate frontend setup; it is not enabled by the supplied configurations

playground airi screenshot

💡 AlphaAvatar is currently developer-first, with a Web Demo available for realtime interaction.

More user-facing web and mobile experiences are under active development.