🐟 Bonito

Unified AI operations — governance, routing, cost management, and deployment across any AI provider.

Bonito gives engineering and platform teams a single operational layer to manage AI workloads across AWS Bedrock, Azure AI Foundry, Google Vertex AI, and more. Connect providers, enforce governance policies, track costs in real time, and manage team access — all from one platform with an AI copilot that helps you move faster.

Why Bonito?

AI adoption is accelerating, but operational tooling hasn't kept up. Teams juggle separate consoles for each cloud provider, have no unified view of costs, and struggle to enforce consistent governance across providers.

Bonito solves this with:

• Operational control — One dashboard for all your AI providers. Manage models, deployments, and routing policies without switching between cloud consoles.
• Governance & compliance — Built-in policy engine for SOC-2, HIPAA, and GDPR compliance checks. Audit logging across every action.
• Cost visibility — Real-time cost aggregation, forecasting, and optimization recommendations across all providers.
• AI Context (Knowledge Base) — Cross-cloud RAG pipeline. Upload company docs, embed with any provider's model, and inject context into any LLM query — vendor-neutral knowledge that works with every model on every cloud.
• Team management — Role-based access control, team seats, and SSO/SAML for enterprise identity management.
• SAML SSO — Enterprise single sign-on with SAML 2.0. Supports Okta, Azure AD, Google Workspace, and custom SAML providers. SSO enforcement, break-glass admin, JIT user provisioning.
• AI copilot — An intelligent assistant that helps with onboarding, configuration, troubleshooting, and infrastructure-as-code generation.
• Multi-cloud gateway — OpenAI-compatible API proxy with intelligent routing, automatic cross-region inference profiles (AWS Bedrock us. prefix handled transparently), and multi-provider failover that catches rate limits, timeouts, 5xx errors, and model unavailability to automatically route to equivalent models on other providers.
• Bonobot — AI Agents — Enterprise AI agent framework with visual canvas (React Flow), project-based organization, built-in tools (KB search, HTTP requests, agent-to-agent invocation), and enterprise security (default deny, budget enforcement, rate limiting, SSRF protection, full audit trail). All agent inference routes through the Bonito gateway for cost tracking and governance.
• Shared Conversational Memory (Memwright) — Automatic per-session memory using SQLite + ChromaDB. Agents recall relevant context before inference and store conversation turns after responding. Enables referential follow-ups ("the third one", "from the list above"). Model tier gating prevents small models from hallucinating on memory context.
• Persistent Agent Memory — Long-term memory system with pgvector similarity search. Agents store and retrieve facts, patterns, interactions, preferences, and contextual information across sessions. AI-powered memory extraction from conversations.
• Org Secrets Store — Secure org-scoped key-value storage backed by HashiCorp Vault. Secrets are injected into agent system prompts at runtime. Full API, CLI, and YAML deploy support.
• VectorBoost (KB Compression) — Adaptive vector compression for knowledge base embeddings. 3.9x to 8x storage reduction with 95-99.5% recall preservation. Three compression methods with API/CLI/YAML configuration.
• Scheduled Autonomous Execution — Cron-based task scheduling with timezone support. Agents run tasks automatically, deliver results via webhook/email/Slack, with full execution history and retry logic.
• Approval Queue / Human-in-the-Loop — Configurable approval workflows for sensitive agent actions. Risk assessment, timeout handling, auto-approval conditions, and comprehensive audit trails for enterprise compliance.
• Origami — conversational platform interface (Phase 1 skeleton, in flight) — Split-pane chat at /origami that knows your org, your tier, and the full platform. Ask in plain English ("what providers do I have?", "build me a support agent backed by these docs"), get a plan card, click Deploy. Built as a customer of Bonito's own gateway — Origami POSTs to /v1/chat/completions like any external user would, proving the platform on every interaction. Tier-gated with upgrade-in-place via inline Stripe Checkout when a build needs a feature your plan doesn't include. Full spec in docs/ORIGAMI-MVP-PLAN.md.

How Bonito Compares

We're not the only platform in this space. Here's an honest look at how we fit:

Where Bonito shines:

• Auto cross-region inference profiles -- Customers register canonical model IDs (e.g. anthropic.claude-sonnet-4-20250514-v1:0) and the gateway transparently routes via AWS cross-region inference profiles (us. prefix) when required. No competitor handles this automatically. When AWS changes their inference profile scheme, you update one function on the platform -- zero customer impact.
• Intelligent multi-provider failover -- Not just rate-limit retries. Bonito detects rate limits, timeouts, server errors (5xx), model unavailability, and overloaded providers, then automatically routes to equivalent models on other providers. A Claude Sonnet request that gets throttled on Anthropic Direct automatically retries on AWS Bedrock. No client code changes needed. Portkey and LiteLLM offer basic retry/fallback, but not cross-provider model equivalence mapping with transparent re-routing.
• Cross-cloud RAG (no competitor has this), integrated governance, IaC generation, and an AI copilot that ties it all together.

Where others lead: Provider breadth (Portkey/LiteLLM support far more providers today), open-source community (LiteLLM), and compliance certifications (Portkey and Helicone have SOC-2 today).

Quick Start

# Clone the repo
git clone <repo-url> && cd bonito

# Copy env file
cp .env.example .env

# Start everything
docker compose up --build -d

# Run database migrations
docker compose exec backend env PYTHONPATH=/app alembic upgrade head

# Open the app
open http://localhost:3001

Architecture

┌─────────────────────────────────────────────────┐
│                   Frontend                       │
│          Next.js 14 · TypeScript · Tailwind      │
│              shadcn/ui · Framer Motion           │
│                  localhost:3001                   │
└──────────────────────┬──────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────┐
│                   Backend                        │
│           FastAPI · Python 3.12 · Async          │
│                  localhost:8001                   │
├──────────┬───────────┬───────────┬──────────────┤
│PostgreSQL│   Redis   │   Vault   │ Cloud APIs   │
│ pgvector │   :6380   │   :8200   │ Bedrock etc  │
│  :5433   │           │           │              │
└──────────┴───────────┴───────────┴──────────────┘

Tech Stack

Project Structure

bonito/
├── frontend/              # Next.js app
│   └── src/
│       ├── app/           # App Router pages
│       └── components/    # UI components
├── backend/               # FastAPI app
│   ├── app/
│   │   ├── api/           # Route handlers
│   │   ├── core/          # Config, DB, Vault client
│   │   ├── models/        # SQLAlchemy models
│   │   ├── schemas/       # Pydantic schemas
│   │   └── services/      # Business logic
│   └── alembic/           # DB migrations
├── vault/                 # Vault init scripts
├── secrets/               # SOPS encrypted secrets
├── docker-compose.yml
└── README.md

Services

Secrets Management

Local dev: SOPS + age for encrypted secrets in git.

# Decrypt secrets
SOPS_AGE_KEY_FILE=secrets/age-key.txt sops decrypt secrets/dev.enc.yaml

# Edit secrets
SOPS_AGE_KEY_FILE=secrets/age-key.txt sops edit secrets/dev.enc.yaml

Vault UI: http://localhost:8200 (token: bonito-dev-token)

Production: HashiCorp Vault with AppRole/Kubernetes auth, HA mode.

API Docs

With the backend running: http://localhost:8001/docs (Swagger UI)

Roadmap

All 18 core phases are complete. Bonito is live at getbonito.com with 12 active deployments across 3 clouds and 171+ gateway requests tracked.

Completed (All 18 Phases) ✅

• ✅ Core platform (auth, RBAC, multi-cloud connections)
• ✅ Cloud integrations (AWS Bedrock, Azure AI Foundry, GCP Vertex AI)
• ✅ AI-powered chat & intelligent routing
• ✅ Compliance & governance engine (SOC-2, HIPAA, GDPR policy checks)
• ✅ Cost intelligence (aggregation, optimization, forecasting)
• ✅ Production deployment (Docker, CI/CD, deployment configs)
• ✅ Onboarding wizard with IaC template generation
• ✅ API Gateway (OpenAI-compatible proxy via LiteLLM)
• ✅ AI Copilot (Groq-powered operations assistant)
• ✅ Engagement & retention (notifications, analytics, digests)
• ✅ Model details & playground (live testing, parameter tuning)
• ✅ Visual routing policy builder (A/B testing, load balancing)
• ✅ Deployment provisioning (cloud endpoints, Terraform, auto-scaling)
• ✅ AI Context (Knowledge Base) — Cross-cloud RAG pipeline with pgvector, document upload/parse/chunk/embed, HNSW vector search, gateway context injection, and source citations
• ✅ Database migration to pgvector PG18.2
• ✅ AI Context onboarding integration (optional KB toggle, storage provider picker)
• ✅ IaC templates updated with KB storage permissions (S3, Azure Blob, GCS)
• ✅ One-click model activation across all 3 clouds

Completed (Recent)

• ✅ Shared Conversational Memory (Memwright) — Automatic per-session memory for Bonobot agents. Recalls relevant context before inference, stores conversation turns after response. Model tier gating (zero memory for small/fast models). 23 tests across 6 groups.
• ✅ Org Secrets Store — Secure org-scoped key-value storage backed by HashiCorp Vault. API, CLI, YAML deploy support. Runtime injection into agent system prompts.
• ✅ VectorBoost (KB Compression) — Adaptive vector compression (3.9-8x reduction, 95-99.5% recall). Scalar 8-bit, Polar 8-bit, Polar 4-bit methods. API, CLI, YAML support.
• ✅ Auto Cross-Region Inference Profiles — Gateway automatically detects newer Bedrock models (Claude Sonnet 4, Opus 4, Llama 3.3/4, Mistral Large 2) and routes via us. cross-region inference profiles. Customers register canonical model IDs; the platform handles routing transparently.
• ✅ Intelligent Multi-Provider Failover — Gateway detects rate limits, timeouts, server errors (500/502/503), model unavailability, and capacity issues, then automatically retries on equivalent models across different providers (e.g. Anthropic Direct -> AWS Bedrock). Model equivalence mapping covers Claude, Llama, Mixtral, and Gemma families.
• ✅ SAML SSO — Enterprise SSO with SAML 2.0 (Okta, Azure AD, Google Workspace, Custom SAML), SSO enforcement, break-glass admin, JIT provisioning
• ✅ Bonobot v1 — AI Agents — Enterprise agent framework with visual canvas, OpenClaw-inspired execution engine, built-in tools, enterprise security (default deny, budget stops, rate limiting, SSRF protection, audit trail)
• ✅ Bonobot Enterprise Features — Persistent Agent Memory (pgvector, 5 memory types, AI extraction), Scheduled Autonomous Execution (cron, timezone, multi-channel delivery), and Approval Queue / Human-in-the-Loop (risk assessment, auto-approve, timeout handling)

Planned

• ~~📋 SSO/SAML integration~~ ✅ Shipped
• 📋 SOC-2 Type II certification — Roadmap
• 📋 Smart routing (complexity-aware model selection)
• 📋 VPC Gateway Agent (enterprise self-hosted data plane)
• 📋 Additional provider integrations (Cohere, Mistral, custom endpoints)
• 📋 Advanced audit log export & SIEM integration

MCP Server

Connect Claude Desktop, Cowork, or any MCP-compatible client to Bonito with the MCP server:

pip install bonito-mcp

Exposes 18 tools covering providers, models, gateway, agents, knowledge bases, and cost monitoring. See mcp-server/ for full documentation and configuration.

Documentation

• AI Context / Knowledge Base — Architecture, API design, and RAG pipeline details
• Secrets Store — Org secrets API, CLI, YAML, and security guide
• VectorBoost — KB compression methods, benchmarks, and configuration
• Bonobot Architecture — Agent framework design and build plan
• Known Issues — Tracking document for known issues and fixes
• Pricing — Plans and pricing structure
• SOC-2 Roadmap — Path to SOC-2 Type II certification
• SSO Scoping — SSO/SAML implementation plan
• Vault Production — Vault hardening guide
• Architectural Patterns — 7 core codebase patterns reference

Built with 🐟 by the Bonito team.