ChatKit Actions Skill

Overview

This skill unlocks the full power of ChatKit's agentic UI capabilities - where AI can render interactive widgets, users can click buttons that trigger both client and server actions, and the conversation becomes a two-way interactive experience.

Core Concepts

Action Handler Types

Widgets can specify where actions are handled:

Widget Lifecycle

1. Agent tool generates widget → yield WidgetItem
2. Widget renders in chat with action buttons
3. User clicks action → action dispatched
4. Handler processes action:
   - client: onAction callback in frontend
   - server: action() method in ChatKitServer
5. Optional: Widget replaced with updated state

Implementation Patterns

Pattern 1: Widget Templates (.widget files)

When: Define reusable widget layouts with dynamic data

Widget Template Format:

{
  "version": "1.0",
  "name": "task_list",
  "template": "{\"type\":\"ListView\",\"children\":[...jinja template...]}",
  "jsonSchema": {
    "type": "object",
    "properties": {
      "tasks": { "type": "array", "items": {...} }
    }
  }
}

Widget Components Available:

• Layout: ListView, ListViewItem, Row, Col, Box
• Content: Text, Title, Image, Icon
• Interactive: Button (with onClickAction)
• Styling: gap, padding, background, border, radius

Example - Task List Widget:

{
  "type": "ListView",
  "children": [
    {
      "type": "ListViewItem",
      "key": "task-1",
      "onClickAction": {
        "type": "task.select",
        "handler": "client",
        "payload": { "taskId": "task-1" }
      },
      "children": [
        {
          "type": "Row",
          "gap": 3,
          "children": [
            { "type": "Icon", "name": "check", "color": "success" },
            { "type": "Text", "value": "Complete review", "weight": "semibold" }
          ]
        }
      ]
    }
  ]
}

Python - Loading Templates:

from chatkit.widgets import WidgetTemplate, WidgetRoot

# Load template from file
task_list_template = WidgetTemplate.from_file("task_list.widget")

def build_task_list_widget(tasks: list[Task]) -> WidgetRoot:
    return task_list_template.build(
        data={
            "tasks": [task.model_dump() for task in tasks],
            "selected": None,
        }
    )

Evidence: cat-lounge/backend/app/widgets/cat_name_suggestions.widget

Pattern 2: Client-Handled Actions

When: Actions that update local state, navigate, or send follow-up messages

Widget Definition (handler: "client"):

{
  "type": "Button",
  "label": "View Article",
  "onClickAction": {
    "type": "open_article",
    "handler": "client",
    "payload": { "id": "article-123" }
  }
}

Frontend Handler:

import { useChatKit, type Widgets } from "@openai/chatkit-react";

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (
      action: { type: string; payload?: Record<string, unknown> },
      widgetItem: { id: string; widget: Widgets.Card | Widgets.ListView }
    ) => {
      switch (action.type) {
        case "open_article":
          // Navigate to article
          navigate(`/article/${action.payload?.id}`);
          break;

        case "more_suggestions":
          // Send follow-up message
          await chatkit.sendUserMessage({ text: "More suggestions, please" });
          break;

        case "select_option":
          // Update local state
          setSelectedOption(action.payload?.optionId);
          break;
      }
    },
  },
});

Evidence: news-guide/frontend/src/components/ChatKitPanel.tsx:55-79

Pattern 3: Server-Handled Actions

When: Actions that mutate data, update widgets, or require backend processing

Widget Definition (handler: "server"):

{
  "type": "ListViewItem",
  "onClickAction": {
    "type": "line.select",
    "handler": "server",
    "payload": { "id": "blue-line" }
  }
}

Backend Handler:

from chatkit.server import ChatKitServer
from chatkit.types import (
    Action,
    WidgetItem,
    ThreadItemReplacedEvent,
    ThreadItemDoneEvent,
    AssistantMessageItem,
    HiddenContextItem,
    ClientEffectEvent,
)

class MyServer(ChatKitServer[dict]):

    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,
        context: dict[str, Any],
    ) -> AsyncIterator[ThreadStreamEvent]:

        if action.type == "line.select":
            line_id = action.payload["id"]

            # 1. Update widget with selection
            updated_widget = build_line_selector_widget(
                lines=self.lines,
                selected=line_id,
            )
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

            # 2. Add hidden context for future agent input
            await self.store.add_thread_item(
                thread.id,
                HiddenContextItem(
                    id=self.store.generate_item_id("ctx", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=f"<LINE_SELECTED>{line_id}</LINE_SELECTED>",
                ),
                context=context,
            )

            # 3. Stream assistant message
            yield ThreadItemDoneEvent(
                item=AssistantMessageItem(
                    id=self.store.generate_item_id("msg", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=[{"text": f"Selected {line_id}. Where to add station?"}],
                )
            )

            # 4. Trigger client effect
            yield ClientEffectEvent(
                name="location_select_mode",
                data={"lineId": line_id},
            )

Evidence: metro-map/backend/app/server.py

Pattern 4: Client-to-Server Action Forwarding

When: Client handles action locally, then notifies server for persistence/widget update

Frontend - Send Custom Action:

widgets: {
  onAction: async (action, widgetItem) => {
    if (action.type === "select_name") {
      // 1. Forward to server for processing
      await chatkit.sendCustomAction(action, widgetItem.id);

      // 2. Optionally refresh local state after server processes
      const data = await refreshCatStatus();
      if (data) {
        handleStatusUpdate(data, `Now called ${data.name}`);
      }
    }
  },
}

Evidence: cat-lounge/frontend/src/components/ChatKitPanel.tsx:68-80

Pattern 5: Entity Tagging (@mentions)

When: Allow users to @mention entities (users, articles, tasks) in messages

Frontend - Entity Configuration:

import { useChatKit, type Entity } from "@openai/chatkit-react";

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  entities: {
    // Search for entities as user types @...
    onTagSearch: async (query: string): Promise<Entity[]> => {
      const results = await fetch(`/api/search?q=${query}`).then(r => r.json());

      return results.map((item) => ({
        id: item.id,
        title: item.name,
        icon: item.type === "person" ? "profile" : "document",
        group: item.type === "People" ? "People" : "Articles",
        interactive: true,
        data: {
          type: item.type,
          article_id: item.id,
        },
      }));
    },

    // Handle entity click (e.g., navigate)
    onClick: (entity: Entity) => {
      if (entity.data?.article_id) {
        navigate(`/article/${entity.data.article_id}`);
      }
    },

    // Render entity preview on hover
    onRequestPreview: async (entity: Entity) => {
      const details = await fetch(`/api/entity/${entity.id}`).then(r => r.json());

      return {
        preview: {
          type: "Card",
          children: [
            { type: "Text", value: entity.title, weight: "bold" },
            { type: "Text", value: details.description, color: "tertiary" },
          ],
        },
      };
    },
  },
});

Backend - Converting Entity Tags:

# thread_item_converter.py
class EntityAwareConverter(BasicThreadItemConverter):
    """Convert entity tags to model-readable markers."""

    async def to_agent_input(self, items: list[ThreadItem]) -> list:
        result = []
        for item in items:
            if isinstance(item, UserMessageItem):
                content = item.content
                # Convert entity tags to XML markers
                for entity in item.entities or []:
                    if entity.type == "article":
                        content = content.replace(
                            f"@{entity.title}",
                            f"<ARTICLE_REFERENCE id='{entity.id}'>{entity.title}</ARTICLE_REFERENCE>"
                        )
                result.append({"role": "user", "content": content})
        return result

Evidence:

• news-guide/frontend/src/components/ChatKitPanel.tsx:122-126
• news-guide/backend/app/thread_item_converter.py
• metro-map/frontend/src/components/ChatKitPanel.tsx:73-117

Pattern 6: Composer Tools (Mode Selection)

When: Let users select different AI modes/tools from the composer

Frontend - Tool Configuration:

const TOOL_CHOICES = [
  {
    id: "general",
    label: "Chat",
    shortLabel: "Chat",
    icon: "sparkle",
    placeholderOverride: "Ask anything...",
    pinned: true,
  },
  {
    id: "event_finder",
    label: "Find Events",
    shortLabel: "Events",
    icon: "calendar",
    placeholderOverride: "What events are you looking for?",
    pinned: true,
  },
  {
    id: "puzzle",
    label: "Word Puzzle",
    shortLabel: "Puzzle",
    icon: "bolt",
    placeholderOverride: "Ready for today's puzzle?",
    pinned: false,
  },
];

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },
  composer: {
    placeholder: "What would you like to do?",
    tools: TOOL_CHOICES,
  },
});

Backend - Routing by Tool Choice:

# server.py
async def respond(self, thread, item, context):
    tool_choice = context.get("tool_choice")

    if tool_choice == "event_finder":
        agent = self.event_finder_agent
    elif tool_choice == "puzzle":
        agent = self.puzzle_agent
    else:
        agent = self.general_agent

    # Run selected agent
    result = Runner.run_streamed(agent, input_items, context=agent_context)
    async for event in stream_agent_response(agent_context, result):
        yield event

Evidence: news-guide/frontend/src/lib/config.ts

Pattern 7: Thread Item Actions (Feedback/Retry/Share)

When: Enable built-in actions on AI messages

Frontend Configuration:

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  threadItemActions: {
    feedback: true,   // Thumbs up/down
    retry: true,      // Regenerate response
    share: true,      // Share message
  },

  onLog: ({ name, data }) => {
    if (name === "message.feedback") {
      // Track feedback analytics
      fetch("/api/analytics/feedback", {
        method: "POST",
        body: JSON.stringify(data),
      });
    }
    if (name === "message.share") {
      // Track share events
      fetch("/api/analytics/share", {
        method: "POST",
        body: JSON.stringify(data),
      });
    }
  },
});

Pattern 8: Widget Streaming from Tools

When: Agent tool generates a widget as part of response

Backend - Tool with Widget Output:

from chatkit.types import WidgetItem
from agents import function_tool

@function_tool
async def show_article_list(ctx: AgentContext, query: str) -> str:
    """Show a list of articles matching the query."""

    articles = await article_store.search(query)

    # Build widget
    widget = build_article_list_widget(articles)

    # Yield widget item
    widget_item = WidgetItem(
        id=ctx.store.generate_item_id("widget", ctx.thread, ctx.request_context),
        thread_id=ctx.thread.id,
        created_at=datetime.now(),
        widget=widget,
    )

    # Save to store
    await ctx.store.add_thread_item(ctx.thread.id, widget_item, ctx.request_context)

    # Yield as event
    yield ThreadItemDoneEvent(item=widget_item)

    return f"Showing {len(articles)} articles"

Evidence: news-guide/backend/app/agents/news_agent.py

Widget Component Reference

Layout Components

Content Components

Interactive Components

Action Structure

interface Action {
  type: string;           // Action identifier
  handler: "client" | "server";
  payload?: Record<string, unknown>;
}

Common Patterns Summary

Critical Implementation Details

Action Object Structure

IMPORTANT: The Action object uses payload, NOT arguments:

# ❌ WRONG - Will cause AttributeError
action.arguments  # 'Action' object has no attribute 'arguments'

# ✅ CORRECT
action.payload    # Access action data via .payload

Action Type Definition:

from chatkit.types import Action

# Action[str, Any] has these fields:
action.type      # str - action identifier (e.g., "task.start")
action.payload   # dict[str, Any] - action data
action.handler   # "client" | "server" - where action is processed

Server Action Handler Signature

CRITICAL: The context parameter is RequestContext, NOT dict[str, Any]

# Type annotation vs runtime reality mismatch
async def action(
    self,
    thread: ThreadMetadata,
    action: Action[str, Any],
    sender: WidgetItem | None,
    context: dict[str, Any],  # ⚠️ Type hint says dict, but runtime is RequestContext!
) -> AsyncIterator[ThreadStreamEvent]:

    # ❌ WRONG - Tries to wrap RequestContext inside RequestContext
    request_context = RequestContext(metadata=context)

    # ✅ CORRECT - Use context directly, it's already RequestContext
    user_id = context.user_id
    metadata = context.metadata

Why this happens: ChatKit SDK passes RequestContext object at runtime, despite type annotations suggesting dict. Always use context directly without wrapping.

UserMessageItem Required Fields

When creating synthetic user messages from actions, ALL these fields are required:

from chatkit.types import UserMessageItem, UserMessageTextContent
from datetime import datetime

# ❌ WRONG - Missing required fields causes ValidationError
synthetic_message = UserMessageItem(
    content=[UserMessageTextContent(type="text", text=message_text)]
)

# ✅ CORRECT - Include all required fields
synthetic_message = UserMessageItem(
    id=self.store.generate_item_id("message", thread, context),
    thread_id=thread.id,
    created_at=datetime.now(),
    content=[UserMessageTextContent(type="input_text", text=message_text)],
    inference_options={},
)

Required fields:

• id: Generate via store.generate_item_id("message", thread, context)
• thread_id: From thread.id parameter
• created_at: Current timestamp via datetime.now()
• content: List of content blocks (UserMessageTextContent)
• inference_options: Empty dict {} if no special options

UserMessageTextContent type values:

• ✅ type="input_text" - User text input (correct)
• ❌ type="text" - Invalid for UserMessageTextContent (causes ValidationError)

Local Tool Wrappers for Widget Streaming

Problem: Agent calls MCP tool successfully, but widget doesn't appear in UI.

Root Cause: Widgets stream via RunHooks pattern. MCP tools alone don't trigger widget rendering - you need local tool wrappers.

Solution Pattern:

# 1. Create local tool wrapper
from agents import function_tool

@function_tool
async def show_task_form(
    ctx: RunContextWrapper[TaskFlowAgentContext],
) -> str:
    """Show interactive task creation form widget."""

    agent_ctx = ctx.context
    mcp_url = agent_ctx.mcp_server_url

    # Call MCP tool via HTTP
    result = await _call_mcp_tool(
        mcp_url,
        "taskflow_show_task_form",
        arguments={"params": {"user_id": agent_ctx.user_id}},
        access_token=agent_ctx.access_token,
    )

    # Return result - RunHooks will intercept and stream widget
    return json.dumps(result)

# 2. Register local wrapper with agent
agent = Agent(
    name="TaskFlow Assistant",
    tools=[
        show_task_form,  # Local wrapper - triggers RunHooks
        # ... other local wrappers
    ],
)

# 3. In RunHooks.on_tool_end() - Stream widget
async def on_tool_end(self, output: str | None, tool_name: str) -> None:
    if tool_name == "show_task_form":
        result = json.loads(output)
        if result.get("action") == "show_form":
            widget = build_task_form_widget()
            yield WidgetItem(...)

Key insight: Direct MCP tools → no widgets. Local wrappers → RunHooks → widgets streamed.

Common Pydantic Validation Errors

Error 1: 'Action' object has no attribute 'arguments'

AttributeError: 'Action[str, Any]' object has no attribute 'arguments'

Fix: Use action.payload instead of action.arguments

Error 2: UserMessageTextContent type mismatch

ValidationError: Input should be 'input_text' [type=literal_error, input_value='text']

Fix: Use type="input_text" for user input, not type="text"

Error 3: UserMessageItem missing required fields

4 validation errors for UserMessageItem
- id: Field required
- thread_id: Field required
- created_at: Field required
- inference_options: Field required

Fix: Include all required fields when creating UserMessageItem (see pattern above)

Error 4: RequestContext wrapping issue

2 validation errors for RequestContext
user_id: Field required
metadata: Input should be a valid dictionary [input_value=RequestContext(...)]

Fix: Don't wrap context - it's already a RequestContext object

Widget Action Testing Checklist

Before claiming widget actions are complete, test:

• [ ] Widget renders with correct data
• [ ] All buttons have clear labels (not just icons)
• [ ] Client actions navigate/update UI correctly
• [ ] Server actions call backend successfully
• [ ] Action payload contains all required data
• [ ] Widget updates after server action completes
• [ ] No AttributeError on action.payload access
• [ ] No ValidationError on UserMessageItem creation
• [ ] Local tool wrappers trigger widget streaming
• [ ] All status transitions have appropriate buttons
• [ ] Test with real user session (not mock data)
• [ ] Check browser console for errors
• [ ] Verify backend logs show action processing
• [ ] Test error cases (network failure, invalid data)

Anti-Patterns to Avoid

1. Mixing handlers - Don't handle same action in both client and server
2. Missing payload - Always include necessary data in action payload
3. Forgetting widget ID - sendCustomAction requires widget ID for updates
4. Not updating widget - Server actions should yield ThreadItemReplacedEvent
5. Blocking in onAction - Keep client handlers fast, offload to server
6. Using action.arguments - Use action.payload (arguments doesn't exist)
7. Wrapping RequestContext - Context is already RequestContext, don't wrap it
8. Missing UserMessageItem fields - Include id, thread_id, created_at, inference_options
9. Wrong content type - Use type="input_text" for user messages
10. No local tool wrappers - MCP tools alone don't stream widgets
11. Not testing thoroughly - Test all actions with real data before claiming done
12. Assuming type hints are correct - ChatKit has type annotation vs runtime mismatches

References

Documentation

• references/widget-templates.md - Widget template syntax
• references/client-vs-server-actions.md - Action routing guide
• references/entity-tagging.md - @mention implementation
• references/composer-tools.md - Tool choice patterns
• references/server-action-handler.py - Complete backend action handler pattern

Widget Template Assets

• assets/line-select.widget - Server action selection list (metro-map pattern)
• assets/name-suggestions.widget - Client action with "more" button (cat-lounge pattern)
• assets/article-list.widget - Rich card layout with images (news-guide pattern)

Evidence Sources

All patterns derived from OpenAI ChatKit advanced samples:

• blueprints/openai-chatkit-advanced-samples-main/examples/cat-lounge/
• blueprints/openai-chatkit-advanced-samples-main/examples/metro-map/
• blueprints/openai-chatkit-advanced-samples-main/examples/news-guide/