Claude Langfuse Plugin

A Claude Code plugin that turns Claude into an expert at Langfuse — instrumentation, setup, evaluation, monitoring, and self-hosting — for Langfuse Cloud and self-hosted deployments, across the Python and JS/TS SDKs and 50+ framework integrations.

It bundles 5 skills, 6 commands, 3 agents, 1 safety hook, and both Langfuse MCP servers, designed on one principle: distill durable judgment, fetch the facts live. The plugin carries the decisions and workflows (which eval method when, how to size a deployment, how to read an experiment); it fetches exact, version-sensitive code from the live Langfuse docs at runtime so it never goes stale.

Features

Skills (auto-activate when relevant)

Commands (you invoke explicitly)

Agents (autonomous, delegated tasks)

Hook

A non-blocking PostToolUse check that warns if you accidentally hardcode a Langfuse secret key (sk-lf-…) in an edited file. High-signal, never blocks your edit.

MCP servers (.mcp.json)

• `langfuse-docs` — public docs search/retrieval (no auth). 3 tools.
• `langfuse-data-platform` — authenticated access to your project's prompts, traces,

observations, scores, datasets, evaluators, metrics, and getHealth. 61 tools.

Installation

1. Prerequisites

• Claude Code installed.
• A Langfuse project (Cloud sign-up at cloud.langfuse.com, or self-hosted) and a

project API key pair (pk-lf-… / sk-lf-…) from Project Settings → API Keys.

2. Install the plugin

Local (development / trying it out):

git clone https://github.com/jbaham2/claude-langfuse-plugin
claude --plugin-dir /path/to/claude-langfuse-plugin

Via a marketplace (shareable install):

/plugin marketplace add jbaham2/claude-langfuse-plugin
/plugin install claude-langfuse-plugin

(Exact marketplace commands can vary by Claude Code version — see the Claude Code plugin docs.)

3. Set credentials (never paste keys into chat)

Copy the template and fill it in:

cp skills/langfuse-setup/assets/.env.example .env

Set in your shell or .env:

export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"   # your region or self-host URL
export LANGFUSE_HOST="$LANGFUSE_BASE_URL"               # some tools read HOST instead

4. Enable the data-platform MCP

The authenticated MCP uses Basic Auth. Generate the token and export it before launching Claude Code (the value is interpolated into .mcp.json at load):

export LANGFUSE_MCP_AUTH="$(printf '%s:%s' "$LANGFUSE_PUBLIC_KEY" "$LANGFUSE_SECRET_KEY" | base64)"

Set the right region endpoint in .mcp.json (the url of langfuse-data-platform):

⚠️ Read + write tools are enabled by default (it can create/delete prompts, scores, datasets, evaluators…). For read-only use, configure a client-side tool allowlist. The bundled diagnostic agents (setup-doctor, trace-reviewer) are read-only by instruction regardless.

>

The Docs MCP needs no auth — it works immediately.

5. Verify

In a Claude Code session inside your project:

/lf-health-check

You should get a green report (API reachable, authenticated, traces arriving). If not, the setup-doctor agent will diagnose it.

Quick start

1. Instrument your app: /lf-setup trace my Python app that uses the OpenAI SDK

→ walks the deployment/region/keys decisions, then hands off to the official langfuse skill for current best-practice tracing code.

1. Confirm it works: /lf-health-check → verifies traces are arriving.
2. Evaluate before you ship a change: /lf-eval gate a prompt change to my support bot

→ designs a dataset + experiment + the right evaluators.

How-to guide

Just describe the goal — the relevant skill activates automatically. Use a command when you want to drive a specific workflow, or an agent for a heavier autonomous task.

Use cases

These mirror Langfuse's own guides, reframed as "ask the plugin."

1. Systematic error analysis (→ Langfuse Error Analysis guide)

"Review the last 24h of traces for my support-bot and tell me what's actually going wrong."

The trace-reviewer agent pulls observations via the data-platform MCP (listObservations, filtered by name/level/time), classifies failures into a taxonomy (hallucination, wrong tool call, format, retrieval miss…), quantifies how often each occurs, and proposes which failing traces to turn into regression test cases — instead of you reading traces one by one.

2. Gate a change before deploy (→ Testing LLM Applications / Experiment Interpretation)

"I changed the system prompt. Make sure quality didn't regress before I ship."

langfuse-evaluation sets up an offline experiment: a dataset (seeded from real production traces), a task, and evaluators. After the run it walks the interpretation funnel — aggregate metrics → baseline diff → item-level regressions → annotate — so a "+4% average" can't hide a "−16% on critical cases."

3. Evaluate a RAG pipeline (→ RAG Observability & Evals)

"Help me evaluate my retrieval-augmented chatbot."

The plugin evaluates retrieval and generation independently — retrieval relevance / context precision & recall on the retrieval step, faithfulness / answer relevancy on the generation — so you learn which half is failing. It can use Ragas (reference-free, runs on production traces) and lets you iterate on chunking cheaply.

4. Evaluate an agent (→ Agent Evaluation)

"How do I evaluate my tool-using agent?"

The eval-designer agent reads your code and produces a plan covering trajectory + per-step + final answer (not just final-output accuracy): tool-selection scores at the observation level, end-to-end trajectory judgment at the trace level, and a phased rollout (manual trace inspection → online evaluators → dataset-based testing).

5. Calibrate an LLM judge (→ Calibrate your LLM-as-a-Judge)

"Is my hallucination judge actually trustworthy?"

langfuse-evaluation sets the judge up; calibration is delegated to the official skill's judge-calibration workflow — run the judge against a labeled dataset as an experiment and report accuracy (or a full confusion matrix / precision-recall) so you know whether to trust it before it gates anything.

6. Stand up production monitoring (→ Automated Evaluations + dashboards)

"Give me a production health and cost view, and alert me if cost spikes."

langfuse-monitoring specs the three standard dashboards (health, cost, quality), answers ad-hoc questions via the Metrics API (queryMetrics), and sets up alerting — clarifying that app-level cost alerts are a scheduled metrics check, while Langfuse Spend Alerts cover your Langfuse bill.

7. Self-host and scale (→ Self-hosting guides)

"Size a Helm deployment for ~5M traces/day and plan our backups."

/lf-deploy gives the components + minimum sizing, the throughput levers (scale workers by CPU, reduce ClickHouse ingestion reads, separate ingestion/UI), the per-component backup strategy, and the security checklist — deferring exact Helm/Terraform values to the live self-hosting docs.

How it's organized

The repo is one plugin plus an authoring workspace. The Claude Code loader only reads the plugin parts; the rest is documentation/build material (most of it git-ignored from a published package — see .gitignore).

claude-langfuse-plugin/
├── .claude-plugin/plugin.json   # manifest
├── .mcp.json                    # both Langfuse MCP servers
├── skills/                      # 5 skills (1 vendored + 4 authored)
├── commands/                    # 6 slash commands
├── agents/                      # 3 subagents
├── hooks/                       # non-blocking secret-key guard
├── README.md  ·  LICENSE
└── meta/                        # architecture docs (ROADMAP, DECISIONS, DISTILLATION-TRACKER)
    # workbench/ — authoring scratch (git-ignored)

Knowledge pipeline: fetch (live .md) → distill durable judgment → skills/<skill>/references/ (shipped) — with stale-prone facts referenced live at runtime. Rationale and full build history are in meta/DECISIONS.md and meta/ROADMAP.md.

Maintenance

• Vendored skill: skills/langfuse/ is the official Langfuse skill, pinned to an upstream commit.

Re-sync periodically using the procedure in skills/langfuse/VENDORED.md.

• Self-describing MCP: the data-platform MCP evolves; the plugin inspects its tools live rather

than hardcoding schemas, so it tolerates upstream changes.

Credits & acknowledgements

This plugin is built on Langfuse's open-source work and would not exist without it:

• The official Langfuse Agent Skill — github.com/langfuse/skills

(© Langfuse GmbH, MIT). It is vendored verbatim into skills/langfuse/ and provides the instrumentation, langfuse-cli, prompt-migration, judge-calibration, error-analysis, and CI/CD workflows. Our four skills are built to complement it, not replace it. Provenance and the upstream commit pin are recorded in skills/langfuse/VENDORED.md.

• The Langfuse documentation, guides, and cookbooks — langfuse.com/docs

and langfuse.com/guides — the source material distilled into the skills' references and the use-cases above.

• The Langfuse MCP servers — the public docs MCP and the authenticated data-platform MCP.

Thank you to the Langfuse team and community.

License

MIT — see LICENSE. The vendored official Langfuse skill (skills/langfuse/) is also MIT, © Langfuse GmbH — see skills/langfuse/LICENSE.upstream and skills/langfuse/VENDORED.md.

Not affiliated with Langfuse; built on their open-source platform and docs.