knowledge-forge

Turn a plain list of links (videos, posts, repos, articles) into a verified, browsable knowledge base — and let your AI coding tool enrich it with grounded, hallucination-checked summaries.

It ships three ways to drive the same pipeline so it works with any AI tool:

• a CLI (knowledge-forge ...)
• an MCP server (knowledge-forge-mcp) for Cursor, Claude Code, Claude Desktop, etc.
• portable skills + editor hooks

What it does

links.txt ──▶ extract (yt-dlp / whisper / youtube-transcript)
          ──▶ classify into categories + pull verified learnings
          ──▶ videos.json ──▶ generate a category/detail canvas (.canvas.tsx)
          ──▶ [optional] rule-based enrichment + an agent queue
          ──▶ agent writes 300-1000 word grounded summaries
          ──▶ validate (anti-hallucination gate) ──▶ apply ──▶ refresh

Key idea: summaries are grounded. A validator enforces word count, per-paragraph token overlap with the source material, source-URL provenance, and a no-engagement-bait rule — so the agent can't invent facts, repos, or URLs.

Capture: send a link, get a knowledge entry

The hardest part of any knowledge base is capturing the link the instant you see its value. knowledge-forge ingest <url> is the one-shot front door: it appends the URL, scrapes + classifies it, and queues it for a grounded summary.

knowledge-forge ingest "https://www.youtube.com/watch?v=..."   # one link, fully processed

Wire it to Telegram so you can capture from your phone while scrolling — share a reel to your bot and it lands in the base automatically. Use the standalone bot in integrations/telegram/, or, if you already run Hermes or any Telegram-connected agent, just give that agent the knowledge-forge ingest <url> tool and let it handle the chat. See integrations/telegram/README.md.

share a link to Telegram ──▶ bot / Hermes extracts the URL
                         ──▶ knowledge-forge ingest <url>
                         ──▶ scrape + classify + queue ──▶ searchable entry

Discover & install (not just enrich)

Once the base is built, search it for a use case and get the recommended best practice plus the tools/skills to install — all grounded in the source videos:

knowledge-forge search "cut Claude Code token cost" --limit 5

Each hit returns a grounded snippet, the source links, any installCommands and recommendedTools extracted verbatim from that video (e.g. a pip install line or a GitHub repo), and a needsScrape flag when the entry is too thin and the source should be transcribed + enriched for a fuller answer. This turns the base into a recommender: "how do I improve X?" → best practice + the exact skill/tool to install, cited to the video. Commands are only ever surfaced when present in the source — never fabricated.

Localhost web UI (browse + search)

pip install "knowledge-forge[all,web]"
knowledge-forge serve                    # http://localhost:8077
knowledge-forge serve --no-bridge        # web only

Category catalogue, detail view, and Mode B search in the browser. Docker: docker compose up -d — see integrations/docker/README.md. Enable Docker Desktop start-on-login for auto-resume + Telegram backlog drain.

Enrich with a model — local or hosted (no IDE/agent required)

The agent-written summary step doesn't have to run inside an AI IDE. A background worker drains the queue on its own, and the same deterministic validator (word count, per-paragraph grounding, no bait, no invented URLs) gates every summary — so only grounded text is ever saved, regardless of which model wrote it.

ollama pull gemma3:4b
knowledge-forge enrich-worker            # loop: queued entries → model → validate → apply
knowledge-forge enrich-worker --once     # single drain pass

Pluggable providers (set KB_LLM_PROVIDER): ollama (default, local), openai (any OpenAI-compatible host — OpenAI, OpenRouter, Groq, Together, vLLM, LM Studio…), or anthropic. Swap in a larger hosted model and longer summaries with env vars only — no code changes:

KB_LLM_PROVIDER=openai KB_ENRICH_MODEL=gpt-4o KB_LLM_API_KEY=sk-... \
KB_ENRICH_MAX_WORDS=2000 knowledge-forge enrich-worker

Combine with the web UI + Telegram bridge (and Docker) for a hands-off loop: send a link → scrape/classify → model writes a grounded summary → live in the UI. Full tunable surface (provider, base URL, API key, word bounds, temperature, context, tokens, timeout, loop cadence) is documented in integrations/docker/README.md.

Install

pip install "knowledge-forge[all]"     # CLI + MCP + extraction + transcription
# or pick extras: .[extract] .[transcribe] .[mcp]

Core has no third-party dependencies; the extras add yt-dlp, youtube-transcript-api, faster-whisper, and mcp.

Quick start

knowledge-forge init                 # scaffold config + folders in the current dir
#  ... add URLs to links.txt (see samples/links.sample.txt) ...
knowledge-forge update               # extract + classify + build the canvas
knowledge-forge run                  # rule-based enrich + build the agent queue
knowledge-forge queue                # see what needs an agent-written summary
# agent writes payload.json, then:
knowledge-forge validate --file payload.json
knowledge-forge apply    --file payload.json

Generate a one-command launcher for your OS:

knowledge-forge bootstrap --write    # writes kf.ps1+kf.bat (Windows) or kf.sh (macOS/Linux)

Configuration

Everything is path-driven via knowledge-forge.json (see config.example.json). All relative paths resolve against the config file's directory — no machine- or OS-specific paths are baked in. The config is auto-discovered by walking up from the CWD, or set KNOWLEDGE_FORGE_PROJECT / KNOWLEDGE_FORGE_CONFIG.

Use it from your AI tool

MCP tools: kf_ingest, kf_search, kf_update, kf_refresh, kf_run, kf_queue, kf_list, kf_validate, kf_apply.

CLI also ships knowledge-forge serve / knowledge-forge-serve for the web UI at localhost:8077, and knowledge-forge enrich-worker / knowledge-forge-enrich for local-model enrichment.

See integrations/<tool>/README.md for copy-paste setup.

Project layout

knowledge_forge/      core package (config, pipeline, classify, verify, enrich, canvas, cli)
mcp_server/           FastMCP stdio server
skills/               tool-agnostic SKILL.md files
integrations/         per-tool hooks + MCP config + setup docs
samples/              anonymized example links + data
config.example.json   starter config

License

MIT — see LICENSE.