social-skills

Social-web skills for AI agents.

Two binaries, one toolkit:

• `social-fetch` — pulls structured data from the social web

and exposes it as CLI output, an MCP server, or a Claude skill.

• `social-ledger` — local SQLite + FTS5 store that auto-caches

every fetch so agents can answer "have we seen this URL?" and "what did we save about X?" without re-fetching. Also tracks an influencer directory — people / companies you flag as topic authorities, with their socials and which channels you subscribe to.

LLMs are great at understanding text but bad at getting it. The social web — HackerNews threads, Reddit comments, GitHub repos, X/Twitter posts, LinkedIn timelines, YouTube transcripts, Bluesky feeds, arXiv papers, Medium / Substack articles, RSS feeds, generic blog posts — lives behind a different API, DOM scraper, auth flow, and rate limit per platform. social-skills hides all of that behind one consistent interface and gives the agent clean Markdown or structured JSON an LLM can actually parse.

Same shape covers grounded web search (Perplexity, Tavily, Brave, SerpAPI, Google, DuckDuckGo) and grounded answer engines (Grok, OpenAI, Anthropic, Gemini), and exposes everything as MCP tools so Claude Desktop, Claude Code, claude.ai, and Perplexity can call into it as a first-class tool — not as another WebFetch round-trip that returns rendered HTML.

Also great as a plain CLI. Every agent capability is also a shell command — social-fetch search "vercel ai sdk" -p auto, social-ledger article search "harness engineering", social-fetch ask "what changed in Go 1.27?" -p perplexity. Pipe into jq, redirect into files, embed in scripts. Agents are the primary audience, but humans get the same toolbox.

social-fetch fetch    https://news.ycombinator.com/item?id=43000000
social-fetch search   "vercel ai sdk" -p auto -n 10 --last 7d
social-fetch ask      "what changed in Go 1.27?" -p perplexity
social-fetch timeline @jedi4ever -p x --last 24h
social-fetch research "tessl harness engineering" -p anthropic
social-ledger article search  "harness engineering"
social-ledger influencer add "Andrej Karpathy" --x karpathy --topics ai
social-ledger influencer list --topic ai --followed

What it is

• One interface for ~20 platforms. Same Item shape whether

you're scraping HN comments, pulling a LinkedIn timeline, or asking Perplexity a recency-filtered question — agents don't need per-platform branching logic.

• Provider chains. -p auto walks a default fallback list, so

whichever subset of API keys you've configured determines coverage and the rest silently no-op. Drop a key in, the agent gets a new capability without code changes.

• MCP server built in. social-fetch mcp exposes every CLI

capability as Model Context Protocol tools, runnable over stdio (Claude Desktop) or Streamable HTTP (claude.ai, Perplexity, Claude Code remote-MCP). Same binary is your CLI and your MCP server.

• Browser bridge for authenticated paths — LinkedIn, Medium /

Substack paywalls — via a small Chrome MV3 extension that brokers between the agent and your real, logged-in browser. Public content still goes direct over HTTP. The bridge is intentionally single-lane (one in-process mutex serializing all calls, plus randomized human-paced pauses between scrolls) so it stays under bot-detection thresholds — at the cost of not scaling for bulk scraping.

• Citations first. Every result carries source, url,

fetched_at, written_at, scores, comment trees — so the agent can ground its answer in something the user can click back to.

Install

social-fetch is built to plug into AI agents — pick whichever channel matches the agent you're working with. Full step-by-step in INSTALL.md.

1. Claude Desktop Extension (.mcpb) — recommended

One-click install with API-key prompts that go straight into the macOS Keychain. Best UX if you live in Claude Desktop.

1. Download social-skills-claude-desktop-extension-<version>-darwin-arm64.mcpb
   from https://github.com/jedi4ever/social-skills/releases/latest
2. Double-click the .mcpb (or drag it onto Claude Desktop →
   Settings → Extensions).
3. Fill in whichever API keys you have — every key is optional.

2. Claude Code plugin (marketplace)

One-line install if you use Claude Code:

/plugin marketplace add jedi4ever/social-skills
/plugin install social-fetch

Requires the social-fetch binary on your PATH separately (the plugin is the skill markdown + manifest, not the binary).

3. Skill (file-based, Claude Desktop or Claude Code)

SKILL.md + binary dropped into ~/.claude/skills/social-fetch/. Useful when you want to manage .env yourself or work offline.

Via npx (no clone, single command) — using vercel-labs/skills, the most-used skill installer (works with Claude Code, Claude Desktop, OpenCode, Codex, and others):

npx skills add jedi4ever/social-skills --skill social-fetch

The social-fetch binary still needs to be on PATH separately — npx skills only installs the markdown skill, not the binary.

Via clone + make — same end state, plus builds the binary:

git clone https://github.com/jedi4ever/social-skills.git
cd social-skills
make skill-install

Other community installers (claude-plugins, agent-skills-cli, add-skill) work similarly — they all read the same SKILL.md files from this repo.

4. Remote MCP server (claude.ai, Perplexity, Claude Code)

social-fetch mcp runs the MCP protocol over Streamable HTTP so cloud-hosted clients can reach your local binary:

# Plain HTTP listener — bring-your-own-tunnel
MCP_AUTH_TOKEN=$(openssl rand -hex 32) \
  social-fetch mcp --http :8080
# expose :8080 publicly with whatever you prefer:
# Cloudflare Tunnel, Tailscale Funnel, fly.io machine, your own
# reverse proxy, etc. Then paste the resulting HTTPS URL +
# token into the client.

If you want a one-line public URL without configuring a tunnel yourself, --ngrok spawns ngrok for you and prints the URL + an auto-generated bearer token (requires ngrok on PATH and one-time ngrok config add-authtoken …):

social-fetch mcp --ngrok           # convenience: prints URL + token

Either way, paste the URL + token into Settings → Connectors / Custom Connector in claude.ai, Perplexity Pro, or claude mcp add http <url> --header "Authorization: Bearer <token>".

API keys stay in your local .env — never sent over the wire.

Bare CLI (shell scripts, library use)

If you don't need any of the above and just want the binary:

go install github.com/jedi4ever/social-skills/cmd/social-fetch@latest
# or download a release tarball from
# https://github.com/jedi4ever/social-skills/releases — files like:
#   social-skills-<version>-darwin-arm64.tar.gz
#   social-skills-<version>-darwin-amd64.tar.gz
#   social-skills-<version>-linux-amd64.tar.gz

Build from source:

git clone https://github.com/jedi4ever/social-skills.git
cd social-skills && make build       # → ./dist/social-fetch

Requires Go 1.26+. Windows is not currently supported (the bridge daemon uses Unix-only syscalls — run via WSL).

Docker container (chromedp + ledger + MCP-over-SSE in one image)

The repo ships a Dockerfile + docker-compose.yml that bundles the three long-running services into one container:

make docker-build         # build social-skills:<version> + :latest
make docker-compose-up    # bring up the stack with named-volume persistence
# … or:
docker compose up --build

Three things to know:

1. `MCP_AUTH_TOKEN` — set in your environment (or .env)

before exposing the container beyond loopback. The MCP HTTP endpoint (5558) requires Authorization: Bearer <token> when the env var is set; without it the endpoint is unauthenticated and only safe on 127.0.0.1.

1. Ledger persistence — the named volume social-skills-ledger

stores ledger.db, the markdown mirror tree, and the screenshots/ directory under /data. docker compose down keeps it; docker compose down -v wipes it.

1. Daytona / remote deploy — clone the repo into the workspace,

make docker-compose-up, and point Claude Desktop / claude.ai at the workspace's tunnel URL for port 5558. The headless container's browser fingerprint stays anonymous; cookies are not honoured (the bridge transport is the path for authenticated sites and it isn't shipped inside the container — see extensions/chrome for the host-side bridge).

The bridge daemon (5555) is intentionally not in the container — it drives a real logged-in Chrome on the user's machine via a companion extension, which has no equivalent inside a container. Bridge-as-screenshot-source is a planned followup.

Social platforms supported

Fetch (URL → structured Item)

Search

Ask (grounded answer engines)

-p auto walks the default chain; -p name1,name2 runs your own fallback order. -p auto skips providers without keys configured.

Timeline (recent activity for a user)

CLI commands

social-fetch fetch     <url> [<url>…]      pull URL(s) into structured Item(s)
social-fetch search    "<query>"           run a search query
social-fetch ask       "<question>"        ask a grounded answer engine
social-fetch timeline  <user-or-url>       recent activity for a user
social-fetch research  "<question>"        EXPERIMENTAL multi-angle research
social-fetch mcp       [flags]             run as an MCP server (stdio or HTTP)
social-fetch bridge    {start|stop|status|run}    Chrome browser-bridge daemon
social-fetch monitor   [--tail N]          tail the global audit log
social-fetch list                          list every fetch / search / ask / timeline provider
social-fetch help      [subcommand]        full flag reference per subcommand
social-fetch version                       print the version

social-fetch help <subcommand> is the authoritative flag reference — every flag has a short and long form; output is shaped to be parseable by agents.

fetch

social-fetch fetch <url>... [-f markdown|json|jsonl] [-o -|FILE|DIR/]
                           [-i FILE|-] [-j N] [--no-comments]
                           [--max-comments N] [--timeout DUR] [-l -|FILE]

Multiple URLs + -f json auto-promotes to jsonl (one item per line). Stdin auto-detected when piped: cat urls.txt | social-fetch fetch. -j N keeps results in input order despite concurrency.

search

social-fetch search "<query>" [-p auto|<provider>|<chain>] [-n N]
                             [--last 7d|--after YYYY-MM-DD|--before …]
                             [--site DOMAIN[,…]] [-f markdown|json|jsonl]

Date filters are native where the provider supports them and client-side otherwise. See HINTS.md for per-provider date-filter quirks.

ask

social-fetch ask "<question>" [-p auto|<provider>|<chain>] [--model NAME]
                             [--recency day|week|month|year]
                             [--max-tokens N] [--instructions "…"]

--recency is honored where the upstream supports it; --instructions is the system-prompt-style preamble (perplexity, grok, openai, anthropic, google).

timeline

social-fetch timeline <handle-or-url> [-p x|linkedin]
                                     [--kind all|tweets|replies|retweets|posts|comments|reactions]
                                     [--last 7d] [-n N] [--expand] [--no-reshares]

Auto-detects the platform from the URL or @-prefix. --expand deep-fetches each post (LinkedIn — slower but fuller content).

research (experimental)

social-fetch research "<question>" [--orchestrator <ask-provider>]
                                  [--max-angles N] [-j N] [--json]

Decomposes the question into 3-8 angles, fans out parallel fetch/search/ask/timeline calls, and synthesizes a markdown answer with citations. Costs ~2 LLM calls + N tool calls per question. Use when you'd otherwise issue 4-8 manual queries.

mcp

social-fetch mcp                       # stdio (Claude Desktop Extension)
social-fetch mcp --http :PORT          # Streamable HTTP (claude.ai, ngrok)
social-fetch mcp --ngrok               # spawn ngrok automatically

Exposes social_fetch_fetch, _search, _ask, _timeline, _research, _list_providers, _bridge_status, _read_file plus the seven social_ledger_* tools as MCP tools. Set MCP_AUTH_TOKEN for HTTP mode (auto-generated when --ngrok is combined with no env var). HTTP-mode tee's tool calls + outbound platform HTTP traffic to stderr live.

Output is file-based by default for the big-payload tools (_fetch, _ask, _research, _ledger_get). The tool result is a small JSON envelope containing metadata + a content_file path pointing at a temp file with the rendered markdown body. Read the body with the agent's built-in Read tool (Claude Code), or with social_fetch_read_file (Claude Desktop / any MCP-only client). This keeps article bodies, research reports, and grounded answers off the JSON-RPC encode path — much faster than streaming 50 KB of escaped markdown through a tool result. Pass inline: true on the producing tool to get the body inline instead.

The _fetch envelope adds a `hint` field when extraction looks suspiciously thin (small body from an article-source page — typically a JS-rendered SPA), nudging the agent to retry via the browser bridge or SOCIAL_FETCH_CHAIN_ARTICLE=jina.

The _ledger_get envelope adds a `provenance` field — auto-fetched when the entry came in via social_fetch_* (we extracted it ourselves), agent-recorded when stored via social_ledger_record (content came from Claude WebFetch / curl / hand-paste). Use it to weigh how much to trust a cached body before quoting from it.

bridge

social-fetch bridge start         # daemonize on :5555
social-fetch bridge status        # exit 0 connected / 1 not connected / 2 not running
social-fetch bridge stop          # SIGTERM
social-fetch bridge run           # foreground (no daemon)

One-time setup: load extensions/chrome/ in chrome://extensions/ (Developer mode → Load unpacked). Required for LinkedIn, Medium / Substack paywall fetches.

monitor

social-fetch monitor                  # follow ~/Library/Caches/social-fetch/audit.jsonl
social-fetch monitor --tail 200       # last N lines then follow

The audit log is always-on across CLI, MCP-stdio, and MCP-HTTP runs. Useful for tailing what an agent is actually invoking from another shell.

Output format

Every output — JSON or markdown — includes both fetched_at (when the data was pulled) and written_at (when this output was rendered) plus author, source, score, tags, and comment trees where applicable. JSON output uses a stable Envelope { written_at, item } shape; JSONL emits one envelope per line.

YouTube transcripts

Three transcript backends, switchable via YOUTUBE_TRANSCRIPT_PROVIDER:

auto (default) tries them in order yt-dlp → innertube → kkdai.

HTML → Markdown (and Jina Reader fallback)

For the catch-all article fetcher and any HTML-rendering source, two knobs control how the page becomes markdown the agent can read:

Converter (HTML2MD_PROVIDER)

Per-platform fetch chain (SOCIAL_FETCH_CHAIN_<PLATFORM>)

Every fetcher walks a configurable chain of transport methods (bridge, http, headless, jina, api, syndication). Operators override per-platform via the env var; defaults live in the platform Go code (internal/platforms/*/fetch.go's defaultChain var) so they don't drift.

The table below mirrors the code at the time of writing — if they ever disagree, the code wins. Run social-fetch hints <platform> for the canonical per-platform recipe (when to override, transport-specific quirks).

Transport quick-reference:

• `bridge` — your real, logged-in Chrome via the

social-fetch extension. Required for auth-walled content (LinkedIn comments, Medium / Substack member-only posts). Needs social-fetch bridge start + extension loaded.

• `headless` — local stealth Chromium via chromedp. Anonymous

but JS-rendering. Faster with the daemon (see "Headless browser pool" below); falls back to per-call spawn when daemon's down.

• `http` — plain HTTP GET. Fastest, no JS. Some sites

return a JS shell to non-browsers — those need headless.

• `jina` — r.jina.ai/<url> remote service. Anti-CF +

headless rendering on Jina's infrastructure. Free tier has rate limits; set JINA_API_KEY for paid.

• `api` / `syndication` — platform's structured API

(Twitter v2, HN Algolia, etc.) where applicable.

Examples:

# air-gapped — never reach for Jina
SOCIAL_FETCH_CHAIN_ARTICLE=http,bridge,headless

# always anonymous LinkedIn (skip the bridge)
SOCIAL_FETCH_CHAIN_LINKEDIN=headless,jina

# bypass Twitter v2 even when keys are set
SOCIAL_FETCH_CHAIN_TWITTER=syndication,jina

Set JINA_API_KEY if any chain includes jina and you want higher rate limits than the free tier.

#### Deprecated: HTML2MD_READER=jina

Pre-dates the per-platform chain. Equivalent today: SOCIAL_FETCH_CHAIN_ARTICLE=jina,http,bridge. Still honoured for one release with a deprecation line in the audit log; will be removed after.

Browser bridge (LinkedIn, Medium / Substack paywalls)

LinkedIn (and the member-only paths of Medium / Substack) only return useful content to an authenticated session. social-fetch's answer is a small Chrome MV3 extension at extensions/chrome/ that opens a WebSocket to a local daemon (social-fetch bridge run) and brokers requests through your real, logged-in browser tab. Public content keeps going direct over HTTP — the bridge is only used when a fetcher explicitly opts in.

# one-time: load extensions/chrome/ in chrome://extensions/
#   (Developer mode → Load unpacked → pick the directory)
social-fetch bridge start         # daemonize on :5555
social-fetch bridge status        # 0 connected / 1 not connected / 2 not running
social-fetch bridge stop          # SIGTERM the daemon

Ledger daemon (sandboxed agents + remote use)

The local SQLite ledger can run as a long-lived HTTP daemon — the sole owner of the database file. Useful when:

• The MCP server runs in a sandbox without filesystem access to

the ledger DB.

• The MCP server runs on a different machine than the ledger.
• You want write coordination across multiple concurrent fetchers

(the daemon serialises so the SQLite WAL contention story stays simple).

social-ledger daemon start [--port 5557] [--bind 0.0.0.0:5557]
social-ledger daemon status     # one-shot snapshot
social-ledger daemon stop

When the daemon is reachable, every caller — social-ledger subcommands, social-fetch's auto-ingest, MCP tools — routes through HTTP. When it's not, callers fall back to opening the SQLite file directly. Switch modes by starting / stopping the daemon; no per-call config knob.

Bonus: in daemon mode, MCP's social_fetch_fetch returns content_url (HTTP pointer to /content?url=…) instead of content_file (local path), so an agent on a different machine or in a sandboxed env can read the body over HTTP.

Env vars:

Local browser bookmarks

social-fetch bookmarks list reads Chrome's local Bookmarks JSON and emits matching entries. Date-range filterable, multi-profile aware, no extension or daemon needed.

social-fetch bookmarks list --since 2026-04-01      # added since April
social-fetch bookmarks list --folder-contains AI    # fuzzy match on folder name
social-fetch bookmarks list --folder "Bookmarks bar/AI"   # exact subtree
# Or set $SOCIAL_FETCH_BOOKMARKS_ROOT_FOLDER once and skip --folder.
social-fetch bookmarks list --all-profiles -f json  # JSON, every profile
social-fetch bookmarks profiles                     # which profiles exist

--platform chrome is today's default + only supported value. Future platforms (Twitter / X bookmarks, Reddit saved posts — server-side, account-scoped) plug in as additional values.

Headless browser pool (anonymous JS-rendered fetches)

Separate from the bridge. The social-browser binary daemonises a pool of warm Chromium browsers via chromedp — anonymous (no session reuse, never touches your real Chrome profile), fast (~3s warm-tab vs ~5-6s cold-spawn), used by article / LinkedIn / Medium / Substack chains when headless is in the chain.

social-browser daemon start --provider local --pool-size 2 --recycle-after 50
curl -s http://127.0.0.1:5560/status | jq .   # snapshot
curl -s http://127.0.0.1:5560/monitor          # text view
social-browser daemon stop

When no daemon is running, social-fetch's headless transport returns a clean error pointing at this start command — there is no in-process fallback. social-fetch autodetects the daemon at http://127.0.0.1:5560 (override with SOCIAL_FETCH_HEADLESS_DAEMON_URL).

For remote pools (run chromedp in Daytona sandboxes, point clients at the local proxy daemon), use --provider daytona — see the social-browser provider section of social-browser help.

Pre-v0.15.0 had a social-fetch headless subcommand; that's gone. Same defaults moved to social-browser daemon start --provider local.

Permissions model — narrow by default, broad on opt-in.

>

The extension's static host_permissions cover only the named social sites: linkedin.com, x.com / twitter.com, medium.com, substack.com. Those are what the install dialog asks for; nothing else.

>

If you want to fetch arbitrary HTTPS pages through your authenticated browser (Reddit, GitHub, Notion, …), open the extension popup → Site permissions → toggle "Allow on all sites". That triggers Chrome's standard runtime-permission prompt and grants https://*/* until you toggle it off again (or remove it from chrome://extensions/). The toggle is reflected back from Chrome's permission state, so an external revoke shows up the next time you open the popup.

>

Without the broad toggle on, asking the bridge to fetch a non-listed site fails fast with a permission_required error — the daemon surfaces it to the CLI so you know exactly what to click.

>

Other practical bits:

>

- The daemon listens on 127.0.0.1:5555 only — not exposed to the network. Anything running locally that can reach that port can still drive the extension, so treat the bridge like any other localhost dev service. - Source is in extensions/chrome/ (~10 small files); audit background.js + content.js if you'd like to see the actual surface. - Toggle the extension off in chrome://extensions/ when you're not using social-fetch and the host permissions stop applying entirely.

Credentials

All API keys are optional — features gated on a missing key degrade gracefully. The binary auto-loads .env files on startup (no override of shell-exported vars):

1. ./.env — current working directory
2. parents of cwd, walked upward
3. <binary_dir>/.env — next to the installed binary

See API_KEYS.md for step-by-step setup per provider — where to sign up, what scope to grant, what's in the free tier, and which env var to set. See HINTS.md for known gotchas (rate-limit cliffs, Cloudflare blocks, auth landmines).

License & disclaimer

Released under the MIT License — free to use, modify, and redistribute. The license itself includes the standard "as is, no warranty, no liability" clauses; what follows is an explicit plain-language version of those for the AI-specific bits.

social-fetch is plumbing for AI agents — and AI agents make mistakes. This tool fetches third-party content (HackerNews, Reddit, X, LinkedIn, articles, …) and routes it through LLMs (Perplexity, OpenAI, Anthropic, Gemini, Grok, …) that can hallucinate, misattribute, paraphrase incorrectly, and surface outdated information. Every answer in this stack is a best-effort synthesis, not a verified fact.

>

Things you should NOT do based solely on social-fetch output, without independent verification:

>

- make legal, medical, or financial decisions - quote the output as if it were the source - act on factual claims that aren't covered by the citations social-fetch returns alongside the answer - assume a missing citation means the claim is unsupported (or the inverse — that a citation means the claim is correct)

>

The source / url / fetched_at metadata on every result exists precisely so you can click back to the original. Do that for anything you care about. Treat social-fetch like a research assistant who's read a lot but might be wrong about any specific detail — useful for breadth, not authoritative on accuracy.

>

The author and contributors accept no liability for decisions made on the basis of output from this tool, third-party API responses it relays, or downstream agents that consume it.

See also

• INSTALL.md — full install guide for all four channels
• API_KEYS.md — per-provider auth setup
• HINTS.md — operator-grade "things that surprise you"
• CLAUDE.md — repo conventions for contributors
• LICENSE — MIT