Content Monitor

Concept of the skill

Building and operating multi-source intelligence pipelines across video, GitHub, Reddit, curated lists, search, and RSS — source adapters, the discovery/transcription/summarization/evaluation phases, deduplication, schedules, per-stage model choice, and actionable brief generation.

Domain Context

What is this skill? This skill provides expertise for a multi-source intelligence pipeline covering YouTube channels, GitHub trending/topic/search feeds, Reddit subreddits, awesome-lists, Google Search, and RSS/Atom feeds (including Hacker News, changelogs, blogs, arXiv, and podcasts). Covers source adapter patterns, unified evaluation pipeline, deduplication, scheduling cadence, and actionable intelligence brief generation. Use when adding new content sources to the pipeline, configuring monitoring schedules, understanding the discover/transcribe/summarize/evaluate phases, extending or debugging the discovery pipeline, or deciding which model to use at each pipeline stage. Do NOT use for SEO keyword research — use the keywords skill. Do NOT use for competitive product analysis — use user-research-synthesis.

Key Files

Workflow

Use the ordered phases, checklists, and guardrails in the sections below as the canonical workflow for this skill. When multiple subsections describe steps, follow them in the order presented.

Coverage

The multi-source intelligence pipeline that tracks AI coding, e-commerce tooling, and agent infrastructure trends. Covers the live source groups configured in scripts/content-monitor/sources.json and scripts/content-monitor/channels.json, the 4-phase pipeline (Discover, Extract, Summarize, Evaluate), the MiniMax/GPT-5.4 model split, source adapter contract, deduplication via seen-items.json (at .content-monitor/), scheduling cadence per source type, evaluation rubric dimensions, and the daily digest output format. Source code lives at scripts/content-monitor/. State files live at .content-monitor/ (the Development repo root).

Philosophy of the skill

Staying current on AI tooling, agent patterns, and e-commerce integrations is essential for a system that competes on engineering velocity. Without this pipeline, the team would manually scan dozens of sources and miss emerging patterns that could inform skill creation, architecture decisions, or competitive positioning. The pipeline automates discovery and extraction (cheap) and reserves expensive model reasoning for evaluation (scarce), following the same constraint-awareness principle that governs the rest of the system.

Location Precision

State persistence is critical for this pipeline. Always refer to:

• Source code and configuration: scripts/content-monitor/ (discover-*.js, evaluate.js, sources.json, run.js)
• State and results: .content-monitor/ at the Development root (seen-items.json, digests, logs, reports)

This separation keeps persistent state separate from executable code, ensuring deduplication survives code updates.

1. When This Skill Applies

1.5. Important: State vs. Source Code Locations

This is a critical distinction that prevents bugs:

• Source code (scripts, adapters, runners): scripts/content-monitor/
• State files (persistent data, results): .content-monitor/ at the Development repo root

The seen-items.json file used for deduplication MUST be read from and written to .content-monitor/seen-items.json, not scripts/content-monitor/. This is the single source of truth for what has already been processed.

2. Pipeline Architecture

The pipeline has 4 sequential phases. Each phase is independently runnable via --discover-only, --skip-evaluate, etc.

Why this model split matters

MiniMax M2.5K is free (200 req/day via OpenRouter) and handles the volume of extraction-to-summary work without cost. GPT-5.4/Gemini is reserved for the evaluation step where quality is the bottleneck — scoring relevance and generating insights that the developer will act on.

Summary JSON schema (produced by Summarize phase)

Each summary contains:

keyFindings         — array of strings, concrete observations
toolsMentioned      — array of tool names
techniquesMentioned — array of technique names
actionableInsights  — array of strings, recommended actions
topicTags           — array of strings for deduplication grouping
oneLineSummary      — string, max 120 chars
estimatedRelevance  — integer 1-10, self-assessed by model

3. Source Types

3.5. Adapter Registry

Maps each source type (from section 3) to the adapter file that handles it. There is no 1-to-1 mapping — discover-rss.js consolidates five source types.

Key architecture note

discover-rss.js is the generic RSS handler. It handles Hacker News, changelogs, blogs, arXiv, podcasts, and can also ingest YouTube RSS — but it lacks YouTube-specific features (transcript extraction, channel metadata). When adding a new RSS-based source, route it through discover-rss.js. When adding a YouTube channel, use discover.js.

There is no `discover-youtube.js`. Do not create one — YouTube RSS is already handled by discover.js with transcript support.

4. State Files

All state files live in .content-monitor/ (the Development repo root directory), NOT in scripts/content-monitor/.

5. Configuration

Config file: scripts/content-monitor/sources.json (multi-source) and channels.json (YouTube legacy).

6. Extending the Pipeline

Before creating a new adapter, check section 3.5 (Adapter Registry). If the new source is RSS/Atom-based (blogs, changelogs, podcasts, arXiv, Hacker News), route it through the existing discover-rss.js — no new file needed. Only create a new discover-<type>.js for source types with fundamentally different discovery mechanics (HTML scraping, vendor APIs, sitemaps).

To add a new source type that needs a new adapter:

1. Create a new adapter file (e.g., discover-reddit.js) in the scripts/content-monitor/ directory implementing the standard adapter interface: fetchNew(config) → Array<{ id, url, title, rawText }>
2. Register the new source in sources.json
3. Implement text extraction in extract-text.js or transcribe.js
4. Update section 3.5 of this skill to add the new adapter to the registry
5. Test with --discover-only
6. Run a full pass and verify output in seen-items.json

Adapter contract: The adapter must return a stable id per item. This is what seen-items.json keys on for deduplication.

Verification

After modifying the content monitor pipeline, verify:

• [ ] New sources have a stable id field for deduplication (test with --discover-only)
• [ ] seen-items.json correctly records discovered items without duplicates
• [ ] Frequency matches the source's update cadence (daily for YouTube/Reddit/blogs, weekly for GitHub topics/awesome-lists)
• [ ] Summarize phase produces all required JSON fields (keyFindings, toolsMentioned, actionableInsights, etc.)
• [ ] Evaluate phase generates a scored digest entry in .content-monitor/digests/YYYY-MM-DD.md
• [ ] No API rate limits are exceeded (GitHub: 60 req/hr unauthenticated, Google: 100/day)
• [ ] Cost remains zero -- MiniMax for summarization, free APIs for discovery

Do NOT Use When