CiteCheck — Paper Citation Verification

Overview

CiteCheck verifies academic paper citations by combining structured parsing with agent-native LLM evaluation. It supports LaTeX source files (preferred) and PDF fallback.

Workflow

1. Parse paper: Call citecheck CLI to read LaTeX (preferred) or PDF, extract references and body text
2. Format check: Call citecheck CLI to validate bibliography entries
3. Queryability verification: Call citecheck CLI to verify existence via Crossref / Semantic Scholar
4. Evaluate thematic relevance: Evaluate directly using the agent's reasoning capabilities — compare cited paper title/abstract/venue against the citing paper
5. Evaluate semantic accuracy: Evaluate directly using the agent's reasoning capabilities — compare in-text citation context against cited source content
6. Generate report: Aggregate all results into a Markdown report

Why matching steps are not done by the CLI The citecheck CLI can run standalone with optional --api-key for external LLM-powered matching. When used as a Skill, the host agent itself possesses LLM reasoning capabilities. Direct evaluation is faster, more consistent, and requires no additional API keys from the user.

1. Parse Paper (CLI)

LaTeX source (preferred)

citecheck path/to/latex_project/ --skip-verification --skip-semantic -o parsed_report.md

Or parse a single file:

citecheck main.tex --skip-verification --skip-semantic -o parsed_report.md

PDF (fallback)

citecheck paper.pdf --skip-verification --skip-semantic -o parsed_report.md

--skip-verification and --skip-semantic are required in Skill mode because steps 3–5 are performed directly by the agent.

2. Format Check (CLI)

Call citecheck to check and report format issues:

Detailed rules: see ./references/format-check-rules.md.

3. Queryability Verification (CLI + Agent Supplement)

Call citecheck to query Crossref and Semantic Scholar public APIs for citation existence.

API details: see ./references/api-reference.md.

If the CLI fails due to network/SSL issues, use WebSearch to directly query suspicious citations (especially those marked "unverifiable"), supplementing the verification results.

4. Evaluate Thematic Relevance (Agent Direct)

Do not call the CLI or any external API. Evaluate directly using the agent's reasoning capabilities.

For each reference, extract:

• Citing paper: title, abstract, keywords
• Cited paper: title, abstract (from API results or WebSearch), venue

Evaluation prompt template: see ./references/thematic-scoring-prompt.md.

5. Evaluate Semantic Accuracy (Agent Direct)

Do not call the CLI or any external API. Evaluate directly using the agent's reasoning capabilities.

For each in-text citation position:

1. Extract 1–2 sentences before and after the citation marker as citing_text
2. Obtain the cited paper's abstract via Semantic Scholar or WebSearch as cited_text
3. Evaluate semantic consistency using the prompt template: see ./references/semantic-matching-prompt.md.

6. Output Report

Generate a Markdown report containing:

Summary

• Total references
• Format issues count
• Query failures / suspicious count
• Average thematic relevance score
• Average semantic accuracy score

Detailed Results Table

Issue Summary

• List all findings and recommendations by severity

Dependencies

pip install CiteCheck

For PDF support:

pip install CiteCheck[pdf]

Notes

• No `--api-key` is required in Skill mode. Thematic and semantic matching are performed directly by the agent, with no external LLM API calls.
• Prioritize LaTeX source files; parsing accuracy is far higher than PDF.
• Add delays and retries to API calls to avoid rate limiting.
• When Semantic Scholar abstracts are missing, fall back to Crossref + title keywords.
• Semantic matching requires the cited paper's abstract; mark "source unreachable" when unavailable.
• If placeholder citations like =? or [?, ?] are found, mark them as "unverifiable references".