Reference Verifier & Citation Auditor

Performs a comprehensive 4-level audit of academic references in uploaded manuscripts. Goes beyond simple existence checks to verify metadata accuracy, topical relevance, and contextual correctness of each citation.

The Four Verification Levels

Level 1: Existence Verification

Does this reference actually exist as a real publication?

Level 2: Metadata Verification

Are the authors, title, year, journal, volume, issue, and pages correct?

Level 3: Topical Relevance

Does the cited paper actually discuss the topic it is being cited for?

Level 4: Contextual Accuracy

Is the citation used correctly? Does the source truly support, contradict, or remain neutral toward the claim made in the manuscript — matching how it is presented?

Workflow

Step 0: Parse the Manuscript

1. Read the uploaded document from /mnt/user-data/uploads/
2. Extract TWO things:

a. The reference list (bibliography section at the end) b. The full manuscript text with in-text citations preserved — this is critical for Levels 3-4

For each in-text citation, identify:

• The citation marker (e.g., [1], [Smith 2023], superscript number)
• The surrounding sentence/paragraph (the citation context)
• What claim is being made at that point
• How the citation is framed (e.g., "Smith et al. demonstrated...", "consistent with previous findings [3]", "in contrast to earlier reports [7]")

Build a citation map: a structured list linking each reference to every place it is cited in the manuscript with its context.

Use Python with appropriate libraries:

• For PDF: pymupdf (fitz) or pdfplumber
• For DOCX: python-docx
• Install with pip install --break-system-packages

Step 1: Level 1 — Existence Verification

For each reference, attempt to find it in databases:

#### 1a. DOI Check (if DOI present)

• Use web_fetch on https://api.crossref.org/works/{DOI}
• Valid response = reference exists

#### 1b. PubMed Search

• Use PubMed:search_articles to search by title keywords and first author
• If found, use PubMed:get_article_metadata to get full details
• Use PubMed:convert_article_ids to get DOI/PMCID if missing

#### 1c. CrossRef Title Search (fallback)

• Use web_fetch on https://api.crossref.org/works?query.bibliographic={encoded_title}&rows=3
• Check if any result matches

#### 1d. Scholar Gateway (additional fallback)

• Use Scholar Gateway:semanticSearch with the reference title as query
• Check if returned results match

Status assignment:

• EXISTS — Found in at least one database
• NOT_FOUND — Not found in any database (possible fabrication)

Step 2: Level 2 — Metadata Verification

For references that exist, compare the manuscript's reference entry against database records:

Use the scripts/ref_utils.py helper for fuzzy matching.

Status assignment:

• METADATA_CORRECT — All fields match
• METADATA_MINOR_ERRORS — 1-2 minor discrepancies (e.g., page range off by one, journal abbreviation difference)
• METADATA_MAJOR_ERRORS — Significant mismatches (wrong year, wrong journal, wrong first author)

For each error, record exactly which fields are wrong and what the correct values are.

Step 3: Level 3 — Topical Relevance Check

For each citation, determine whether the cited paper actually covers the topic it is cited for.

#### 3a. Retrieve the cited paper's content Try in this order:

1. Full text via PMC: Use PubMed:convert_article_ids to get PMCID, then PubMed:get_full_text_article
2. Abstract via PubMed: Use PubMed:get_article_metadata (always has abstract)
3. Scholar Gateway: Use Scholar Gateway:semanticSearch with a query combining the paper's title and the topic it's cited for
4. bioRxiv/medRxiv: If it's a preprint, use bioRxiv:get_preprint

#### 3b. Identify the cited topic From the citation map (Step 0), determine what specific topic/claim the manuscript cites this reference for. For example:

• Manuscript says: "The incidence of NEC is approximately 7% in VLBW infants [12]."
• Cited topic: NEC incidence rate in VLBW infants

#### 3c. Check topical coverage Analyze the retrieved content (full text or abstract) to determine:

• Does the cited paper discuss this specific topic?
• Is the topic a central focus or just briefly mentioned?

Status assignment:

• TOPIC_CONFIRMED — The cited paper clearly covers the cited topic
• TOPIC_PERIPHERAL — The paper mentions the topic but it is not a main focus
• TOPIC_NOT_FOUND — The paper does not appear to discuss the cited topic at all
• TOPIC_UNVERIFIABLE — Could not retrieve enough content to assess (only title available)

Step 4: Level 4 — Contextual Accuracy Check

The most critical level. Determine whether the citation is used correctly in the manuscript context.

#### 4a. Classify citation usage in the manuscript Read how the citation is used and classify the manuscript's claim:

• SUPPORTS: The manuscript presents the citation as supporting/confirming a claim
• Phrases: "as shown by", "consistent with", "demonstrated that", "confirmed by", "in agreement with"
• CONTRADICTS: The manuscript presents the citation as opposing/contradicting
• Phrases: "in contrast to", "unlike", "however, X found that", "contradicts", "despite findings by"
• NEUTRAL: Citation is used for background, definition, or without directional claim
• Phrases: "has been reported", "reviewed by", "defined as", "according to"
• METHODOLOGICAL: Citation is used to justify a method or approach
• Phrases: "as described by", "following the protocol of", "using the method of"
• QUANTITATIVE: Citation is used to cite a specific number, rate, or statistic
• Phrases: "reported a rate of X%", "found that N=", "estimated at"

#### 4b. Verify against source content Using the retrieved content from Level 3, check:

• Does the cited paper actually say what the manuscript claims it says?
• If a specific statistic is cited, does the source contain that number?
• If the manuscript says "Smith et al. found X", did they actually find X?
• Is the directionality correct (support vs. contradict)?

Status assignment:

• CITATION_CORRECT — The source supports the way it is cited
• CITATION_MISLEADING — The source is real and relevant, but the manuscript misrepresents its findings (e.g., claims support when the source actually found the opposite)
• CITATION_EXAGGERATED — The source partially supports the claim but the manuscript overstates it
• CITATION_IMPRECISE — Minor inaccuracies in how the source is represented (e.g., slightly wrong statistic)
• CITATION_UNVERIFIABLE — Cannot retrieve enough content to verify contextual accuracy
• CITATION_METHOD_ONLY — Cited for methodology; cannot assess claim accuracy

For each non-correct status, provide a specific explanation of the discrepancy.

Output Generation

Output 1: verified_references.bib

BibTeX file containing all references that passed Level 1 (existence), with corrected metadata from Level 2. Use scripts/ref_utils.py for generation.

@article{Smith2024,
  author    = {Smith, John and Doe, Jane},
  title     = {Corrected Title From Database},
  journal   = {Correct Journal Name},
  year      = {2024},
  volume    = {10},
  number    = {2},
  pages     = {100--110},
  doi       = {10.xxxx/xxxxx},
  pmid      = {12345678},
}

Output 2: reference_audit_report.md

Comprehensive markdown report organized as follows:

# Reference Audit Report

## Executive Summary
- Total references: N
- Level 1 (Existence): X verified, Y not found
- Level 2 (Metadata): X correct, Y with errors
- Level 3 (Topical): X confirmed, Y questionable
- Level 4 (Context): X correct, Y problematic

## Critical Issues (Action Required)
[References with NOT_FOUND, TOPIC_NOT_FOUND, CITATION_MISLEADING, or METADATA_MAJOR_ERRORS]

## Detailed Audit by Reference

### Reference [1]: Smith et al. (2024)
**Level 1 — Existence**: ✅ EXISTS (PMID: 12345678)
**Level 2 — Metadata**: ⚠️ MINOR ERRORS
  - Year in manuscript: 2023 → Correct: 2024
  - Pages in manuscript: 100-108 → Correct: 100-110
**Level 3 — Topical Relevance**:
  - Cited for: NEC incidence in preterm infants
  - Assessment: ✅ TOPIC_CONFIRMED — The paper's primary focus is NEC epidemiology
**Level 4 — Contextual Accuracy**:
  - Citation 1: "Smith et al. reported a NEC incidence of 7% [1]"
    - Usage type: QUANTITATIVE
    - Assessment: ⚠️ CITATION_IMPRECISE — Source reports 6.8%, manuscript rounds to 7%
    - Recommendation: Consider citing the exact figure (6.8%)
  - Citation 2: "These findings support early intervention [1]"
    - Usage type: SUPPORTS
    - Assessment: ✅ CITATION_CORRECT — Source concludes early intervention improves outcomes

[Repeat for each reference]

## Metadata Corrections Table
| Ref # | Field | In Manuscript | Correct Value |
|-------|-------|--------------|---------------|
| [3]   | Year  | 2023         | 2024          |
| [7]   | Pages | 45-50        | 45-52         |

## References Not Found
[Details on references that could not be verified]

Where to write the files

Write both outputs to the user's current working directory (not /mnt/user-data/outputs/ — that path only exists in the Claude.ai web sandbox and will not work in local CLI installs).

• reference_audit_report.md — the markdown report shown above
• verified_references.bib — the cleaned BibTeX

If the input manuscript file is named e.g. paper.pdf, prefix the outputs with the manuscript stem: paper_audit.md and paper_refs.bib. Otherwise fall back to the generic names.

Optional: HTML report

If the user asks for HTML (phrases like "as html", "html report", "open in browser"), also produce <stem>_audit.html next to the markdown. Use the first method that works:

1. pandoc (preferred):

   pandoc <stem>_audit.md -o <stem>_audit.html --standalone --metadata title="Reference Audit"

1. Python `markdown` library (fallback — pip install --break-system-packages markdown):

   python3 -c "import markdown; h=markdown.markdown(open('<stem>_audit.md').read(),extensions=['tables','fenced_code']); print('<!DOCTYPE html><html><head><meta charset=\"utf-8\"><title>Reference Audit</title><style>body{font-family:system-ui,sans-serif;max-width:900px;margin:2em auto;padding:0 1em;line-height:1.5}table{border-collapse:collapse}td,th{border:1px solid #ccc;padding:6px 10px}code{background:#f4f4f4;padding:2px 4px}</style></head><body>'+h+'</body></html>')" > <stem>_audit.html

After writing

In the chat reply, output:

• A one-line summary (e.g. "Saved audit → paper_audit.md, paper_refs.bib")
• The absolute path of every file written
• The Executive Summary block from the report (counts only)
• Highlight any critical issues (NOT_FOUND / CITATION_MISLEADING /

METADATA_MAJOR_ERRORS / RETRACTED)

Do not dump the full per-reference detail into chat after writing — point the user at the file.

Processing Strategy

For small reference lists (≤15 references)

Process all four levels for each reference sequentially. Give a progress update every 5 references.

For large reference lists (>15 references)

1. Run Level 1 (existence) for ALL references first
2. Run Level 2 (metadata) for all existing references
3. Ask the user which references to deep-audit at Levels 3-4, or do all if they request comprehensive audit
4. Levels 3-4 require fetching content for each cited paper — this is the slowest step

Rate limiting

• Add 0.5s delays between PubMed API calls
• Add 1s delays between CrossRef API calls
• Process in batches if needed

Important Notes

• Full text availability: Not all papers have full text in PMC. When only abstracts are available, Level 3-4 assessments are less certain — always note this limitation clearly.
• Citation context extraction: Pay careful attention to surrounding text. A single reference may be cited multiple times in different contexts — check ALL occurrences separately.
• Multiple citations in one bracket: When a manuscript cites [3-5] or [2,7,12], each reference needs its own contextual assessment.
• Preprints: Check bioRxiv/medRxiv for preprints. Note if a preprint has since been published in final form — the published version may differ.
• Non-English references: Flag for manual review if content cannot be retrieved in English.
• Self-citation: Note when the manuscript's authors cite their own work (not an error, but useful metadata).
• Retracted papers: If a cited paper has been retracted, flag this prominently as a CRITICAL issue.
• Review articles vs. primary sources: Note if a citation points to a review article when the context implies a primary source finding.

Dependencies

• Python packages: pymupdf, python-docx, pdfplumber (install as needed)
• Tools: PubMed MCP tools, Scholar Gateway, bioRxiv tools, web_fetch (for CrossRef API)
• Helper script: scripts/ref_utils.py — fuzzy matching, BibTeX generation, report formatting