pdf-to-md

Turn a PDF into Markdown. The right path depends on what the document is and whether an OCR API key is available:

• Scientific paper → produce the canonical paper-to-md bundle (Markdown +

section_audit.json + article.json) so it can feed csag-extraction. Convert with the OCR API when a key is set, else LiteParse v2 locally.

• Any other PDF (reports, slides, letters, forms) → just convert to Markdown

with LiteParse v2 for a fast, local, no-key result. Stop there.

LiteParse must be v2 (run-llama/liteparse, the Rust rewrite with the LiteParse Python API and lit CLI). LiteParse v1 is a different, unsupported API. liteparse_to_md.py pins liteparse>=2,<3 and refuses to run on anything else, so uv run always provisions the right per-platform v2 binary inside the wheel — nothing to vendor or compile, and no API key. OCR is on by default (bundled Tesseract).

LiteParse output is a draft, not the deliverable. LiteParse is a mechanical parser: it has no native Markdown, infers headings from font size/weight, and introduces artifacts (split words, broken hyphenation, dropped author blocks, merged columns). Whenever LiteParse is the engine, the LLM running this skill is responsible for shaping that draft into the right form — see "Shape the LiteParse output" below. The OCR API engine needs far less shaping.

Instructions

Step 0 — Classify the document and pick a path

Check for a key with printenv OCR_API_KEY NELLI_API_KEY. If the paper is scanned or layout-heavy and no key is set, LiteParse v2 still works (it OCRs locally), but layout fidelity is lower than the OCR API.

Mode A — Scientific paper (full bundle)

Produces, beside the input, for stem <stem>: <stem>.md, <stem>.section_audit.json, <stem>.article.json (and optionally <stem>.ocr.json, <stem>.job.json, figure_review/).

1. Convert to Markdown with the first engine that fits.

OCR API (preferred when a key is available; best layout fidelity):

   uv run skills/pdf-to-md/scripts/ocr_api_job.py \
     /path/to/input.pdf --output-dir /path/to/output-dir

The helper auto-detects a local OCR host (http://127.0.0.1:8002/ocr) and otherwise uses the remote API (https://api.newlineages.com/ocr). Override with OCR_BASE_URL or --base-url.

LiteParse v2 fallback (no key required):

   uv run skills/pdf-to-md/scripts/liteparse_to_md.py \
     /path/to/input.pdf --output-dir /path/to/output-dir

If you used the LiteParse engine, shape `<stem>.md` before continuing — see "Shape the LiteParse output" below. The downstream steps only work as well as the Markdown they read.

1. Build the section audit:

   uv run skills/pdf-to-md/scripts/build_section_audit.py /path/to/output-dir/<stem>.md

1. Populate the first-pass article JSON (also writes the audit):

   uv run skills/pdf-to-md/scripts/populate_article_json.py /path/to/output-dir/<stem>.md

This is a first pass. Review and complete fields the heuristics miss (authors with superscripts, methods, references, figure interpretation) against the Markdown and the article schema.

1. Render figure pages when figure/table captions are present, then fill

figure_interpretation from captions plus the rendered pages:

   uv run skills/pdf-to-md/scripts/render_pdf_pages_to_png.py \
     /path/to/input.pdf --output-dir /path/to/output-dir/figure_review

1. Validate against the schema and the section audit:

   uv run skills/pdf-to-md/scripts/validate_article_json.py \
     /path/to/output-dir/<stem>.article.json \
     --scientific-paper \
     --section-audit /path/to/output-dir/<stem>.section_audit.json

Resolve every reported error before stopping. A missing field that is genuinely absent from the source is fixed by confirming absence, not by inventing content.

You may also start Mode A from a Markdown file you already trust — skip step 1 and run steps 2–5 on that .md.

Mode B — Any other PDF (fast Markdown)

One step, fully local, no key:

uv run skills/pdf-to-md/scripts/liteparse_to_md.py \
  /path/to/input.pdf --output-dir /path/to/output-dir

Useful flags: --no-ocr (faster on text-based PDFs), --ocr-server-url URL (higher-accuracy OCR server), --target-pages "1-5,10", --max-pages N, --password PW. The converter detects the title and section headings from font size and weight, filters page furniture (watermarks, running headers, repeated footers), and reflows text into paragraphs — then shape the result (next section).

Shape the LiteParse output (required when LiteParse is the engine)

LiteParse v2 gives a fast first draft. Because it is mechanical, you (the LLM running this skill) must read <stem>.md against the rendered pages and bring it into the right shape before treating the conversion as done. Do not hand back raw script output. Fix what the heuristics cannot:

• Title — confirm # is the real title, not a journal banner, DOI line, or

"Downloaded from…" watermark; set it correctly if wrong or missing.

• Headings — promote section headings the font heuristic missed (## Abstract,

## Introduction, ## Methods, ## Results, ## Discussion, ## References, etc.) and demote false positives; keep reading order.

• Broken words — rejoin words split mid-token (e.g. "Berke ley" → "Berkeley")

and fix hyphenation that did not rejoin across line breaks.

• Front matter — reconstruct the author list and affiliations, which LiteParse

often drops or scrambles around superscripts and email addresses.

• Captions & tables — keep one figure/table caption per block; rebuild simple

tables that collapsed into runs of text.

• Residual furniture — delete any leftover running headers, page numbers, or

license boilerplate the filter missed.

• References — ensure each reference is its own entry, not one merged blob.

For Mode A, after this Markdown cleanup run populate_article_json.py, then complete every article.json field the first-pass heuristics leave empty (authors, affiliations, methods, references, figure_interpretation) from the shaped Markdown and rendered pages, so validation passes for the right reasons — never by inventing content. For Mode B, the shaped Markdown is the deliverable.

Quick Reference

All commands run from the repo root with uv run skills/pdf-to-md/scripts/<name>. liteparse_to_md.py and render_pdf_pages_to_png.py carry PEP 723 inline dependencies (liteparse, pypdfium2) that uv run installs automatically; the remaining scripts are standard-library only.

Input Requirements

• A PDF, or a format LiteParse converts to PDF first (DOCX/PPTX/XLSX/ODT/CSV via

LibreOffice; JPG/PNG/TIFF/etc. via ImageMagick).

• For Mode A from existing Markdown: a .md with a clear # Title, an

author/affiliation block, recognizable section headings (Abstract, Introduction, Methods, Results, Discussion, Conclusion, References), and figure/table captions starting with Fig./Figure/Table.

• For the OCR API engine: OCR_API_KEY or NELLI_API_KEY, plus curl.
• A writable --output-dir (keep it outside this repository).

Output

• Mode B: <stem>.md, plus <stem>.ocr.json and <stem>.job.json provenance.
• Mode A: the above plus <stem>.section_audit.json and <stem>.article.json;

optionally figure_review/ PNGs. csag-extraction consumes <stem>.md and <stem>.article.json; everything else is provenance.

• The article JSON has exactly these keys: title, authors, affiliations,

abstract, main, methods, figure_legends (list), figure_interpretation, references (list). See references/article_schema.md and references/article.yaml.

Quality Gates

• The conversion engine is LiteParse v2 (or the OCR API); <stem>.job.json

records tool_version 2.x for the LiteParse engine.

• When LiteParse was the engine, the Markdown has been shaped (title, headings,

rejoined words, front matter, captions, references) — not handed back raw.

• Mode B Markdown is non-empty, has a sensible # title (or none, never a

watermark), and is free of repeated page furniture.

• Mode A: validate_article_json.py --scientific-paper returns OK.
• title, authors, and main are populated for a real paper, or their absence

is confirmed against the source (do not fabricate).

• When figure/table captions exist, figure_legends is populated and

figure_interpretation is filled (or an explicit no-interpretation note is recorded).

• Provenance (<stem>.job.json) records the engine, tool version, and OCR setting.
• No test inputs or outputs are written inside this repository.

Examples

Fast Markdown from a non-paper PDF:

uv run skills/pdf-to-md/scripts/liteparse_to_md.py report.pdf --output-dir /tmp/out --no-ocr
# -> /tmp/out/report.md  (+ report.ocr.json, report.job.json)

Full paper bundle with no OCR key (LiteParse v2 engine):

DIR=/tmp/paper
uv run skills/pdf-to-md/scripts/liteparse_to_md.py paper.pdf --output-dir "$DIR"
uv run skills/pdf-to-md/scripts/populate_article_json.py "$DIR/paper.md"
uv run skills/pdf-to-md/scripts/validate_article_json.py \
  "$DIR/paper.article.json" --scientific-paper \
  --section-audit "$DIR/paper.section_audit.json"

Troubleshooting

• `liteparse is not installed`: run the script with uv run skills/pdf-to-md/scripts/liteparse_to_md.py (not uv run python ...), so uv reads the PEP 723 inline dependency. Offline: pip install 'liteparse>=2,<3' first.
• `pdf-to-md requires LiteParse v2`: a v1 (or non-v2) liteparse is installed. Reinstall with pip install 'liteparse>=2,<3'; the skill targets the run-llama Rust rewrite (LiteParse Python API), not the old v1 API.
• Title is a journal banner, watermark, or "Downloaded from…" line: the converter filters furniture and repeated headers; if one slips through, remove it in the Markdown before step 2, or note that article_extraction re-derives the title from the body.
• `authors`/`methods`/`references` empty on a real paper: the first-pass heuristics miss superscript-heavy author lines and short note formats. Fill them by hand from the Markdown; this is expected, not a converter failure.
• Scanned/image-only PDF gives little text: keep OCR enabled (default) and raise --dpi, or point --ocr-server-url at EasyOCR/PaddleOCR; for best fidelity use the OCR API engine.
• `Missing OCR API key`: set OCR_API_KEY/NELLI_API_KEY, or use the LiteParse v2 engine instead.
• Garbled equations or merged columns: LiteParse is the fast path; for layout-heavy papers prefer the OCR API engine.