Skip to content

Improve PyMuPDF4LLM OCR pipeline and benchmark results - #86

Open
hashiromer wants to merge 1 commit into
run-llama:mainfrom
pymupdf:pymupdf4llm-rapidtess
Open

Improve PyMuPDF4LLM OCR pipeline and benchmark results#86
hashiromer wants to merge 1 commit into
run-llama:mainfrom
pymupdf:pymupdf4llm-rapidtess

Conversation

@hashiromer

Copy link
Copy Markdown

Summary

Improves the existing pymupdf4llm_markdown pipeline and enables layout and table evaluation.

This PR also updates the pipeline documentation, dependency setup, tests, and leaderboard results. No additional PyMuPDF4LLM pipeline variants are added.

Benchmark Results

The full ParseBench run completed successfully:

  • Total examples: 2,078
  • Successful: 2,078
  • Failed: 0
Metric Score
Overall 50.27
Tables 57.05
Charts 2.29
Content Faithfulness 75.25
Semantic Formatting 58.39
Visual Grounding 58.38

Reproducing the Results

The submitted results were produced in the following environment:

  • CPU: 8 Intel Haswell-class vCPUs
  • Memory: 24 GiB RAM
  • Operating system: Ubuntu 26.04 LTS
  • Python: 3.12
  • Tesseract: 5.5.0
  • Maximum concurrency: 1
  • Full-run time: approximately 1 hour

Install Tesseract with English language data:

sudo apt-get install tesseract-ocr tesseract-ocr-eng

Install the pipeline dependencies:

uv sync --python 3.12 --extra pymupdf4llm

Run the small test benchmark:

uv run --python 3.12 --extra pymupdf4llm parse-bench run pymupdf4llm_markdown --test --max_concurrent 1

Run the full benchmark used to produce the submitted scores:

uv run --python 3.12 --extra pymupdf4llm parse-bench run pymupdf4llm_markdown --max_concurrent 1

Validation

  • Full benchmark completed without inference failures
  • 297 tests passed
  • Ruff and mypy checks passed
  • Lockfile validation passed

@hashiromer
hashiromer force-pushed the pymupdf4llm-rapidtess branch from 0a1e33e to 0c922ef Compare July 28, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant