Skip to content

fix: rewrite PDF extraction -- real bytes, one clean read, geometry-based markup flag - #82

Merged
tamara-builds merged 1 commit into
mainfrom
fix/pdf-extraction-clean-and-visual-markup
Jul 21, 2026
Merged

fix: rewrite PDF extraction -- real bytes, one clean read, geometry-based markup flag#82
tamara-builds merged 1 commit into
mainfrom
fix/pdf-extraction-clean-and-visual-markup

Conversation

@tamara-builds

@tamara-builds tamara-builds commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Three fixes to `actions/extract`'s PDF handling, all surfaced while investigating why a bill's extracted text was duplicated (ID H0493):

  1. Raw PDF storage bug: the "original content" file (e.g. `H0493_Bill_Text.pdf`) was never actually the real PDF -- it was the already-extracted text, re-saved under a `.pdf`-looking filename, written in text mode. `extract_pdf()` now returns the genuine `raw_bytes` alongside the extracted text, written in binary mode. This matters for the plan to hand real PDFs to a vision-capable model for redline-heavy bills.
  2. Duplicated output: every extracted `.txt` wrote the same content twice, once as a "Section N:" breakdown and once as "Raw Text:". Section-splitting rarely produced more than one section for real bills (the PDF regex broke on any state that numbers every line, i.e. most of them), so the whole document became "Section 1" and then got printed again as the raw dump. Dropped section-splitting entirely (across XML/HTML/PDF) in favor of one clean read.
  3. Strikethrough detection replaced with a geometry-based visual-markup flag. The old heuristic guessed off font name/character spacing/color and produced unreadable garbage even when it fired. Replaced with a check for actual drawn line/rect objects (pdfplumber `page.lines`/`page.rects`) overlapping a character's vertical span -- catches both strikethrough and underline, which matter equally as a signal that a document needs full-fidelity (e.g. vision-LLM) handling rather than plain text extraction. This is a routing signal (`has_visual_markup`), not an attempt to reconstruct the marked text.

Test plan

  • Verified against ID H0493 (real redline bill): raw bytes start with genuine `%PDF-` magic bytes, text is a single clean 2545-char read with no duplication, `has_visual_markup` correctly `True`
  • Verified `has_visual_markup` is `False` on a synthetic clean PDF with no drawn marks
  • No remaining references to removed functions (`extract_text_from_pdf`, `download_pdf_content`, `extract_text_with_strikethroughs`, `debug_pdf_structure`, `is_likely_strikethrough`)
  • No new dependencies -- pdfplumber/PyPDF2/PyMuPDF already in Pipfile
  • Live run against a real state's extract-text workflow to confirm end-to-end

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

@tamara-builds @claude
fix: rewrite PDF extraction -- real bytes, one clean read, geometry-b…
a926480 
…ased markup flag

Three fixes to actions/extract's PDF handling, all found while investigating
why a bill's extracted text was duplicated:

1. Raw PDF storage bug: the "original content" file (e.g.
   H0493_Bill_Text.pdf) was never actually the real PDF -- it was the
   already-extracted text, re-saved under a .pdf-looking filename, written
   in text mode. The real downloaded bytes were discarded after in-memory
   processing. extract_pdf() now returns the genuine raw_bytes alongside
   the extracted text, and text_extraction.py writes them in binary mode.
   This matters because handing the real PDF to a vision-capable model is
   the plan for redline-heavy bills going forward -- can't do that with a
   mislabeled text file.

2. Duplicated output: every extracted .txt file wrote the same content
   twice, once as a "Section N:" breakdown and once as "Raw Text:". Section
   detection (in all three extractors -- XML, HTML, PDF) never reliably
   produced more than one section for most real bills (the PDF regex in
   particular broke on any state that numbers every line, i.e. most of
   them), so the whole document became "Section 1" and then got printed
   again in full as the raw text dump. Dropped section-splitting entirely
   in favor of one clean read -- arbitrary formatting-based splits don't
   reliably capture anything meaningful across 56 states' differing bill
   formats anyway.

3. Strikethrough detection replaced with a geometry-based visual-markup
   flag. The old is_likely_strikethrough() guessed off font name/character
   spacing/color -- unreliable proxies that don't actually correspond to
   how most legislative PDFs render strikethrough, and its character-by-
   character text reconstruction (joined with no separators) produced
   unreadable garbage even when it did fire. Replaced with a check for
   actual drawn line/rect objects (pdfplumber page.lines/page.rects)
   overlapping a character's vertical span -- catches both strikethrough
   (deleted text) and underline (added text), which matter equally for
   knowing a document needs full-fidelity handling rather than plain text
   extraction. This is a routing signal (has_visual_markup), not an attempt
   to reconstruct the marked text ourselves.

Verified against ID H0493 (a real redline bill): raw bytes now start with
the genuine %PDF- magic bytes, text is a single clean 2545-char read with
no duplication, and has_visual_markup correctly detects true. Verified
false on a synthetic clean PDF with no drawn marks.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@tamara-builds
tamara-builds merged commit 43c7bf9 into main Jul 21, 2026
5 checks passed
@tamara-builds
tamara-builds deleted the fix/pdf-extraction-clean-and-visual-markup branch July 21, 2026 04:15
@tamara-builds tamara-builds mentioned this pull request Jul 21, 2026
Merged
4 tasks
tamara-builds added a commit that referenced this pull request Jul 21, 2026
@tamara-builds @claude
feat(pipeline-manager): script to reset govbot-data repos without tou…
f0495fd 
…ching session config (#90)

## Summary
- Adds `rebuild_data_repos.py`: archives a locale's current
`govbot-data` repo (transfer to `govbot-archive`, full git history
preserved, then marked read-only) and recreates it fresh, so
format/extract-text rebuild from scratch with no inherited incremental
state to trust or distrust.
- Motivated by lingering doubt about incremental-skip state after
tonight's DC/MA/NY ad hoc repo deletions and the PDF-extraction bug fix
(#89 / PR #82).
- Reuses `close_session.py`'s transfer/rename/archive primitives and
`apply.py`'s `create_repo`, but deliberately never writes
`locales.<locale>.session` in either config file — this isn't a
legislative session closing, and writing that field would plant a
misleading marker for real close-session decisions later.
- Renames **before** transferring (reverse of `close_session.py`'s
order): `govbot-archive` already holds a same-named placeholder repo for
every locale from this morning's migration, and GitHub's transfer API
rejects a transfer outright if the source name already exists at the
destination — no atomic rename-during-transfer option.
- Transfers retry with backoff — confirmed via an IL pilot that GitHub
can return a transient "previous repository operation in progress" lock
immediately after the pre-transfer rename.
- Detects and resumes a locale whose rename succeeded but transfer
didn't (rather than treating it as missing) — the exact state the IL
pilot landed in mid-run.

## Test plan
- [x] Dry run against a single untouched locale (`ca`) — all 5 steps
report correctly
- [x] Ran for real against `il` (the pilot) — hit and fixed the two
issues above live; final state verified: `govbot-data/il-legislation`
fresh and active, `govbot-archive/il-legislation-pre-rebuild-2026-07-21`
archived read-only with full history, this morning's
`govbot-archive/il-legislation` placeholder untouched
- [x] Re-ran dry run afterward — correctly refuses on `il` (already
done) and proceeds cleanly on `ca`
- [ ] Not yet run at `--all-states` scale — planned as a follow-up once
this PR is in

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tamara-builds