diff --git a/docex/intake/README.md b/docex/intake/README.md new file mode 100644 index 0000000..acf1e09 --- /dev/null +++ b/docex/intake/README.md @@ -0,0 +1,221 @@ +# PDF Intake - Commercial Real Estate Invoice Reconciliation + +Read a commercial-real-estate (CRE) invoice PDF of arbitrary layout, extract its +fields, and reconcile it against what we *expected* to be billed. The use case: +a landlord or property manager sends an invoice as a PDF, formatted however their +system happens to format it, and we need to answer one question reliably - **does +this bill match the lease?** Base rent, CAM, tax and insurance recoveries, +square footage, pro-rata share, totals: each is checked against our recorded +actuals so an overcharge is caught before it is paid. + +The design goal beyond correctness is **cost and latency**: do the cheap, +deterministic work first and only reach for a language model when the cheap work +genuinely cannot answer. + +## How it works + +```text +PDF ──► text ──► extract (cascade) ──► match ground truth ──► reconcile ──► escalate? ──► learn + pdfminer heuristic, then by invoice / PO per-field LLM on only record + LLM only for gaps w/ tolerances disputed fields confirmed labels +``` + +### 1. Text extraction (`pdf.py`) + +`pdfminer.six` turns the PDF into text. This is the only module that touches the +PDF binary, so everything above it is plain-text and trivially testable. The +dependency is optional (`pip install docex[pdf]`). + +### 2. The extraction cascade (`extractors/`) + +Fields are extracted in cost order, escalating only what the cheaper tier could +not resolve: + +- **Tier 1 - Heuristic (`heuristic.py`)**: free and deterministic. It scans for + each field's label aliases (from the canonical registry) at the start of a + line and reads the value beside or below it, and it parses the charges table + into typed line items. It resolves the great majority of fields on well-formed + invoices. +- **Tier 2 - LLM (`llm.py`)**: the last resort. The model is **caller-provided** + (`llm_fn`, a callable returning JSON), so the core takes no hard dependency on + any provider - exactly like DocEX's `embedding_fn` pattern. It is invoked only + for *gaps* (required fields the heuristic missed) and, after reconciliation, + for *disputed* fields. A clean invoice never reaches it. + +The cascade (`cascade.py`) owns this ordering and the merge logic. + +#### Why there is no embedding-similarity tier + +An obvious middle tier would map an unseen label phrasing to a field by +embedding similarity, to avoid an LLM call. We deliberately left it out: + +- The **learning loop (below) already removes that cost.** The first time a novel + phrasing appears the LLM resolves it; once confirmed, the heuristic learns it + permanently and every later occurrence is free at Tier 1. An embedding tier + would only save the single LLM call on the *first* sighting. +- An embedding match is a similarity score, not an auditable label-and-value on + the page. It can confidently bind the wrong line to a field, and a wrong value + that coincidentally matched ground truth would be learned as a real alias - + poisoning the heuristic. +- It adds a hard dependency on a caller-supplied embedding model whose quality we + cannot guarantee. + +A one-time cost saving is not worth a false-positive risk to a loop we already +built. If the heuristic cannot read a field, we go straight to the authoritative +tier. + +### 3. Ground truth (`ground_truth.py`) + +A `GroundTruthInvoice` is our recorded actuals for a lease and billing period - +the schema mirrors the canonical field registry one-to-one. Two stores ship: + +- `InMemoryGroundTruthStore` for tests and small in-process use. +- `DocEXGroundTruthStore`, which persists each record as a JSON document in a + DocEX basket and mirrors the lookup keys (invoice number, PO, lease, account) + into document metadata, so retrieval is an indexed metadata query. + +### 4. Reconciliation (`reconcile.py`) + +`GroundTruthMatcher` finds the record to compare against by stable identifier +(invoice number, then PO) - never by fuzzy totals, because reconciling against +the wrong lease is worse than reporting no match. `Reconciler` then compares each +field with type-aware tolerances (a cent on money, configurable days on dates, +case/whitespace-insensitive strings) and rolls line items up by charge type so a +vendor's "CAM" line is compared against the lease's expected CAM regardless of +wording. The verdict is `matched`, `discrepancy`, `incomplete`, or `unresolved`. + +#### Fuzzy ground-truth retrieval (`embedding_match.py`) + +Identifier matching cannot help when the incoming invoice has no clean +identifier - a typo in the invoice number, a vendor using their own numbering, +or a number the extractor could not read. `EmbeddingGroundTruthMatcher` embeds a +short fingerprint of each ground-truth record (tenant, property, suite, lease, +totals, charge descriptions) and the same fingerprint of the invoice, and ranks +records by cosine similarity (`embedding_fn` is caller-provided, the same one +you would use for DocEX vector indexing; record embeddings are cached by id). + +Pass `embedding_fn` to the pipeline and it becomes an automatic fallback: an +invoice that matches no record by identifier is reconciled against the closest +record by similarity instead of being reported as unresolved. Crucially this is +*retrieval, not a verdict* - the invoice is still reconciled against the +retrieved record, so a wrong guess surfaces as a discrepancy (for example the +typo'd invoice number is flagged) rather than being silently trusted. This is +why embeddings are safe for finding the record but were deliberately kept out of +extracting field values. You can also call the matcher directly for a ranked +`candidates(...)` list to suggest matches in a UI. + +### 5. The self-improving learning loop (`learning.py`) + +Every extracted field carries the **label phrase** that identified it. After +reconciliation, the label of every field that *matched* ground truth is recorded +with a running count. This does two things: + +- **Trend**: how customers actually phrase each field (analytics). +- **Learned alias**: a phrasing the registry did not know - usually surfaced by + the LLM on a messy invoice - is promoted into the heuristic's alias set, so the + next invoice that uses it is solved for free at Tier 1. + +Only ground-truth-confirmed labels are ever learned, so the loop cannot teach the +heuristic a wrong mapping. Over time the cheap tier absorbs the long tail of +vendor phrasings and the LLM is needed less and less - the explicit goal being a +system that, for a stable set of vendors, may not need the LLM at all. + +### 6. DocEX integration (`processor.py`) + +`InvoiceIntakeProcessor` wraps the pipeline as a DocEX `BaseProcessor`, so an +invoice already stored in a basket can be reconciled in place with the verdict +written back to its metadata. It is imported separately from the core so the +pipeline stays free of any database dependency. + +## Usage + +```python +import asyncio +from docex.intake import InvoiceIntakePipeline, InMemoryGroundTruthStore + +store = InMemoryGroundTruthStore() +store.add(my_ground_truth_invoice) # your recorded actuals + +# Heuristic-only (fully offline): +pipeline = InvoiceIntakePipeline() + +# With an LLM fallback + persistent learning (recommended for production): +from docex.intake import JsonFileLearningStore +from examples.integrations.anthropic.invoice_intake_llm import make_claude_llm_fn + +pipeline = InvoiceIntakePipeline( + llm_fn=make_claude_llm_fn(), + learning_store=JsonFileLearningStore("learned_labels.json"), +) + +outcome = asyncio.run(pipeline.process_pdf("invoice.pdf", store)) +print(outcome.status) # matched | discrepancy | incomplete | unresolved +print(outcome.reconciliation.mismatches) # the lines that disagree with the lease +``` + +## How it was tested + +The suite lives in `tests/intake/` and is built so the bulk runs fast and +offline, with the binary and provider boundaries exercised separately. + +- **Unit tests** pin down each layer: amount/date/percent normalization across + real-world formats (US and European grouping, parenthesised negatives, spelled + dates), the charge taxonomy, heuristic extraction traps (a scalar `tax` field + must not read a "Real Estate Tax Recovery" charge line; labelled metric rows + must not become phantom charges; a specific label beats a generic one), the + reconciler's tolerances and statuses, and the matcher. +- **The learning loop** is tested end to end: a novel label forces one LLM call, + is confirmed against ground truth and learned, and the *same* invoice then + reconciles with zero further LLM calls. +- **Randomized scenarios** (`test_random.py`) generate dozens of seeded invoices + with varied layouts, label phrasings, date formats, and values. Every clean + invoice must reconcile as matched; every overcharged one must be caught. A + failure reports the seed that produced it. +- **The LLM tier** is covered two ways: deterministic stub tests that run + everywhere (JSON parsing, value normalization, label capture, line items), and + a **live** test against Claude that is skipped unless `ANTHROPIC_API_KEY` is + set. +- **Real PDFs**: two committed, human-viewable invoices in + `example_docs/cre_invoices/` - a positive one that matches, and a negative one + that overstates CAM by $750 - are run through the full pipeline (pdfminer + included). A synthetic reportlab round-trip covers the write/read boundary too. + +Run the offline suite: + +```sh +python -m pytest tests/intake/ +``` + +Run the live-LLM tests (after setting a key): + +```sh +ANTHROPIC_API_KEY=sk-... python -m pytest tests/intake/ -k live -s +``` + +## Key assumptions + +These are the boundaries within which the heuristic is reliable; outside them, +extraction falls through to the LLM, and reconciliation simply reports what it +could not confirm. + +1. **Text-based PDFs.** Extraction relies on a text layer. A scanned image with + no embedded text yields no text; such invoices need an OCR step ahead of the + intake (out of scope here). +2. **Labels lead their line.** The heuristic treats a label as valid only when it + starts a "label: value" row. Values embedded mid-sentence, or two fields + sharing one line, are left to the LLM. This is what makes a generic label like + "property" safe - it cannot match the word inside "Summit Property Group". +3. **Monetary amounts are formatted as money.** A charge amount carries a + currency symbol, decimals, or thousands grouping. A bare integer (a suite + number, a square-foot count) is never mistaken for a charge. +4. **US conventions by default.** Ambiguous amounts and dates default to US + formatting (`,` groups thousands, `.` is the decimal point, dates are + month-first) unless the value itself disambiguates (both separators present, + or a date component greater than 12). +5. **Ground truth is matched by identifier.** An invoice is reconciled against + the record whose invoice number (or PO) it carries. Without one of those, the + result is `unresolved` rather than a guess. +6. **The LLM is authoritative but optional.** When present it overrides the + heuristic on disputed fields and teaches new labels; when absent the pipeline + runs fully offline and reports unresolved fields plainly. +``` diff --git a/docex/intake/__init__.py b/docex/intake/__init__.py new file mode 100644 index 0000000..ba8fd60 --- /dev/null +++ b/docex/intake/__init__.py @@ -0,0 +1,84 @@ +""" +DocEX PDF Intake + +Reads commercial-real-estate invoice PDFs of arbitrary layout, extracts their +fields using a cost-ordered cascade (a free heuristic first, an LLM only as a +last resort), and reconciles the extracted invoice against a stored +ground-truth record. Confirmed field labels feed a learning loop so the cheap +heuristic keeps improving and the LLM is needed less over time. + +See ``README.md`` in this package for the full design, testing notes, and the +assumptions the intake makes about its inputs. + +The DocEX ``BaseProcessor`` glue lives in :mod:`docex.intake.processor` and is +imported separately so that the core pipeline stays free of any database +dependency. +""" + +from docex.intake.charges import ChargeType, classify_charge +from docex.intake.embedding_match import ( + EmbeddingGroundTruthMatcher, + GroundTruthMatch, + extracted_fingerprint, + ground_truth_fingerprint, +) +from docex.intake.fields import FIELDS, FieldSpec, FieldType +from docex.intake.ground_truth import ( + DocEXGroundTruthStore, + GroundTruthInvoice, + GroundTruthStore, + InMemoryGroundTruthStore, +) +from docex.intake.learning import ( + FieldObservation, + InMemoryLearningStore, + JsonFileLearningStore, + LearningStore, +) +from docex.intake.models import ( + ExtractedField, + ExtractedInvoice, + ExtractionTier, + FieldComparison, + LineItem, + LineItemComparison, + MatchStatus, + ReconciliationResult, + ReconciliationStatus, +) +from docex.intake.pipeline import IntakeOutcome, InvoiceIntakePipeline +from docex.intake.reconcile import GroundTruthMatcher, Reconciler, TolerancePolicy + +__all__ = [ + "FIELDS", + "FieldSpec", + "FieldType", + "ChargeType", + "classify_charge", + "ExtractedField", + "ExtractedInvoice", + "ExtractionTier", + "LineItem", + "FieldComparison", + "LineItemComparison", + "MatchStatus", + "ReconciliationResult", + "ReconciliationStatus", + "GroundTruthInvoice", + "GroundTruthStore", + "InMemoryGroundTruthStore", + "DocEXGroundTruthStore", + "LearningStore", + "InMemoryLearningStore", + "JsonFileLearningStore", + "FieldObservation", + "Reconciler", + "TolerancePolicy", + "GroundTruthMatcher", + "EmbeddingGroundTruthMatcher", + "GroundTruthMatch", + "ground_truth_fingerprint", + "extracted_fingerprint", + "InvoiceIntakePipeline", + "IntakeOutcome", +] diff --git a/docex/intake/charges.py b/docex/intake/charges.py new file mode 100644 index 0000000..4ad7afa --- /dev/null +++ b/docex/intake/charges.py @@ -0,0 +1,89 @@ +""" +Charge taxonomy for commercial-real-estate invoice line items. + +CRE invoices break the amount due into recurring and pass-through charges: +base rent, CAM, real estate tax and insurance recoveries, utilities, parking, +management fees, and periodic reconciliations. Classifying each line item into +a canonical charge type lets the reconciler compare like-for-like against the +ground-truth lease schedule instead of relying on free-text descriptions. +""" + +from __future__ import annotations + +from enum import Enum +from typing import Dict, Tuple + + +class ChargeType(str, Enum): + """Canonical category for a CRE invoice line item.""" + + BASE_RENT = "base_rent" + ADDITIONAL_RENT = "additional_rent" + PERCENTAGE_RENT = "percentage_rent" + CAM = "cam" + OPERATING_EXPENSES = "operating_expenses" + REAL_ESTATE_TAX = "real_estate_tax" + INSURANCE = "insurance" + UTILITIES = "utilities" + HVAC = "hvac" + PARKING = "parking" + MANAGEMENT_FEE = "management_fee" + JANITORIAL = "janitorial" + SECURITY = "security" + LATE_FEE = "late_fee" + TENANT_IMPROVEMENT = "tenant_improvement" + ESCALATION = "escalation" + PREPAID_RENT = "prepaid_rent" + SECURITY_DEPOSIT = "security_deposit" + CAM_RECONCILIATION = "cam_reconciliation" + OTHER = "other" + + +# Aliases are checked most-specific first so that, e.g., "CAM reconciliation" +# classifies as CAM_RECONCILIATION rather than CAM. +_LABELS: Tuple[Tuple[ChargeType, Tuple[str, ...]], ...] = ( + (ChargeType.CAM_RECONCILIATION, ("cam reconciliation", "cam true-up", "cam true up", "year-end adjustment", "expense reconciliation", "opex reconciliation")), + (ChargeType.PERCENTAGE_RENT, ("percentage rent", "overage rent", "% rent")), + (ChargeType.BASE_RENT, ("base rent", "minimum rent", "fixed rent", "base monthly rent", "minimum monthly rent")), + (ChargeType.ADDITIONAL_RENT, ("additional rent", "supplemental rent")), + (ChargeType.OPERATING_EXPENSES, ("operating expense", "operating cost", "opex", "common area expense")), + (ChargeType.CAM, ("common area maintenance", "cam charge", "cam", "common area")), + (ChargeType.REAL_ESTATE_TAX, ("real estate tax", "property tax", "real property tax", "re tax", "tax recovery")), + (ChargeType.INSURANCE, ("insurance recovery", "property insurance", "liability insurance", "insurance")), + (ChargeType.UTILITIES, ("electricity", "water", "sewer", "natural gas", "utility", "utilities")), + (ChargeType.HVAC, ("hvac", "air conditioning", "after-hours hvac", "heating")), + (ChargeType.PARKING, ("parking", "parking stall", "garage")), + (ChargeType.MANAGEMENT_FEE, ("management fee", "property management fee", "administrative fee")), + (ChargeType.JANITORIAL, ("janitorial", "cleaning", "day porter")), + (ChargeType.SECURITY, ("security service", "security")), + (ChargeType.LATE_FEE, ("late fee", "late charge", "penalty", "interest charge")), + (ChargeType.TENANT_IMPROVEMENT, ("tenant improvement", "ti amortization", "build-out", "build out", "ti charge")), + (ChargeType.ESCALATION, ("escalation", "annual increase", "cpi adjustment", "rent step")), + (ChargeType.PREPAID_RENT, ("prepaid rent", "rent in advance")), + (ChargeType.SECURITY_DEPOSIT, ("security deposit", "deposit")), +) + + +_LABEL_INDEX: Dict[str, ChargeType] = { + alias: charge_type for charge_type, aliases in _LABELS for alias in aliases +} + + +def classify_charge(description: str) -> ChargeType: + """Map a free-text line-item description to a canonical charge type. + + Args: + description: The raw description text from the invoice line. + + Returns: + The best-matching :class:`ChargeType`, or ``ChargeType.OTHER`` when no + alias is recognised. + """ + if not description: + return ChargeType.OTHER + + text = description.strip().lower() + for charge_type, aliases in _LABELS: + if any(alias in text for alias in aliases): + return charge_type + return ChargeType.OTHER diff --git a/docex/intake/embedding_match.py b/docex/intake/embedding_match.py new file mode 100644 index 0000000..4f5a4fb --- /dev/null +++ b/docex/intake/embedding_match.py @@ -0,0 +1,135 @@ +""" +Find the ground-truth record an invoice most likely belongs to, by embedding +similarity. + +:class:`~docex.intake.reconcile.GroundTruthMatcher` matches on a stable +identifier (invoice number, then PO). That is the right default - exact and +auditable - but it cannot help when the incoming invoice has no clean identifier +to match on: a typo in the invoice number, a vendor using their own numbering, +or a field the extractor simply could not read. + +This matcher fills that gap. It embeds a short fingerprint of each ground-truth +record (tenant, property, suite, lease, totals, charge descriptions) and the +same fingerprint of the extracted invoice, and ranks records by cosine +similarity. It is a *retrieval* aid, not a verdict: the pipeline still reconciles +against whatever record this returns, so a wrong guess shows up as a discrepancy +rather than being silently trusted - which is exactly why embeddings are safe +here and were left out of field extraction. + +The embedding function is caller-provided (sync or async), the same one you +would use for DocEX vector indexing. Record embeddings are cached by record id, +so the cost is one embedding per record per process, not per invoice. +""" + +from __future__ import annotations + +import inspect +from dataclasses import dataclass +from typing import Awaitable, Callable, Dict, List, Optional, Union + +from docex.intake.ground_truth import GroundTruthInvoice, GroundTruthStore +from docex.intake.models import ExtractedInvoice + +EmbeddingFn = Callable[[str], Union[List[float], Awaitable[List[float]]]] + +_FINGERPRINT_FIELDS = ( + "invoice_number", + "po_number", + "landlord_name", + "tenant_name", + "tenant_account", + "property_name", + "property_address", + "suite_number", + "lease_number", + "total", +) +_DEFAULT_THRESHOLD = 0.8 + + +@dataclass +class GroundTruthMatch: + """A candidate ground-truth record and its similarity to the invoice.""" + + record: GroundTruthInvoice + score: float + + +def ground_truth_fingerprint(record: GroundTruthInvoice) -> str: + """The text an embedding model sees for a ground-truth record.""" + parts = [str(record.get(name)) for name in _FINGERPRINT_FIELDS if record.get(name) is not None] + parts.extend(item.description for item in record.line_items if item.description) + return " ".join(parts) + + +def extracted_fingerprint(extracted: ExtractedInvoice) -> str: + """The matching fingerprint for an extracted invoice.""" + parts = [str(extracted.value(name)) for name in _FINGERPRINT_FIELDS if extracted.value(name) is not None] + parts.extend(item.description for item in extracted.line_items if item.description) + return " ".join(parts) + + +class EmbeddingGroundTruthMatcher: + """Ranks ground-truth records by embedding similarity to an invoice.""" + + def __init__(self, embedding_fn: EmbeddingFn, threshold: float = _DEFAULT_THRESHOLD) -> None: + """ + Args: + embedding_fn: Sync or async callable mapping text to an embedding. + threshold: Minimum cosine similarity for :meth:`find` to accept a + match. :meth:`candidates` ignores it and returns a ranked list. + """ + if not callable(embedding_fn): + raise ValueError("embedding_fn is required and must be callable") + self._embedding_fn = embedding_fn + self._threshold = threshold + self._cache: Dict[str, List[float]] = {} + + async def find(self, extracted: ExtractedInvoice, store: GroundTruthStore) -> Optional[GroundTruthMatch]: + """Return the single best match above the threshold, or ``None``.""" + ranked = await self.candidates(extracted, store, top_k=1) + best = ranked[0] if ranked else None + return best if best and best.score >= self._threshold else None + + async def candidates( + self, + extracted: ExtractedInvoice, + store: GroundTruthStore, + top_k: int = 5, + ) -> List[GroundTruthMatch]: + """Return up to ``top_k`` records ranked by similarity, best first.""" + fingerprint = extracted_fingerprint(extracted) + if not fingerprint: + return [] + + query = await self._embed(fingerprint) + scored = [ + GroundTruthMatch(record=record, score=_cosine(query, await self._record_vector(record))) + for record in store.all() + ] + scored.sort(key=lambda match: match.score, reverse=True) + return scored[:top_k] + + async def _record_vector(self, record: GroundTruthInvoice) -> List[float]: + if record.id is not None and record.id in self._cache: + return self._cache[record.id] + vector = await self._embed(ground_truth_fingerprint(record)) + if record.id is not None: + self._cache[record.id] = vector + return vector + + async def _embed(self, text: str) -> List[float]: + embedding = self._embedding_fn(text) + if inspect.isawaitable(embedding): + embedding = await embedding + if not isinstance(embedding, list) or not embedding: + raise ValueError("embedding_fn must return a non-empty list of floats") + return [float(value) for value in embedding] + + +def _cosine(left: List[float], right: List[float]) -> float: + """Cosine similarity of two equal-length vectors, 0.0 when either is zero.""" + dot = sum(a * b for a, b in zip(left, right)) + norm_left = sum(a * a for a in left) ** 0.5 + norm_right = sum(b * b for b in right) ** 0.5 + return dot / (norm_left * norm_right) if norm_left and norm_right else 0.0 diff --git a/docex/intake/extractors/__init__.py b/docex/intake/extractors/__init__.py new file mode 100644 index 0000000..7205224 --- /dev/null +++ b/docex/intake/extractors/__init__.py @@ -0,0 +1,22 @@ +""" +Field extraction strategies, ordered cheapest-first by the cascade. + +There are deliberately only two tiers: a free, deterministic heuristic and a +paid LLM fallback. We considered an intermediate embedding-similarity tier and +chose not to build it - see :mod:`docex.intake.extractors.cascade` for the +rationale (the learning loop already makes it redundant and it carries a +false-positive risk the cascade cannot cheaply audit). +""" + +from docex.intake.extractors.base import ExtractionContext, FieldExtractor +from docex.intake.extractors.cascade import CascadingExtractor +from docex.intake.extractors.heuristic import HeuristicExtractor +from docex.intake.extractors.llm import LLMExtractor + +__all__ = [ + "ExtractionContext", + "FieldExtractor", + "HeuristicExtractor", + "LLMExtractor", + "CascadingExtractor", +] diff --git a/docex/intake/extractors/base.py b/docex/intake/extractors/base.py new file mode 100644 index 0000000..7e1a6ea --- /dev/null +++ b/docex/intake/extractors/base.py @@ -0,0 +1,122 @@ +""" +Shared extraction primitives. + +Every extractor consumes an :class:`ExtractionContext` (the page text plus the +set of fields still worth attempting) and returns the fields it could resolve. +The cascade decides which extractor runs next based on what came back, so an +extractor only needs to answer "given this text, what can I find?". + +The value matchers here turn a labelled span into a typed value. They live next +to the extractors rather than in :mod:`docex.intake.normalize` because they +encode *where on a line a value sits*, which is an extraction concern, not a +parsing one. +""" + +from __future__ import annotations + +import re +from abc import ABC, abstractmethod +from dataclasses import dataclass, field +from typing import Dict, List, Optional, Tuple + +from docex.intake.fields import FieldType +from docex.intake.models import ExtractedField, ExtractionTier, LineItem +from docex.intake.normalize import normalize_text, parse_value + +_MONEY_TOKEN = re.compile(r"\(?-?\s*[$€£]?\s*\d[\d,]*(?:\.\d+)?\s*\)?-?") +_DATE_TOKEN = re.compile( + r"\d{1,4}[/\-.]\d{1,2}[/\-.]\d{1,4}" + r"|[A-Za-z]{3,9}\.?\s+\d{1,2}(?:st|nd|rd|th)?,?\s+\d{4}" + r"|\d{1,2}(?:st|nd|rd|th)?\s+[A-Za-z]{3,9}\.?,?\s+\d{4}" +) +_PERCENT_TOKEN = re.compile(r"\d[\d.,]*\s*%?") +_NUMBER_TOKEN = re.compile(r"\d[\d,]*(?:\.\d+)?") + +_TYPE_TOKENS = { + FieldType.AMOUNT: _MONEY_TOKEN, + FieldType.DATE: _DATE_TOKEN, + FieldType.PERCENT: _PERCENT_TOKEN, + FieldType.NUMBER: _NUMBER_TOKEN, +} + + +@dataclass +class ExtractionContext: + """Everything an extractor needs to do one pass over an invoice. + + Attributes: + raw_text: The full page text from the PDF. + target_fields: Canonical field names still worth attempting. The cascade + narrows this to only the unresolved or disputed fields when it + escalates, so costly tiers never re-extract what cheap tiers nailed. + want_line_items: Whether to parse charge lines. Set only on the first, + full pass; escalations target individual scalar fields. + """ + + raw_text: str + target_fields: Tuple[str, ...] + want_line_items: bool = True + lines: List[str] = field(init=False) + + def __post_init__(self) -> None: + self.lines = [normalize_text(line) for line in self.raw_text.splitlines() if line.strip()] + + +@dataclass +class ExtractionResult: + """What one extractor produced: resolved fields and any parsed charges.""" + + fields: Dict[str, ExtractedField] = field(default_factory=dict) + line_items: List[LineItem] = field(default_factory=list) + + +class FieldExtractor(ABC): + """One strategy for turning invoice text into typed fields.""" + + tier: ExtractionTier + + @abstractmethod + async def extract(self, context: ExtractionContext) -> ExtractionResult: + """Return the fields and charges this strategy could resolve.""" + + +def find_typed_value(field_type: FieldType, text: str) -> Tuple[Optional[object], Optional[str]]: + """Find the first value of ``field_type`` in ``text``. + + Returns a ``(value, raw_span)`` pair, both ``None`` when nothing matched. + For string fields the whole trimmed text is the value, since a label's tail + is the value (a property name, a suite, a vendor). + """ + if field_type in (FieldType.STRING, FieldType.CURRENCY): + cleaned = normalize_text(text) + if not cleaned: + return None, None + value = parse_value(field_type, cleaned) + return (value, cleaned) if value else (None, None) + + token = _TYPE_TOKENS[field_type] + for match in token.finditer(text): + raw_span = match.group(0).strip() + value = parse_value(field_type, raw_span) + if value is not None: + return value, raw_span + return None, None + + +def make_field( + name: str, + value: object, + tier: ExtractionTier, + confidence: float, + raw_text: Optional[str], + label: Optional[str] = None, +) -> ExtractedField: + """Construct an :class:`ExtractedField`, clamping confidence to ``[0, 1]``.""" + return ExtractedField( + name=name, + value=value, + confidence=max(0.0, min(1.0, confidence)), + tier=tier, + raw_text=raw_text, + label=label, + ) diff --git a/docex/intake/extractors/cascade.py b/docex/intake/extractors/cascade.py new file mode 100644 index 0000000..82a0aaa --- /dev/null +++ b/docex/intake/extractors/cascade.py @@ -0,0 +1,104 @@ +""" +The extraction cascade: cheap first, expensive only when needed. + +Order of operations: + +1. **Heuristic pass** over the whole invoice - free and deterministic. +2. **LLM pass**, but only for *gaps* - required fields the heuristic could not + read at all. Without those (an invoice number, a total) we cannot even match + a ground-truth record, so they are worth one call up front. +3. Later, the pipeline calls :meth:`repair` to re-extract *specific* fields the + reconciler flagged as wrong - again, the LLM touches only those fields. + +A clean invoice costs zero LLM calls. A messy one costs one call for the gaps +plus, at most, one repair call for disputed fields. + +Why there is no embedding-similarity tier between the two +-------------------------------------------------------- +An embedding tier would exist to map a never-before-seen label phrasing to a +canonical field without paying for an LLM. We deliberately left it out: + +* The learning loop already removes that cost. The first time a novel phrasing + appears, the LLM resolves it and - once confirmed against ground truth - the + heuristic learns it permanently (:mod:`docex.intake.learning`). Every later + occurrence is then free at Tier 1. The embedding tier would only save the + *single* LLM call on the *first* sighting. +* That saving comes with a real downside: an embedding match is a similarity + score, not an auditable label-and-value on the page. It can confidently bind + the wrong line to a field, and a wrong value that happens to match ground + truth would be learned as a true alias, poisoning the heuristic. +* It also adds a hard dependency on a caller-supplied embedding model whose + quality we cannot guarantee. + +A minor, one-time cost saving is not worth a false-positive risk to a learning +loop we already built. If the heuristic cannot read a field, we go straight to +the authoritative tier. +""" + +from __future__ import annotations + +from typing import Optional, Tuple + +from docex.intake.extractors.base import ( + ExtractionContext, + ExtractionResult, + FieldExtractor, +) +from docex.intake.fields import required_fields + + +class CascadingExtractor: + """Runs the heuristic first and the LLM only for what it leaves behind.""" + + def __init__(self, heuristic: FieldExtractor, llm: Optional[FieldExtractor] = None) -> None: + """ + Args: + heuristic: The Tier 1 extractor (always present). + llm: The Tier 2 extractor, or ``None`` to run heuristic-only (for + offline or cost-capped deployments). + """ + self._heuristic = heuristic + self._llm = llm + + async def extract(self, raw_text: str) -> ExtractionResult: + """Full heuristic pass, escalating only missing required fields.""" + result = await self._heuristic.extract( + ExtractionContext(raw_text=raw_text, target_fields=(), want_line_items=True) + ) + + gaps = self._required_gaps(result) + if gaps and self._llm is not None: + llm_result = await self._llm.extract( + ExtractionContext( + raw_text=raw_text, + target_fields=gaps, + want_line_items=not result.line_items, + ) + ) + result = self._merge(result, llm_result) + return result + + async def repair(self, raw_text: str, fields: Tuple[str, ...], base: ExtractionResult) -> ExtractionResult: + """Re-extract specific disputed fields with the LLM and merge them in. + + Returns ``base`` unchanged when there is no LLM or nothing to repair, so + the pipeline can call this unconditionally. + """ + if not fields or self._llm is None: + return base + llm_result = await self._llm.extract( + ExtractionContext(raw_text=raw_text, target_fields=fields, want_line_items=False) + ) + return self._merge(base, llm_result) + + @staticmethod + def _required_gaps(result: ExtractionResult) -> Tuple[str, ...]: + return tuple(name for name in required_fields() if name not in result.fields) + + @staticmethod + def _merge(base: ExtractionResult, addition: ExtractionResult) -> ExtractionResult: + """Overlay ``addition`` onto ``base``; the newer tier wins per field.""" + merged_fields = dict(base.fields) + merged_fields.update(addition.fields) + line_items = base.line_items or addition.line_items + return ExtractionResult(fields=merged_fields, line_items=line_items) diff --git a/docex/intake/extractors/heuristic.py b/docex/intake/extractors/heuristic.py new file mode 100644 index 0000000..0f92a39 --- /dev/null +++ b/docex/intake/extractors/heuristic.py @@ -0,0 +1,269 @@ +""" +Tier 1: heuristic extraction. + +Free, fast, and deterministic. It scans for a field's label aliases and reads +the value that sits beside (or just below) the label, then parses each charge +line into a typed :class:`LineItem`. It resolves the great majority of fields on +well-formed invoices; whatever it leaves unresolved or low-confidence is what +the cascade escalates. + +Two assumptions keep it honest, both documented in the package README: + +* Labels are read longest-first and each matched label span is claimed, so + ``tax id`` wins over ``tax`` and ``invoice date`` over ``date``. +* A charge amount is money-formatted (currency symbol, decimals, or thousands + grouping). Bare integers such as ``Suite 400`` are never mistaken for charges. +""" + +from __future__ import annotations + +import re +from typing import Dict, List, Optional, Tuple + +from docex.intake.charges import ChargeType, classify_charge +from docex.intake.extractors.base import ( + ExtractionContext, + ExtractionResult, + FieldExtractor, + find_typed_value, + make_field, +) +from docex.intake.fields import FIELDS, FieldType +from docex.intake.learning import LearningStore +from docex.intake.models import ExtractedField, ExtractionTier, LineItem +from docex.intake.normalize import detect_currency, parse_amount + +_SAME_LINE_CONFIDENCE = 0.9 +_NEXT_LINE_CONFIDENCE = 0.7 +_STRING_PENALTY = 0.05 +_CURRENCY_FALLBACK_CONFIDENCE = 0.6 +_NEXT_LINE_LOOKAHEAD = 2 + +_LABEL_SEPARATORS = " \t:#.-–" +_COLUMN_GAP = re.compile(r"\s{2,}") +_MONEY_LIKE = re.compile( + r"[$€£]\s?\d[\d,]*(?:\.\d+)?" # currency-symbol amounts + r"|\d{1,3}(?:,\d{3})+(?:\.\d+)?" # comma-grouped thousands + r"|\d+\.\d{2}" # plain decimal money +) +_QUANTITY_RATE = re.compile(r"(\d[\d,.]*)\s*(?:sf\s*)?(?:@|x)\s*[$€£]?\s*(\d[\d,.]*)", re.IGNORECASE) + +_SUMMARY_KEYWORDS = ( + "subtotal", "total", "amount due", "balance", "payment", "prior balance", + "grand total", "amount payable", "current charges", "tax", "vat", "gst", + "currency", "invoice", "date", "page", "remit", "account", +) + + +class HeuristicExtractor(FieldExtractor): + """Label-proximity field extraction plus charge line parsing. + + When given a :class:`~docex.intake.learning.LearningStore`, it also scans the + labels learned from past ground-truth-confirmed invoices, so phrasings the + LLM once discovered are now resolved here for free. + """ + + tier = ExtractionTier.HEURISTIC + + def __init__(self, learning_store: Optional[LearningStore] = None) -> None: + self._learning_store = learning_store + + async def extract(self, context: ExtractionContext) -> ExtractionResult: + claimed: Dict[int, List[Tuple[int, int]]] = {} + result = ExtractionResult() + + # Claim recognised charge lines before scanning scalar fields, so a + # field like ``tax`` reads the "Tax" summary row, not a "Real Estate + # Tax Recovery" charge line that merely contains the word. + charges = self._charge_lines(context.lines) + known_charges = [(index, item) for index, item in charges if item.charge_type != ChargeType.OTHER] + for index, _ in known_charges: + claimed[index] = [(0, len(context.lines[index]))] + + for name in self._fields_by_label_specificity(context.target_fields): + extracted = self._extract_field(name, context, claimed) + if extracted: + result.fields[name] = extracted + + if self._currency_wanted(context, result): + currency = self._infer_currency(context.raw_text) + if currency: + result.fields["currency"] = currency + + if context.want_line_items: + result.line_items = self._assemble_line_items(charges, known_charges, claimed) + + return result + + @staticmethod + def _currency_wanted(context: ExtractionContext, result: ExtractionResult) -> bool: + targets_currency = not context.target_fields or "currency" in context.target_fields + return targets_currency and "currency" not in result.fields + + def _fields_by_label_specificity(self, target_fields: Tuple[str, ...]) -> List[str]: + """Order fields so those with longer, more specific labels resolve first.""" + names = list(target_fields) or list(FIELDS) + return sorted(names, key=lambda name: -max(len(label) for label in FIELDS[name].labels)) + + def _extract_field( + self, + name: str, + context: ExtractionContext, + claimed: Dict[int, List[Tuple[int, int]]], + ) -> Optional[ExtractedField]: + spec = FIELDS[name] + for label in self._labels_for(name): + for line_index, line in enumerate(context.lines): + span = self._match_label(line, label, claimed.get(line_index, [])) + if span is None: + continue + field = self._read_value(name, spec.type, context, line_index, span, label) + if field: + # Claim the whole line, not just the label span, so a generic + # label (e.g. "property") cannot later match a word embedded + # in this line's value ("Meridian Property Management"). + claimed.setdefault(line_index, []).append((0, len(line))) + return field + return None + + def _labels_for(self, name: str) -> List[str]: + """Registry labels plus any learned for this field, longest-first.""" + labels = set(FIELDS[name].labels) + if self._learning_store: + labels.update(self._learning_store.learned_labels(name)) + return sorted(labels, key=len, reverse=True) + + def _match_label(self, line: str, label: str, claimed: List[Tuple[int, int]]) -> Optional[Tuple[int, int]]: + """Match ``label`` only when it leads the line, outside claimed spans. + + Invoice labels head a "label: value" row; requiring the label at the + start of the (already whitespace-normalised) line stops a generic label + like "property" from matching a word embedded in another field's value + ("Summit Property Group"). + """ + lowered = line.lower() + if not lowered.startswith(label): + return None + end = len(label) + if self._ends_on_boundary(lowered, end) and not self._overlaps(0, end, claimed): + return 0, end + return None + + @staticmethod + def _ends_on_boundary(text: str, end: int) -> bool: + return end >= len(text) or not text[end].isalnum() + + @staticmethod + def _overlaps(start: int, end: int, claimed: List[Tuple[int, int]]) -> bool: + return any(start < c_end and c_start < end for c_start, c_end in claimed) + + def _read_value( + self, + name: str, + field_type: FieldType, + context: ExtractionContext, + line_index: int, + span: Tuple[int, int], + label: str, + ) -> Optional[ExtractedField]: + tail = context.lines[line_index][span[1]:].lstrip(_LABEL_SEPARATORS) + if field_type == FieldType.STRING: + tail = _COLUMN_GAP.split(tail, maxsplit=1)[0] + + value, raw = find_typed_value(field_type, tail) + if value is not None: + confidence = _SAME_LINE_CONFIDENCE - (_STRING_PENALTY if field_type == FieldType.STRING else 0.0) + return make_field(name, value, self.tier, confidence, raw, label) + + return self._read_following_lines(name, field_type, context, line_index, label) + + def _read_following_lines( + self, + name: str, + field_type: FieldType, + context: ExtractionContext, + line_index: int, + label: str, + ) -> Optional[ExtractedField]: + for offset in range(1, _NEXT_LINE_LOOKAHEAD + 1): + next_index = line_index + offset + if next_index >= len(context.lines): + break + candidate = context.lines[next_index] + if field_type == FieldType.STRING: + candidate = _COLUMN_GAP.split(candidate, maxsplit=1)[0] + value, raw = find_typed_value(field_type, candidate) + if value is not None: + return make_field(name, value, self.tier, _NEXT_LINE_CONFIDENCE, raw, label) + return None + + def _infer_currency(self, raw_text: str) -> Optional[ExtractedField]: + currency = detect_currency(raw_text) + if not currency: + return None + return make_field("currency", currency, self.tier, _CURRENCY_FALLBACK_CONFIDENCE, None) + + def _charge_lines(self, lines: List[str]) -> List[Tuple[int, LineItem]]: + """Every line that parses as a charge, paired with its line index.""" + charges = [] + for index, line in enumerate(lines): + item = self._parse_charge_line(line) + if item: + charges.append((index, item)) + return charges + + @staticmethod + def _assemble_line_items( + charges: List[Tuple[int, LineItem]], + known_charges: List[Tuple[int, LineItem]], + claimed: Dict[int, List[Tuple[int, int]]], + ) -> List[LineItem]: + """Recognised charges, plus unknown charges from lines no field claimed. + + A line a scalar field consumed (a labelled "Pro Rata Share" or + "Rentable Square Feet" row) is excluded here so it is never also + reported as a phantom charge. + """ + items = [item for _, item in known_charges] + items.extend( + item + for index, item in charges + if item.charge_type == ChargeType.OTHER and index not in claimed + ) + return items + + def _parse_charge_line(self, line: str) -> Optional[LineItem]: + money_matches = list(_MONEY_LIKE.finditer(line)) + if not money_matches: + return None + + amount_match = money_matches[-1] + description = line[: amount_match.start()].strip(" \t:-") + if not description: + return None + + charge_type = classify_charge(description) + if charge_type.value == "other" and self._is_summary_line(description): + return None + + quantity, unit_price = self._parse_quantity_rate(description) + return LineItem( + description=description, + charge_type=charge_type, + quantity=quantity, + unit_price=unit_price, + amount=parse_amount(amount_match.group(0)), + confidence=0.8, + ) + + @staticmethod + def _is_summary_line(description: str) -> bool: + lowered = description.lower() + return any(keyword in lowered for keyword in _SUMMARY_KEYWORDS) + + @staticmethod + def _parse_quantity_rate(description: str): + match = _QUANTITY_RATE.search(description) + if not match: + return None, None + return parse_amount(match.group(1)), parse_amount(match.group(2)) diff --git a/docex/intake/extractors/llm.py b/docex/intake/extractors/llm.py new file mode 100644 index 0000000..24262da --- /dev/null +++ b/docex/intake/extractors/llm.py @@ -0,0 +1,147 @@ +""" +Tier 2 (last resort): LLM extraction. + +Invoked only on the fields the heuristic could not resolve or that reconciled +badly - never on a whole invoice that already parsed cleanly. The model is +caller-provided (``llm_fn``), exactly like the embedding function pattern +elsewhere in DocEX, so the core package takes no hard dependency on any provider +SDK. See ``examples/`` for a concrete Claude adapter. + +The prompt asks the model to return, for each field, both the value and the +*label phrase as printed on the invoice*. That label feeds the learning loop: +once a novel phrasing is confirmed against ground truth, the free heuristic tier +learns it and this expensive tier is not needed for it again. +""" + +from __future__ import annotations + +import inspect +import json +import re +from typing import Awaitable, Callable, Dict, List, Optional, Tuple, Union + +from docex.intake.charges import classify_charge +from docex.intake.extractors.base import ( + ExtractionContext, + ExtractionResult, + FieldExtractor, + make_field, +) +from docex.intake.fields import FIELDS +from docex.intake.models import ExtractionTier, LineItem +from docex.intake.normalize import parse_amount, parse_value + +LLMFn = Callable[[str], Union[str, Awaitable[str]]] + +_LLM_CONFIDENCE = 0.95 +_JSON_BLOCK = re.compile(r"\{.*\}", re.DOTALL) + + +class LLMExtractor(FieldExtractor): + """Extracts fields by prompting a caller-provided language model.""" + + tier = ExtractionTier.LLM + + def __init__(self, llm_fn: LLMFn) -> None: + if not callable(llm_fn): + raise ValueError("llm_fn is required and must be callable") + self._llm_fn = llm_fn + + async def extract(self, context: ExtractionContext) -> ExtractionResult: + targets = context.target_fields or tuple(FIELDS) + prompt = self._build_prompt(context.raw_text, targets, context.want_line_items) + response = await self._call(prompt) + data = self._parse_response(response) + if data is None: + return ExtractionResult() + + result = ExtractionResult() + for name, entry in self._extracted_fields(data, targets): + result.fields[name] = entry + if context.want_line_items: + result.line_items = self._extracted_line_items(data) + return result + + def _build_prompt(self, raw_text: str, targets: Tuple[str, ...], want_line_items: bool) -> str: + field_lines = "\n".join( + f"- {name} ({FIELDS[name].type.value}): the invoice's {name.replace('_', ' ')}" + for name in targets + ) + line_item_clause = ( + '\n "line_items": [{"description": "...", "amount": "..."}],' if want_line_items else "" + ) + return ( + "You extract fields from a commercial real estate invoice.\n" + "Return ONLY minified JSON, no prose, in this exact shape:\n" + "{\n" + ' "fields": {"": {"value": "...", "label": ""}},' + f"{line_item_clause}\n" + "}\n" + "Use null for any field you cannot find. The 'label' is the heading or caption that " + "sits next to the value on the page; it is how we learn new invoice formats.\n\n" + f"Fields to extract:\n{field_lines}\n\n" + f"Invoice text:\n{raw_text}" + ) + + async def _call(self, prompt: str) -> str: + response = self._llm_fn(prompt) + if inspect.isawaitable(response): + response = await response + if not isinstance(response, str): + raise ValueError("llm_fn must return a JSON string (or an awaitable that resolves to one)") + return response + + def _parse_response(self, response: str) -> Optional[Dict]: + match = _JSON_BLOCK.search(response) + if not match: + return None + try: + return json.loads(match.group(0)) + except json.JSONDecodeError: + return None + + def _extracted_fields(self, data: Dict, targets: Tuple[str, ...]): + fields = data.get("fields", {}) + if not isinstance(fields, dict): + return + for name in targets: + entry = fields.get(name) + value = self._normalize_entry(name, entry) + if value is not None: + label = entry.get("label") if isinstance(entry, dict) else None + yield name, make_field(name, value, self.tier, _LLM_CONFIDENCE, _raw(entry), label) + + def _normalize_entry(self, name: str, entry) -> Optional[object]: + raw = entry.get("value") if isinstance(entry, dict) else entry + if raw is None or raw == "": + return None + return parse_value(FIELDS[name].type, str(raw)) + + def _extracted_line_items(self, data: Dict) -> List[LineItem]: + raw_items = data.get("line_items", []) + if not isinstance(raw_items, list): + return [] + items = [] + for raw in raw_items: + if not isinstance(raw, dict): + continue + description = (raw.get("description") or "").strip() + amount = parse_amount(str(raw.get("amount", ""))) + if not description and amount is None: + continue + items.append( + LineItem( + description=description or None, + charge_type=classify_charge(description), + amount=amount, + confidence=_LLM_CONFIDENCE, + ) + ) + return items + + +def _raw(entry) -> Optional[str]: + if isinstance(entry, dict): + value = entry.get("value") + return str(value) if value is not None else None + return str(entry) if entry is not None else None diff --git a/docex/intake/fields.py b/docex/intake/fields.py new file mode 100644 index 0000000..0bcf95e --- /dev/null +++ b/docex/intake/fields.py @@ -0,0 +1,191 @@ +""" +Canonical commercial-real-estate (CRE) invoice field registry. + +This is the single source of truth for the scalar header fields the intake +knows how to extract and reconcile. Every layer (heuristic extractor, embedding +extractor, LLM prompt, reconciler) reads from this registry, so adding a field +is a one-line change here rather than an edit in five places. + +The model is CRE-first: alongside generic invoice identity and totals it covers +the lease, property, billing-period, and pro-rata concepts that appear on rent +statements and operating-expense bills. Per-charge detail (base rent, CAM, real +estate tax pass-throughs, and so on) lives in line items, classified by the +charge taxonomy in :mod:`docex.intake.charges`. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import Dict, Tuple + + +class FieldType(str, Enum): + """How a field's value is parsed and compared.""" + + STRING = "string" + AMOUNT = "amount" + DATE = "date" + CURRENCY = "currency" + PERCENT = "percent" + NUMBER = "number" + + +@dataclass(frozen=True) +class FieldSpec: + """Describes one canonical invoice field. + + Attributes: + name: Canonical machine name (also the key used everywhere). + type: How the value is normalized and compared. + labels: Human label aliases that may precede the value on a page. + Used by the heuristic extractor for label-proximity matching and + by the embedding extractor as the prototype text for a field. + required: Whether the field is expected on every invoice. Required + fields that cannot be extracted drive escalation up the cascade. + """ + + name: str + type: FieldType + labels: Tuple[str, ...] + required: bool = False + + +_SPECS: Tuple[FieldSpec, ...] = ( + # --- Invoice identity ------------------------------------------------ + FieldSpec( + name="invoice_number", + type=FieldType.STRING, + labels=("invoice number", "invoice no", "invoice #", "invoice id", "statement number", "bill number"), + required=True, + ), + FieldSpec( + name="po_number", + type=FieldType.STRING, + labels=("purchase order", "po number", "po no", "po #", "p.o.", "order number"), + ), + # --- Parties --------------------------------------------------------- + FieldSpec( + name="landlord_name", + type=FieldType.STRING, + labels=("landlord", "lessor", "owner", "billed by", "remit to", "from"), + ), + FieldSpec( + name="landlord_tax_id", + type=FieldType.STRING, + labels=("tax id", "ein", "vat number", "abn", "gst number"), + ), + FieldSpec( + name="property_manager", + type=FieldType.STRING, + labels=("property manager", "managing agent", "management company", "managed by"), + ), + FieldSpec( + name="tenant_name", + type=FieldType.STRING, + labels=("tenant", "lessee", "bill to", "billed to", "occupant"), + ), + FieldSpec( + name="tenant_account", + type=FieldType.STRING, + labels=("tenant id", "account number", "account no", "tenant code", "customer number"), + ), + # --- Property and lease --------------------------------------------- + FieldSpec( + name="property_name", + type=FieldType.STRING, + labels=("property", "building", "property name", "building name", "premises", "project"), + ), + FieldSpec( + name="property_address", + type=FieldType.STRING, + labels=("property address", "premises address", "building address", "site address"), + ), + FieldSpec( + name="suite_number", + type=FieldType.STRING, + labels=("suite", "unit", "suite number", "unit number", "space", "floor"), + ), + FieldSpec( + name="lease_number", + type=FieldType.STRING, + labels=("lease number", "lease id", "lease no", "lease reference", "lease #"), + ), + # --- Measures -------------------------------------------------------- + FieldSpec( + name="rentable_square_feet", + type=FieldType.NUMBER, + labels=("rentable square feet", "rentable sf", "rsf", "rentable area", "leased area", "square feet", "sq ft"), + ), + FieldSpec( + name="pro_rata_share", + type=FieldType.PERCENT, + labels=("pro rata share", "pro-rata share", "proportionate share", "tenant share", "percentage share"), + ), + # --- Billing period -------------------------------------------------- + FieldSpec( + name="invoice_date", + type=FieldType.DATE, + labels=("invoice date", "statement date", "date of issue", "billing date", "date"), + ), + FieldSpec( + name="due_date", + type=FieldType.DATE, + labels=("due date", "payment due", "pay by", "due"), + ), + FieldSpec( + name="billing_period_start", + type=FieldType.DATE, + labels=("billing period from", "period start", "service period from", "from", "period beginning"), + ), + FieldSpec( + name="billing_period_end", + type=FieldType.DATE, + labels=("billing period to", "period end", "service period to", "to", "period ending"), + ), + # --- Money ----------------------------------------------------------- + FieldSpec( + name="currency", + type=FieldType.CURRENCY, + labels=("currency", "ccy"), + ), + FieldSpec( + name="prior_balance", + type=FieldType.AMOUNT, + labels=("prior balance", "previous balance", "balance forward", "beginning balance"), + ), + FieldSpec( + name="payments_received", + type=FieldType.AMOUNT, + labels=("payments received", "less payments", "payments applied", "amount paid"), + ), + FieldSpec( + name="current_charges", + type=FieldType.AMOUNT, + labels=("current charges", "current period charges", "charges this period", "new charges"), + ), + FieldSpec( + name="subtotal", + type=FieldType.AMOUNT, + labels=("subtotal", "sub total", "net amount", "net total", "amount before tax"), + ), + FieldSpec( + name="tax", + type=FieldType.AMOUNT, + labels=("tax", "vat", "gst", "sales tax", "tax amount"), + ), + FieldSpec( + name="total", + type=FieldType.AMOUNT, + labels=("total amount due", "total due", "amount due", "balance due", "grand total", "total", "amount payable"), + required=True, + ), +) + + +FIELDS: Dict[str, FieldSpec] = {spec.name: spec for spec in _SPECS} + + +def required_fields() -> Tuple[str, ...]: + """Names of fields expected on every invoice.""" + return tuple(name for name, spec in FIELDS.items() if spec.required) diff --git a/docex/intake/ground_truth.py b/docex/intake/ground_truth.py new file mode 100644 index 0000000..776b37a --- /dev/null +++ b/docex/intake/ground_truth.py @@ -0,0 +1,198 @@ +""" +Ground-truth invoices: our recorded actuals and where they live. + +A ground-truth record is what the lease *should* bill, against which an +incoming vendor PDF is reconciled. The schema mirrors the canonical field +registry one-to-one so the reconciler can read an expected value by name with +``record.get("base_rent_field")`` and never special-case anything. + +Two stores ship here: + +* :class:`InMemoryGroundTruthStore` - a dict-backed store for tests and small + in-process use. +* :class:`DocEXGroundTruthStore` - persists each record as a JSON document in a + DocEX basket, mirroring the lookup keys into document metadata so retrieval is + an indexed metadata query rather than a scan. +""" + +from __future__ import annotations + +import json +import tempfile +from abc import ABC, abstractmethod +from datetime import date +from decimal import Decimal +from pathlib import Path +from typing import Any, Dict, List, Optional + +from pydantic import BaseModel, ConfigDict, Field + +from docex.intake.charges import ChargeType +from docex.intake.models import LineItem + + +class GroundTruthInvoice(BaseModel): + """The expected (actual) invoice for a lease and billing period. + + Attribute names match the canonical field registry so the reconciler can + look up an expected value by field name via :meth:`get`. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + id: Optional[str] = None + + # Identity + invoice_number: str + po_number: Optional[str] = None + + # Parties + landlord_name: Optional[str] = None + landlord_tax_id: Optional[str] = None + property_manager: Optional[str] = None + tenant_name: Optional[str] = None + tenant_account: Optional[str] = None + + # Property and lease + property_name: Optional[str] = None + property_address: Optional[str] = None + suite_number: Optional[str] = None + lease_number: Optional[str] = None + + # Measures + rentable_square_feet: Optional[Decimal] = None + pro_rata_share: Optional[Decimal] = None + + # Billing period + invoice_date: Optional[date] = None + due_date: Optional[date] = None + billing_period_start: Optional[date] = None + billing_period_end: Optional[date] = None + + # Money + currency: str = "USD" + prior_balance: Optional[Decimal] = None + payments_received: Optional[Decimal] = None + current_charges: Optional[Decimal] = None + subtotal: Optional[Decimal] = None + tax: Optional[Decimal] = None + total: Decimal + + line_items: List[LineItem] = Field(default_factory=list) + + def get(self, field_name: str) -> Optional[Any]: + """Return the expected value for a canonical field, or ``None``.""" + return getattr(self, field_name, None) + + def charge_total(self, charge_type: ChargeType) -> Optional[Decimal]: + """Sum the line-item amounts for one charge type, or ``None`` if absent.""" + amounts = [item.amount for item in self.line_items if item.charge_type == charge_type and item.amount is not None] + return sum(amounts, Decimal("0")) if amounts else None + + def lookup_keys(self) -> Dict[str, Optional[str]]: + """The metadata keys a store mirrors for fast retrieval.""" + return { + "invoice_number": self.invoice_number, + "po_number": self.po_number, + "lease_number": self.lease_number, + "tenant_account": self.tenant_account, + } + + +class GroundTruthStore(ABC): + """Retrieval interface over recorded ground-truth invoices.""" + + @abstractmethod + def add(self, record: GroundTruthInvoice) -> GroundTruthInvoice: + """Persist a record and return it (with ``id`` populated).""" + + @abstractmethod + def get_by_invoice_number(self, invoice_number: str) -> Optional[GroundTruthInvoice]: + """Return the record with this invoice number, if any.""" + + @abstractmethod + def get_by_po_number(self, po_number: str) -> Optional[GroundTruthInvoice]: + """Return the record with this PO number, if any.""" + + @abstractmethod + def all(self) -> List[GroundTruthInvoice]: + """Return every stored record.""" + + +class InMemoryGroundTruthStore(GroundTruthStore): + """A dict-backed store, indexed by invoice and PO number.""" + + def __init__(self) -> None: + self._by_invoice: Dict[str, GroundTruthInvoice] = {} + self._by_po: Dict[str, GroundTruthInvoice] = {} + + def add(self, record: GroundTruthInvoice) -> GroundTruthInvoice: + if record.id is None: + record = record.model_copy(update={"id": f"gt_{len(self._by_invoice) + 1:06d}"}) + self._by_invoice[record.invoice_number] = record + if record.po_number: + self._by_po[record.po_number] = record + return record + + def get_by_invoice_number(self, invoice_number: str) -> Optional[GroundTruthInvoice]: + return self._by_invoice.get(invoice_number) + + def get_by_po_number(self, po_number: str) -> Optional[GroundTruthInvoice]: + return self._by_po.get(po_number) + + def all(self) -> List[GroundTruthInvoice]: + return list(self._by_invoice.values()) + + +class DocEXGroundTruthStore(GroundTruthStore): + """Persists ground-truth invoices as JSON documents in a DocEX basket. + + Each record becomes one JSON document; its lookup keys are mirrored into + document metadata so retrieval is an indexed metadata query. The full record + is reconstructed from the document's JSON content on read. + """ + + _METADATA_MARKER = "ground_truth_invoice" + + def __init__(self, basket: Any) -> None: + """ + Args: + basket: A DocEX ``DocBasket`` dedicated to ground-truth records. + """ + self._basket = basket + + def add(self, record: GroundTruthInvoice) -> GroundTruthInvoice: + payload = record.model_dump(mode="json") + metadata = {key: value for key, value in record.lookup_keys().items() if value is not None} + metadata["record_type"] = self._METADATA_MARKER + + with tempfile.NamedTemporaryFile( + mode="w", suffix=f"_{record.invoice_number}.json", delete=False + ) as handle: + json.dump(payload, handle) + temp_path = handle.name + try: + document = self._basket.add(temp_path, document_type="ground_truth", metadata=metadata) + finally: + Path(temp_path).unlink(missing_ok=True) + + return record.model_copy(update={"id": document.id}) + + def get_by_invoice_number(self, invoice_number: str) -> Optional[GroundTruthInvoice]: + return self._first_match({"invoice_number": invoice_number}) + + def get_by_po_number(self, po_number: str) -> Optional[GroundTruthInvoice]: + return self._first_match({"po_number": po_number}) + + def all(self) -> List[GroundTruthInvoice]: + documents = self._basket.find_documents_by_metadata({"record_type": self._METADATA_MARKER}) + return [self._load(document) for document in documents] + + def _first_match(self, metadata: Dict[str, str]) -> Optional[GroundTruthInvoice]: + documents = self._basket.find_documents_by_metadata(metadata, limit=1) + return self._load(documents[0]) if documents else None + + def _load(self, document: Any) -> GroundTruthInvoice: + payload = document.get_content(mode="json") + record = GroundTruthInvoice.model_validate(payload) + return record.model_copy(update={"id": document.id}) diff --git a/docex/intake/learning.py b/docex/intake/learning.py new file mode 100644 index 0000000..b7a1c31 --- /dev/null +++ b/docex/intake/learning.py @@ -0,0 +1,138 @@ +""" +The self-improving label loop. + +Every time an extracted field is confirmed against ground truth, the label +phrase that identified it is recorded with a running count. That yields two +things: + +* A *trend*: how customers actually phrase each field (``total`` arrives as + "Amount Due" far more often than "Balance Payable"), useful for analytics. +* A *learned alias*: a phrasing the static registry did not know about - often + surfaced by the LLM on a messy invoice - is promoted into the heuristic's + alias set, so the next invoice that uses it is solved for free by Tier 1. + +Only ground-truth-validated labels are recorded, so the loop cannot teach the +heuristic a wrong mapping. Over time the cheap tier absorbs the long tail of +vendor phrasings and the LLM is needed less and less. +""" + +from __future__ import annotations + +import json +from abc import ABC, abstractmethod +from pathlib import Path +from typing import Dict, List, Tuple + +from pydantic import BaseModel + +from docex.intake.fields import FIELDS +from docex.intake.models import ExtractedInvoice, MatchStatus, ReconciliationResult +from docex.intake.normalize import normalize_for_compare + + +class FieldObservation(BaseModel): + """How often one label phrasing has been confirmed for a canonical field.""" + + field_name: str + label: str + count: int + + +class LearningStore(ABC): + """Persists confirmed (field, label) observations and their counts.""" + + @abstractmethod + def record(self, field_name: str, label: str) -> None: + """Increment the confirmed-count for a (field, label) pairing.""" + + @abstractmethod + def label_counts(self, field_name: str) -> Dict[str, int]: + """Confirmed label phrasings for a field, mapped to their counts.""" + + @abstractmethod + def observations(self) -> List[FieldObservation]: + """Every observation, descending by count - the formatting trend report.""" + + def learned_labels(self, field_name: str, min_count: int = 1) -> Tuple[str, ...]: + """Labels confirmed at least ``min_count`` times that the registry lacks. + + Registry labels are excluded because the heuristic already scans those; + this returns only the *new* phrasings worth teaching it. + """ + known = {normalize_for_compare(label) for label in FIELDS[field_name].labels} if field_name in FIELDS else set() + learned = [ + label + for label, count in self.label_counts(field_name).items() + if count >= min_count and label not in known + ] + return tuple(learned) + + +class InMemoryLearningStore(LearningStore): + """A counter-backed store for tests and single-process use.""" + + def __init__(self) -> None: + self._counts: Dict[Tuple[str, str], int] = {} + + def record(self, field_name: str, label: str) -> None: + key = (field_name, normalize_for_compare(label)) + if not key[1]: + return + self._counts[key] = self._counts.get(key, 0) + 1 + + def label_counts(self, field_name: str) -> Dict[str, int]: + return {label: count for (name, label), count in self._counts.items() if name == field_name} + + def observations(self) -> List[FieldObservation]: + rows = [ + FieldObservation(field_name=name, label=label, count=count) + for (name, label), count in self._counts.items() + ] + return sorted(rows, key=lambda row: row.count, reverse=True) + + +class JsonFileLearningStore(InMemoryLearningStore): + """An :class:`InMemoryLearningStore` that persists to a JSON file. + + Counts survive across runs so the heuristic keeps everything it has learned. + The file is small (one entry per confirmed phrasing) and written on each + record; for high write volumes swap in a database-backed store. + """ + + def __init__(self, path: str | Path) -> None: + super().__init__() + self._path = Path(path) + self._load() + + def record(self, field_name: str, label: str) -> None: + super().record(field_name, label) + self._save() + + def _load(self) -> None: + if not self._path.exists(): + return + payload = json.loads(self._path.read_text()) + for row in payload: + self._counts[(row["field_name"], row["label"])] = row["count"] + + def _save(self) -> None: + payload = [obs.model_dump() for obs in self.observations()] + self._path.parent.mkdir(parents=True, exist_ok=True) + self._path.write_text(json.dumps(payload, indent=2)) + + +def record_confirmed_labels( + store: LearningStore, + extracted: ExtractedInvoice, + result: ReconciliationResult, +) -> None: + """Record the label of every field that reconciled cleanly against ground truth. + + This is the write side of the learning loop: it is called by the pipeline + after reconciliation so that only confirmed mappings are ever learned. + """ + matched = {comparison.field for comparison in result.field_comparisons if comparison.status == MatchStatus.MATCH} + for name in matched: + field = extracted.fields.get(name) + if field and field.label: + store.record(name, field.label) diff --git a/docex/intake/models.py b/docex/intake/models.py new file mode 100644 index 0000000..98ea97b --- /dev/null +++ b/docex/intake/models.py @@ -0,0 +1,167 @@ +""" +Data models for the PDF intake. + +These models carry an invoice from raw extraction through reconciliation. They +deliberately separate *what* was extracted from *how confident* the intake is +and *which tier* produced it, so the pipeline can escalate only the fields that +need it and the caller can see exactly how much each result cost. +""" + +from __future__ import annotations + +from decimal import Decimal +from enum import Enum +from typing import Any, Dict, List, Optional + +from pydantic import BaseModel, ConfigDict, Field + +from docex.intake.charges import ChargeType + + +class ExtractionTier(str, Enum): + """The extraction strategy that produced a value, cheapest to most costly.""" + + NONE = "none" + HEURISTIC = "heuristic" + LLM = "llm" + + +class ExtractedField(BaseModel): + """A single extracted value with its provenance. + + ``value`` is already normalized to its target type (``Decimal`` for money, + ``date`` for dates, ``str`` otherwise), so downstream layers never re-parse. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + name: str + value: Optional[Any] + confidence: float = Field(ge=0.0, le=1.0) + tier: ExtractionTier + raw_text: Optional[str] = None + label: Optional[str] = None # the label phrase on the page that identified this value + + +class LineItem(BaseModel): + """One charge line on a CRE invoice.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + description: Optional[str] = None + charge_type: ChargeType = ChargeType.OTHER + quantity: Optional[Decimal] = None + unit_price: Optional[Decimal] = None + amount: Optional[Decimal] = None + confidence: float = 1.0 + + +class ExtractedInvoice(BaseModel): + """The result of extracting one invoice, with per-field provenance. + + Scalar fields live in ``fields`` keyed by canonical name (see + :data:`docex.intake.fields.FIELDS`); charge detail lives in ``line_items``. + """ + + fields: Dict[str, ExtractedField] = Field(default_factory=dict) + line_items: List[LineItem] = Field(default_factory=list) + + def value(self, name: str) -> Optional[Any]: + """Return the normalized value for a field, or ``None`` if unextracted.""" + field = self.fields.get(name) + return field.value if field else None + + def confidence(self, name: str) -> float: + """Return the extraction confidence for a field (0.0 if unextracted).""" + field = self.fields.get(name) + return field.confidence if field else 0.0 + + def tier(self, name: str) -> ExtractionTier: + """Return the tier that produced a field (NONE if unextracted).""" + field = self.fields.get(name) + return field.tier if field else ExtractionTier.NONE + + def has(self, name: str) -> bool: + """Whether a field was extracted with a non-null value.""" + field = self.fields.get(name) + return field is not None and field.value is not None + + def put(self, field: ExtractedField) -> None: + """Insert or replace a field by name.""" + self.fields[field.name] = field + + def tiers_used(self) -> set[ExtractionTier]: + """Set of tiers that contributed at least one field.""" + return {field.tier for field in self.fields.values() if field.tier != ExtractionTier.NONE} + + +class MatchStatus(str, Enum): + """Outcome of comparing one extracted field against ground truth.""" + + MATCH = "match" + MISMATCH = "mismatch" + MISSING = "missing" # ground truth had a value, extraction did not + + +class FieldComparison(BaseModel): + """The reconciliation verdict for a single field.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + field: str + expected: Optional[Any] + actual: Optional[Any] + status: MatchStatus + confidence: float = 0.0 + tier: ExtractionTier = ExtractionTier.NONE + note: Optional[str] = None + + +class LineItemComparison(BaseModel): + """The reconciliation verdict for one charge type across the invoice.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + charge_type: ChargeType + expected: Optional[Decimal] + actual: Optional[Decimal] + status: MatchStatus + note: Optional[str] = None + + +class ReconciliationStatus(str, Enum): + """Overall verdict for an invoice against its ground-truth record.""" + + MATCHED = "matched" # every compared field agrees + DISCREPANCY = "discrepancy" # at least one field disagrees + INCOMPLETE = "incomplete" # fields missing but nothing disagrees + UNRESOLVED = "unresolved" # no ground-truth record could be matched + + +class ReconciliationResult(BaseModel): + """The full comparison of an extracted invoice against ground truth.""" + + status: ReconciliationStatus + ground_truth_id: Optional[str] = None + field_comparisons: List[FieldComparison] = Field(default_factory=list) + line_item_comparisons: List[LineItemComparison] = Field(default_factory=list) + tiers_used: List[ExtractionTier] = Field(default_factory=list) + + def by_status(self, status: MatchStatus) -> List[FieldComparison]: + """All field comparisons with the given status.""" + return [c for c in self.field_comparisons if c.status == status] + + @property + def mismatches(self) -> List[FieldComparison]: + """Field comparisons where extracted and expected values disagree.""" + return self.by_status(MatchStatus.MISMATCH) + + @property + def missing(self) -> List[FieldComparison]: + """Field comparisons where extraction failed to find an expected value.""" + return self.by_status(MatchStatus.MISSING) + + @property + def is_clean(self) -> bool: + """True when the invoice fully reconciles with no discrepancies.""" + return self.status == ReconciliationStatus.MATCHED diff --git a/docex/intake/normalize.py b/docex/intake/normalize.py new file mode 100644 index 0000000..7623afd --- /dev/null +++ b/docex/intake/normalize.py @@ -0,0 +1,212 @@ +""" +Value normalization for messy invoice text. + +Vendors format the same number a dozen ways: ``$1,234.56``, ``1.234,56``, +``(125.00)`` for a credit, ``USD 1,234``. Dates are worse. These helpers turn +raw spans into normalized Python types (``Decimal``, ``date``, ISO currency +codes) so every layer above compares apples to apples. + +Assumption: amounts and dates default to US conventions (``,`` groups +thousands, ``.`` is the decimal point, dates are month-first) unless the text +itself disambiguates (for example a component greater than 12 in a date, or a +value that has both separators). The reconciler never re-parses; it trusts the +normalized value carried on each field. +""" + +from __future__ import annotations + +import re +from datetime import date +from decimal import Decimal, InvalidOperation +from typing import Optional + +from docex.intake.fields import FieldType + +_CURRENCY_SYMBOLS = { + "$": "USD", + "€": "EUR", + "£": "GBP", + "¥": "JPY", + "A$": "AUD", + "C$": "CAD", +} + +_CURRENCY_CODES = {"USD", "EUR", "GBP", "JPY", "AUD", "CAD", "NZD", "CHF"} + +_MONTHS = { + "jan": 1, "january": 1, "feb": 2, "february": 2, "mar": 3, "march": 3, + "apr": 4, "april": 4, "may": 5, "jun": 6, "june": 6, "jul": 7, "july": 7, + "aug": 8, "august": 8, "sep": 9, "sept": 9, "september": 9, "oct": 10, + "october": 10, "nov": 11, "november": 11, "dec": 12, "december": 12, +} + + +def normalize_text(value: str) -> str: + """Collapse runs of whitespace and trim. ``None``-safe via empty string.""" + return re.sub(r"\s+", " ", value or "").strip() + + +def normalize_for_compare(value: str) -> str: + """Casefold and collapse whitespace for tolerant string comparison.""" + return normalize_text(value).casefold() + + +def parse_amount(raw: str) -> Optional[Decimal]: + """Parse a monetary string into a signed ``Decimal``. + + Handles currency symbols and codes, thousands grouping, both US and + European decimal separators, and parenthesised or trailing-minus negatives. + Returns ``None`` when no numeric value is present. + """ + if raw is None: + return None + text = raw.strip() + if not text: + return None + + negative = False + if text.startswith("(") and text.endswith(")"): + negative = True + text = text[1:-1] + if text.endswith("-"): + negative = True + text = text[:-1] + if text.startswith("-"): + negative = True + text = text[1:] + + digits = re.sub(r"[^0-9.,]", "", text) + if not re.search(r"\d", digits): + return None + + digits = _unify_decimal_separator(digits) + try: + amount = Decimal(digits) + except InvalidOperation: + return None + return -amount if negative else amount + + +def _unify_decimal_separator(digits: str) -> str: + """Reduce a grouped numeric string to a bare ``Decimal``-parseable string.""" + has_dot = "." in digits + has_comma = "," in digits + + if has_dot and has_comma: + # The rightmost separator is the decimal point; the other groups thousands. + decimal_sep = "." if digits.rfind(".") > digits.rfind(",") else "," + thousands_sep = "," if decimal_sep == "." else "." + digits = digits.replace(thousands_sep, "").replace(decimal_sep, ".") + elif has_comma: + digits = _resolve_single_separator(digits, ",") + elif has_dot: + digits = _resolve_single_separator(digits, ".") + return digits + + +def _resolve_single_separator(digits: str, sep: str) -> str: + """Decide whether a lone separator groups thousands or marks the decimal.""" + if digits.count(sep) > 1: + return digits.replace(sep, "") # only thousands grouping repeats + fractional_digits = len(digits.split(sep)[1]) + if fractional_digits == 3 and sep == ",": + return digits.replace(sep, "") # "1,234" is one thousand two hundred + return digits.replace(sep, ".") + + +def detect_currency(raw: str) -> Optional[str]: + """Return the ISO currency code implied by a string, if any.""" + if not raw: + return None + text = raw.upper() + for code in _CURRENCY_CODES: + if code in text: + return code + for symbol, code in _CURRENCY_SYMBOLS.items(): + if symbol in raw: + return code + return None + + +def parse_percent(raw: str) -> Optional[Decimal]: + """Parse a percentage into its numeric value (``"12.5%"`` -> ``Decimal('12.5')``).""" + if not raw: + return None + return parse_amount(raw.replace("%", " ").replace("percent", " ")) + + +def parse_number(raw: str) -> Optional[Decimal]: + """Parse a plain (possibly grouped) number such as a square-foot figure.""" + return parse_amount(raw) + + +def parse_date(raw: str) -> Optional[date]: + """Parse a date in any of the common invoice formats. + + Supports ISO (``2024-01-15``), slash/dot/dash numeric dates with US + month-first defaulting, and spelled-out months (``Jan 15, 2024`` or + ``15 January 2024``). Returns ``None`` if no date can be read. + """ + if not raw: + return None + text = normalize_text(raw) + + spelled = _parse_spelled_date(text) + if spelled: + return spelled + return _parse_numeric_date(text) + + +def _parse_spelled_date(text: str) -> Optional[date]: + """Parse dates that name the month, e.g. ``January 15, 2024``.""" + match = re.search(r"([A-Za-z]+)\.?\s+(\d{1,2})(?:st|nd|rd|th)?,?\s+(\d{4})", text) + if match: + month = _MONTHS.get(match.group(1).lower()) + if month: + return _safe_date(int(match.group(3)), month, int(match.group(2))) + + match = re.search(r"(\d{1,2})(?:st|nd|rd|th)?\.?\s+([A-Za-z]+)\.?,?\s+(\d{4})", text) + if match: + month = _MONTHS.get(match.group(2).lower()) + if month: + return _safe_date(int(match.group(3)), month, int(match.group(1))) + return None + + +def _parse_numeric_date(text: str) -> Optional[date]: + """Parse all-numeric dates, defaulting to month-first when ambiguous.""" + match = re.search(r"(\d{1,4})[/\-.](\d{1,2})[/\-.](\d{1,4})", text) + if not match: + return None + first, second, third = (int(g) for g in match.groups()) + + if first > 31: # leading four-digit year: YYYY-MM-DD + return _safe_date(first, second, third) + + year = third if third > 99 else 2000 + third + month, day = first, second + if first > 12 >= second: # first component cannot be a month + month, day = second, first + return _safe_date(year, month, day) + + +def _safe_date(year: int, month: int, day: int) -> Optional[date]: + try: + return date(year, month, day) + except ValueError: + return None + + +def parse_value(field_type: FieldType, raw: str) -> Optional[object]: + """Normalize a raw span according to its canonical field type.""" + if field_type == FieldType.AMOUNT: + return parse_amount(raw) + if field_type == FieldType.DATE: + return parse_date(raw) + if field_type == FieldType.CURRENCY: + return detect_currency(raw) + if field_type == FieldType.PERCENT: + return parse_percent(raw) + if field_type == FieldType.NUMBER: + return parse_number(raw) + return normalize_text(raw) diff --git a/docex/intake/pdf.py b/docex/intake/pdf.py new file mode 100644 index 0000000..6a61fdd --- /dev/null +++ b/docex/intake/pdf.py @@ -0,0 +1,42 @@ +""" +PDF to text, the only place that touches pdfminer. + +Keeping the binary-to-text boundary in one small module means every layer above +operates on plain text and is testable without generating PDFs. The heavy +``pdfminer.six`` dependency is optional (``pip install docex[pdf]``); importing +this module without it succeeds, and the failure only surfaces when extraction +is actually attempted. +""" + +from __future__ import annotations + +import io +from pathlib import Path +from typing import Union + +try: + from pdfminer.high_level import extract_text as _pdfminer_extract_text + + HAS_PDFMINER = True +except ImportError: # pragma: no cover - exercised only without the extra + HAS_PDFMINER = False + + def _pdfminer_extract_text(*args: object, **kwargs: object) -> str: + raise ImportError( + "PDF text extraction requires 'pdfminer.six'. Install it with: pip install docex[pdf]" + ) + + +def extract_text_from_pdf(source: Union[str, Path, bytes]) -> str: + """Extract the full text of a PDF given a path or raw bytes. + + Args: + source: A filesystem path or the PDF's bytes. + + Returns: + The document's text. Empty when the PDF carries no extractable text + (for example a pure image scan), which callers treat as "needs OCR". + """ + if isinstance(source, (str, Path)): + return _pdfminer_extract_text(str(source)) or "" + return _pdfminer_extract_text(io.BytesIO(source)) or "" diff --git a/docex/intake/pipeline.py b/docex/intake/pipeline.py new file mode 100644 index 0000000..3d3faa8 --- /dev/null +++ b/docex/intake/pipeline.py @@ -0,0 +1,144 @@ +""" +The intake pipeline: PDF in, reconciliation out. + +This ties the pieces together in the cost-minimal order the whole design is +built around: + +1. Extract cheaply (heuristic), escalating only missing required fields. +2. Match the invoice to a ground-truth record by stable identifier. +3. Reconcile against that record. +4. *Only if* the reconciliation shows disputes, ask the LLM to re-extract just + those fields, then reconcile once more. This is the "final check if + everything else shows an issue" step - a clean invoice never reaches it. +5. Record the labels of confirmed fields so the heuristic keeps improving. + +The LLM is optional. With no ``llm_fn`` the pipeline runs fully offline and +simply reports the fields the heuristic could not resolve. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional, Tuple, Union + +from docex.intake.embedding_match import EmbeddingFn, EmbeddingGroundTruthMatcher +from docex.intake.extractors.cascade import CascadingExtractor +from docex.intake.extractors.heuristic import HeuristicExtractor +from docex.intake.extractors.llm import LLMExtractor, LLMFn +from docex.intake.ground_truth import GroundTruthInvoice, GroundTruthStore +from docex.intake.learning import LearningStore, record_confirmed_labels +from docex.intake.models import ( + ExtractedInvoice, + MatchStatus, + ReconciliationResult, + ReconciliationStatus, +) +from docex.intake.pdf import extract_text_from_pdf +from docex.intake.reconcile import GroundTruthMatcher, Reconciler, TolerancePolicy + +GroundTruth = Union[GroundTruthInvoice, GroundTruthStore] + + +@dataclass +class IntakeOutcome: + """The result of running one invoice through the pipeline.""" + + extracted: ExtractedInvoice + reconciliation: ReconciliationResult + ground_truth: Optional[GroundTruthInvoice] + + @property + def status(self) -> ReconciliationStatus: + return self.reconciliation.status + + @property + def is_clean(self) -> bool: + return self.reconciliation.is_clean + + +class InvoiceIntakePipeline: + """Extracts, matches, reconciles, and (only as needed) escalates to an LLM.""" + + def __init__( + self, + llm_fn: Optional[LLMFn] = None, + learning_store: Optional[LearningStore] = None, + tolerance: Optional[TolerancePolicy] = None, + embedding_fn: Optional[EmbeddingFn] = None, + match_threshold: float = 0.8, + ) -> None: + """ + Args: + llm_fn: Optional caller-provided language model (see + :class:`~docex.intake.extractors.llm.LLMExtractor`). Omit to run + heuristic-only. + learning_store: Optional store that records confirmed label phrasings + so the heuristic improves over time. + tolerance: Optional reconciliation tolerances; sensible defaults + otherwise (a cent on money, exact on dates). + embedding_fn: Optional embedding function enabling fuzzy ground-truth + retrieval. When set, an invoice that does not match any record by + identifier falls back to the closest record by embedding + similarity instead of being reported as unresolved. + match_threshold: Minimum cosine similarity for the embedding fallback + to accept a record. + """ + llm = LLMExtractor(llm_fn) if llm_fn else None + self._cascade = CascadingExtractor(HeuristicExtractor(learning_store), llm) + self._reconciler = Reconciler(tolerance) + self._learning_store = learning_store + self._can_escalate = llm is not None + self._embedding_matcher = ( + EmbeddingGroundTruthMatcher(embedding_fn, match_threshold) if embedding_fn else None + ) + + async def process_pdf(self, source, ground_truth: GroundTruth) -> IntakeOutcome: + """Extract text from a PDF (path or bytes) and run the pipeline.""" + return await self.process_text(extract_text_from_pdf(source), ground_truth) + + async def process_text(self, raw_text: str, ground_truth: GroundTruth) -> IntakeOutcome: + """Run the pipeline on already-extracted invoice text.""" + extraction = await self._cascade.extract(raw_text) + extracted = ExtractedInvoice(fields=extraction.fields, line_items=extraction.line_items) + + record = await self._resolve_ground_truth(extracted, ground_truth) + if record is None: + return IntakeOutcome(extracted, _unresolved(), None) + + result = self._reconciler.reconcile(extracted, record) + if self._should_escalate(result): + extraction = await self._cascade.repair(raw_text, self._disputed_fields(result), extraction) + extracted = ExtractedInvoice(fields=extraction.fields, line_items=extraction.line_items) + result = self._reconciler.reconcile(extracted, record) + + if self._learning_store is not None: + record_confirmed_labels(self._learning_store, extracted, result) + + return IntakeOutcome(extracted, result, record) + + async def _resolve_ground_truth(self, extracted: ExtractedInvoice, ground_truth: GroundTruth) -> Optional[GroundTruthInvoice]: + if isinstance(ground_truth, GroundTruthInvoice): + return ground_truth + + exact = GroundTruthMatcher(ground_truth).find(extracted) + if exact is not None or self._embedding_matcher is None: + return exact + + # No identifier matched; fall back to the closest record by similarity. + match = await self._embedding_matcher.find(extracted, ground_truth) + return match.record if match else None + + def _should_escalate(self, result: ReconciliationResult) -> bool: + return self._can_escalate and result.status != ReconciliationStatus.MATCHED + + @staticmethod + def _disputed_fields(result: ReconciliationResult) -> Tuple[str, ...]: + return tuple( + comparison.field + for comparison in result.field_comparisons + if comparison.status in (MatchStatus.MISMATCH, MatchStatus.MISSING) + ) + + +def _unresolved() -> ReconciliationResult: + return ReconciliationResult(status=ReconciliationStatus.UNRESOLVED) diff --git a/docex/intake/processor.py b/docex/intake/processor.py new file mode 100644 index 0000000..7701991 --- /dev/null +++ b/docex/intake/processor.py @@ -0,0 +1,79 @@ +""" +DocEX processor integration for invoice intake. + +Wraps :class:`~docex.intake.pipeline.InvoiceIntakePipeline` as a DocEX +``BaseProcessor`` so an invoice already stored in a basket can be reconciled in +place, with the verdict written back to the document's metadata. + +The pipeline dependencies (the ground-truth store, the optional LLM, the +learning store) are passed to the constructor rather than through the +JSON-serialisable processor config, mirroring how +:class:`~docex.processors.vector.VectorIndexingProcessor` takes its +``embedding_fn``. A tenant-aware ``db`` is optional: without it the processor +still returns the full reconciliation, it just does not persist metadata. +""" + +from __future__ import annotations + +from typing import Any, Dict, Optional + +from docex.db.connection import Database +from docex.document import Document +from docex.intake.extractors.llm import LLMFn +from docex.intake.ground_truth import GroundTruthStore +from docex.intake.learning import LearningStore +from docex.intake.pipeline import IntakeOutcome, InvoiceIntakePipeline +from docex.intake.reconcile import TolerancePolicy +from docex.processors.base import BaseProcessor, ProcessingResult +from docex.services.metadata_service import MetadataService + + +class InvoiceIntakeProcessor(BaseProcessor): + """Reconciles a stored invoice PDF against ground truth, recording the verdict.""" + + def __init__( + self, + ground_truth_store: GroundTruthStore, + llm_fn: Optional[LLMFn] = None, + learning_store: Optional[LearningStore] = None, + tolerance: Optional[TolerancePolicy] = None, + db: Optional[Database] = None, + store_in_metadata: bool = True, + ) -> None: + self.config = {"store_in_metadata": store_in_metadata} + self.db = db + self._store = ground_truth_store + self._pipeline = InvoiceIntakePipeline(llm_fn, learning_store, tolerance) + self._store_in_metadata = store_in_metadata + + def can_process(self, document: Document) -> bool: + return document.name.lower().endswith(".pdf") + + async def process(self, document: Document) -> ProcessingResult: + try: + pdf_bytes = self.get_document_bytes(document) + outcome = await self._pipeline.process_pdf(pdf_bytes, self._store) + metadata = self._build_metadata(outcome) + + if self._store_in_metadata and self.db is not None and outcome.ground_truth is not None: + MetadataService(self.db).update_metadata(document.id, metadata) + + return ProcessingResult(success=True, content=outcome.status.value, metadata=metadata) + except Exception as exc: # noqa: BLE001 - surfaced as a processing failure + return ProcessingResult(success=False, error=str(exc)) + + def _build_metadata(self, outcome: IntakeOutcome) -> Dict[str, Any]: + reconciliation = outcome.reconciliation + return { + "intake_status": reconciliation.status.value, + "intake_ground_truth_id": reconciliation.ground_truth_id, + "intake_invoice_number": outcome.extracted.value("invoice_number"), + "intake_total": _as_text(outcome.extracted.value("total")), + "intake_mismatched_fields": [comparison.field for comparison in reconciliation.mismatches], + "intake_missing_fields": [comparison.field for comparison in reconciliation.missing], + "intake_tiers_used": [tier.value for tier in reconciliation.tiers_used], + } + + +def _as_text(value: Any) -> Optional[str]: + return str(value) if value is not None else None diff --git a/docex/intake/reconcile.py b/docex/intake/reconcile.py new file mode 100644 index 0000000..4af63f7 --- /dev/null +++ b/docex/intake/reconcile.py @@ -0,0 +1,189 @@ +""" +Reconciliation: does the extracted invoice agree with our recorded actuals? + +The reconciler compares an :class:`~docex.intake.models.ExtractedInvoice` +against a :class:`~docex.intake.ground_truth.GroundTruthInvoice` field by field, +applying type-aware tolerances (a cent of rounding on money, a day on dates), +and rolls per-charge line items up by category so a vendor's "CAM" line is +compared against the lease's expected CAM regardless of wording. + +It only judges fields the ground truth actually specifies - a value we never +recorded an expectation for cannot be right or wrong, so it is not reported. +""" + +from __future__ import annotations + +from datetime import date +from decimal import Decimal +from typing import List, Optional + +from docex.intake.charges import ChargeType +from docex.intake.fields import FIELDS, FieldType +from docex.intake.ground_truth import GroundTruthInvoice, GroundTruthStore +from docex.intake.models import ( + ExtractedInvoice, + FieldComparison, + LineItemComparison, + MatchStatus, + ReconciliationResult, + ReconciliationStatus, +) +from docex.intake.normalize import normalize_for_compare + + +class TolerancePolicy: + """How much disagreement still counts as a match, per value type.""" + + def __init__( + self, + amount_abs: Decimal = Decimal("0.01"), + amount_rel: float = 0.0, + date_days: int = 0, + percent_abs: Decimal = Decimal("0.01"), + ) -> None: + self.amount_abs = amount_abs + self.amount_rel = amount_rel + self.date_days = date_days + self.percent_abs = percent_abs + + def amounts_match(self, expected: Decimal, actual: Decimal) -> bool: + allowed = max(self.amount_abs, Decimal(str(self.amount_rel)) * abs(expected)) + return abs(expected - actual) <= allowed + + def percents_match(self, expected: Decimal, actual: Decimal) -> bool: + return abs(expected - actual) <= self.percent_abs + + def dates_match(self, expected: date, actual: date) -> bool: + return abs((expected - actual).days) <= self.date_days + + +class Reconciler: + """Compares an extracted invoice against ground truth with tolerances.""" + + def __init__(self, tolerance: Optional[TolerancePolicy] = None) -> None: + self._tolerance = tolerance or TolerancePolicy() + + def reconcile(self, extracted: ExtractedInvoice, ground_truth: GroundTruthInvoice) -> ReconciliationResult: + field_comparisons = [ + comparison + for name in FIELDS + if (comparison := self._compare_field(name, extracted, ground_truth)) is not None + ] + line_item_comparisons = self._compare_line_items(extracted, ground_truth) + + return ReconciliationResult( + status=self._overall_status(field_comparisons), + ground_truth_id=ground_truth.id, + field_comparisons=field_comparisons, + line_item_comparisons=line_item_comparisons, + tiers_used=sorted(extracted.tiers_used(), key=lambda tier: tier.value), + ) + + def _compare_field( + self, + name: str, + extracted: ExtractedInvoice, + ground_truth: GroundTruthInvoice, + ) -> Optional[FieldComparison]: + expected = ground_truth.get(name) + if expected is None: + return None # no recorded expectation -> nothing to judge + + actual = extracted.value(name) + status = self._field_status(FIELDS[name].type, expected, actual) + return FieldComparison( + field=name, + expected=expected, + actual=actual, + status=status, + confidence=extracted.confidence(name), + tier=extracted.tier(name), + ) + + def _field_status(self, field_type: FieldType, expected: object, actual: object) -> MatchStatus: + if actual is None: + return MatchStatus.MISSING + return MatchStatus.MATCH if self._values_match(field_type, expected, actual) else MatchStatus.MISMATCH + + def _values_match(self, field_type: FieldType, expected: object, actual: object) -> bool: + if field_type == FieldType.AMOUNT: + return self._tolerance.amounts_match(expected, actual) + if field_type == FieldType.NUMBER: + return self._tolerance.amounts_match(expected, actual) + if field_type == FieldType.PERCENT: + return self._tolerance.percents_match(expected, actual) + if field_type == FieldType.DATE: + return self._tolerance.dates_match(expected, actual) + return normalize_for_compare(str(expected)) == normalize_for_compare(str(actual)) + + def _compare_line_items( + self, + extracted: ExtractedInvoice, + ground_truth: GroundTruthInvoice, + ) -> List[LineItemComparison]: + charge_types = self._charge_types(extracted, ground_truth) + comparisons = [] + for charge_type in charge_types: + expected = ground_truth.charge_total(charge_type) + actual = self._extracted_charge_total(extracted, charge_type) + comparisons.append( + LineItemComparison( + charge_type=charge_type, + expected=expected, + actual=actual, + status=self._charge_status(expected, actual), + ) + ) + return comparisons + + def _charge_types(self, extracted: ExtractedInvoice, ground_truth: GroundTruthInvoice) -> List[ChargeType]: + seen = {item.charge_type for item in ground_truth.line_items} + seen.update(item.charge_type for item in extracted.line_items) + return sorted(seen, key=lambda charge: charge.value) + + @staticmethod + def _extracted_charge_total(extracted: ExtractedInvoice, charge_type: ChargeType) -> Optional[Decimal]: + amounts = [ + item.amount + for item in extracted.line_items + if item.charge_type == charge_type and item.amount is not None + ] + return sum(amounts, Decimal("0")) if amounts else None + + def _charge_status(self, expected: Optional[Decimal], actual: Optional[Decimal]) -> MatchStatus: + if expected is None or actual is None: + return MatchStatus.MISSING + return MatchStatus.MATCH if self._tolerance.amounts_match(expected, actual) else MatchStatus.MISMATCH + + @staticmethod + def _overall_status(comparisons: List[FieldComparison]) -> ReconciliationStatus: + statuses = {comparison.status for comparison in comparisons} + if MatchStatus.MISMATCH in statuses: + return ReconciliationStatus.DISCREPANCY + if MatchStatus.MISSING in statuses: + return ReconciliationStatus.INCOMPLETE + return ReconciliationStatus.MATCHED + + +class GroundTruthMatcher: + """Finds the ground-truth record an extracted invoice should match. + + Matching is by stable identifiers only - invoice number first, then PO + number. We never guess a record from fuzzy totals: reconciling against the + wrong lease is worse than reporting that no record was found. + """ + + def __init__(self, store: GroundTruthStore) -> None: + self._store = store + + def find(self, extracted: ExtractedInvoice) -> Optional[GroundTruthInvoice]: + invoice_number = extracted.value("invoice_number") + if invoice_number: + match = self._store.get_by_invoice_number(str(invoice_number)) + if match: + return match + + po_number = extracted.value("po_number") + if po_number: + return self._store.get_by_po_number(str(po_number)) + return None diff --git a/example_docs/cre_invoices/README.md b/example_docs/cre_invoices/README.md new file mode 100644 index 0000000..4dbc627 --- /dev/null +++ b/example_docs/cre_invoices/README.md @@ -0,0 +1,34 @@ +# Sample Commercial Real Estate Invoices + +These two PDFs are the sample invoices the PDF intake is tested against. Open +them in any PDF viewer to see exactly what the intake reads. + +Both invoices are billed for the **same lease** (invoice `INV-2024-0042`, +tenant Acme Retail LLC, Harbor Point Tower, Suite 1200). Our recorded ground +truth for that lease expects: + +| Charge | Expected amount | +| ----------------------- | --------------: | +| Base Rent | $20,833.33 | +| Common Area Maintenance | $5,000.00 | +| Real Estate Tax Recovery| $1,250.00 | +| **Total Amount Due** | **$27,083.33** | + +### `positive_invoice_matches_ground_truth.pdf` + +What a correct invoice looks like. Every charge matches the lease, so the intake +reconciles it as **matched** - no action needed. + +### `negative_invoice_overcharged_cam.pdf` + +The same invoice, but the landlord has overstated **Common Area Maintenance** by +$750 (billing $5,750.00 instead of $5,000.00), which inflates the total to +$27,833.33. The intake reconciles it as a **discrepancy**, flagging the CAM line +and the total so the bill can be disputed before payment. + +These files are generated from `tests/intake/realistic_invoice.py`. To +regenerate them after a change, run: + +```sh +python -m tests.intake.realistic_invoice +``` diff --git a/example_docs/cre_invoices/negative_invoice_overcharged_cam.pdf b/example_docs/cre_invoices/negative_invoice_overcharged_cam.pdf new file mode 100644 index 0000000..b1f4d6c Binary files /dev/null and b/example_docs/cre_invoices/negative_invoice_overcharged_cam.pdf differ diff --git a/example_docs/cre_invoices/positive_invoice_matches_ground_truth.pdf b/example_docs/cre_invoices/positive_invoice_matches_ground_truth.pdf new file mode 100644 index 0000000..3e51acc Binary files /dev/null and b/example_docs/cre_invoices/positive_invoice_matches_ground_truth.pdf differ diff --git a/examples/integrations/anthropic/invoice_intake_llm.py b/examples/integrations/anthropic/invoice_intake_llm.py new file mode 100644 index 0000000..8503577 --- /dev/null +++ b/examples/integrations/anthropic/invoice_intake_llm.py @@ -0,0 +1,64 @@ +""" +Claude adapter for the DocEX invoice intake (copy-adapt example). + +The intake's LLM tier (:class:`docex.intake.extractors.llm.LLMExtractor`) takes a +provider-neutral ``llm_fn`` - a callable that maps a prompt string to a JSON +string. This module builds that callable on top of the Anthropic SDK so the +intake stays free of any hard provider dependency. + +It is intentionally an example, not core API. Copy it into your project and +adapt the model, credentials, and error handling to your needs. + + from docex.intake import InvoiceIntakePipeline + from examples.integrations.anthropic.invoice_intake_llm import make_claude_llm_fn + + pipeline = InvoiceIntakePipeline(llm_fn=make_claude_llm_fn()) + outcome = await pipeline.process_pdf("invoice.pdf", ground_truth_store) + +Install the SDK first: ``pip install anthropic``. +""" + +from __future__ import annotations + +from typing import Optional + +try: + import anthropic +except ImportError as exc: # pragma: no cover - example module + raise ImportError("This example requires the 'anthropic' package: pip install anthropic") from exc + +from docex.intake.extractors.llm import LLMFn + +# The intake calls the LLM only as a last resort, on a handful of unresolved or +# disputed fields, so the request is small. Opus 4.8 is the default for accuracy +# on messy layouts; switch to "claude-haiku-4-5" or "claude-sonnet-4-6" if you +# would rather trade a little accuracy for lower cost on this fallback path. +_DEFAULT_MODEL = "claude-opus-4-8" +_MAX_TOKENS = 4096 + + +def make_claude_llm_fn( + model: str = _DEFAULT_MODEL, + client: Optional["anthropic.Anthropic"] = None, +) -> LLMFn: + """Build an ``llm_fn`` for the intake backed by the Anthropic Messages API. + + Args: + model: The Claude model id. Defaults to ``claude-opus-4-8``. + client: An existing ``anthropic.Anthropic`` client, or ``None`` to build + one from the ``ANTHROPIC_API_KEY`` environment variable. + + Returns: + A callable suitable for ``InvoiceIntakePipeline(llm_fn=...)``. + """ + client = client or anthropic.Anthropic() + + def llm_fn(prompt: str) -> str: + message = client.messages.create( + model=model, + max_tokens=_MAX_TOKENS, + messages=[{"role": "user", "content": prompt}], + ) + return "".join(block.text for block in message.content if block.type == "text") + + return llm_fn diff --git a/tests/intake/__init__.py b/tests/intake/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/intake/conftest.py b/tests/intake/conftest.py new file mode 100644 index 0000000..44e320d --- /dev/null +++ b/tests/intake/conftest.py @@ -0,0 +1,15 @@ +"""Shared fixtures for the intake test suite.""" + +import asyncio + +import pytest + + +@pytest.fixture +def run(): + """Run a coroutine to completion without requiring pytest-asyncio.""" + + def _run(coro): + return asyncio.run(coro) + + return _run diff --git a/tests/intake/realistic_invoice.py b/tests/intake/realistic_invoice.py new file mode 100644 index 0000000..8a1995b --- /dev/null +++ b/tests/intake/realistic_invoice.py @@ -0,0 +1,178 @@ +""" +A realistic commercial-real-estate invoice PDF fixture. + +The bulk of the suite tests on text. This module builds a genuinely formatted +invoice - landlord letterhead, billing and property blocks, a ruled charges +table, and a totals section - so the intake is proven against a document that +looks like something a property manager would actually send, not a plain text +dump. + +``SAMPLE_GROUND_TRUTH`` is the recorded actuals for this invoice. The committed +fixture ``fixtures/sample_cre_invoice.pdf`` is produced by +:func:`build_realistic_invoice_pdf` (run ``python -m tests.intake.realistic_invoice`` +to regenerate it). The renderer lays each label/value on its own baseline and +right-aligns charge amounts so pdfminer recovers clean "label: value" and +"description ... amount" lines. +""" + +from __future__ import annotations + +from decimal import Decimal +from pathlib import Path + +from docex.intake.charges import ChargeType +from docex.intake.ground_truth import GroundTruthInvoice +from docex.intake.models import LineItem + +# The sample PDFs live under example_docs/ so a non-technical reader can open +# and view them directly; the tests read the same files. +_SAMPLE_DIR = Path(__file__).resolve().parents[2] / "example_docs" / "cre_invoices" +FIXTURE_PATH = _SAMPLE_DIR / "positive_invoice_matches_ground_truth.pdf" +OVERCHARGED_FIXTURE_PATH = _SAMPLE_DIR / "negative_invoice_overcharged_cam.pdf" +OVERCHARGE_DELTA = Decimal("750.00") # CAM the negative fixture overstates by + +SAMPLE_GROUND_TRUTH = GroundTruthInvoice( + invoice_number="INV-2024-0042", + po_number="PO-778812", + landlord_name="Summit Property Group", + property_manager="Meridian Property Management", + tenant_name="Acme Retail LLC", + tenant_account="TEN-4821", + property_name="Harbor Point Tower", + suite_number="1200", + lease_number="LS-5567", + rentable_square_feet=Decimal("12500"), + pro_rata_share=Decimal("8.25"), + invoice_date=None, # printed on the invoice but not reconciled in the fixture test + due_date=None, + currency="USD", + subtotal=Decimal("27083.33"), + tax=Decimal("0.00"), + total=Decimal("27083.33"), + line_items=[ + LineItem(description="Base Rent", charge_type=ChargeType.BASE_RENT, amount=Decimal("20833.33")), + LineItem(description="Common Area Maintenance", charge_type=ChargeType.CAM, amount=Decimal("5000.00")), + LineItem(description="Real Estate Tax Recovery", charge_type=ChargeType.REAL_ESTATE_TAX, amount=Decimal("1250.00")), + ], +) + + +_TABLE_WIDTH = 64 # characters across the monospaced body column + + +def _row(left_text: str, right_text: str = "") -> str: + """One monospaced row, ``right_text`` flushed to the right margin.""" + if not right_text: + return left_text + pad = max(1, _TABLE_WIDTH - len(left_text) - len(right_text)) + return f"{left_text}{' ' * pad}{right_text}" + + +def build_realistic_invoice_pdf(gt: GroundTruthInvoice = SAMPLE_GROUND_TRUTH) -> bytes: + """Render ``gt`` as a formatted one-page invoice PDF (requires reportlab). + + A styled Helvetica letterhead and "INVOICE" banner sit above a monospaced + body. The body is drawn as one text object so each logical row (a labelled + field, a charge line, a total) is a single line of text - the way many + property-management systems emit invoices, and what lets pdfminer recover + the rows intact instead of splitting columns. + """ + import io + + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + width, height = letter + buffer = io.BytesIO() + pdf = canvas.Canvas(buffer, pagesize=letter) + left = 54 + right = width - 54 + + # Styled letterhead and banner. + pdf.setFont("Helvetica-Bold", 18) + pdf.drawString(left, height - 60, gt.landlord_name) + pdf.setFont("Helvetica", 9) + pdf.drawString(left, height - 74, "1 Harbor Plaza, 30th Floor, Boston, MA 02210") + pdf.drawString(left, height - 86, f"Managed by {gt.property_manager}") + pdf.setFont("Helvetica-Bold", 22) + pdf.drawRightString(right, height - 64, "INVOICE") + pdf.line(left, height - 96, right, height - 96) + + rule = "-" * _TABLE_WIDTH + body = [ + _row("Invoice Number:", gt.invoice_number), + _row("Purchase Order:", gt.po_number), + _row("Invoice Date:", "January 01, 2024"), + _row("Due Date:", "January 15, 2024"), + "", + _row("Bill To:", gt.tenant_name), + _row("Tenant ID:", gt.tenant_account), + _row("Property:", gt.property_name), + _row("Suite:", gt.suite_number), + _row("Lease Number:", gt.lease_number), + _row("Rentable Square Feet:", f"{gt.rentable_square_feet:,.0f}"), + _row("Pro Rata Share:", f"{gt.pro_rata_share}%"), + "", + _row("Description", "Amount"), + rule, + ] + body.extend(_row(item.description, f"${item.amount:,.2f}") for item in gt.line_items) + body.extend( + [ + rule, + _row("Subtotal", f"${gt.subtotal:,.2f}"), + _row("Tax", f"${gt.tax:,.2f}"), + _row("Total Amount Due", f"${gt.total:,.2f}"), + "", + _row("Remit To:", gt.landlord_name), + "Please reference the invoice number on your payment.", + ] + ) + + text_object = pdf.beginText(left, height - 130) + text_object.setFont("Courier", 10) + text_object.setLeading(15) + for line in body: + text_object.textLine(line) + pdf.drawText(text_object) + + pdf.showPage() + pdf.save() + return buffer.getvalue() + + +def as_billed_overcharge(gt: GroundTruthInvoice = SAMPLE_GROUND_TRUTH) -> GroundTruthInvoice: + """The same invoice with CAM (and its totals) overstated by ``OVERCHARGE_DELTA``. + + Same invoice number, so it still matches the ground-truth record - but the + printed charges no longer agree with it, which is exactly the + vendor-overcharge case the intake must catch. + """ + inflated_items = [ + item.model_copy(update={"amount": item.amount + OVERCHARGE_DELTA}) + if item.charge_type == ChargeType.CAM + else item + for item in gt.line_items + ] + inflated_total = sum((item.amount for item in inflated_items), Decimal("0")) + return gt.model_copy( + update={"line_items": inflated_items, "subtotal": inflated_total, "total": inflated_total} + ) + + +def build_overcharged_invoice_pdf() -> bytes: + """Render the negative (overcharged) invoice as a realistic PDF.""" + return build_realistic_invoice_pdf(as_billed_overcharge()) + + +def regenerate_fixtures() -> tuple[Path, Path]: + """Write both committed PDF fixtures to disk and return their paths.""" + FIXTURE_PATH.parent.mkdir(parents=True, exist_ok=True) + FIXTURE_PATH.write_bytes(build_realistic_invoice_pdf()) + OVERCHARGED_FIXTURE_PATH.write_bytes(build_overcharged_invoice_pdf()) + return FIXTURE_PATH, OVERCHARGED_FIXTURE_PATH + + +if __name__ == "__main__": + for path in regenerate_fixtures(): + print(f"Wrote {path}") diff --git a/tests/intake/synthetic.py b/tests/intake/synthetic.py new file mode 100644 index 0000000..53dfff8 --- /dev/null +++ b/tests/intake/synthetic.py @@ -0,0 +1,194 @@ +""" +Synthetic commercial-real-estate invoice generator for the intake tests. + +The generator produces a matched pair: a :class:`GroundTruthInvoice` (our +recorded actuals) and the text of a vendor invoice that should reconcile to it. +Both are driven by a seeded ``random.Random`` so every test is reproducible, and +the renderer deliberately varies label phrasing, date format, and column layout +so the heuristic extractor is exercised against the messiness real invoices +carry - without ever needing a real PDF. + +Helpers are also provided for the specific edge cases the test suite asserts on +(overcharges, missing fields, novel labels, European amount formatting). +""" + +from __future__ import annotations + +import random +from datetime import date, timedelta +from decimal import ROUND_HALF_UP, Decimal +from typing import Tuple + +from docex.intake.charges import ChargeType +from docex.intake.ground_truth import GroundTruthInvoice +from docex.intake.models import LineItem + +_TENANTS = ["Acme Retail LLC", "Borealis Trading Co", "Cedar & Vine Hospitality", "Delphi Analytics Inc"] +_LANDLORDS = ["Harbor Point Holdings", "Summit Property Group", "Northgate Realty Trust"] +_PROPERTIES = ["Harbor Point Tower", "Summit Plaza", "Northgate Commons", "Riverside Exchange"] +_MANAGERS = ["Meridian Property Management", "BlueStone Asset Services"] + +# Phrasings the registry already knows, used to vary clean invoices. +_INVOICE_NO_LABELS = ["Invoice Number", "Invoice No", "Statement Number"] +_TOTAL_LABELS = ["Total Amount Due", "Amount Due", "Balance Due", "Total Due"] +_TENANT_LABELS = ["Bill To", "Tenant", "Billed To"] +_DATE_STYLES = ["iso", "us_slash", "spelled"] + + +def _money(value: Decimal) -> Decimal: + return value.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) + + +def _format_money(value: Decimal) -> str: + return f"${value:,.2f}" + + +def _format_date(value: date, style: str) -> str: + if style == "iso": + return value.isoformat() + if style == "us_slash": + return value.strftime("%m/%d/%Y") + return value.strftime("%B %d, %Y") + + +def random_ground_truth(rng: random.Random) -> GroundTruthInvoice: + """Build a self-consistent CRE ground-truth invoice for one lease period.""" + year = rng.choice([2023, 2024, 2025]) + invoice_date = date(year, rng.randint(1, 12), 1) + due_date = invoice_date + timedelta(days=rng.choice([15, 30])) + + rentable_sf = Decimal(rng.choice([5000, 8200, 12500, 18750, 24000])) + annual_base_psf = Decimal(rng.choice([24, 26, 30, 38, 42])) + base_rent = _money(rentable_sf * annual_base_psf / 12) + cam = _money(rentable_sf * Decimal(rng.choice([3, 4, 5])) / 12) + real_estate_tax = _money(rentable_sf * Decimal(rng.choice([1, 2])) / 12) + + line_items = [ + LineItem(description="Base Rent", charge_type=ChargeType.BASE_RENT, amount=base_rent), + LineItem(description="Common Area Maintenance", charge_type=ChargeType.CAM, amount=cam), + LineItem(description="Real Estate Tax Recovery", charge_type=ChargeType.REAL_ESTATE_TAX, amount=real_estate_tax), + ] + if rng.random() < 0.5: + insurance = _money(rentable_sf * Decimal("0.5") / 12) + line_items.append(LineItem(description="Insurance Recovery", charge_type=ChargeType.INSURANCE, amount=insurance)) + + subtotal = _money(sum((item.amount for item in line_items), Decimal("0"))) + building_sf = rentable_sf * Decimal(rng.choice([8, 10, 12])) + pro_rata = _money(rentable_sf / building_sf * 100) + + return GroundTruthInvoice( + invoice_number=f"INV-{year}-{rng.randint(1000, 9999)}", + po_number=f"PO-{rng.randint(100000, 999999)}", + landlord_name=rng.choice(_LANDLORDS), + property_manager=rng.choice(_MANAGERS), + tenant_name=rng.choice(_TENANTS), + tenant_account=f"TEN-{rng.randint(1000, 9999)}", + property_name=rng.choice(_PROPERTIES), + suite_number=str(rng.choice([100, 250, 400, 1200, 2150])), + lease_number=f"LS-{rng.randint(1000, 9999)}", + rentable_square_feet=rentable_sf, + pro_rata_share=pro_rata, + invoice_date=invoice_date, + due_date=due_date, + currency="USD", + subtotal=subtotal, + tax=Decimal("0.00"), + total=subtotal, + line_items=line_items, + ) + + +def render_invoice(gt: GroundTruthInvoice, rng: random.Random) -> str: + """Render a clean invoice that should reconcile exactly to ``gt``. + + Label phrasing, date format, and spacing are randomized so repeated calls + exercise different layouts the heuristic must cope with. + """ + date_style = rng.choice(_DATE_STYLES) + gap = " " * rng.choice([2, 4, 8]) + + def row(label: str, value: str) -> str: + return f"{label}:{gap}{value}" + + lines = [ + gt.property_name.upper(), + "Monthly Rent Statement", + "", + row(rng.choice(_INVOICE_NO_LABELS), gt.invoice_number), + row("Purchase Order", gt.po_number), + row("Landlord", gt.landlord_name), + row("Property Manager", gt.property_manager), + row("Property", gt.property_name), + row("Invoice Date", _format_date(gt.invoice_date, date_style)), + row("Due Date", _format_date(gt.due_date, date_style)), + row(rng.choice(_TENANT_LABELS), gt.tenant_name), + row("Tenant ID", gt.tenant_account), + row("Suite", gt.suite_number), + row("Lease Number", gt.lease_number), + row("Rentable Square Feet", f"{gt.rentable_square_feet:,.0f}"), + row("Pro Rata Share", f"{gt.pro_rata_share}%"), + "", + ] + for item in gt.line_items: + lines.append(f"{item.description}{gap}{_format_money(item.amount)}") + lines.extend( + [ + "", + f"Subtotal{gap}{_format_money(gt.subtotal)}", + f"Tax{gap}{_format_money(gt.tax)}", + f"{rng.choice(_TOTAL_LABELS)}{gap}{_format_money(gt.total)}", + ] + ) + return "\n".join(lines) + + +def matched_pair(seed: int) -> Tuple[GroundTruthInvoice, str]: + """A ground-truth record and a clean invoice that reconciles to it.""" + rng = random.Random(seed) + gt = random_ground_truth(rng) + return gt, render_invoice(gt, rng) + + +def overcharged_invoice(seed: int, charge_type: ChargeType, delta: Decimal) -> Tuple[GroundTruthInvoice, str]: + """A pair where the rendered invoice overstates one charge by ``delta``. + + The ground truth is returned unchanged; the invoice text inflates the named + charge (and its totals) so reconciliation must report a discrepancy. + """ + rng = random.Random(seed) + gt = random_ground_truth(rng) + inflated = [ + LineItem( + description=item.description, + charge_type=item.charge_type, + amount=(item.amount + delta) if item.charge_type == charge_type else item.amount, + ) + for item in gt.line_items + ] + inflated_total = _money(sum((item.amount for item in inflated), Decimal("0"))) + invoice = GroundTruthInvoice( + **{**gt.model_dump(), "line_items": [item.model_dump() for item in inflated], "subtotal": inflated_total, "total": inflated_total} + ) + return gt, render_invoice(invoice, rng) + + +def lines_to_pdf_bytes(text: str) -> bytes: + """Render invoice text into a minimal one-page PDF using reportlab. + + Raises ``ImportError`` if reportlab is not installed; callers skip the test. + """ + import io + + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + buffer = io.BytesIO() + pdf = canvas.Canvas(buffer, pagesize=letter) + text_object = pdf.beginText(50, 740) + text_object.setFont("Courier", 9) + for line in text.split("\n"): + text_object.textLine(line) + pdf.drawText(text_object) + pdf.showPage() + pdf.save() + return buffer.getvalue() diff --git a/tests/intake/test_cascade.py b/tests/intake/test_cascade.py new file mode 100644 index 0000000..88c1a9c --- /dev/null +++ b/tests/intake/test_cascade.py @@ -0,0 +1,59 @@ +"""The cascade: heuristic first, LLM only for gaps, repair on demand.""" + +import json +from decimal import Decimal + +from docex.intake.extractors.cascade import CascadingExtractor +from docex.intake.extractors.heuristic import HeuristicExtractor +from docex.intake.extractors.llm import LLMExtractor +from docex.intake.models import ExtractionTier + + +def _llm(payload): + return LLMExtractor(lambda prompt: json.dumps(payload)) + + +def test_heuristic_only_leaves_required_gaps(run): + cascade = CascadingExtractor(HeuristicExtractor()) # no LLM + result = run(cascade.extract("Invoice Number: INV-1\nMystery line with no total")) + assert "total" not in result.fields + + +def test_llm_fills_a_required_gap(run): + cascade = CascadingExtractor( + HeuristicExtractor(), + _llm({"fields": {"total": {"value": "$500.00", "label": "Net Payable"}}}), + ) + # "Net Payable" is not a label the heuristic knows, so total is a gap. + result = run(cascade.extract("Invoice Number: INV-1\nNet Payable: $500.00")) + assert result.fields["total"].value == Decimal("500.00") + assert result.fields["total"].tier == ExtractionTier.LLM + assert result.fields["invoice_number"].tier == ExtractionTier.HEURISTIC + + +def test_no_llm_call_when_required_fields_present(run): + def explode(prompt): + raise AssertionError("LLM must not be called when the heuristic resolved all required fields") + + cascade = CascadingExtractor(HeuristicExtractor(), LLMExtractor(explode)) + result = run(cascade.extract("Invoice Number: INV-1\nTotal Amount Due: $500.00")) + assert result.fields["total"].tier == ExtractionTier.HEURISTIC + + +def test_repair_overrides_disputed_field(run): + cascade = CascadingExtractor( + HeuristicExtractor(), + _llm({"fields": {"total": {"value": "$999.00", "label": "Total"}}}), + ) + base = run(cascade.extract("Invoice Number: INV-1\nTotal Amount Due: $100.00")) + assert base.fields["total"].value == Decimal("100.00") + + repaired = run(cascade.repair("...", ("total",), base)) + assert repaired.fields["total"].value == Decimal("999.00") + assert repaired.fields["total"].tier == ExtractionTier.LLM + + +def test_repair_is_noop_without_llm(run): + cascade = CascadingExtractor(HeuristicExtractor()) + base = run(cascade.extract("Invoice Number: INV-1\nTotal Amount Due: $100.00")) + assert run(cascade.repair("...", ("total",), base)) is base diff --git a/tests/intake/test_charges.py b/tests/intake/test_charges.py new file mode 100644 index 0000000..8610c85 --- /dev/null +++ b/tests/intake/test_charges.py @@ -0,0 +1,36 @@ +"""The charge taxonomy must classify CRE line items, most-specific first.""" + +import pytest + +from docex.intake.charges import ChargeType, classify_charge + + +@pytest.mark.parametrize( + "description, expected", + [ + ("Base Rent", ChargeType.BASE_RENT), + ("Minimum Monthly Rent", ChargeType.BASE_RENT), + ("Common Area Maintenance", ChargeType.CAM), + ("CAM Charge", ChargeType.CAM), + ("Real Estate Tax Recovery", ChargeType.REAL_ESTATE_TAX), + ("Property Tax", ChargeType.REAL_ESTATE_TAX), + ("Insurance Recovery", ChargeType.INSURANCE), + ("Parking - 10 stalls", ChargeType.PARKING), + ("Property Management Fee", ChargeType.MANAGEMENT_FEE), + ("Late Charge", ChargeType.LATE_FEE), + ("Percentage Rent", ChargeType.PERCENTAGE_RENT), + ("After-Hours HVAC", ChargeType.HVAC), + ("Holiday Decorations", ChargeType.OTHER), + ], +) +def test_classify_charge(description, expected): + assert classify_charge(description) == expected + + +def test_cam_reconciliation_beats_plain_cam(): + # "CAM reconciliation" must not fall through to the broader CAM bucket. + assert classify_charge("Annual CAM Reconciliation") == ChargeType.CAM_RECONCILIATION + + +def test_blank_description_is_other(): + assert classify_charge("") == ChargeType.OTHER diff --git a/tests/intake/test_embedding_match.py b/tests/intake/test_embedding_match.py new file mode 100644 index 0000000..969e8f8 --- /dev/null +++ b/tests/intake/test_embedding_match.py @@ -0,0 +1,137 @@ +"""Fuzzy ground-truth retrieval by embedding similarity. + +A deterministic bag-of-words embedding stands in for a real model so these tests +run everywhere: each token hashes to a fixed dimension and increments it, so +text overlap drives cosine similarity exactly as a sentence embedding would, +without any provider. +""" + +import hashlib +from decimal import Decimal + +from docex.intake.charges import ChargeType +from docex.intake.embedding_match import EmbeddingGroundTruthMatcher +from docex.intake.ground_truth import GroundTruthInvoice, InMemoryGroundTruthStore +from docex.intake.models import ( + ExtractedField, + ExtractedInvoice, + ExtractionTier, + LineItem, + ReconciliationStatus, +) +from docex.intake.pipeline import InvoiceIntakePipeline + +_DIMS = 128 + + +def bag_of_words(text: str) -> list: + """A stable token-frequency embedding (deterministic across runs).""" + vector = [0.0] * _DIMS + for token in text.lower().split(): + index = int(hashlib.md5(token.encode()).hexdigest(), 16) % _DIMS + vector[index] += 1.0 + return vector + + +def _lease(invoice_number, tenant, property_name, total, charge) -> GroundTruthInvoice: + return GroundTruthInvoice( + invoice_number=invoice_number, + tenant_name=tenant, + property_name=property_name, + total=Decimal(total), + line_items=[LineItem(description=charge, charge_type=ChargeType.BASE_RENT, amount=Decimal(total))], + ) + + +def _three_lease_store(): + store = InMemoryGroundTruthStore() + a = store.add(_lease("INV-A", "Acme Retail", "Harbor Point Tower", "1000", "Base Rent")) + b = store.add(_lease("INV-B", "Borealis Trading", "Summit Plaza", "2000", "Base Rent")) + c = store.add(_lease("INV-C", "Cedar Vine", "Northgate Commons", "3000", "Base Rent")) + return store, a, b, c + + +def _extracted(**values) -> ExtractedInvoice: + invoice = ExtractedInvoice() + line_items = values.pop("line_items", []) + for name, value in values.items(): + invoice.put(ExtractedField(name=name, value=value, confidence=0.9, tier=ExtractionTier.HEURISTIC)) + invoice.line_items = line_items + return invoice + + +def test_candidates_rank_the_right_lease_first(run): + store, _a, b, _c = _three_lease_store() + # Wrong invoice number, but tenant/property/total point unmistakably at B. + extracted = _extracted( + invoice_number="INV-TYPO", + tenant_name="Borealis Trading", + property_name="Summit Plaza", + total=Decimal("2000"), + line_items=[LineItem(description="Base Rent", charge_type=ChargeType.BASE_RENT, amount=Decimal("2000"))], + ) + ranked = run(EmbeddingGroundTruthMatcher(bag_of_words).candidates(extracted, store, top_k=3)) + assert ranked[0].record.id == b.id + assert ranked[0].score >= ranked[1].score >= ranked[2].score + + +def test_find_returns_best_above_threshold(run): + store, _a, b, _c = _three_lease_store() + extracted = _extracted( + tenant_name="Borealis Trading", + property_name="Summit Plaza", + total=Decimal("2000"), + ) + match = run(EmbeddingGroundTruthMatcher(bag_of_words, threshold=0.5).find(extracted, store)) + assert match is not None and match.record.id == b.id + + +def test_find_returns_none_below_threshold(run): + store, _a, b, _c = _three_lease_store() + extracted = _extracted(tenant_name="Borealis Trading", property_name="Summit Plaza", total=Decimal("2000")) + assert run(EmbeddingGroundTruthMatcher(bag_of_words, threshold=0.999).find(extracted, store)) is None + + +def test_empty_invoice_has_no_candidates(run): + store, *_ = _three_lease_store() + assert run(EmbeddingGroundTruthMatcher(bag_of_words).candidates(ExtractedInvoice(), store)) == [] + + +def test_record_embeddings_are_cached(run): + store, *_ = _three_lease_store() + calls = [] + + def counting_embedding(text): + calls.append(text) + return bag_of_words(text) + + matcher = EmbeddingGroundTruthMatcher(counting_embedding, threshold=0.0) + extracted = _extracted(tenant_name="Borealis Trading", total=Decimal("2000")) + run(matcher.find(extracted, store)) + after_first = len(calls) + run(matcher.find(extracted, store)) + # Second run re-embeds only the query, not the three cached records. + assert len(calls) == after_first + 1 + + +def test_pipeline_falls_back_to_embedding_when_identifier_is_wrong(run): + store, _a, b, _c = _three_lease_store() + # Invoice text whose number is a typo absent from the store, so identifier + # matching fails; tenant/property/total still identify lease B. + text = ( + "Invoice Number: INV-TYPO-999\n" + "Bill To: Borealis Trading\n" + "Property: Summit Plaza\n" + "Base Rent $2,000.00\n" + "Total Amount Due $2,000.00\n" + ) + + without_embeddings = run(InvoiceIntakePipeline().process_text(text, store)) + assert without_embeddings.status == ReconciliationStatus.UNRESOLVED + + with_embeddings = run(InvoiceIntakePipeline(embedding_fn=bag_of_words, match_threshold=0.4).process_text(text, store)) + assert with_embeddings.ground_truth is not None + assert with_embeddings.ground_truth.id == b.id + # The right lease was found; the typo'd invoice number is now a flagged discrepancy. + assert with_embeddings.status == ReconciliationStatus.DISCREPANCY + assert "invoice_number" in {c.field for c in with_embeddings.reconciliation.mismatches} diff --git a/tests/intake/test_ground_truth.py b/tests/intake/test_ground_truth.py new file mode 100644 index 0000000..9d2913d --- /dev/null +++ b/tests/intake/test_ground_truth.py @@ -0,0 +1,66 @@ +"""The in-memory store and the matcher that finds the right record.""" + +from decimal import Decimal + +from docex.intake.ground_truth import GroundTruthInvoice, InMemoryGroundTruthStore +from docex.intake.models import ExtractedField, ExtractedInvoice, ExtractionTier +from docex.intake.reconcile import GroundTruthMatcher + + +def _record(**overrides) -> GroundTruthInvoice: + base = dict(invoice_number="INV-1", po_number="PO-9", total=Decimal("100.00")) + base.update(overrides) + return GroundTruthInvoice(**base) + + +def _extracted(**values) -> ExtractedInvoice: + invoice = ExtractedInvoice() + for name, value in values.items(): + invoice.put(ExtractedField(name=name, value=value, confidence=0.9, tier=ExtractionTier.HEURISTIC)) + return invoice + + +def test_add_assigns_id_and_indexes_by_invoice_and_po(): + store = InMemoryGroundTruthStore() + stored = store.add(_record()) + assert stored.id is not None + assert store.get_by_invoice_number("INV-1") is stored + assert store.get_by_po_number("PO-9") is stored + + +def test_matcher_prefers_invoice_number(): + store = InMemoryGroundTruthStore() + store.add(_record(invoice_number="INV-1", po_number="PO-9")) + other = store.add(_record(invoice_number="INV-2", po_number="PO-OTHER")) + + match = GroundTruthMatcher(store).find(_extracted(invoice_number="INV-2", po_number="PO-9")) + assert match.id == other.id # invoice number wins over the conflicting PO + + +def test_matcher_falls_back_to_po_number(): + store = InMemoryGroundTruthStore() + stored = store.add(_record(invoice_number="INV-1", po_number="PO-9")) + + match = GroundTruthMatcher(store).find(_extracted(po_number="PO-9")) + assert match.id == stored.id + + +def test_matcher_returns_none_when_nothing_identifies_the_record(): + store = InMemoryGroundTruthStore() + store.add(_record()) + assert GroundTruthMatcher(store).find(_extracted()) is None + + +def test_charge_total_sums_by_type(): + from docex.intake.charges import ChargeType + from docex.intake.models import LineItem + + record = _record( + line_items=[ + LineItem(charge_type=ChargeType.CAM, amount=Decimal("100")), + LineItem(charge_type=ChargeType.CAM, amount=Decimal("50")), + LineItem(charge_type=ChargeType.BASE_RENT, amount=Decimal("900")), + ] + ) + assert record.charge_total(ChargeType.CAM) == Decimal("150") + assert record.charge_total(ChargeType.PARKING) is None diff --git a/tests/intake/test_heuristic.py b/tests/intake/test_heuristic.py new file mode 100644 index 0000000..2d4f923 --- /dev/null +++ b/tests/intake/test_heuristic.py @@ -0,0 +1,84 @@ +"""Tier 1 extraction across the layouts and traps real invoices contain.""" + +from decimal import Decimal + +from docex.intake.charges import ChargeType +from docex.intake.extractors.base import ExtractionContext +from docex.intake.extractors.heuristic import HeuristicExtractor + + +def extract(text: str, run): + return run(HeuristicExtractor().extract(ExtractionContext(raw_text=text, target_fields=()))) + + +def test_reads_labelled_colon_values(run): + text = "Invoice Number: INV-7\nTotal Amount Due: $1,500.00\nSuite: 400" + result = extract(text, run) + assert result.fields["invoice_number"].value == "INV-7" + assert result.fields["total"].value == Decimal("1500.00") + assert result.fields["suite_number"].value == "400" + + +def test_reads_value_on_following_line(run): + text = "Invoice Number\nINV-42\nAmount Due\n$2,000.00" + result = extract(text, run) + assert result.fields["invoice_number"].value == "INV-42" + assert result.fields["total"].value == Decimal("2000.00") + + +def test_scalar_tax_ignores_real_estate_tax_charge_line(run): + text = ( + "Invoice Number: INV-1\n" + "Real Estate Tax Recovery $1,250.00\n" + "Tax $0.00\n" + "Total Amount Due $5,000.00\n" + ) + result = extract(text, run) + assert result.fields["tax"].value == Decimal("0.00") # not the 1,250 charge line + + +def test_labelled_metric_lines_are_not_phantom_charges(run): + text = ( + "Invoice Number: INV-1\n" + "Rentable Square Feet: 12,500\n" + "Pro Rata Share: 8.25%\n" + "Base Rent $10,000.00\n" + "Total Amount Due $10,000.00\n" + ) + result = extract(text, run) + assert result.fields["rentable_square_feet"].value == Decimal("12500") + assert result.fields["pro_rata_share"].value == Decimal("8.25") + charge_types = {item.charge_type for item in result.line_items} + assert charge_types == {ChargeType.BASE_RENT} # no OTHER phantoms from the metric rows + + +def test_specific_label_beats_generic_one(run): + # "Tax ID" must not be read as the "tax" amount. + text = "Invoice Number: INV-1\nTax ID: 99-1234567\nTax: $50.00\nTotal Amount Due: $100.00" + result = extract(text, run) + assert result.fields["landlord_tax_id"].value == "99-1234567" + assert result.fields["tax"].value == Decimal("50.00") + + +def test_currency_inferred_from_symbol_when_unlabelled(run): + text = "Invoice Number: INV-1\nTotal Amount Due: $1,000.00" + result = extract(text, run) + assert result.fields["currency"].value == "USD" + + +def test_unknown_charge_line_kept_as_other(run): + text = ( + "Invoice Number: INV-1\n" + "Holiday Decoration Fee $500.00\n" + "Total Amount Due $500.00\n" + ) + result = extract(text, run) + other = [item for item in result.line_items if item.charge_type == ChargeType.OTHER] + assert len(other) == 1 + assert other[0].amount == Decimal("500.00") + + +def test_extracted_fields_carry_label_for_learning(run): + text = "Invoice Number: INV-1\nTotal Amount Due: $1,000.00" + result = extract(text, run) + assert result.fields["total"].label == "total amount due" diff --git a/tests/intake/test_learning.py b/tests/intake/test_learning.py new file mode 100644 index 0000000..02cc79c --- /dev/null +++ b/tests/intake/test_learning.py @@ -0,0 +1,76 @@ +"""The self-improving label loop: confirmed phrasings teach the heuristic.""" + +from decimal import Decimal + +from docex.intake.extractors.base import ExtractionContext +from docex.intake.extractors.heuristic import HeuristicExtractor +from docex.intake.fields import FIELDS +from docex.intake.ground_truth import GroundTruthInvoice +from docex.intake.learning import ( + InMemoryLearningStore, + JsonFileLearningStore, + record_confirmed_labels, +) +from docex.intake.models import ( + ExtractedField, + ExtractedInvoice, + ExtractionTier, +) +from docex.intake.reconcile import Reconciler + + +def test_records_normalised_counts(): + store = InMemoryLearningStore() + store.record("total", "Amount Due") + store.record("total", "amount due") # same phrasing, different case + assert store.label_counts("total") == {"amount due": 2} + + +def test_learned_labels_exclude_registry_phrasings(): + store = InMemoryLearningStore() + store.record("total", "total amount due") # already a registry label + store.record("total", "net payable this cycle") # genuinely new + learned = store.learned_labels("total") + assert "net payable this cycle" in learned + assert "total amount due" not in learned + + +def test_only_matched_fields_are_recorded(): + store = InMemoryLearningStore() + extracted = ExtractedInvoice() + extracted.put(ExtractedField(name="total", value=Decimal("100"), confidence=0.95, tier=ExtractionTier.LLM, label="Net Payable This Cycle")) + extracted.put(ExtractedField(name="tax", value=Decimal("9"), confidence=0.95, tier=ExtractionTier.LLM, label="Levy")) + + gt = GroundTruthInvoice(invoice_number="INV-1", total=Decimal("100"), tax=Decimal("0")) + result = Reconciler().reconcile(extracted, gt) + record_confirmed_labels(store, extracted, result) + + # total matched -> learned; tax mismatched -> not learned. + assert store.learned_labels("total") == ("net payable this cycle",) + assert store.learned_labels("tax") == () + + +def test_heuristic_uses_a_learned_label(run): + store = InMemoryLearningStore() + store.record("total", "net payable this cycle") + extractor = HeuristicExtractor(store) + + text = "Invoice Number: INV-1\nNet Payable This Cycle: $4,200.00" + result = run(extractor.extract(ExtractionContext(raw_text=text, target_fields=()))) + assert result.fields["total"].value == Decimal("4200.00") + + +def test_json_file_store_persists(tmp_path): + path = tmp_path / "learned.json" + JsonFileLearningStore(path).record("total", "net payable this cycle") + + reloaded = JsonFileLearningStore(path) + assert reloaded.label_counts("total") == {"net payable this cycle": 1} + + +def test_registry_unchanged_by_learning(): + # Learning must not mutate the static registry it reads from. + before = FIELDS["total"].labels + store = InMemoryLearningStore() + store.record("total", "brand new phrase") + assert FIELDS["total"].labels == before diff --git a/tests/intake/test_llm_extractor.py b/tests/intake/test_llm_extractor.py new file mode 100644 index 0000000..2794f6c --- /dev/null +++ b/tests/intake/test_llm_extractor.py @@ -0,0 +1,125 @@ +"""The LLM tier. + +Two layers of coverage: + +* Deterministic stub tests that run everywhere (CI included) and pin down the + contract: JSON parsing, value normalization, label capture, line items. +* A live test against Claude, skipped unless ``ANTHROPIC_API_KEY`` is set, that + proves the real model produces output the extractor can consume. +""" + +import json +import os +from datetime import date +from decimal import Decimal + +import pytest + +from docex.intake.charges import ChargeType +from docex.intake.extractors.base import ExtractionContext +from docex.intake.extractors.llm import LLMExtractor +from docex.intake.models import ExtractionTier + + +def _stub(payload): + def llm_fn(prompt): + return json.dumps(payload) + + return llm_fn + + +def test_extracts_fields_with_value_and_label(run): + payload = { + "fields": { + "total": {"value": "$4,200.00", "label": "Net Payable This Cycle"}, + "invoice_number": {"value": "INV-9", "label": "Invoice Number"}, + } + } + result = run(LLMExtractor(_stub(payload)).extract(ExtractionContext(raw_text="...", target_fields=("total", "invoice_number")))) + + assert result.fields["total"].value == Decimal("4200.00") + assert result.fields["total"].label == "Net Payable This Cycle" + assert result.fields["total"].tier == ExtractionTier.LLM + assert result.fields["invoice_number"].value == "INV-9" + + +def test_normalizes_typed_values(run): + payload = {"fields": {"invoice_date": {"value": "January 15, 2024", "label": "Date"}}} + result = run(LLMExtractor(_stub(payload)).extract(ExtractionContext(raw_text="...", target_fields=("invoice_date",)))) + assert result.fields["invoice_date"].value == date(2024, 1, 15) + + +def test_parses_line_items(run): + payload = { + "fields": {}, + "line_items": [ + {"description": "Base Rent", "amount": "$10,000.00"}, + {"description": "CAM", "amount": "$1,200.00"}, + ], + } + result = run(LLMExtractor(_stub(payload)).extract(ExtractionContext(raw_text="...", target_fields=("total",), want_line_items=True))) + assert {item.charge_type for item in result.line_items} == {ChargeType.BASE_RENT, ChargeType.CAM} + + +def test_tolerates_prose_around_json(run): + def llm_fn(prompt): + return 'Sure:\n{"fields": {"total": {"value": "100.00", "label": "Total"}}}\nHope that helps!' + + result = run(LLMExtractor(llm_fn).extract(ExtractionContext(raw_text="...", target_fields=("total",)))) + assert result.fields["total"].value == Decimal("100.00") + + +def test_returns_empty_on_unparseable_response(run): + result = run(LLMExtractor(lambda prompt: "no json here").extract(ExtractionContext(raw_text="...", target_fields=("total",)))) + assert result.fields == {} + + +def test_supports_async_llm(run): + async def llm_fn(prompt): + return json.dumps({"fields": {"total": {"value": "50.00", "label": "Total"}}}) + + result = run(LLMExtractor(llm_fn).extract(ExtractionContext(raw_text="...", target_fields=("total",)))) + assert result.fields["total"].value == Decimal("50.00") + + +def test_null_values_are_skipped(run): + payload = {"fields": {"total": {"value": None, "label": "Total"}, "po_number": {"value": "PO-1", "label": "PO"}}} + result = run(LLMExtractor(_stub(payload)).extract(ExtractionContext(raw_text="...", target_fields=("total", "po_number")))) + assert "total" not in result.fields + assert result.fields["po_number"].value == "PO-1" + + +_LIVE_INVOICE = """ +HARBOR POINT TOWER - Monthly Rent Statement +Invoice Number: INV-2024-0042 +Bill To: Acme Retail LLC +Suite: 1200 +Base Rent $20,833.33 +Common Area Maintenance $5,000.00 +Real Estate Tax Recovery $1,250.00 +Total Amount Due $27,083.33 +""" + + +@pytest.mark.skipif(not os.getenv("ANTHROPIC_API_KEY"), reason="ANTHROPIC_API_KEY not set; skipping live LLM test") +def test_live_claude_extraction(run): + """End-to-end against a real Claude model (Haiku, to keep the call cheap).""" + anthropic = pytest.importorskip("anthropic") + client = anthropic.Anthropic() + + def llm_fn(prompt: str) -> str: + message = client.messages.create( + model="claude-haiku-4-5", + max_tokens=1024, + messages=[{"role": "user", "content": prompt}], + ) + return "".join(block.text for block in message.content if block.type == "text") + + result = run( + LLMExtractor(llm_fn).extract( + ExtractionContext(raw_text=_LIVE_INVOICE, target_fields=("invoice_number", "total"), want_line_items=False) + ) + ) + + assert result.fields["invoice_number"].value == "INV-2024-0042" + assert abs(result.fields["total"].value - Decimal("27083.33")) <= Decimal("0.01") diff --git a/tests/intake/test_normalize.py b/tests/intake/test_normalize.py new file mode 100644 index 0000000..8023678 --- /dev/null +++ b/tests/intake/test_normalize.py @@ -0,0 +1,87 @@ +"""Normalization is the foundation; these cases pin down the messy formats.""" + +from datetime import date +from decimal import Decimal + +import pytest + +from docex.intake.fields import FieldType +from docex.intake.normalize import ( + detect_currency, + parse_amount, + parse_date, + parse_percent, + parse_value, +) + + +@pytest.mark.parametrize( + "raw, expected", + [ + ("$1,234.56", Decimal("1234.56")), + ("USD 1,234.56", Decimal("1234.56")), + ("1234.56", Decimal("1234.56")), + ("1,234", Decimal("1234")), + ("$27,083.33", Decimal("27083.33")), + ("(125.00)", Decimal("-125.00")), + ("125.00-", Decimal("-125.00")), + ("-125.00", Decimal("-125.00")), + ("1.234,56", Decimal("1234.56")), # European grouping + ("€2.000,00", Decimal("2000.00")), + ("0.00", Decimal("0.00")), + ], +) +def test_parse_amount_handles_real_world_formats(raw, expected): + assert parse_amount(raw) == expected + + +@pytest.mark.parametrize("raw", ["", " ", "n/a", "—", None]) +def test_parse_amount_returns_none_for_non_amounts(raw): + assert parse_amount(raw) is None + + +@pytest.mark.parametrize( + "raw, expected", + [ + ("2024-01-15", date(2024, 1, 15)), + ("01/15/2024", date(2024, 1, 15)), + ("1/5/2024", date(2024, 1, 5)), + ("January 15, 2024", date(2024, 1, 15)), + ("Jan 15 2024", date(2024, 1, 15)), + ("15 January 2024", date(2024, 1, 15)), + ("25/12/2024", date(2024, 12, 25)), # day-first disambiguated by 25 > 12 + ("2024/03/09", date(2024, 3, 9)), + ], +) +def test_parse_date_handles_common_formats(raw, expected): + assert parse_date(raw) == expected + + +@pytest.mark.parametrize("raw", ["", "not a date", "13/13/2024"]) +def test_parse_date_returns_none_when_unparseable(raw): + assert parse_date(raw) is None + + +@pytest.mark.parametrize( + "raw, expected", + [("8.25%", Decimal("8.25")), ("12.5 percent", Decimal("12.5")), ("3", Decimal("3"))], +) +def test_parse_percent(raw, expected): + assert parse_percent(raw) == expected + + +@pytest.mark.parametrize( + "raw, expected", + [("$1,000", "USD"), ("€500", "EUR"), ("£10", "GBP"), ("USD 5", "USD"), ("plain", None)], +) +def test_detect_currency(raw, expected): + assert detect_currency(raw) == expected + + +def test_parse_value_dispatches_by_field_type(): + assert parse_value(FieldType.AMOUNT, "$1,000.00") == Decimal("1000.00") + assert parse_value(FieldType.DATE, "2024-01-15") == date(2024, 1, 15) + assert parse_value(FieldType.PERCENT, "8.25%") == Decimal("8.25") + assert parse_value(FieldType.NUMBER, "12,500") == Decimal("12500") + assert parse_value(FieldType.CURRENCY, "$100") == "USD" + assert parse_value(FieldType.STRING, " Acme Corp ") == "Acme Corp" diff --git a/tests/intake/test_pdf_roundtrip.py b/tests/intake/test_pdf_roundtrip.py new file mode 100644 index 0000000..9a2d3b7 --- /dev/null +++ b/tests/intake/test_pdf_roundtrip.py @@ -0,0 +1,31 @@ +"""Real PDF round-trip through pdfminer. + +The bulk of the suite works on text so it stays fast and dependency-free. This +module proves the one boundary the text tests cannot: that a genuine PDF, parsed +by pdfminer, still flows through the pipeline. It is skipped when either +optional dependency (reportlab to write, pdfminer to read) is absent. +""" + +import pytest + +from docex.intake.ground_truth import InMemoryGroundTruthStore +from docex.intake.models import ReconciliationStatus +from docex.intake.pdf import HAS_PDFMINER +from docex.intake.pipeline import InvoiceIntakePipeline +from tests.intake.synthetic import lines_to_pdf_bytes, matched_pair + +reportlab = pytest.importorskip("reportlab", reason="reportlab not installed; skipping real-PDF test") +pytestmark = pytest.mark.skipif(not HAS_PDFMINER, reason="pdfminer.six not installed; skipping real-PDF test") + + +def test_pdf_is_extracted_and_reconciled(run): + gt, text = matched_pair(seed=7) + pdf_bytes = lines_to_pdf_bytes(text) + + store = InMemoryGroundTruthStore() + store.add(gt) + + outcome = run(InvoiceIntakePipeline().process_pdf(pdf_bytes, store)) + assert outcome.ground_truth is not None + assert outcome.status in (ReconciliationStatus.MATCHED, ReconciliationStatus.INCOMPLETE) + assert outcome.extracted.value("invoice_number") == gt.invoice_number diff --git a/tests/intake/test_pipeline.py b/tests/intake/test_pipeline.py new file mode 100644 index 0000000..cc12035 --- /dev/null +++ b/tests/intake/test_pipeline.py @@ -0,0 +1,90 @@ +"""End-to-end pipeline behaviour, the contract the whole package exists for.""" + +import json +from decimal import Decimal + +from docex.intake.charges import ChargeType +from docex.intake.ground_truth import GroundTruthInvoice, InMemoryGroundTruthStore +from docex.intake.learning import InMemoryLearningStore +from docex.intake.models import ExtractionTier, MatchStatus, ReconciliationStatus +from docex.intake.pipeline import InvoiceIntakePipeline +from tests.intake.synthetic import matched_pair, overcharged_invoice + + +def _store(*records) -> InMemoryGroundTruthStore: + store = InMemoryGroundTruthStore() + for record in records: + store.add(record) + return store + + +def test_clean_invoice_matches_with_zero_llm_calls(run): + gt, text = matched_pair(seed=1) + store = _store(gt) + + def explode(prompt): + raise AssertionError("a clean invoice must not reach the LLM") + + outcome = run(InvoiceIntakePipeline(llm_fn=explode).process_text(text, store)) + assert outcome.status == ReconciliationStatus.MATCHED + assert outcome.reconciliation.tiers_used == [ExtractionTier.HEURISTIC] + + +def test_overcharge_is_reported_as_discrepancy(run): + gt, text = overcharged_invoice(seed=2, charge_type=ChargeType.CAM, delta=Decimal("250.00")) + outcome = run(InvoiceIntakePipeline().process_text(text, _store(gt))) + + assert outcome.status == ReconciliationStatus.DISCREPANCY + assert "total" in {c.field for c in outcome.reconciliation.mismatches} + cam = next(c for c in outcome.reconciliation.line_item_comparisons if c.charge_type == ChargeType.CAM) + assert cam.status == MatchStatus.MISMATCH + + +def test_unmatched_invoice_is_unresolved(run): + store = _store(GroundTruthInvoice(invoice_number="INV-KNOWN", total=Decimal("100.00"))) + outcome = run(InvoiceIntakePipeline().process_text("Invoice Number: INV-UNKNOWN\nTotal Amount Due: $100.00", store)) + assert outcome.status == ReconciliationStatus.UNRESOLVED + assert outcome.ground_truth is None + + +def test_llm_escalation_repairs_a_missing_field(run): + # Ground truth expects a tenant name the invoice never labels, so the + # heuristic leaves it missing and the pipeline escalates just that field. + gt = GroundTruthInvoice(invoice_number="INV-1", total=Decimal("100.00"), tenant_name="Acme Retail LLC") + text = "Invoice Number: INV-1\nTotal Amount Due: $100.00\nAcme Retail LLC" # name present, unlabelled + + def llm_fn(prompt): + return json.dumps({"fields": {"tenant_name": {"value": "Acme Retail LLC", "label": "Customer"}}}) + + outcome = run(InvoiceIntakePipeline(llm_fn=llm_fn).process_text(text, _store(gt))) + assert outcome.status == ReconciliationStatus.MATCHED + assert outcome.extracted.tier("tenant_name") == ExtractionTier.LLM + + +def test_learning_loop_eliminates_the_second_llm_call(run): + gt = GroundTruthInvoice(invoice_number="INV-1", total=Decimal("100.00")) + store = _store(gt) + text = "Invoice Number: INV-1\nNet Payable This Cycle: $100.00" # novel total label + + calls = [] + + def llm_fn(prompt): + calls.append(prompt) + return json.dumps({"fields": {"total": {"value": "$100.00", "label": "Net Payable This Cycle"}}}) + + pipeline = InvoiceIntakePipeline(llm_fn=llm_fn, learning_store=InMemoryLearningStore()) + + first = run(pipeline.process_text(text, store)) + assert first.status == ReconciliationStatus.MATCHED + assert len(calls) == 1 # LLM needed to read the novel label + + second = run(pipeline.process_text(text, store)) + assert second.status == ReconciliationStatus.MATCHED + assert len(calls) == 1 # heuristic learned the label; no new call + assert second.reconciliation.tiers_used == [ExtractionTier.HEURISTIC] + + +def test_heuristic_only_pipeline_runs_without_an_llm(run): + gt, text = matched_pair(seed=3) + outcome = run(InvoiceIntakePipeline().process_text(text, _store(gt))) + assert outcome.status == ReconciliationStatus.MATCHED diff --git a/tests/intake/test_random.py b/tests/intake/test_random.py new file mode 100644 index 0000000..e5d7909 --- /dev/null +++ b/tests/intake/test_random.py @@ -0,0 +1,36 @@ +"""Property-style coverage: many randomized invoices, one invariant each. + +These run the heuristic-only pipeline over dozens of seeded invoices with +varied layouts, labels, and values. A clean invoice must always reconcile; an +overcharged one must always be caught. If a layout the generator can produce +ever breaks the heuristic, one of these fails with the seed that did it. +""" + +from decimal import Decimal + +import pytest + +from docex.intake.charges import ChargeType +from docex.intake.ground_truth import InMemoryGroundTruthStore +from docex.intake.models import ReconciliationStatus +from docex.intake.pipeline import InvoiceIntakePipeline +from tests.intake.synthetic import matched_pair, overcharged_invoice + + +@pytest.mark.parametrize("seed", range(40)) +def test_clean_invoices_always_reconcile(run, seed): + gt, text = matched_pair(seed) + store = InMemoryGroundTruthStore() + store.add(gt) + outcome = run(InvoiceIntakePipeline().process_text(text, store)) + assert outcome.status == ReconciliationStatus.MATCHED, f"seed {seed} failed: {outcome.reconciliation.mismatches or outcome.reconciliation.missing}" + + +@pytest.mark.parametrize("seed", range(20)) +def test_overcharged_invoices_are_always_caught(run, seed): + charge = [ChargeType.CAM, ChargeType.BASE_RENT, ChargeType.REAL_ESTATE_TAX][seed % 3] + gt, text = overcharged_invoice(seed, charge_type=charge, delta=Decimal("175.00")) + store = InMemoryGroundTruthStore() + store.add(gt) + outcome = run(InvoiceIntakePipeline().process_text(text, store)) + assert outcome.status == ReconciliationStatus.DISCREPANCY, f"seed {seed} not caught" diff --git a/tests/intake/test_realistic_pdf.py b/tests/intake/test_realistic_pdf.py new file mode 100644 index 0000000..72781fc --- /dev/null +++ b/tests/intake/test_realistic_pdf.py @@ -0,0 +1,66 @@ +"""The committed realistic-invoice PDFs: a positive and a negative case. + +These read the same files a human can open in ``example_docs/cre_invoices/``. +They need only pdfminer (to read); the PDFs themselves are committed, so +reportlab is not required at test time. A live-LLM variant runs when +``ANTHROPIC_API_KEY`` is set. +""" + +import os + +import pytest + +from docex.intake.charges import ChargeType +from docex.intake.ground_truth import InMemoryGroundTruthStore +from docex.intake.models import MatchStatus, ReconciliationStatus +from docex.intake.pdf import HAS_PDFMINER +from docex.intake.pipeline import InvoiceIntakePipeline +from tests.intake.realistic_invoice import ( + FIXTURE_PATH, + OVERCHARGED_FIXTURE_PATH, + SAMPLE_GROUND_TRUTH, +) + +pytestmark = pytest.mark.skipif(not HAS_PDFMINER, reason="pdfminer.six not installed; skipping real-PDF tests") + + +def _store() -> InMemoryGroundTruthStore: + store = InMemoryGroundTruthStore() + store.add(SAMPLE_GROUND_TRUTH) + return store + + +def test_positive_invoice_reconciles_clean(run): + outcome = run(InvoiceIntakePipeline().process_pdf(FIXTURE_PATH.read_bytes(), _store())) + assert outcome.status == ReconciliationStatus.MATCHED + assert outcome.extracted.value("invoice_number") == "INV-2024-0042" + + +def test_negative_invoice_is_flagged_as_overcharge(run): + outcome = run(InvoiceIntakePipeline().process_pdf(OVERCHARGED_FIXTURE_PATH.read_bytes(), _store())) + + assert outcome.status == ReconciliationStatus.DISCREPANCY + assert "total" in {c.field for c in outcome.reconciliation.mismatches} + cam = next(c for c in outcome.reconciliation.line_item_comparisons if c.charge_type == ChargeType.CAM) + assert cam.status == MatchStatus.MISMATCH + + +@pytest.mark.skipif(not os.getenv("ANTHROPIC_API_KEY"), reason="ANTHROPIC_API_KEY not set; skipping live LLM test") +def test_live_llm_on_realistic_pdf(run): + """Drive the full pipeline with a real Claude model as the LLM tier.""" + anthropic = pytest.importorskip("anthropic") + client = anthropic.Anthropic() + + def llm_fn(prompt: str) -> str: + message = client.messages.create( + model="claude-haiku-4-5", + max_tokens=1024, + messages=[{"role": "user", "content": prompt}], + ) + return "".join(block.text for block in message.content if block.type == "text") + + positive = run(InvoiceIntakePipeline(llm_fn=llm_fn).process_pdf(FIXTURE_PATH.read_bytes(), _store())) + assert positive.status == ReconciliationStatus.MATCHED + + negative = run(InvoiceIntakePipeline(llm_fn=llm_fn).process_pdf(OVERCHARGED_FIXTURE_PATH.read_bytes(), _store())) + assert negative.status == ReconciliationStatus.DISCREPANCY diff --git a/tests/intake/test_reconcile.py b/tests/intake/test_reconcile.py new file mode 100644 index 0000000..4145e46 --- /dev/null +++ b/tests/intake/test_reconcile.py @@ -0,0 +1,106 @@ +"""Reconciliation: tolerances, statuses, and per-charge comparison.""" + +from datetime import date +from decimal import Decimal + +from docex.intake.charges import ChargeType +from docex.intake.ground_truth import GroundTruthInvoice +from docex.intake.models import ( + ExtractedField, + ExtractedInvoice, + ExtractionTier, + LineItem, + MatchStatus, + ReconciliationStatus, +) +from docex.intake.reconcile import Reconciler, TolerancePolicy + + +def _extracted(fields=None, line_items=None) -> ExtractedInvoice: + invoice = ExtractedInvoice(line_items=line_items or []) + # Ground truth always carries a currency (defaults to USD), so supply it + # unless a test overrides it; otherwise every case reports currency missing. + merged = {"currency": "USD", **(fields or {})} + for name, value in merged.items(): + invoice.put(ExtractedField(name=name, value=value, confidence=0.9, tier=ExtractionTier.HEURISTIC)) + return invoice + + +def _ground_truth(**overrides) -> GroundTruthInvoice: + base = dict(invoice_number="INV-1", total=Decimal("1000.00")) + base.update(overrides) + return GroundTruthInvoice(**base) + + +def test_exact_match_is_matched(): + gt = _ground_truth(total=Decimal("1000.00")) + result = Reconciler().reconcile(_extracted({"invoice_number": "INV-1", "total": Decimal("1000.00")}), gt) + assert result.status == ReconciliationStatus.MATCHED + assert not result.mismatches + + +def test_amount_within_tolerance_matches(): + gt = _ground_truth(total=Decimal("1000.00")) + extracted = _extracted({"invoice_number": "INV-1", "total": Decimal("1000.01")}) + assert Reconciler().reconcile(extracted, gt).status == ReconciliationStatus.MATCHED + + +def test_amount_beyond_tolerance_is_discrepancy(): + gt = _ground_truth(total=Decimal("1000.00")) + extracted = _extracted({"invoice_number": "INV-1", "total": Decimal("1000.50")}) + result = Reconciler().reconcile(extracted, gt) + assert result.status == ReconciliationStatus.DISCREPANCY + assert [c.field for c in result.mismatches] == ["total"] + + +def test_missing_value_is_incomplete_not_discrepancy(): + gt = _ground_truth(total=Decimal("1000.00"), tenant_name="Acme") + extracted = _extracted({"invoice_number": "INV-1", "total": Decimal("1000.00")}) + result = Reconciler().reconcile(extracted, gt) + assert result.status == ReconciliationStatus.INCOMPLETE + assert [c.field for c in result.missing] == ["tenant_name"] + + +def test_date_tolerance_is_configurable(): + gt = _ground_truth(due_date=date(2024, 1, 15)) + extracted = _extracted({"invoice_number": "INV-1", "total": Decimal("1000.00"), "due_date": date(2024, 1, 16)}) + + strict = Reconciler().reconcile(extracted, gt) + assert strict.status == ReconciliationStatus.DISCREPANCY + + lenient = Reconciler(TolerancePolicy(date_days=2)).reconcile(extracted, gt) + assert lenient.status == ReconciliationStatus.MATCHED + + +def test_string_comparison_ignores_case_and_whitespace(): + gt = _ground_truth(tenant_name="Acme Retail LLC") + extracted = _extracted({"invoice_number": "INV-1", "total": Decimal("1000.00"), "tenant_name": "acme retail llc"}) + assert Reconciler().reconcile(extracted, gt).status == ReconciliationStatus.MATCHED + + +def test_only_fields_with_expectations_are_judged(): + gt = _ground_truth(total=Decimal("1000.00")) # no suite expectation + extracted = _extracted({"invoice_number": "INV-1", "total": Decimal("1000.00"), "suite_number": "400"}) + result = Reconciler().reconcile(extracted, gt) + judged = {c.field for c in result.field_comparisons} + assert "suite_number" not in judged + + +def test_line_items_reconcile_by_charge_type(): + gt = _ground_truth( + line_items=[ + LineItem(charge_type=ChargeType.BASE_RENT, amount=Decimal("900")), + LineItem(charge_type=ChargeType.CAM, amount=Decimal("100")), + ] + ) + extracted = _extracted( + {"invoice_number": "INV-1", "total": Decimal("1000.00")}, + line_items=[ + LineItem(charge_type=ChargeType.BASE_RENT, amount=Decimal("900")), + LineItem(charge_type=ChargeType.CAM, amount=Decimal("130")), # overcharged + ], + ) + result = Reconciler().reconcile(extracted, gt) + by_type = {c.charge_type: c.status for c in result.line_item_comparisons} + assert by_type[ChargeType.BASE_RENT] == MatchStatus.MATCH + assert by_type[ChargeType.CAM] == MatchStatus.MISMATCH