Structured Data Extraction Pipeline

Defines the InvoiceExtractionTool — the tool definition passed to tools parameter in messages.create(). Contains both the JSON Schema (API-level syntactic enforcement) and the Pydantic models (application-level semantic enforcement). Two separate validation layers for two different failure modes.

• Required fields (6): vendor_name, invoice_number, invoice_date, total_amount, currency, confidence_score — without any of these the extraction is useless
• Nullable fields: currency_detail, line_items, payment_terms — "I don't know" is a legitimate real-world answer; null > hallucinated value
• "other" enum + currency_detail pattern: prevents silent data loss for unlisted currencies (JPY, CAD, etc.); free-form strings would create normalization chaos downstream
• Pydantic catches: date not in ISO 8601 format, confidence_score outside [0,1], currency=="other" requires currency_detail, line_item arithmetic (quantity × unit_price = line_total ± 0.01), sum(line_items) ≈ total_amount ±1%

# Design rule for nullable vs required
# Make nullable when "I don't know" is a valid answer
"payment_terms": {"type": ["string", "null"]}

# Required because extraction is useless without it
"total_amount": {"type": "number"}  # in "required" array

Builds the extraction prompt with 4 few-shot examples and a normalization rules block. Few-shot outperforms textual instructions for document extraction because the model pattern-matches to demonstrated transformations under uncertainty rather than re-deriving abstract rules.

• NORMALIZATION_RULES block embedded in every prompt: dates→YYYY-MM-DD, strip currency symbols, written-out numbers→numeric, null over fabrication
• Prompt order: task description → normalization rules → examples → target document → closing instruction
• XML tags delimit examples: <example_document>, <example_extraction>, <example_notes>

Core extraction module. Uses tool_choice = {"type": "tool", "name": "extract_invoice_data"} to force Claude to always call exactly that tool. No prose output, no markdown, no choice — guaranteed structured output every call.

• tool_choice: {"type": "tool"} — Claude MUST call this exact tool; stop_reason is always "tool_use"; block.input is already a Python dict
• tool_choice: {"type": "auto"} — Claude may write text instead of calling a tool
• tool_choice: {"type": "any"} — Claude must call one of the tools but can choose which
• MockClaudeClient has 3 modes: normal (happy path), force_error (line items sum ≠ total, tests retry), force_semantic_error (confidence=0.97 for ambiguous doc, proves tool_use ≠ semantic correctness)

# Problems solved by forced tool use:
# 1. Claude might write a text response instead of JSON
# 2. Claude might add explanatory text before/after JSON
# 3. JSON might be wrapped in ```json ... ``` code blocks

tool_choice = {"type": "tool", "name": "extract_invoice_data"}

Two-layer validation architecture and retry loop. Layer 1 (JSON Schema, API-level) catches syntax errors before you receive the response. Layer 2 (Pydantic, application-level) catches semantic errors after. The retry loop provides specific, quantified feedback to maximize correction success.

• Good retry feedback: "sum(line_items)=$145, total_amount=$200, difference=$55. Look for additional line items, fees, or taxes."
• Bad retry feedback: "Your extraction failed. Please try again." — Claude has nothing to work with
• Retry helps: arithmetic errors, date format errors — info exists in document but was missed
• Retry doesn't help: missing invoice number, ambiguous amounts — information doesn't exist; retrying risks hallucination
• max_retries=2 — beyond 2 attempts, rare improvement; adds cost and latency

Routes extractions to AUTO_PROCESS, HUMAN_REVIEW, or REJECT based on confidence score, validation errors/warnings, and invoice amount. Confidence scores are self-assessments, not calibrated probabilities — requires regular auditing.

• 5% random audit of AUTO_PROCESS items catches systematic miscalibration before it compounds
• 97% overall accuracy can hide 40% error rate on a specific document subtype (3% of volume)
• Stratified sampling: audit across document types, not just from the full pool proportionally
• NOT valid routing signals: sentiment/tone of document, response length, secondary classifiers

Batch processing with selective retry. Uses the Anthropic Message Batches API for ~50% cost reduction on large-volume, non-time-critical workloads.

• custom_id pattern: "doc-{document_type}-{index:04d}" — encodes type (targeted failure analysis) + zero-padded index (lexicographic sorting)
• Selective retry: 1,000 docs, 5% failure = 50 failures → re-submit only 50; saves 950 API calls (95% retry cost reduction)
• context_length_exceeded failure: apply 50% truncation (headers at top), resubmit with "{original_id}-retry"
• SLA calculation: deadline 30h from now → submit within 6h. Deadline 18h → batch API cannot guarantee delivery; use synchronous API

Demonstrates the full pipeline on 3 sample documents from sample_docs/, covering the complete range of extraction scenarios.

• invoice_standard.txt (Northstar Consulting) — clean professional invoice, happy path, confidence ~0.95
• invoice_informal.txt (Dan Kowalski) — handwritten-style, written-out numbers, no formal structure, confidence ~0.78
• invoice_ambiguous.txt (Waverly Creative Studio) — draft with missing fields, approximate amounts, "DO NOT PROCESS" warning, routes to REJECT

Pipeline configuration: routing thresholds (0.85 for auto-process, 0.60 for reject), retry limits (max_retries=2), batch settings, model selection. Centralizes all tunable parameters so thresholds can be adjusted as ground truth calibration data accumulates.

• Thresholds are starting points, not universal constants — calibrate against labeled data
• Lower auto-process threshold (e.g., 0.90) for high-stakes contexts (regulatory, healthcare)
• Increase audit rate (above 5%) for document types with historically higher error rates