Demo Requirements: The Refinery

Pick a format

This demo can run in two modes. Decide on Saturday morning at kickoff, after the group has read The Contract together.

The rest of this document works for either mode. Where a section differs by mode, both variants are shown.

1. Purpose Locked

Prove the refinery works. One sample credit memo goes in; verified structured data comes out. Every field traces back to a page in the original PDF. Every number reconciles against the proofreader's arithmetic checks. Nothing silently wrong.

This is the credibility anchor for the broader supply chain. If we can show end-to-end extraction with verification on one document in a weekend, the business case for building the full platform writes itself.

2. Supply chain position Locked

3. The sample document Locked

A 22-page MBFS Credit Memorandum for a $514,500 multifamily real estate refinance in Philadelphia. It contains the full mixed-content profile the refinery needs to handle: form fields, multi-year financial tables, analyst narrative, a third-party industry infographic, site-visit photographs, and a signed decision form.

4. Scope Locked

5. Success criteria Locked

At the end of Sunday, the demo must produce the following observable outcomes, regardless of how the system is built.

1. Running the system on Sample-Enhanced-Memo.pdf produces a JSON document matching the schema in §7.
2. Every populated field carries a source-page reference back to the PDF.
3. All mandatory checks in §8 pass, and the validation report is visible in the demo surface.
4. At least one recovered error is visible in the output — a case where an initial extraction attempt produced an incorrect value, the incorrect value was detected, and a corrected value was produced.
5. A reviewer can point at any field in the output and be shown the specific page of the source PDF it came from.
6. A five-minute walkthrough maps each demo step back to the strategic brief's four-stage supply chain.

6. Functional outcomes Locked

What must be true of the system when the demo runs, regardless of how the system is built. An implementation that uses a single LLM pass, a specialist-team architecture, a state machine, or any other pattern is acceptable as long as every outcome below is observable in the final output.

7. Output schema Locked

The schema below is a Python/Pydantic expression. Teams using other languages must produce JSON that is shape-compatible — same field names, same structure, same nesting. The schema is deliberately flat and minimal.

from pydantic import BaseModel, Field
from typing import Literal

class SourceRef(BaseModel):
    page: int
    agent: str          # e.g. "form", "table", "narrative", "visual", or custom
    method: str         # e.g. "pdfplumber-lattice", "claude-vision", "regex"
    confidence: float = 1.0

class Field_(BaseModel):
    value: str | float | int | None
    source: SourceRef

class Guarantor(BaseModel):
    name: Field_
    credit_score: Field_
    ownership_pct: Field_
    stated_net_worth: Field_ | None = None

class LoanTerms(BaseModel):
    amount: Field_
    rate: Field_
    rate_type: Field_
    term_years: Field_
    amortization_years: Field_

class Collateral(BaseModel):
    asset_type: Field_
    address: Field_
    estimated_value: Field_
    ltv_pct: Field_
    lien_position: Field_

class FinancialTable(BaseModel):
    name: str
    rows: list[dict]    # column name -> value
    source: SourceRef

class ValidationCheck(BaseModel):
    name: str
    formula: str
    expected: float | str
    observed: float | str
    status: Literal["pass", "fail", "warning"]
    source_pages: list[int]

class CreditMemo(BaseModel):
    document_id: str
    memo_date: Field_
    borrower_name: Field_
    loan: LoanTerms
    collateral: Collateral
    guarantors: list[Guarantor]
    sources: list[dict] = Field(default_factory=list)
    uses: list[dict] = Field(default_factory=list)
    tables: list[FinancialTable] = Field(default_factory=list)
    strengths: list[str] = Field(default_factory=list)
    weaknesses: list[str] = Field(default_factory=list)
    analyst_notes: str | None = None
    validation: list[ValidationCheck]

8. Proofreader checks Locked

Mandatory arithmetic checks against the sample document. Expected values are the ground truth — if an implementation produces something different, either the extraction is wrong or the check is wrong. Either way, the demo does not ship until both agree. Tolerance: ±1 unit for integer dollar totals, ±0.01 for ratios.

9. Deliverables Locked

Every implementation — whether in unified-team or bake-off mode — delivers the following by the end of the weekend.

• Git repository with a one-command bootstrap (make demo, npm run demo, or equivalent)
• README.md with setup, run instructions, and how to regenerate the sample output
• A short statement of which stack, framework, and architecture the implementation used and why
• Generated sample-output.json committed alongside the source PDF, matching the §7 schema
• Generated validation report showing all §8 checks with expected, observed, and status
• A demo surface (CLI, UI, or static HTML) capable of running the five-minute walkthrough
• A five-minute recorded demo video (insurance policy against demo-day failures)
• A one-page retrospective: what worked, what didn't, what to build next

10. Reference architecture Reference

The strategic brief describes a specialist-team pattern — an orchestrator that routes pages, a set of focused extraction agents, a validator (the proofreader), and a reconciler. This is the reference architecture for the refinery. Teams may adopt it as-is, modify it, or use a different decomposition entirely.

Alternative patterns that would also satisfy The Contract include: a single-LLM-pass approach with a separate validation step, a state-machine approach where each page advances through explicit extraction and verification states, or a pipeline-per-content-type approach. Pick what matches the team's comfort and the problem.

11. Reference tech stack Reference

Recommended defaults. Substitute freely — the only hard requirement is that an LLM is used somewhere in the extraction path, because semantic understanding of the document is the reason this approach works at all.

12. Reference timeline Reference

A suggested schedule for a team that has not run this before. Bake-off pairs may run their own schedules in parallel.

13. Reference team roles Reference

For unified-team mode. Bake-off pairs split responsibilities however they like inside each pair.

14. Reference demo script Reference

A suggested five-minute walkthrough. Teams may adapt the structure; the content points are what matters.

1. (30s) The pitch. One sentence: "This is Stage 2 of the supply chain from the strategic brief — the refinery that turns a PDF into trustworthy structured data."
2. (45s) The input. Show the sample PDF. Flip through form pages, spreadsheets, narrative, infographic, photos. "One document, every content type."
3. (1m) The run. Drop the file into the demo surface. Show the routing decisions. Show the JSON output appearing.
4. (90s) The proofreader. Open the validation report. Walk through three checks: Sources = Uses, G1 net worth, global DSCR. Show one check that initially failed and how the system recovered. This is the money moment.
5. (30s) Provenance. Click any field in the output and show the source page it came from.
6. (45s) The bigger picture. Return to the brief's supply chain diagram. "We just proved Stage 2. Stages 3 and 4 are the business case." Close.

Bake-off variant. Each pair runs the same 5-minute script. After all pairs have demoed, the group spends 15 minutes on a comparative retrospective: which stack extracted fastest, which caught the hardest edge case (the red ($2,918) on page 12 is a good benchmark), which recovery path was cleverest, and what each approach would cost to scale to 20,000 memos.

15. Stretch goals Reference

• Cross-page consistency checks beyond §8 (narrative gross rents vs. transactional table, etc.)
• Actually parsing the IBISWorld infographic on page 10 (no tool handles this cleanly — vision is the only real option)
• Side-by-side PDF viewer with extracted fields highlighted on their source pages
• A second retry path using a different LLM or library for the same extraction
• CSV export of the financial tables
• Load a second hand-modified memo to show the approach is robust to template drift
• Cost tracking per extraction (tokens, API calls, wall time)

16. Risks and mitigations

Document types

Phase 2 introduces a loan file containing several document types beyond the credit memo:

Cross-document checks

The core challenge of Phase 2 is verifying consistency across documents:

• PFS net worth vs. credit memo guarantor net worth (C2/C3/C4 values should trace to PFS line items)
• Rent roll unit rents vs. credit memo gross rents (C8/C9 values should match rent roll totals)
• Appraisal value vs. credit memo collateral value and LTV calculation
• Income figures on PFS vs. cash flow tables in credit memo
• Property address and description consistency across all documents

Timeline dimension

Phase 2 adds an origination vs. servicing distinction:

• Origination: Documents as they existed at loan closing — the baseline truth
• Servicing: Updated documents during the life of the loan (annual reviews, renewed PFS, updated rent rolls)
• The system must track which version of a document it is processing and flag inconsistencies between origination and servicing snapshots

Phase 2 tiers

Relationship to Phase 1

Phase 1 tiers (Bronze/Silver/Gold based on checks C1–C9) remain the entry point. A team must achieve at least Phase 1 Silver (6+ checks passing) before attempting Phase 2. The Phase 1 credit memo extraction becomes one component of the larger Phase 2 loan file pipeline.