The Method Stack

01The corpus, in numbers

Verified counts from the published shards.

On this shard alone: 4,533 sheets, 66,201 rows detected and boxed, 41,590 with a fully resolved date. Every row you see on the overlays pages is a real extraction, scored against the sheet it came from.

02The pipeline, end to end

Each sheet — a phone photo or a scanned PDF — passes through the same stack.

1. Render & normalise. PDF pages rasterised at 150 dpi and scaled to ~2,000 px; photographs auto-rotated and deskewed (2,002 sheets deskewed, 329 re-oriented on this shard) so the grid is level before anything reads it.
2. Three-model vision union. The page is read independently by three local vision models; their answers are reconciled field-by-field (see §3). No single model is trusted alone.
3. Second-pass geometry. The vision models return values but no coordinates. A separate Tesseract + OpenCV pass finds the real table grid and pins each extracted row to its actual pixels (see §4).
4. Compute & reconcile hours. Worked hours are recomputed from time-in / time-out / break with shift-aware logic, and cross-checked against the sheet's own stated total — disagreements are flagged, never silently overwritten.
5. Classify & gate. A classifier confirms the page is actually a timesheet before its hours count — printed email threads and contracts are held out (see §5, learning of 4 Sep).
6. Score & publish. Each row gets a confidence tier and a date-resolution status, then the whole corpus is published to the overlay galleries for row-by-row inspection.

03Why three models, not one

Any single vision model hallucinates on a bad photo — a smudged "3" becomes an "8", a blank cell gets a confident guess. Three models rarely make the same mistake. We read every page with all three and reconcile by first-non-null agreement keyed on the canonical date: a field is filled from the first model that read it, and a model's correct blank read is respected rather than overwritten by another's guess.

Accuracy metric — null-matched. We score only cells where a ground-truth value exists: a model is never penalised for leaving genuinely-blank cells blank. This is the honest denominator — it measures reading, not guessing.

04Where the boxes come from

The overlays are not decoration and they are not invented. Vision models return text with no coordinates, so a second pass — Tesseract word-boxes (TSV) plus OpenCV horizontal/vertical line detection — reconstructs the actual table grid, and each reconciled row is assigned to the grid band it physically occupies. A row we could extract but not locate is drawn as an inferred band and marked as such. Green = row extracted and date resolved; amber = extracted but the date could not be resolved from the sheet.

05What we learned — the last ten days

Every item here changed the pipeline. Most were paid for in wrong answers first.

• 27 Aug · the LoRA verdict
  The union beats a fine-tune.
  A task-specific LoRA lost to the plain three-model union on five separate evaluations. We retired the fine-tune: reconciliation across diverse models generalises better than one specialised head, and needs no retraining as templates change.
• 29 Aug · duplicate rows
  One date, written three ways, was counted three times.
  01/08/26, 01/08/2026 and 1/8/26 were splitting one shift into three rows and inflating totals. A canonical norm_date key now collapses format variants to a single row — the row still displays the sheet's own date text.
• 31 Aug · geometry, honestly
  Stop fabricating evenly-spaced boxes.
  Early overlays synthesised bands from the row count when no coordinates existed — they drifted off the real rows. Replaced entirely with the Tesseract + OpenCV grid pass (§4); a row that can't be located is labelled inferred, not faked.
• 1 Sep · the 24B bottleneck
  The heavyweight model was the whole cost.
  The 24B arbiter runs ~100 s/page and dominated throughput. When GPUs are contended we drop to a two-model 7B union (~3× faster) and restore the full three-model union the moment the fleet frees up — accuracy back at will, speed when needed.
• 2 Sep · one GPU, many threads
  Parallelism can go backwards.
  Running the union's models concurrently on a single GPU thrashed VRAM and dropped throughput to near zero. The rule now: sequential models within a page when they share a GPU; parallel across pages and across boxes when the models sit on separate GPUs.
• 3 Sep · the fleet is not uniform
  The Jetson is ~5× slower at vision than the Macs.
  Vision OCR is GPU-bound and the aarch64 Jetson is far slower per page — so batch extraction is balanced across the fast Mac boxes by residue-sharded work lists, with a supervisor that respawns any shard until its remaining count hits zero.
• 3 Sep · the network drops under load
  Every model call and Drive fetch now retries.
  Under heavy fleet contention, connections drop mid-request and silently lost a model's vote (or a whole file). Every OCR call and Google API call now retries with backoff, so a transient drop no longer costs an answer.
• 4 Sep · not everything is a timesheet
  A printed email thread will confabulate a perfect grid.
  A 125-hour "timesheet" turned out to be a printed Gmail approval thread — the model invented a uniform grid from prose. A classifier now gates every page: emails and contracts are held out of the hour totals and shown separately, so fabricated hours never reach a report.

06Principles we won't give back

• On-fleet only. Every page is read by local Ollama models on the Elitez fleet — no timesheet leaves the estate, and there is no per-page cloud cost.
• Reconcile, never overwrite. Disagreements between models, and between a computed total and a stated total, are surfaced for review — the pipeline flags, it does not paper over.
• Show the working. Every extracted row is drawn on the sheet it came from, at real coordinates, so any number can be traced back to the pixels that produced it.
• Honest denominators. Accuracy is measured only where a true value exists; date-resolution and confidence are reported per row, not as a single flattering headline.