PaperNest OCR improvement report — 2026-10-06

Scope and baseline

Ten labeled local images in tests/TestRecipts were run through the existing filename-based benchmark. Python 3.12, Pillow and pytesseract ran on the development host with Tesseract 5.5.0 / Leptonica 1.86.0 isolated under /tmp; these are desktop measurements, not phone measurements. The unchanged pre-edit OCR service was saved and loaded separately for baseline measurement. No receipt data or source images were changed.

Initial pre-change run: 80% dates, 100% amounts, 100% merchants, 80% all three; mean 2764.5 ms, median 3374.0 ms, p95 5054.5 ms. Baseline was repeated with the final instrumentation after adding merchant-decision parsing timing; that comparable run is below. The final sweep overlapped Android toolchain work on the host. Repeated baseline variance is substantial: do not interpret one sweep as a speed improvement.

Metric	Baseline, final instrumentation	After
Receipts	10.0	10.0
Date correct	80%	80%
Amount correct	100%	100%
Merchant correct	100%	100%
All three correct	80%	80%
First-pass success	40%	40%
Fallback frequency	60%	60%
Extraction failure	0%	0%
Average OCR passes	7.1	7.1
Mean latency (ms)	2486.7	3010.3
Median latency (ms)	2851.2	3003.1
p95 latency (ms)	4673.7	8713.5
Mean preprocessing (ms)	524.5	608.8
Mean provider call (ms)	1941.7	2354.4
Mean parsing (ms)	18.6	45.0

One sweep per configuration, sequential sorted images, includes the first recognition call but excludes engine setup. p95 uses nearest rank and therefore equals the slowest of ten samples. First-pass success means a correct result returned after exactly one provider call. Fallback frequency means more than one recognition call. Failure means an exception or no suggestions, distinct from incorrect fields. Android recognition timing in the benchmark includes bitmap decode and task wait; it is not an isolated ML Kit inference timer. Preprocessing and parsing do not account for every orchestration or image-open operation.

Receipt	Baseline ms	After ms	OCR passes, both
2026-06-01_26.60_Triple_O’s.jpeg	4662.4	8713.5	17
2026-06-01_46.86_Uber.png	318.5	350.3	1
2026-06-02_10.76_Jugo_Juice.jpeg	281.4	311.2	1
2026-06-02_15.75_Olly_Fresco.jpeg	3936.6	4267.3	10
2026-06-02_60.00_IL_Chianti.jpeg	2235.7	2411.6	7
2026-06-03_11.39_Tim_Hortons.jpeg	4649.3	5013.1	10
2026-06-03_18.48_Olly_Fresco.jpeg	4673.7	4792.2	13
2026-06-03_25.95_HMSHost.jpeg	3466.7	3594.6	10
2026-06-03_41.97_Uber.png	323.3	318.1	1
2026-06-03_94.80_Uber.png	319.6	331.2	1

Verified changes

The synchronous Android service previously generated three fallback stages before scanning the original. An isolated mock of the original orchestration measured 3 preparation factories called for a confident original; the new orchestration calls 0. Recognition remains one pass. This is eliminated work, not a fabricated phone latency claim. The UI already used staged helpers; this fix applies to direct service callers.
Best core evidence is retained when a later, shorter pass supplies only the missing merchant; that now stops unnecessary further stages. A failed fallback-preparation stage is skipped, keeping the best usable result; all registered temporary files are cleaned.
Explicit valid year-first numeric dates keep their month/day order. Before: 2026-03-06 selected 2026-06-03 because of recency ranking. After: only 2026-03-06. Regression cases cover hyphen, slash and dot separators. This fixes a deterministic parser defect; the ten-image aggregate remains unchanged.
Suggestions now expose missing, review, or strong evidence separately for date, amount and merchant, plus normalized *_confidence_score evidence in [0, 1], weakest-field overall_confidence_score, and review_fields. Strong labels use a 0.8 evidence threshold. These are conservative heuristics, not calibrated confidence probabilities. Ambiguous dates and weak or competing amount evidence require review. Existing constructor arguments and saved receipt schema remain compatible.
Expiry, redemption and best-before date lines are excluded from transaction-date candidates, including a standalone expiry label followed by a date. Transaction-labeled lines remain eligible.
Short receipts with strong field evidence can stop without the old 180-character gate; a mocked original scan verifies one pass and no fallback. Existing successful stopping paths remain available. Requiring the new numeric confidence in addition to the old gate was experimentally rejected: unchanged accuracy, but average passes rose from 7.1 to 15.5, first-pass success fell from 40% to 10%, and mean latency rose to 5867.9 ms. The rejected strategy is preserved in benchmarks/ocr-rejected-strict-confidence.json.
The existing benchmark now reports latency distribution, stage timing, pass counts, failure/fallback/first-pass rates and optional JSON. A failed image is recorded and does not abort the remaining run.

No new image filters, kernel changes, downsampling changes or speculative document scanner integration were added. There is no demonstrated desktop speedup and no improvement in aggregate fixture accuracy. The two remaining misses are Triple O’s (no legible date in selected OCR text) and the 15.75 Olly Fresco receipt (recognized September 2 rather than June 2). Hard-coding expected dates or guessing 9 as 6 is not justified.

Android worker safety

The default Accurate/Smart path prepared images in a worker but then called the blocking recognizer from a Kivy Clock callback. Recognition, decoding, scoring and final parsing now run in a sequential worker; Clock dispatches continuations and updates only. Tests hold recognition blocked and verify the dispatcher returns, scoring stays off the UI thread, late results cannot unlock a newer scan, and failure paths detach/clean up before posting their error continuation.

Physical-phone testing found that PyJNIus 1.7.0 has no exported attach function and uses JNI FindClass for some ML Kit type conversions. Setting a worker context class loader alone did not prevent native aborts. Before dispatch, Kivy now preloads the Task and Text Java wrappers and creates the reusable recognizer; the device agent verified this priming permits real worker recognition. This preparation does not decode an image, run recognition or wait for a task. Optional explicit JVM attachment no longer prevents class-loader setup when attach is unavailable. The separate device report contains the full phone evidence and remaining release checks.

Phone-observed merchant corrections

The first actual Pixel 9 Pro comparison (benchmarks/android-baseline.json / android-after.json) preserved date 9/10 and amount 10/10 but exposed four merchant misses hidden by desktop recognition. Device output read JUGO SUICE, 01ly Fresco's twice, and PPR BEEF for the HMSHost receipt.

After preserving that comparison, normalization was extended only for the observed whole-line JUGO SUICE alias and an alternate digit 1 → lowercase l decoding when the entire result matches an already-known merchant name. This recovers 01ly Fresco's without rewriting arbitrary text. Regression cases preserve unrelated OILY FRESCO'S / Oil Painting Studio and reject item descriptions containing the brand. HMSHost is not guessed from PPR BEEF. A subsequent unmeasured per-pass device capture revealed htfps://w .hmsho:st.com/Ewntact and htfps://4P.hmsho:st.Com/contact. Only this complete known hostname is now repaired before the existing domain extractor; unrelated hosts and lookalike suffixes are excluded. The three actual raw attempts are retained in tests/fixtures/ocr/hmshost_android_passes.json. Regression tests recover HMSHost from the enhanced attempts while preserving the date and total; the original, more damaged text is left uncorrected.

The subsequent desktop run (benchmarks/ocr-alias-after.json) retains 8/10 dates, 10/10 amounts and merchants, with 7.1 average passes. The alias-only phone rebenchmark measured merchant 9/10 (up from 6/10), with date 9/10, amount 10/10 and all-three 8/10. The final physical-phone run with the scoped HMSHost domain correction measured date 9/10, amount 10/10, merchant 10/10 and all-three 9/10. Mean latency was 2602 ms, median 3185 ms and p95 4754 ms, compared with baseline 3830/3653/4838 ms. Average passes were 3.3 versus 3.4; first-pass success was 30% versus 0%. See the complete comparison and raw final measurements. The remaining Android date miss is Triple O’s. These merchant fixes were informed by the same ten-image corpus; independent receipts are still needed to estimate generalization.

Synthetic short-receipt experiment

A fixed 900×500 white image with three clear lines (ACME MARKET, 2025-06-01, TOTAL $12.00) isolates the old minimum-text-length stop condition. This is synthetic desktop evidence, not a real store receipt or phone measurement. The exact input is retained in benchmarks/synthetic-short/2025-06-01_12.00_ACME_MARKET.png and measured with the same benchmark/provider.

Metric	Baseline	After
All three fields correct	Yes	Yes
Recognition passes	17	1
Total latency	1475.6 ms	100.2 ms

This provides measured support for the additional normalized-evidence early exit on short, clear receipts. It does not imply similar gains on the ten real receipts, whose pass counts remain unchanged. Tests explicitly reject short receipts containing only subtotal/tax, an unlabelled item amount, or competing totals. JSON details: benchmarks/ocr-short-baseline.json, benchmarks/ocr-short-after.json. Reproduce the final synthetic scan with python tools/benchmark_ocr_receipts.py docs/benchmarks/synthetic-short --json-output /tmp/ocr-short.json.

Research and decisions

Applied the requested external preprocessing-decisions skill and thresholding-strategy skill selectively: lighting/noise should determine an operation, and filter speed/detail tradeoffs need measured outcomes. No additional filter earned inclusion from this benchmark.

Google’s ML Kit input guidance recommends enough pixels per character (about 16×16), with little recognition benefit beyond 24×24. Smaller images can improve latency if text remains sufficiently resolved. A fixed aggressive resize cannot guarantee that for tiny receipt totals, so existing resolution limits were retained pending a phone accuracy/latency comparison.

ML Kit Document Scanner supplies a scanner flow with JPEG/PDF results, requires API21+ and at least 1.7 GB device RAM, and downloads its models/UI through Google Play services. A Kivy integration would need intent-result lifecycle handling, a first-use download state and fallback capture. There is no measured scanner-versus-current comparison in this work, so the existing capture/import route is retained. Requesting only JPEG would avoid unnecessary PDF generation in a future controlled trial.

Validation and reproduction

Before edits: 34 OCR tests passed. After changes: 57 OCR-related tests passed (51 service/benchmark cases and 6 Android worker cases), including new lazy-stage/cleanup, cross-pass evidence, ISO-date, review-state, benchmark-error-continuation and p95 cases. Providers in regression tests are mocked. Real desktop fixtures returned exit code 2 because 2/10 dates miss the expected values; this is explicitly not an all-green accuracy benchmark.

python -m unittest discover -s tests -p 'test*ocr*.py'
python tools/benchmark_ocr_receipts.py tests/TestRecipts --json-output /tmp/ocr-report.json

Machine-readable retained artifacts: docs/benchmarks/ocr-baseline.json and docs/benchmarks/ocr-after.json; test logs /tmp/papernest-ocr-baseline-tests.log and /tmp/papernest-ocr-after-tests.log. Ruflo memory namespace papernest stores baseline and verified decisions. The Android/device report is maintained separately by the device-testing agent; no on-device verification is implied by desktop results.