V4-M1b — composite scoring (retrieval + OCR legend + softmax)
The eval run that kept the old model in production.
A retrieval-first classifier was built to replace the shipped one. It was measured against a gate written before the work started, it came in 5.4 points short, and it did not ship. These are the numbers that decided it.
Gate
≥ 70.0% top-one on the real-capture holdout
Result
64.6% FAILS
Best pre-registered policy (P4), on the honest holdout.
6 ways to combine the signals. None of them cleared 70%.
Every policy below was registered before the run, so none of them is a post-hoc winner. Percentages are micro top-one accuracy on the same 48-capture holdout.
- Shipped model, in production 40.0%
Measured from modelTelemetry on live traffic, not from this harness
- P0 · retrieval alone 47.9% · macro 42.7%
Centroid retrieval top-one, multi-anchor, full 67 classes
- P3 · shortlist → legend 52.1% · macro 48.4%
Centroid top-five shortlist, then legend unique-match tie-break
- P2 · shortlist → softmax 56.2% · macro 46.4%
Centroid top-five shortlist, then highest softmax probability
- P5 · catalog-only shortlist 58.3% · macro 70.3%
P4's rule, but the shortlist comes from catalog-only retrieval
- P1 · softmax alone 60.4% · macro 56.0%
Argmax over real classes, no retrieval stage at all
- P4 · composite Winner 64.6% · macro 61.7%
Shortlist, then legend unique-match wins, else softmax argmax. Best pre-registered policy, and still short of the gate
- Oracle · top-five shortlist recall 81.2%
The ceiling every shortlist policy is bounded by. This, not fusion, is the binding constraint
The oracle row is the useful one. Top-five shortlist recall caps every shortlist policy at 81.2%, so no amount of additional fusion on top of that shortlist could have reached the gate. The constraint was retrieval, not the tie-break.
The version of this that could ship answers less.
Sweeping the accept threshold trades coverage for precision. At 33.3% coverage the classifier is right 93.8% of the time, which clears the production precision bar. It just declines to answer two thirds of the captures, and every declined capture costs a cloud call.
That is the shape of the real decision, and it is why the gate is written as precision at a coverage floor rather than as accuracy alone.
Show all 16 threshold rows
| Threshold | Coverage | Precision |
|---|---|---|
| 0.619 | 8.3% | 75.0% |
| 0.638 | 14.6% | 85.7% |
| 0.674 | 20.8% | 90.0% |
| 0.679 | 27.1% | 92.3% |
| 0.694 | 33.3% | 93.8% |
| 0.724 | 39.6% | 84.2% |
| 0.732 | 45.8% | 86.4% |
| 0.741 | 52.1% | 84.0% |
| 0.747 | 58.3% | 78.6% |
| 0.752 | 64.6% | 77.4% |
| 0.761 | 70.8% | 73.5% |
| 0.772 | 77.1% | 70.3% |
| 0.788 | 83.3% | 70.0% |
| 0.818 | 89.6% | 67.4% |
| 0.834 | 95.8% | 67.4% |
| 0.991 | 100.0% | 64.6% |
Where the 17 misses came from.
Every miss is assigned to exactly one bucket, in priority order, so the counts sum rather than overlap. More than half never had a chance: the right answer was not in the shortlist at all.
- 9
Truth not in the shortlist
Retrieval never surfaced the right class in its top five, so no downstream stage could recover it. This is the 81.2% ceiling showing up as misses.
- 4
In shortlist, no legend phrase
The right class was available but had no curated legend phrase to match against.
- 3
Legend phrase existed, OCR missed it
Text recognition did not read the phrase off the plate, usually glare, angle, or a cropped banner.
- 1
Legend matched the wrong candidate
The fusion stage actively made this one worse.
Result that contradicted the plan
Adding real captures as anchors did not help top-one accuracy
The plan assumed multi-anchor classes would beat a single catalog render. Held against the same 48-capture holdout, catalog-only scored 50.0% micro top-one and multi-anchor centroid scored 47.9%. Multi-anchor won decisively on top-five recall, 81.2% against 62.5%, which is why the shortlist policies use it. At n = 48 the 2.1-point top-one gap sits inside the noise band, and it is reported that way rather than reshaped to fit the hypothesis.
| Anchor strategy | Top-one | Top-five |
|---|---|---|
| Catalog-only, one anchor per class | 50.0% | 62.5% |
| Multi-anchor centroid | 47.9% | 81.2% |
Per class, with the sample sizes left in.
31 of 48 holdout captures correct. Most classes carry a single capture, so one miss moves that row by a hundred points. The counts are shown next to every percentage for exactly that reason.
The 46 classes absent from this table had no accumulated real captures yet. They are untested here, not passing.
| Class | Correct | Anchors | Accuracy |
|---|---|---|---|
| passenger | 11/16 | 43 | 69% |
| cape-and-islands | 3/5 | 19 | 60% |
| massachusetts-250-years-of-independence | 3/3 | 23 | 100% |
| new-england-patriots | 1/3 | 10 | 33% |
| red-sox-jimmy-fund | 2/3 | 12 | 67% |
| right-whale-roseate-tern | 2/3 | 19 | 67% |
| als-one | 1/1 | 1 | 100% |
| atlantic-white-shark-conservancy | 1/1 | 4 | 100% |
| blackstone-valley | 0/1 | 0 | 0% |
| bronze-star | 0/1 | 4 | 0% |
| bruins-youth-hockey | 1/1 | 13 | 100% |
| conquer-cancer | 1/1 | 4 | 100% |
| electric-vehicle | 1/1 | 22 | 100% |
| fresh-and-local | 0/1 | 0 | 0% |
| low-speed | 1/1 | 1 | 100% |
| martha-s-vineyard | 0/1 | 1 | 0% |
| nantucket-island | 1/1 | 1 | 100% |
| news-photographer-plates | 1/1 | 0 | 100% |
| pan-mass-challenge | 0/1 | 0 | 0% |
| plymouth-400 | 0/1 | 1 | 0% |
| purple-heart | 1/1 | 2 | 100% |
How the run was set up.
The protocol matters more than the score. If any of this were loose, the 64.6% would not be worth reporting.
- Holdout
- 48 real captures across 21 classes, provenance real_holdout
- Anchors
- 67 catalog renders, one per class, plus 222 real-train captures across 32 classes
- Classes
- 67 in classes.json; the label set matched the shipped model exactly
- Leakage
- Leave-one-out re-verified independently: 0 dHash violations (Hamming ≤ 10) between any train anchor and any holdout query
- Embeddings
- Vision feature print revision pinned to 2, 768-dimensional. Recomputed L2 matched Vision's own computeDistance to 1.52e-07
- Tuning
- One knob tuned, the legend fuzzy-match ratio, chosen on the 222 train captures and never on the holdout
Model under test ma-v2.0.1. Regenerated with
uv run eval-composite.py, which rewrites the
results file from current inputs, so the numbers on this page can be reproduced
rather than taken on trust.
This is what I mean by knowing whether it works.
A gate agreed in advance, an honest holdout, one tuned knob, and a published result when the answer was no. I would run a customer's deployment the same way.