Detector quality baseline — 2026-10-05

This is the frozen development reference for the detector-retraining plan. It describes the currently served ONNX path, not just the weights. The score below was reproduced locally from the available model and validation files and matched the published evaluation.json field for field. The model and labeled screenshots are not tracked in this Git checkout; their hashes identify the required bytes. No new model was trained or released in this step.

Frozen inputs and inference contract

The model input is a static 1280 × 1280 tensor. The evaluator sends each full image through the server’s letterbox/ONNX decoder, applies the served confidence thresholds, then the client’s same-class NMS at IoU 0.50. Confidence floors are 0.35 by default, 0.25 for berry bush and villager, 0.20 for deer, relic, and sheep, and 0.55 for mill. There is no tracker, camera cache, crop pass, or agent action in this score. The full thresholds and all 60 class results are in evaluation.json.

Reproduced development result

The evaluation set contains 32 real_* screenshots at 1280 × 831 pixels and 1,212 labeled boxes. Its ordered image-and-label manifest SHA-256 is 3a21b93f150f155d0538a45c0fa6da8fed0ec4466703cf21560badfb38d2bfd0. These January 2026 frames share capture sessions with training and therefore are not an independent or release-quality test. Exact zoom settings were not recorded for this set, so no zoom-robustness claim follows from it. The reproduced registry report itself has SHA-256 72e9502b1136566dfc8c24291551c2231b2d3ceda784794e99551ec33380cc7d.

For each image and class, detections are sorted by confidence and greedily matched one-to-one to labels at box IoU ≥ 0.50. Unmatched detections are false positives; unmatched labels are false negatives. Micro precision is TP / (TP + FP), recall is TP / (TP + FN), and F1 is their harmonic mean. False positives from classes with no labeled examples still count in the overall score. FP/frame is total false positives divided by evaluated frames. This is a single operating point, not mAP. A class with no ground-truth boxes has no interpretable recall; support of six or seven boxes is too small to claim reliable behavior.

Latency diagnostic

The new detection_server.benchmark_latency command measured the same full-image path on this Mac: 32 decoded images, three warmup calls, one timed call per image, ONNX Runtime CPU provider. It measured 109.5 ms median and 147.2 ms p95 from decoded image to postprocessed boxes, on macOS 26.6.2 arm64 with Python 3.11.9, ONNX Runtime 1.26.0, and Pillow 12.2.0. These are a single-pass, machine-specific diagnostic, not a performance claim for the Windows VM. JPEG decoding, screen capture, network transfer, tracking, and agent scheduling are excluded. Requested target-refresh p50/p95 cannot be measured from this Mac baseline; record them on the Windows runtime in step 2.

Reproduction

Run from the repository root with the local model and validation data present:

uv run --no-sync python -m detection_server.evaluate \
  --model packages/detection/src/inference/models/aoe2_yolo_v9.onnx \
  --data packages/detection/src/training_data_v9_slim \
  --output tmp/evaluations/current-development.json

uv run --no-sync python -m detection_server.benchmark_latency \
  --model packages/detection/src/inference/models/aoe2_yolo_v9.onnx \
  --data packages/detection/src/training_data_v9_slim \
  --output tmp/evaluations/current-latency.json

Compare the evaluator’s parsed JSON to the registry’s evaluation.json and verify the model/schema/data hashes before interpreting a different result. CPU latency will vary by host and load; compare it only with a candidate run on the same host, provider, input set, and measurement scope.

Predeclared comparison for later candidates

1. Keep this reference untouched. On a future session-held-out real-image test, run both reference and candidate over the same entire frames with the same class schema, label convention, and one-to-one IoU 0.50 scorer. Never tune thresholds on the final-test sessions.
2. For a weights/data experiment, hold preprocessing, postprocessing, confidence thresholds, and full-frame inference fixed so changes can be attributed to the checkpoint. If the inference package changes (for example, adds crops), report that as a separate package comparison with its extra detections, duplicate boxes, coordinate errors, and latency included.
3. The primary quality target is higher held-out micro precision, recall, and F1 than this reference measured on those same new sessions. Also report all 60 per-class results and support, especially the action-critical classes above; FP/frame; metrics by actual zoom and model-input object-size band; and missed/wrong action targets. Do not mask a food-target regression behind a gain from common tree boxes. If support is too small, call that class unqualified, not improved.
4. Measure decoded-image CPU latency as above and whole requested-refresh p50/p95 on Windows, including capture, transport, detection, and target handoff. A candidate must meet the existing ≤2-second p95 requested-refresh target and show no increase in wrong-click targets before deployment.

The independent test, zoom coverage, and Windows requested-refresh baseline are deliberately pending; they require the captures and instrumentation of later steps. An apparent gain on the 32 development images alone does not meet the release gate.