US-4, US-5 · FR-006, FR-013, FR-014
Skill or luck? The numbers
How often the imagery model's calls are right against labels made without seeing its output, how often it
sees things that are not there, what every door cost, and whether the score ranked homes that later pulled permits or
sold. Everything here is computed from files in the repo: rerun with make eval.
Imagery signals · SC-003
Precision and recall per signal
Bars show the point estimate; the whisker is the 95% interval when the report has one. Each signal is drawn once per label set. The dashed lines on pool against the blind labels are the SC-003 target (precision 0.85, recall 0.80).
Loading…
Hallucination probes
Does it see things that are not there?
Blank, noise and off-parcel images (roads, fields, parking, water) are sent to the same prompt with every question; any "yes" is a hallucination. Target: 5% or less of probes with any yes. Which signals were answered yes is read from the report below.
Loading…
Backtest · SC-005
Score as of 2024-01-01 against what happened next
Outcome, as backtest.json defines it: loading…. Lift above 1 (the dashed line on the lift chart) means the top of the ranking found more of them than chance; the naive baseline ranks by home age only.
Loading…
Calibration · SC-005
Score v1.0 against v1.1 and the home-age baseline
From eval/calibration.json: the same test window scored by both score versions and the naive baseline, each with its 95% interval.
Loading…
Cost ledger · NFR-001, SC-004
What it cost
Every paid call is charged to the ledger before it is made and stops at the spend cap. Read paths (this site, the map, MCP) make no paid calls.
Loading…