Get startedModels · Quality evidence
Evaluate a model release
Measure one executable model version on a declared benchmark or custom held-out dataset with frozen thresholds, metrics, uncertainty, and sample evidence.
Source reviewed 2026-09-16. Availability depends on your installation, permissions, and compatible runtimes. A supported path is not a guarantee of model quality or production readiness.
Outcome
Make a defensible promote, reject, or investigate decision for one exact model and dataset version.
Start here
What you need to know first
Check the inputs below before starting. If you are new, begin with the first-project guide. A dataset holds media and labels; a model produces results; a deployment makes a selected model version callable. Creating one does not create the others.
Bring these inputs
- — One exact model version
- — An immutable benchmark and metric configuration
Expected output and limits
- — Evaluation run with metrics and examples
- — A release decision or failure-analysis queue
Core workflow
Choose the evidence path
Use a reviewed benchmark or declare a custom validation, test, stress, or out-of-distribution contract.
Validate the contract
Confirm the held-out split, distribution, provenance, labels or explicit class mapping, thresholds, seed, and acceptance criteria before queueing.
Interrogate failures
Inspect AP50-95, AP75, average recall, per-class performance, confidence reliability, size slices, failure types, thresholds, and examples. Image previews start at the saved report confidence; adjusting the preview does not change the saved evaluation or carry into another report.
Record the decision
Attach the measured run to a release gate or feed hard examples back into data work.
What this surface supports
COCO-style AP and average recall with an explicit protocol
Threshold and confidence-reliability analysis
Bootstrap confidence intervals
Wrong-class, localization, duplicate, background, and missed-object diagnostics
Per-class and diagnostic object-size metrics
Measured p50/p95/p99 latency and serial inference rate
Saved reproducible configurations via Save setup for reuse
Expert section
Contracts, signals, and failure modes
Use this section when you are defining acceptance criteria, automating the surface, or reviewing whether its output is safe to promote downstream.
Quality and operating signals
- — Primary metric with confidence interval
- — AP75, average recall, per-class and slice quality
- — Failure decomposition and confidence reliability
- — Measured tail latency, serial throughput, and timing scope
Common failure modes
- — Benchmark contamination
- — Optimizing only a global average
- — Uncalibrated thresholds
- — Comparing runs with changed preprocessing
Expert release checklist
□ Inputs and dependencies are pinned to immutable versions.
□ Acceptance metrics include critical classes and operating slices.
□ Failure, retry, cost, and rollback behavior are understood.
□ The resulting artifact has an owner and a downstream review path.
Engineering safeguards
- — Held-out splits fail closed
- — Incompatible protocols are excluded from rankings
- — Mask metrics require real masks; provider telemetry can be unavailable