Get startedModels · When training stops
Recover a failed training attempt
Find the cause and keep the exact data selection instead of starting another run blindly.
Source reviewed 2026-09-16. Availability depends on your installation, permissions, and compatible runtimes. A supported path is not a guarantee of model quality or production readiness.
Outcome
A reviewed replacement setup, not an automatic restart or a fabricated ready model.
Start here
What you need to know first
Check the inputs below before starting. If you are new, begin with the first-project guide. A dataset holds media and labels; a model produces results; a deployment makes a selected model version callable. Creating one does not create the others.
Bring these inputs
- — Failed attempt and its recorded dataset version
- — Named image or provider failure detail
Expected output and limits
- — A reviewed retry setup
- — A new attempt only when explicitly started
Core workflow
Open the failed attempt
Open its model page or Runs. Read the failure and inspect retained work; dismissing a notification does not delete the attempt.
Repair the right input
For invalid class or box coordinates, use Review training annotations and inspect the named image. Changing compute does not fix annotations. Provider failures need a provider/runtime check instead.
Preserve the version
Review dataset and retry retains the recorded source version. A missing saved version needs an explicit selection; it must not silently switch to latest.
Review, then restart
Retry your first model initializes from foundation weights when no ready incumbent exists. Earlier ready models remain unchanged. Check independent provider jobs before launching a replacement.
What this surface supports
Specific recovery actions
Pinned source dataset
First-model retry
Retained failure history
Expert section
Contracts, signals, and failure modes
Use this section when you are defining acceptance criteria, automating the surface, or reviewing whether its output is safe to promote downstream.
Quality and operating signals
- — Failure category
- — Retained artifact state
- — Exact source version
Common failure modes
- — Retrying invalid labels unchanged
- — Selecting a failed model as ready
- — Starting duplicate provider work
Expert release checklist
□ Inputs and dependencies are pinned to immutable versions.
□ Acceptance metrics include critical classes and operating slices.
□ Failure, retry, cost, and rollback behavior are understood.
□ The resulting artifact has an owner and a downstream review path.
Engineering safeguards
- — Retry creates another attempt
- — A failed version is not runnable
- — Worker retries are bounded; the error inventory is not continuous alerting