How the daily run works
- Synopsis
- Full Details
In briefThe daily run is what happens after a workflow is built: its jobs move through statuses as dependencies are satisfied, run on agents, and report back. Every job status belongs to a family — held, waiting, running, pending-term, terminal, under review or lost in failover — and a finished job passes briefly through pending-term before its dependents are released.
WhereOperations › Processes, where operators watch and act on the run. The statuses reference lists every status.
When
- You are reading a job's status and want to know whether it is working as designed.
- A job seems stuck and you need to tell a wait from a fault.
- A workflow will not complete although its jobs look finished.
How
- Identify the job's status family.
- For a waiting job, find what it waits on — a dependency, its start time, a resource or an agent.
- For a failed job, respond from the Operator response guide: mark it fixed, restart, skip, cancel, mark it finished OK or failed, or close the workflow.
- Pending-term is transitional: completion is being made durable before dependents are released. Cancelled, initialization-error and missed-start-time jobs skip it and settle at once.
- The run survives a restart; every status change is saved as it happens.
- On hold is intentional until released.
- Under review is not terminal. The workflow stays in process until you resolve the job or close the workflow.
- Lost in Failover is not terminal either. Check the job on its machine, then mark it finished OK or failed; until then its dependents wait.
- A job that stays in pending-term for a long time is worth investigating; a momentary one is not.
For the design side, see How workflows work.
The daily run is what happens after a workflow is built: its jobs move through a series of statuses as dependencies are satisfied, work is dispatched to agents, and results come back. Operators watch and act on the run in the Processes view.
Lifecycle
Status families
Every job status falls into one of these families (the statuses reference lists them all):
| Family | Meaning |
|---|---|
| Held | Held; won't run until released. |
| Waiting | Eligible but waiting on something — a dependency, its start time, a resource, or an agent. |
| Running | Actively starting or running. |
| Pending-term | Finished, but completion is still being processed. Transitional, not stuck. |
| Terminal | Done — finished OK, marked finished OK, fixed, failed, marked failed, initialization error, missed start time, cancelled, or skipped. |
| Under review | A failed job an operator has flagged for review. Not terminal: the workflow does not complete while a job is under review. |
| Lost in Failover | A relay could not confirm the job's result after a failover or a restart, or the Universal Agent running it stopped responding. Not terminal, and it releases no dependents until a real result arrives or you mark it Finished OK or Failed — see Lost in Failover. |
Under review holds the workflow open. Flagging a failed job Under Review releases dependents that wait for it to fail (or to finish either way), but the workflow stays in process until you resolve the job — Mark Fixed, Restart, Skip, Cancel, Mark Finished OK or Mark Failed — or Close the workflow.
Two-phase completion
When a job finishes, it first enters a pending-term status (e.g. Finished OK (Pending), which the Processes status badge shows as Finished OK; Processing Events) while completion is made durable, then settles to its final status. Dependent jobs are not released until that settles — so a brief pending-term state is normal, not a hang. (A cancelled job, an initialization error and a missed start time have no pending-term status: they settle at once and their dependents are checked straight away. A cancelled job or a missed start time satisfies no job dependency on it, whatever outcome the dependency waits for.)
Recovery
The run survives a restart. Every status change is saved to the job's record as it happens, so when the service starts again it reloads the active workflow instances and their jobs from those records and carries on from where they were. In-flight jobs are reconciled rather than lost.
- A job waiting is usually working as designed — identify what it's waiting on (dependency, start time, resource, machine) before treating it as a fault.
- Pending-term is transitional. If a job is stuck there for a long time, that's worth investigating; a momentary pending-term is not.
- On hold is intentional until released.
Related topics