Workflow Container job
- Synopsis
- Full Details
In briefA Workflow Container job runs another workflow as one step of the current one. The nested workflow is built during its parent's build, waits in Wait Container Job until the container job runs, and the container stays running until the nested run finishes, then settles to match. A container whose sub-workflow can't be built does not fail the parent's build: the reason is kept on the job and in its Status History.
WhereIn the Design module, add a Workflow Container job to a workflow and choose the workflow to embed in its Sub-Workflow picker. The picker offers only workflows marked Sub-Workflow — a switch at the head of the Multi-Instance section on the Overview tab of the workflow's Workflow Settings panel. At run time the nested run appears in Processes, named for its place in the hierarchy; a container that came up empty carries a warning icon on the workflow diagram and in the Job View list, and its reason is on the job's Status History tab.
WhenUse a container when:
- one workflow should run another as a single step, and wait for it to finish;
- the same sequence of jobs is shared by several parent workflows;
- you need to find out why a container failed, has nothing to run, or is still running.
To run one workflow after another without embedding it, use a cross-workflow job dependency instead.
How
- Mark the workflow to embed as Sub-Workflow on the Overview tab of its Workflow Settings panel, and deploy it to every environment and date range its parent runs in.
- In the parent, add a Workflow Container job and pick the sub-workflow.
- Commit and deploy the parent, then build it: the nested run is built with it.
- If a container has nothing to run, read the reason on the warning banner, the job's warning icon or its Status History, fix it, and rebuild with overwrite.
The full steps are on Embed another workflow with a Workflow Container.
- Turning Sub-Workflow off is refused while a container job in another workflow embeds the workflow, and nothing is saved.
- The reference is resolved against the workflows deployed for the parent's schedule date, by id first and otherwise by name ignoring case.
- Nesting is limited to 10 levels, and one build creates at most 250 nested runs within 25 seconds.
- A running container does not take one of the environment's job slots.
- When a container ends while its nested run still has jobs running, those jobs finish first; then the nested run is cancelled.
- A job instance cannot change its container's Sub-Workflow: change it on the job definition and rebuild.
- A sub-workflow must be deployed wherever, and for whenever, its parent is — otherwise the container has nothing to run and fails when it starts.
- A job-level event naming a workflow that exists only as a nested run can act on that nested run.
- Deleting or overwriting a parent removes its whole container subtree, and is refused while any nested run in it still has jobs running.
Task walkthrough: Embed another workflow with a Workflow Container. This page is the full configuration and troubleshooting reference.
The Workflow Container embeds another workflow as a step inside the current one. The referenced
workflow is built when the parent is built, and runs when the container job runs; the container stays
running until that nested workflow finishes. It's an internal job type
(internal:workflow-container) — it runs inside the platform, with no agent and no connection.
When you review a workflow, a single job may be an entire child workflow. Follow the container to the nested instance.
| Property | Value |
|---|---|
| Job type | Workflow Container (workflow-container) |
| Runs on | The platform — no agent |
| Connections required | None |
| Category | Internal |
- Configuration Reference
- Sub-Workflows
- How It Works
- Resolving the Reference
- Nested Runs
- Outcomes and Troubleshooting
Configuration Reference
The field is labelled Sub-Workflow on the job, and what it holds is a reference to a workflow rather than a copy of one.
| Parameter | Required | Notes |
|---|---|---|
| Sub-Workflow | Yes | The workflow to embed. The reference holds the workflow's name, which is required, and its id, which is optional and used for a faster, exact lookup. Picking a workflow in the Design module stores both. |
Choosing the sub-workflow
In the Design module, the reference is a searchable picker — Search sub-workflows… — that searches the tenant's workflows as you type, and stores both the name and the id of the one you pick. The picker offers only workflows marked Sub-Workflow (see the Sub-Workflows tab), and the workflow you are currently editing is excluded from the list, so the picker cannot be used to create a trivial self-reference. (Deeper cycles are still caught at run time — see the Resolving the Reference tab.)
The search runs only while the list is open, and the query resets each time you open it, so a field that already holds a workflow opens onto the full list rather than filtering down to its own value. If the search itself fails, the picker says so rather than showing an empty list — an empty result and an unreachable search look identical otherwise.
In the Processes module the reference is shown as an object rather than a picker, with the workflow's id beside its name, and it is read-only. A job instance runs the sub-workflow it was built with: to run a different one, change Sub-Workflow on the job definition and rebuild the workflow. A change to it on a job instance is refused — "The sub-workflow cannot be changed on a job instance. Change it on the job definition and rebuild the workflow."
Sub-Workflows
A workflow that exists to be embedded can be marked as one. On the Overview tab of the workflow's Workflow Settings panel in the Design module, the Multi-Instance section leads with the Sub-Workflow switch, which declares that this workflow runs only as a container's child — "Runs only from a Workflow Container Job in another workflow." Turning it on:
- Hides the workflow from the Schedule Workflow dialog, so it can't be picked for a direct run.
- Refuses a direct build — building it on its own reports "This workflow is marked as a sub-workflow and can only be run by a container job."
- Forces Allow Multi-Instance on, and locks it. One parent can invoke the same sub-workflow more than once on a date, so a sub-workflow has to be able to carry more than one instance. Turning Sub-Workflow back off leaves Allow Multi-Instance on — clear it yourself if you want it off.
- Puts the workflow in the container picker. The container job's picker lists sub-workflows only.
- Removes the AutoBuild Settings section, and clears what it held. A sub-workflow runs only when a container job runs it, so the nightly build skips it — an auto-build setting on one was a promise the platform never kept. The section is absent rather than disabled, and turning the switch on clears the stored auto-build settings in the same change that forces Allow Multi-Instance on, so the stored configuration says what the panel shows. Turning Sub-Workflow back off brings the section back with Auto Build off: re-enabling it is a decision you make with the section in view, the same reasoning that leaves Allow Multi-Instance alone on that path.
The Sub-Workflow switch sits above Allow Multi-Instance because turning it on is what locks the switch below it, and both carry their explanation whether or not the flag is set.
Saving a version warns you which parents it affects
Committing a version of a sub-workflow changes how every workflow embedding it behaves on its next build. The Save to a New Version dialog says so, above the deployment notice: Changes apply to every workflow that uses this sub-workflow, followed by a line such as Used by: Parent A, Parent B, Parent C, +1 more.
- Only container references count. A cross-workflow dependency on this workflow is not an embedding and is not listed.
- Parents are named, de-duplicated and sorted, with the first three shown and the rest as a count.
- It appears only where there is something to warn about: a sub-workflow no container embeds gets no warning, and neither does an ordinary workflow.
- Save is never blocked by the lookup. If the referencing workflows can't be read, the sentence is shown without names rather than being suppressed — a failed lookup is not evidence that nothing depends on this workflow.
- The warning is driven by the flag as stored as well as as edited, so the save that turns Sub-Workflow off is checked too — and is refused while a container embeds the workflow, as described next.
- The first version commits with no dialog, and so carries no warning. See Workflow versions and deployments.
Turning Sub-Workflow off is refused while a container uses it
Turning Sub-Workflow off on a workflow that a container job embeds would leave the parent's container pointing at a workflow that is no longer marked as embeddable-only, so the save is refused and nothing is saved:
- In the workflow editor, the Save to a New Version dialog replaces the usual warning with "Sub-workflow cannot be turned off". It names the workflow, the number of container jobs and the parents (the first three, then a count), and Save is disabled — while the lookup runs, and while it finds container jobs. Remove those container jobs or point them at another sub-workflow, or turn Sub-Workflow back on. If the lookup itself fails, the dialog warns without blocking and the platform's own check on save decides; its refusal is shown in the same dialog until you change the configuration.
- A parent that embeds the workflow blocks the save. The container's reference counts by id, or by name when it carries no id (capitalisation and surrounding spaces are ignored). A workflow's own container, a plain dependency on it, and a parent that has been deleted do not block.
- A parent in a workspace you cannot view still blocks, and is shown only as a count — "2 workflows in a workspace you cannot view" — never by name.
- A deployed parent blocks too, in each environment where this workflow is deployed on Latest — including a parent pinned to an older version that still embeds it. The message names the environments and says to turn Sub-Workflow back on.
- Renaming the workflow in the same save is checked under both names. A save that renames the workflow and turns Sub-Workflow off is refused when a container names it by its old name or its new one.
- Moving a deployment to a version with Sub-Workflow off is refused too. Changing a live deployment of this workflow, or rolling it back, so that it serves a version with Sub-Workflow off is refused while a deployed parent in that environment embeds it: "This sub-workflow is used by deployed workflows in this environment and cannot be changed to a regular workflow. Used by: …"
- Turning Sub-Workflow on, or saving without changing it, is never refused. A workflow that is already off while something embeds it stays editable.
- An import is held to the same rule. An import in overwrite mode that turns Sub-Workflow off is refused with a validation error when, once the whole bundle is applied, container jobs still embed the workflow. A bundle that also removes or retargets those container jobs succeeds. A refused import writes nothing. Its message lists each parent once and names each environment.
There is no dry run for this check.
Sub-workflows in the workflow list
The Design module's workflow picker separates the two kinds rather than mixing them:
- Results are grouped under Workflows and Sub-Workflows, each loading more of its own kind.
- Filter chips for the two kinds narrow the list. Both are on to begin with; un-pressing one leaves the other, and un-pressing the last one is ignored, so the list is never empty. The choice is remembered for the browser session only — it is a browsing convenience, not a setting.
- A sub-workflow row carries a Sub-Workflow badge after its name, so a workflow named without a suffix convention is still identifiable.
- A Recent section lists recently opened workflows, narrowed by the same chips.
Following a container to its sub-workflow
From a Workflow Container job on the Design canvas, Open in New Tab opens the sub-workflow it references in a new browser tab. It is offered from the job card's own menu and from the canvas context menu, on a single selected container job that holds a reference.
Two things about it are worth knowing:
- It survives read-only. Opening changes nothing, so it is the one action still offered when the canvas cannot be edited — where the card's menu then holds just that item.
- It reads the reference you can see. The reference is taken from the job as it stands in the editor, so a target you changed moments ago in the job editor is the one that opens, not the one last saved.
A reference that names a workflow by name alone is resolved the same way the build resolves it — trimmed and case-insensitive. When that cannot be settled, the page says which case it hit rather than opening the wrong workflow: the reference names nothing (Sub-workflow “X” was not found.), too many workflows have similar names to tell them apart, the job no longer references a sub-workflow at all, or the browser blocked the tab.
An embedding counts as a reference to the sub-workflow
A workflow's Cross Reference lists every job that refers to it. That used to mean cross-workflow dependencies only, so a workflow embedded by three container jobs and depended on by none reported nothing referring to it — and could be deleted out from under all three.
A Workflow Container job that embeds the workflow is now one of those references. It appears in the Cross Reference list alongside any cross-workflow predecessors, it counts toward the total, and it blocks the delete the same way a dependency does. A job that both depends on the workflow and embeds it is listed once.
The reference is matched the way the build matches it: by stored id when the container holds one, otherwise by name, trimmed and case-insensitive. So "what refers to this workflow" and "what would run this workflow" are the same answer — which is the point. A container that holds an id but no name is counted too, even though the build would refuse it, because the author did point the job here.
A container job's sub-workflow must be marked Sub-Workflow when you add or change the reference: see Saving a container that points at a workflow you cannot view. A container that already points at an ordinary workflow, and that a save leaves as it is, keeps working and does not stop the workflow from saving, so you can adopt the flag on the workflows you want protected without revisiting every container first.
A deploy transform rule can't rewrite the Sub-Workflow flag — it's a protected field. A rule that tried to add, remove, or rename it is rejected, because the Schedule Workflow dialog reads the stored workflow while the build reads the transformed one, and a rule that changed the flag would make the two disagree about whether a workflow can be run. The sub-workflow reference itself is protected for the same class of reason — see below.
Saving a container that points at a workflow you cannot view
When a workflow is saved, each container job's sub-workflow reference that the save adds or changes is checked; a reference the save leaves as it is, or removes, is not.
| What your save does | Result |
|---|---|
| Points a container job at a workflow that does not exist or has been deleted | Refused: Workflow 'Month End' not found, or you do not have permission to view it. |
| Points a container job at a workflow in a workspace your role cannot view | Refused with the same message, whether or not the workflow exists |
| Points a container job at a hidden workflow that the saved workflow already uses — through another container job or a cross-workflow dependency | Refused with the same message: Workflow 'Month End' not found, or you do not have permission to view it. |
| Points a container job at a workflow you can view that is not marked Sub-Workflow | Refused: "Month End" is not a sub-workflow. Select a sub-workflow, or turn on Sub-Workflow for "Month End". Earlier builds saved it |
| Leaves or removes a container job whose sub-workflow you cannot view | Allowed. |
So you can edit the rest of a workflow you can view even when its containers run workflows you cannot see, and Copy Workflow keeps the container references it copies as they are. The Save to a New Version dialog's check reports the same refusals before you save. Cross-workflow dependencies follow the same rule — see A dependency on a workflow you cannot view.
An embedding constrains where each workflow can be deployed
Because the parent cannot run its container job unless the child is deployed alongside it, the platform holds the two deployments together:
- A parent cannot be deployed to an environment where a container job's sub-workflow is not deployed, or is deployed only for part of the parent's date window.
- A child's deployment cannot be removed, deactivated, or narrowed while a live parent deployment in that environment uses it.
- Renaming the child is refused while a deployed parent references it by name alone; deleting it is refused while any deployed parent references it, including a Pinned parent serving an older version that still embeds it.
- A deploy transform rule cannot retarget the reference (
taskDefinition.parameters.workflow, or theidornamebeneath it). The deployment check reads the reference as stored and a rule is applied after it, so a rule here would point the build at a workflow nobody checked.
The Deploy modal shows this per environment before you act, as a Sub-Workflows section on a parent's cards and a Used By section on a child's. For the coverage rule, the exact messages, and what each card disables, see A sub-workflow must be deployed wherever its parent is.
How It Works
How it works
A container's nested workflow is built during its parent's build, not when the container job dispatches. When you build a workflow that contains a container job, the platform builds that container's sub-workflow as part of the same build, and recurses — so a three-level design is fully materialized, and every nested run is visible, the moment the build finishes.
Each nested run is created waiting on its container job (Wait Container Job). It exists and can be inspected, but nothing in it starts until the container job that owns it actually runs.
When the container job then runs, the platform:
- Adopts the nested run built for it. The build already created it, so the container attaches to that run rather than building a second one. The same adoption covers a nested workflow that hasn't reached a final state after a recurrence or a service restart.
- Builds the nested workflow now, if it doesn't have one. A container built before build-time expansion existed, or one whose child is missing, falls back to building at dispatch — checking the nesting depth (default 10 levels), resolving the reference against the workflows deployed for the run's schedule date, and checking for a circular reference.
- Moves the container job to running.
The container job remains running until the nested workflow completes, then settles to the matching outcome. A resource the container job holds is released as soon as it settles, so the next job waiting for that resource can start. Earlier builds kept it until a periodic check freed it, so that job waited in Waiting: Resource Dependency for up to a minute, and longer when the finished instance had been rebuilt with overwrite.
A running container does not occupy a job slot. The environment has a cap on how many jobs may be running at once (100 by default), and a container sits in running for as long as its nested workflow does — which can be hours — while holding no agent capacity of its own. Counting it against the cap deadlocked the environment: once enough containers were running to fill it, no slot was left for their own nested jobs, so the children waited to start indefinitely, the containers never finished, and nothing else in the environment could dispatch either. Running containers are now excluded from that count. A container waiting to start still takes an ordinary place in the queue, and a null job still counts — it is only running for a moment.
What the build reports
A build of a workflow with container jobs reports three things alongside its job counts: how many containers were expanded in this workflow, how many nested instances were created beneath it (counted through the whole hierarchy), and any container failures.
A container that can't be built doesn't fail the build
If a container's sub-workflow can't be built — a missing or unreadable reference, nothing deployed for the schedule date, an ambiguous reference, a cycle, too deep, or nothing in it qualifying for the date (no frequency fired, or every job in it is Disabled) — the parent build still succeeds. The parent's own jobs are complete and runnable, so destroying them because one child couldn't be built would be the wrong trade. Instead:
- The reason is written to the container job's termination description, so it's visible on the job.
- The container job is listed in the build result's container failures, naming the job, the workflow instance that owns it, the workflow it referenced, and the reason.
- The container job's Status History gets a Container Expansion Failed entry, whose details hold the full reason, the schedule date, the owning workflow and the workflow it referenced. The details show in a tooltip that wraps a long reason and scrolls when it is tall.
The Status History entry is the one that lasts. The build result is shown once, and only to a
build that returned one — a build that timed out before answering, or a container built when it
started, has none to show. The termination description is cleared once the container dispatches, or
replaced by the dispatch error if it fails. It is also cleared when a container that never ran is
cancelled, skipped or marked failed without a reason, so the build's reason is never mistaken for
the job's outcome: an Exit Description event does not
compare it. A container at Missed Start Time keeps the reason on screen, so you can still see
why it never ran, but its Exit Description events compare it as having no description. A
Completion Expression event, or an event command,
that reads [[JI.$JOB TERMINATION]] gets it as empty too.
The history entry stays, so the reason can still be read after either. Earlier builds kept only the
first two.
A grandchild's failure is reported the same way, in the same list, because nothing else surfaces a
nested build's result to you. The list is tree-wide, which is why each entry names its own owning
instance: a container job name is unique only within its own workflow, so the job name alone
wouldn't say which schedule to go and look at. The owner is named with the same composed
Workflow_Instance$nnnn convention the Processes grids use, so two fan-out instances of one
workflow that failed the same container are told apart, and so is a repeat build of one instance.
A container that failed to expand is not de-duplicated across fan-out instances: each instance expands on its own, so each one that came up empty is its own entry.
Where you're told a container has nothing to run
You don't have to open the job to find out. Three surfaces report it:
A warning on the Processes page, after the build. Scheduling a workflow whose build left any container empty raises a warning banner above the list on the Processes page — not inside the Schedule Workflow dialog, which closes on a build that succeeded. It reads "1 container job has no sub-workflow to run:" and lists, per container, the owning schedule instance → the container job → the sub-workflow it referenced, with the build's own reason beneath each one. When one submission covered a range of dates, each entry is prefixed with the schedule date it belongs to. It closes with what the consequence actually is:
1 container job has no sub-workflow to run:
• HB-Demo-Parent → New Job → HB-Demo-Child
No deployment found for workflow 'HB-Demo-Child' on 2026-09-11
Each one fails when it starts unless its sub-workflow is deployed for that date first.
The referenced workflow is left out of the path when the reference itself was unreadable — which is exactly the case where there's no name to report.
The banner has no timer: it stays until you dismiss it with its close control, because it's the only statement of which containers came up empty. The ordinary scheduling confirmation still appears alongside it when it has something to add — instances that failed or were skipped, and instances an overwrite deleted, none of which the banner covers. When it would add nothing, the banner is shown on its own.
A marker on the container job, for as long as the warning is still true. The container job carries a warning icon — beside its status badge on the workflow diagram, and beside its name in the Job View list. Hovering it shows the build's reason. It's on the job itself, so it covers a nested level too: each nested instance's own diagram and list mark their own containers, which the one-shot build response can't do.
The marker is shown only when all of these hold, so it can't contradict what the job's badge says:
- the job is a Workflow Container;
- it carries a termination description — the build's reason or, once the container has failed, the reason it failed;
- its status is one where that reason is still true of the job — any waiting status, On Hold, Failed, Failed (Pending) or Initialization Error.
The job's own termination description, on its detail panel, which is where the reason has always been.
The markers earn their keep on a future-dated build, where the container sits in Wait Start Time on Processes and nothing else says anything. On a same-day build with no start-time constraint the container dispatches within about a second and fails, and the dispatch error replaces the build's reason — the job then shows the red Failed badge, and the marker shows the dispatch error.
Resolving a container clears its build warning
Cancel, Mark Finished OK and Skip clear the build's reason from a container job they resolve, so the marker goes with the action rather than standing on a job you've closed out.
| Action | Clears the build's reason |
|---|---|
| Cancel, Mark Finished OK | When the container was still waiting — any waiting status, or On Hold. |
| Skip | Always, when the skip defers to the engine (the job goes to Job to Be Skipped). On a terminal skip, the waiting rule above applies. |
| Restart | Always, as it always has. |
| Mark Failed | Never — it writes your own reason into the same field, and the job leaves the statuses the marker reads. |
The reason is deliberately kept when the container is resolved from Failed, Initialization Error or Marked Failed: the text there is the engine's account of a real dispatch failure, or your own, and clearing it would erase it.
A container left at Job to Be Skipped by its frequency — rather than by an operator — keeps its marker, because it really does have no sub-schedule to run.
Two limits bound one build
Depth has always been capped. Build-time expansion adds two more bounds, because depth alone doesn't limit how much work one build does:
| Limit | Default | What happens when it's reached |
|---|---|---|
| Sub-workflow instances created across the whole hierarchy | 250 | The next container is recorded as a failure naming the limit. Everything already built stands. |
| Wall-clock budget for expansion | 25 seconds | Same — the next container is recorded as a failure naming the deadline, and the build returns. |
| Nesting depth | 10 levels | The container fails with the depth error. |
Both new limits are deliberately reported, not silent: the build succeeds, and the containers that weren't expanded say so. Containers that failed to build anything don't consume the instance budget, so one bad reference can't push a later valid container over a limit nothing really reached.
Resolving the Reference
How the reference is resolved
The reference is resolved against the deployments effective for the schedule date the parent is running for — not against the list of workflows that exist. A workflow that exists but isn't deployed for that date is just as unbuildable as one that was never created, and the error says so by naming the date.
Within that set:
- If the reference carries the workflow's id, the id decides. The stored name is not consulted, so a workflow that has been renamed since the container was authored still resolves.
- If it carries only a name, the name is matched case-insensitively and ignoring surrounding whitespace. A case-only rename of the target therefore doesn't break a container that still spells the name the old way.
- Renaming another workflow onto a name a container uses by name is refused, so a container can
never quietly start running a different workflow: Cannot rename to "
<name>": N container jobs in M workflows already refer to a workflow by that name. Choose another name, or retarget those container jobs first. Container jobs whose reference carries an id are not counted, and nor is a rename that only changes capitalisation or surrounding spaces. Containers in workflows you cannot view are counted too.
The reference is read from the container job instance's own configuration, so a container that was fanned out by a multi-instance group or an agent pool — or whose job name was edited after the build — resolves its target the same way a plain one does.
The check that runs before the build and the build itself resolve the reference by the same rule, so a reference the pre-check accepts is one the build can find.
The resolution messages name the workflow, in quotes, and never its identifier — no deployment for the date, an ambiguous reference, a sub-workflow that does not qualify for the date — wherever they appear: in the build's container failures, in the container job's termination description, in the markers' tooltips, and in the error a container fails with at dispatch. A reference that carries an id used to be reported by that id, which told you nothing you could search for. A failure that isn't about resolving the reference — an unexpected error while building the nested workflow, for instance — reports a generic message such as "Failed to build nested workflow. See logs for details."
Circular references and depth
The cycle check walks the container's full chain of ancestors, comparing by id when the reference
has one and by normalized name otherwise. That catches indirect cycles — A embeds B, which
embeds A — not only a workflow embedding itself, and it isn't defeated by a rename or a difference
in capitalization. The depth limit is a separate backstop for genuinely deep nesting.
When the platform can't answer yet
A container distinguishes "this reference is wrong" from "the platform couldn't tell me":
- A configuration problem — no such deployment for the date, a cycle, too deep — fails the job.
- A transient platform condition — a service that didn't answer, a database failover, or another builder already partway through building the same nested run — defers the job. The container keeps the status it had, releases any resources it had acquired so other jobs aren't blocked behind it, and tries again on the next scheduling pass.
A deferring container keeps its planned start time, so if the job has a late-to-start time, it goes to Late to Start once that passes, like any other job that hasn't started on time.
Nested Runs
Deleting or rebuilding a parent
Deleting a workflow instance, or rebuilding it with overwrite, cascades to its whole container subtree — every nested run beneath it, at every level, goes with it. Every nested run in that subtree faces the same checks as the instance itself, before anything is removed: if any of them is in a status that can't be deleted or overwritten, or still has jobs running or finishing, the whole delete or overwrite is refused and nothing is removed. The refusal names the nested sub-schedule — for a delete, "Cannot delete workflow: … job(s) in nested sub-schedule '…' are still running or finishing."
When an overwrite goes through, the old nested runs stop being scheduled at the same moment as the parent they belonged to, at every level, and only the replacement's nested runs go on. Earlier builds stopped only the old parent, so its old nested runs could go on starting jobs after the overwrite.
A nested run can't be deleted on its own: "Cannot delete a nested sub-schedule directly; it is managed by its parent workflow." Delete or overwrite the parent instead.
A nested run whose container can never run
A container job that goes terminal without ever dispatching — cancelled, or failed on a dependency — would leave its nested run waiting forever. Such a run is now reaped: the platform notices the container can no longer drive it and cancels it, both as it happens and in a pass at startup that collects any that predate this. A nested run with jobs still running is left alone.
A nested run whose container has ended
When a container job reaches a final status while its nested run is still active, the nested run can no longer be driven, and it is cancelled. If any of its jobs are still running, or still finishing, it is not cancelled out from under them: it moves back to Wait Container Job, where those jobs finish but nothing new starts, and it is cancelled once they are done.
Earlier builds cancelled the nested run at once, with its jobs still running on their agents.
The same holds at every level below it. A nested run inside that nested run — a grandchild — starts no new jobs either. It is cancelled once its own running jobs finish, which ends its container job as cancelled, and the run above it is then cancelled in turn. Earlier builds let a grandchild go on starting jobs under a run that was waiting to be cancelled, so neither ever ended.
After a restart of the platform, a nested run whose container job is not loaded is checked against the stored container job. When that shows the container has ended, the nested run is handled as above: it starts nothing new and is cancelled once its running jobs finish, without waiting for another restart. Earlier builds left such a run waiting until the platform was restarted again.
The same check runs while the platform is up. About once a minute, a nested run whose container job has not been loaded for five minutes is checked against the stored container job in the same way. So a nested run stranded while the platform is running — its container job gone, as after a delete that stopped part-way, or ended — is cancelled without waiting for a restart, once any of its jobs still running have finished.
When the stored container job does not show that it has ended, the nested run is not cancelled. One already waiting in Wait Container Job stays there and starts nothing, and so does one whose container is waiting or held.
A container job cannot be killed — see Which actions a job status accepts. To stop a nested run, act on its jobs.
How a nested run is named
A nested run is named for its place in the hierarchy, so two runs of the same sub-workflow under different containers are told apart:
- Its full name is
<parent's full name>_<container job name>[<sub-workflow name>]. This is recursive, so a three-level hierarchy readsTop_RunMiddle[Middle]_RunLeaf[Leaf]. - Its hierarchy path is the top-level workflow's name followed by the chain of container
jobs that leads to it, separated by
\—Top\RunMiddle\RunLeaf. Below the top level it names the jobs rather than the workflows, so it stays short and reads as a location.
Both are derived from the sub-workflow's resolved name, so a container pointing at a renamed workflow is named for what the workflow is called now, not for the spelling stored on the reference.
A full name is fitted to 255 characters by shortening the prefix, not the sub-workflow name — the sub-workflow name is what identifies which child this is, and the prefix is context. (Only a sub-workflow name so long that it nearly fills the 255 characters on its own leaves the whole name to be shortened.) A shortened prefix ends in a checksum of what was dropped, so two long siblings that agree on their opening characters don't collapse into one name.
Neither value replaces the workflow's own name: a nested run still records the sub-workflow's canonical name too, which is what cross-workflow dependencies and child reuse match on. Top-level runs have no full name or hierarchy path — their name is just the workflow's name.
Recurring containers
When a container job is set to recur, the nested workflow is recycled in place rather than rebuilt — the same nested instance runs again for each recurrence. Two consequences are worth knowing:
- The nested workflow fires its completion events on every recurrence, not just the first.
- Each recycle is a workflow status transition on the nested run, from completed back to waiting on its container, so a notification trigger that matches on that transition sees each one.
A recycle is not cancelled part-way through. While a recurring container resets its nested run, and any nested runs below it, for the next recurrence, the platform's check for nested runs whose container has ended leaves them alone. Earlier builds could cancel the nested run, or a grandchild, that had just been reset, because the container still read as finished while the reset was under way.
That holds across a platform restart too. A reset the platform was in the middle of when it stopped is finished when it starts again, so the nested run goes on to its next recurrence. Earlier builds kept the reset's protection only in memory, so a restart during a reset could cancel the nested run, and the container did not recur.
After a service restart
If the platform restarts while a container is running, the container is reconnected to its nested run on reload. A nested workflow that reached a final state during the outage is reconciled and handed to the same completion path a live finish takes, so the container settles rather than staying running against a run that already finished.
A workflow the platform loads later is linked up too. When a workflow instance is loaded after the platform has started — for example, one whose first load failed — its container jobs are connected to the nested runs already loaded beneath them, and the instance itself to the container job above it. So the container's failure retry and its next recurrence reach the nested run. Earlier builds made those links only in the load at startup, so a nested run loaded before its container's workflow was never reached, and stayed parked.
A container whose nested run finished while it was not loaded completes. If a running container's workflow is loaded after its nested run has already reached a final state, the platform hands that nested run to the same completion path a live finish takes, so the container settles instead of staying running. Only the newest build of the nested run can complete the container, never an older one. And when a container has run again, it links to its newest nested run: a nested run still under way is preferred over a finished one, and among finished runs the newest build wins, whichever loaded first. Earlier builds could leave the container linked to a nested run that was no longer loaded, so a nested run that loaded later was never linked and the container stayed running.
How a nested run relates to the rest of the schedule
A container's nested workflow is a private sub-schedule, not another run on the daily schedule, and the platform keeps the two apart:
- Schedule-level events skip it.
$SCHEDULE:HOLD,$SCHEDULE:RELEASE,$SCHEDULE:CANCELand$JOB:ADDresolve to the standalone run you named, never to a container's nested run. If a workflow name and date exist only as a container's nested run, the event reports that the target wasn't found — which is the safer answer than quietly acting on a container's child and breaking its roll-up. - Cross-workflow dependencies prefer the standalone run. If a workflow is both on the daily schedule and embedded by a container on the same date, a dependency in another workflow resolves to the standalone instance.
- A nested run's own dependencies resolve as a top-level run's do. A job in the nested workflow with a same-day dependency on another workflow, naming no instance, waits on that workflow's first instance — the same one it would wait on if its workflow ran on its own. See When the workflow builds as several schedule instances.
A job-level event is not filtered the same way: one naming a workflow that exists only as a container's nested run can resolve to a job in that nested run.
Outcomes and Troubleshooting
Outcomes
| Result | Status | When |
|---|---|---|
| Nested run adopted | Running | The nested workflow was built with the parent, and the container attaches to it. This is the normal case. |
| Nested workflow building/running | Running | Reference resolved and the nested workflow built at dispatch — a container that predates build-time expansion, or one repairing a missing child. |
| Existing nested run adopted | Running | The container already had a live nested run — after a recurrence or a restart. |
| Never dispatched, nested run reaped | Terminal (as set) | The container went terminal without dispatching, so its nested run was cancelled rather than left waiting. |
| Container ended, nested run cancelled | Terminal (as set) | The container reached a final status while its nested run was active. The nested run waits in Wait Container Job until its running jobs finish, then is cancelled. |
| Deferred, retried next pass | Unchanged | A service didn't answer, or another builder holds the same nested run. |
| Failed — missing reference | Failed | The job carries no workflow name. |
| Failed — no deployment for the date | Failed | Nothing deployed for the schedule date matches the reference. The error names the date. |
| Failed — ambiguous reference | Failed | More than one deployment matches the reference. |
| Failed — circular reference | Failed | The referenced workflow is already an ancestor (the error names the cycle path). |
| Failed — nesting too deep | Failed | The nesting depth limit (default 10) was exceeded. |
Troubleshooting
| Symptom | Likely cause | Resolution |
|---|---|---|
| "No deployment found for workflow … on …" (the message names the schedule date) | The target isn't deployed for that schedule date, or the name/id is wrong | Deploy the target workflow so it's effective for that date, or correct the reference (Builder). |
| "Ambiguous workflow reference: … deployments match" | More than one deployment answers the reference | Point the container at a specific workflow with the picker, so it stores the id (Builder). |
| "Circular workflow reference detected" | The embedded workflow references one of its own ancestors | Remove the cycle; an embedded workflow can't include a parent (Builder). |
| "Maximum nested workflow depth … exceeded" | Containers nested more than the limit (default 10) | Flatten the design so nesting stays within the limit (Builder). |
| Container job stays running for a long time | The nested workflow hasn't finished | Open the nested workflow instance and troubleshoot it there (Operator). |
| Container sits in its pre-run status across several passes, and may go Late to Start | It's deferring on a transient condition rather than failing | Check whether the platform's services are healthy; the container retries on its own once they are (Operator). |
| "Workflow-container job missing workflow reference" | The job has no workflow selected | Set the workflow reference on the job (Builder). |
| The build succeeded, but a warning banner says a container job has no sub-workflow to run | The container couldn't be expanded at build. The banner names the owning instance, the job and the reason; the same reason is on the job and behind its warning icon | Fix what the reason names — most often, deploy the referenced workflow for that date — then rebuild with overwrite (Builder/Operator). |
| A container job carries a warning icon beside its name or status badge | Its sub-schedule couldn't be built, and the job is still waiting, held or failed | Hover the icon for the reason. Resolve the reference and rebuild, or Cancel, Skip or Mark Finished OK the container, which clears the warning (Operator). |
| "Build-time expansion limit of … sub-workflow instances reached for this build" | The hierarchy is larger than one build's instance cap | The container is left waiting, and builds its sub-workflow when it starts. To keep every nested run visible after the build, split the design so one build creates fewer (Builder). |
| "Build-time expansion deadline of …ms reached for this build" | Expansion ran out of its time budget before reaching this container | The container is left waiting, and builds its sub-workflow when it starts. To avoid it, split the design so one build does less (Builder). |
| "This workflow is marked as a sub-workflow and can only be run by a container job" | A direct build was attempted on a workflow marked Sub-Workflow | Build the parent workflow that contains the container, or clear the flag if the workflow is meant to run on its own and no container embeds it (Builder). |
| "Sub-workflow cannot be turned off" in the save dialog, or Cannot turn off Sub-Workflow: "…" is used by … container jobs in … workflows from the API | A container job in another workflow still embeds this workflow, or a deployed parent does where this workflow runs on Latest | Remove or retarget those container jobs, or turn Sub-Workflow back on. Nothing was saved (Builder). |
| A workflow you want to embed isn't in the container's picker | The picker lists sub-workflows only | Mark the target workflow Sub-Workflow on the Overview tab of its Workflow Settings panel (Builder). |
| "Job task definition not found" / "… carries no parameters" | The job instance's stored configuration is missing or unusable | Set the reference on the job definition and rebuild the workflow (Builder). |
| "The sub-workflow cannot be changed on a job instance. Change it on the job definition and rebuild the workflow." | A job instance edit changed the container's Sub-Workflow | Change Sub-Workflow on the job definition and rebuild the workflow. Nothing was saved (Builder). |
| A nested run went back to Wait Container Job after its container job ended, and nothing new in it starts | Expected — its jobs that were already running are allowed to finish | It is cancelled once they finish. Kill those jobs too if you do not want to wait (Operator). |
| A nested run is still active although its container job, or the parent workflow, has gone — deleted, or ended | It is left with nothing that can run it | The platform finds such a nested run within about six minutes, without a restart, and cancels it, along with any nested runs beneath it; one whose own jobs are still running is left until they finish. It shows as Cancelled rather than disappearing. Earlier builds cleaned it up only after the platform's runtime restarted (Operator). |
| The build's reason for an empty container is gone from the job's termination description | The container dispatched, which replaces it | Open the container job's Status History and read the Container Expansion Failed entry's details (Operator). |
| A hold/release/cancel event "didn't find" the workflow | The only instance for that name and date is a container's nested run | Target the standalone run, or act on the container's parent workflow instead (Operator). |
| A delete or overwrite of a workflow instance is refused, naming a nested sub-schedule | A nested run beneath it is still running, or still has jobs running or finishing | Wait for that nested run to finish, or deal with its running jobs first, then retry (Operator). |
Contact support when
- The referenced workflow exists, is deployable, isn't circular, and is within the depth limit, but the container still fails to build it.
- A container reports its nested workflow finished, but its own status doesn't settle to match.
Include the parent workflow instance, the container job name, the referenced workflow name/ID, and the nested workflow instance ID if one was created.