Notification Manager (groups, triggers, channels)

BETA Preview

This feature is still being finalized by Development.

Task walkthrough: Manage notifications. This page is the full configuration and troubleshooting reference.

Notifications tell people when something happens to their automation — an agent goes offline, a job fails, a schedule finishes — through notification groups that watch objects and fire actions over channels.

Notification group

Top-level, tenant-scoped container. Its type is set at creation and immutable — it governs valid member kinds, triggers, and actions.

Setting	Type	Required	Default	Notes
name	string (3–255)	Yes	—	Unique per tenant; follows the platform naming rules
type	AGENT | RELAY | WORKFLOW_INSTANCE | JOB_INSTANCE	Yes (create)	—	Immutable; UI labels Agent / Relay / Workflow / Job
description	string	No	null

Membership (what the group watches)

Allowed member kinds depend on group type:

Group type	Member kinds
Agent / Relay	EXPLICIT, NAME_PATTERN
Workflow	EXPLICIT, NAME_PATTERN
Job	EXPLICIT, NAME_PATTERN, TAG

• EXPLICIT — a specific object, bound to its identity rather than its name, since instances are ephemeral: an Agent or Relay by object id, a Workflow by definition id, and a Job by its definition's persistent GUID.
• NAME_PATTERN — a glob (*, ?). At most 255 characters and 32 wildcards. For a Workflow or Job it matches the definition name; for an Agent or Relay — which have no definition — it matches the object name. Matching is case-sensitive: EOD-* does not match a job named eod-close.
• TAG (Job only) — the definition's tags include the tag.
• Optional per member: environmentId — restrict matching to one environment (null = tenant-wide).

Watching every agent or relay

Agent and Relay groups take a name pattern, so * is how you watch all of them and PROD-* is how you watch a subset. Previously those two types could only name objects one at a time, which meant a group had to be edited every time an agent was added.

An Agent or Relay member cannot be scoped to an environment, and one that tries is rejected rather than saved. Agent and relay events carry no environment at all, so a scoped member of either type would match nothing — it would enrol happily and then never fire. Workflow and Job members are unaffected: scope those to one environment whenever you want to.

Naming one specific job

A Job group can now name a single job, from a picker listing every job as workflow / job. The workflow name is there to tell two jobs of the same name apart — the job's GUID is globally unique, so matching needs nothing beside it.

Because the member binds to that GUID:

• Renaming the job does not sever the notification. The stored workflow / job label is a display cache and goes stale on a rename; nothing consults it when matching. Re-open the member to see the current name.
• Moving the job between environments keeps it matched, which is why a deployment transformation rule can't touch a job's id.
• A job whose workflow version predates persistent job GUIDs is not offered. There would be nothing to store, and a member with nothing to match on would enrol happily and then never fire. Create a new version of the workflow to give its jobs GUIDs.

The other member kinds stay name- and attribute-based on purpose. A NAME_PATTERN should stop matching a job renamed out of its pattern — that is what a name pattern means.

Triggers (catalog, opt-in)

Triggers are a built-in catalog keyed by group type — users toggle catalog entries (opt-in; default disabled), they don't invent event types.

• Agent (connectivity): Offline, Online · Relay: Offline, Online
• Agent (operator marks): Marked Offline, Marked Draining, Mark Cleared — raised when someone takes an agent out of service or returns it. These are a separate stream from the connectivity triggers above and never cross-fire: clearing a mark raises Mark Cleared, not Online, so an agent that never went offline can't page you as though it had.

The Agent "Disabled" trigger was retired

There used to be a third connectivity trigger, Disabled. Nothing in the platform ever set an agent's observed status to disabled, so the trigger could never fire — and beside the real Marked Offline trigger it read as a duplicate of it. Taking an agent out of service is the operator mark, not an observed status, so Marked Offline is the trigger to use.

A group that had Disabled switched on had that trigger, and any actions on it, removed on upgrade — each recorded in the configuration audit log first. Nothing else about the group changed. Without the removal a group holding the trigger could no longer be saved at all, because a save re-checks every trigger on the group against the catalog.

• Workflow: Schedule Start, Schedule Complete, Schedule Placed On Hold, Schedule Released From Hold
• Job: Job Wait Machine, Job Late to Start, Job Late to Finish, Job Submitted, Job Still Attempting to Start, Job Initialization Error, Job Prerun Failed, Job Missed Start Time, Job Running, Job to be Killed, Job Finished OK, Job Failed, Job Marked Finished OK, Job Marked Failed, Job Cancelled, Job Restarted, Job Skipped, Job Under Review, Job Fixed, Job Exceeded Max Runtime (the only derived condition).

These are the labels exactly as the trigger list shows them: Workflow triggers carry a Schedule prefix and Job triggers a Job prefix. Agent and Relay triggers carry none.

Job Exceeded Max Runtime now actually raises

It is the only trigger in that list that isn't a job status: a job that runs past its Max Run Time is labelled rather than transitioned. Nothing in earlier builds compared a running job's elapsed time against its limit, so this trigger could be enabled and never raise. It now raises once per run, while the job is still running, and the job is not stopped. See Exceeding Max Run Time.

Per trigger: enabled. A trigger sends its actions once, when it fires.

A trigger that fires with no enabled action is recorded as sent, and nothing is sent. The trigger still matches and still creates a notification, which is finalized as SENT although nobody was told. Give every enabled trigger at least one enabled action, or switch the trigger off.

A trigger's actions can be sent on demand, without waiting for the event — see Testing a trigger.

Saving a group and its contents together

A group, its members and its triggers-with-actions can be created or updated in one save, and that save is all-or-nothing: if any part of it is invalid, none of it is written. A half-written group — one that exists and does nothing — is never left behind.

What a save leaves alone is deliberately not uniform, because members and triggers are different kinds of thing:

	On save
Members	Replaced. A stored member you don't send is removed. Sending an empty list removes every member; not sending the list at all leaves the members untouched. A member you send that matches a stored one — same kind, same target, same environment — keeps its existing row.
Triggers	Merged. A trigger you send is created or updated; one you don't send is left exactly as it was. Sending an empty list changes nothing. Turn a trigger off with enabled: false rather than by omitting it.
A trigger's actions	Replaced, if you send them. Leave them out and that trigger's actions are untouched.

The asymmetry matters for anything mid-flight: a member is a selection, so removing the row is the only way to stop it matching, while a trigger is configuration attached to a fixed catalog entry.

What the dialog requires before Save comes alive. A name and at least one member — a group that watches nothing can never fire. Save also stays inactive until something actually changes, so re-opening a group and closing it cannot rewrite it.

A group with every trigger switched off can be saved. It is a valid configuration, not a half-built one — switching a trigger off for a maintenance window or for a job that is noisy this week and saving the group in that state is the normal way to use it. The group simply matches nothing and fires nothing until a trigger is turned back on. A group with no triggers at all saves too. Earlier builds required one enabled trigger and left Save inactive with nothing on screen saying why.

Escalation

Escalation can't be set up in the Notifications interface. A trigger sends its actions once, when it fires.

Channels / actions

An action = channel + config (validated per channel). Three channels can be chosen:

Channel	Config (required)	Delivered as
EMAIL	to[], subjectTemplate (opt: cc/bcc, bodyTemplate)	Email, through your environment's email service
OPCON_EVENT	eventType, environmentId, parameters (the type's named field values; the command line is derived from them)	An OpCon event submitted to the platform
IN_APP	one of roles/users, titleTemplate (opt bodyTemplate, severity)	A row in each recipient's in-app feed

Email and OpCon Event retry on transient failures; the In-App channel does not — it writes a feed row, and a retry would post it twice with nothing to de-duplicate on.

An email needs a subject, not a body. bodyTemplate is optional — plenty of alerts say everything they need to in the subject line. subjectTemplate is not optional: an email with neither carries nothing.

Addressing an email action

To, Cc and Bcc each take a list, comma-separated, and an entry may be a literal address, a [[Property]] token, or free text. Recipients used to be validated as addresses; they no longer are, because the address is often only known when the notification is sent — which is what lets one action page whoever the on-call property names today.

That moves the real check to send time, and what happens there is worth knowing:

1. Each entry is resolved, then split on commas. A property holding primary@acme.com, secondary@acme.com is a distribution list, and it stays one — each address is checked on its own.
2. Each resulting address must be exactly one address: one @, something either side of it, and no spaces. A token that did not resolve, or resolved to free text like On-call Rota, is not one.
3. An entry that is not a deliverable address is dropped, and the rest still goes out — in To as well as in Cc and Bcc. A token that resolves for most jobs and not for a few should not silence the alert for exactly the jobs nobody thought to check, and a real address standing beside an unset token should still be paged.
4. The send fails only when To has nothing deliverable left in it. A surviving Cc does not rescue it: the addressee is what the notification is addressed to, and delivering to the courtesy copies alone would quietly make a Cc recipient the only recipient.

Every drop is recorded — the field and how many entries were dropped, never the addresses themselves, which are resolved property values and as personal as any address. So a Cc that reached nobody leaves a trace rather than passing as a clean send.

Punctuation that could add a recipient is refused when you save

A recipient may not contain ; : < > " ( ) \ or a line break — the punctuation that turns one recipient into several, or hides a second address behind a display name. A comma is allowed on purpose, because a comma-separated list is the intended shape here and every part of one is checked separately. The same rule is applied again after a token resolves, since resolution substitutes text from elsewhere — a job name, an agent name, a property somebody edited — straight into a mail header.

OpCon Event actions

An OPCON_EVENT action submits an OpCon event back into the platform when the trigger fires, so a notification can act rather than only tell someone — restart the job that failed, tag it for review, set a property. It is the one channel whose message is an instruction to the platform instead of a message to a person, and everything below follows from that.

It takes an Environment — the environment the event is submitted to, picked from the list — and then the event itself, built on the standard event form: you pick the Event type once, and the form renders that type's own fields.

Setting	What it holds
Environment	The environment the event is submitted to, picked from the list. Required
Event	The event to submit — $JOB:RESTART, $SCHEDULE:BUILD, and so on — chosen from the catalogue grouped by category, each name shown with its $TYPE code
The type's own fields	One control per field the type declares, with the type's own required-field and value rules. A field that takes text accepts a [[…]] property token from the global, job, schedule instance and agent scopes
Generated Event	The line the fields render to, shown read-only below them, so you can see exactly what will be submitted

You no longer type the command line. Earlier builds asked for a free-text Event type and then a hand-written Payload template that had to repeat that same type as its first segment. The line is now derived from the fields and regenerated every time you save, so it cannot disagree with the type or the values it came from. It is kept for display, and it is not what the action reads when it fires — the named fields are.

The rendered line is capped at 10,000 characters. The form measures the line your field values render to and refuses a longer one.

The event type is a fixed list, not a pattern. Only a type the platform actually routes is accepted, and only one it can also apply. Of the 58 routed types, 18 are refused when you save rather than left to fail on every firing, and they are refused for two different reasons:

Why it is refused	How many	Which
No handler yet — the type is meant to work and does not	10	$JOB:RESCHEDULE, $JOB:RESCHEDHLD, $JOB:USER, $JOB:MACHGRP, $SCHEDULE:START, $SCHEDULE:DELETE, $SCHEDULE:CHECK, $SCHEDULE:CHECKALL, $MACHINE:STATUS, $MACHINE:MAXJOBS
Not supported, and will not be — it depended on something this platform does not have	8	$NOTIFY:COMMAND, $NOTIFY:LOG, $NOTIFY:NETSEND, $NOTIFY:SNMP, $NOTIFY:SPOAL, $NOTIFY:SPOCO, $NOTIFY:TASKS, $NOTIFY:TEXTMSG

That leaves 40 types available. A well-formed type that does not exist is rejected with "Not an OpCon event type the platform routes"; one that exists but is refused is rejected by name.

$NOTIFY:EMAIL is available, and it is the one $NOTIFY type that is. An OpCon Event action carrying it sends mail — see Sending mail with $NOTIFY:EMAIL. The other eight are no longer offered in the picker at all, on this channel or any other event picker in the product; an action already saved with one still shows its stored type rather than reading as blank.

caution

$EVENT:ACTION is not an event type

Earlier builds hinted that the field wanted the shape $PREFIX:ACTION, and offered $EVENT:ACTION as the example. No such event exists and it was never routable. If an action of yours holds it, that action was never delivering anything — see what changed for an action configured before this build.

The generated line is positional, and the order is the contract. The values after the type are read off one by one against that type's own field list. That is why the line is generated rather than typed: a hand-written line that declared a different type from the one picked would be read against the wrong field list and produce a complete, wrong command.

An action configured before this build keeps firing, until you edit it

There is no data migration. An action saved before the standard event form keeps firing its stored command line exactly as it always has. What changes is what you see when you open it:

• A line the form can read opens as its type and its fields. Saving it re-derives the line from those fields.
• A line the form cannot read — an unknown type, or values that do not line up with the type's fields — opens empty, with the saved line shown and a note that it keeps firing as saved until you choose the event again here. There is no free-text command mode to fall back to: the event has to be re-entered before that action can be saved again.

Every save is checked against the event's own rules

Saving an OpCon Event action runs the same event rules the job editor, Self Service and Vision apply, and the server runs them again. A problem in the event answers 422 with each field named; a malformed request still answers 400.

What is refused, whether you touched the field or not:

• A required field of the type left empty.
• A value the field cannot read — a non-number in a number field, a value outside an option list, a malformed date.
• A comma inside a value, where the line's own separator would read it as the start of the next field. The exception is a trailing list field such as tags, which is a comma list by design.
• A value over the field's length, or a rendered line over 10,000 characters.

One exception: an action you have not changed, whose event type is listed but not built yet, is accepted as a warning rather than blocking the save. This keeps an unrelated edit to the same trigger from being held up by it.

A stored action can fail a rule it was saved before

These rules did not exist when some actions were saved, and they are applied to the whole action on every save — not only to what you edited. So an action that has been sitting there can block the first save you make after this build, naming a field you did not touch. The fix is the field it names; nothing is lost, and the action keeps firing as stored in the meantime.

What an OpCon Event action refuses to send

Every other channel renders a template leniently: a token that cannot be resolved arrives as its own name and the message still goes out, because a degraded message to a person still beats silence. This channel renders strictly, for the opposite reason — a degraded instruction to a machine is not a lesser instruction, it is a different one. [[OI.OncallTag]] arriving as the literal text OI.OncallTag would be a perfectly valid tag applied to a real job.

So the delivery attempt fails, with the field named and nothing submitted, when:

• a token does not resolve — a typo, a property that does not exist, or a scope the group type cannot resolve;
• a substituted value contains a comma, which the command line cannot escape and which would be read as the start of the next field. The exception is a type's trailing field, whose content the platform rejoins: in $JOB:TAGADD and $JOBMASTER:TAGADD the trailing field is a tag list, so a property holding oncall,prod-critical there deliberately adds both tags;
• a substituted value contains a field's own list separator — a properties field is split on ; into Name=Value pairs, so a value carrying a ; would add an entry the template never wrote. Write the list into the template instead of carrying it in one property;
• an encrypted property is substituted anywhere the event log would keep it in the clear. The platform stores the payload it receives and serves it back from the event log, so an encrypted property may only be placed in the value of $PROPERTY:SET or $PROPERTY:ADD. A secret in a job name or a properties slot is refused;
• the property an encrypted value would be written into is not itself encrypted. Even in the one slot that accepts a secret, the target has to be able to hold it: $PROPERTY:SET refuses a plaintext property, $PROPERTY:ADD creates the property encrypted, and an instance-scoped target is refused outright because it has no encrypted form at all. This channel used to be the gap in that rule — see below;
• the rendered line is longer than the cap. It is refused rather than truncated: a truncated command line is a different command line.

A refusal like these is permanent — retrying renders the same template — so it is recorded once against that action rather than retried. A failure the platform could not be asked about (a timeout, a restart) is transient and is retried under a stable key, so a retry cannot apply the same event twice.

Accepted is not applied

All but one of the available types are applied while the submission is open, so "accepted" and "applied" are the same answer. $CONSOLE:DISPLAY is the exception — it is queued, and pending is its final answer. For every other type, a pending reply means an earlier attempt is still in flight and the outcome is not known yet; the attempt is failed so the next one gets the real answer, rather than recording "accepted" as "done".

Check an existing action that copies an encrypted property

This channel renders its own payload and submits the finished line, which means it arrived past the point where the encrypted-value rules were applied. They therefore did not run on it, and an action reading $PROPERTY:ADD,TMP,[[OI.DB_PASSWORD]] created TMP as an ordinary plaintext property holding the decrypted password — readable by anyone who can read properties, and a value [[OI.TMP]] could then carry anywhere. $PROPERTY:SET into a plaintext property was allowed for the same reason.

The action now declares that its value came from an encrypted property, so the rules above apply to it like any other event.

The fix cannot undo what an earlier firing created. If you have an OpCon Event action that substitutes an encrypted property into a $PROPERTY event, find the target property: either delete it, or switch Encrypted on and re-enter the value. Then check what else read it.

Sending mail with $NOTIFY:EMAIL

$NOTIFY:EMAIL is the one $NOTIFY type the platform carries out, and it sends mail through the same Email channel a notification group uses — the same provider, the same delivery attempts and the same retry behavior. It does not need a notification group, a trigger or a watched object: the event carries its own recipients and its own text.

Its command line is positional like every other event type:

$NOTIFY:EMAIL,{to},{cc},{bcc},{subject},{message},{attachments},{excludePrefix}

Field	Required	What it takes
to	Yes	Recipient addresses. Separate them with ; — a bare comma is read as the next field, so it cannot be a separator here. A [[…]] token that resolves to a comma-separated list is still fanned out.
cc, bcc	No	Same rule.
subject	Yes	Up to 500 characters in the authoring form. An empty subject is sent as OpCon Notification.
message	Yes	The body, up to 32,000 characters — Classic's own ceiling for an assembled $NOTIFY.
attachments	No	Must be empty. See below.
excludePrefix	No	Y suppresses the context footer; anything else leaves it on.

The message carries a short footer unless excludePrefix is Y — Sent by OpCon with the event type, who triggered it, and the environment name where those are known. It is appended after the body, separated by a -- line.

Attachments are refused. Classic read attachment paths off the OpCon server's own filesystem, and there is no such filesystem here, so a non-empty attachments field fails the event rather than sending the mail without them. If the event was authored as a single command line and the failure names attachments, the usual cause is a comma inside the message shifting the value into that field — the error message says so.

At most 100 recipients across to, cc and bcc together. An individual address that is not deliverable is skipped and the rest of the message still goes out, as in Classic; a to with nothing deliverable in it fails the event, because the message would reach nobody it was addressed to.

Re-firing or retrying the same event does not send twice. The send is keyed on the event, so a replay finds the message the first attempt already queued instead of queueing another. The event completes once the send is queued — the same point at which Classic marked a $NOTIFY processed — so a delivery that later fails at the provider shows on the notification, not on the event.

What changed for an action configured before this build

Until this build the channel did not submit anything: it recorded every attempt as sent and finalized the notification as sent, in every environment, while nothing ever reached the platform. Nothing configured here has ever submitted an event, so there is no working behavior to preserve — but three things follow on upgrade:

• An OpCon Event action may start reporting FAILED. The channel needs service credentials configured for your deployment; without them it is disabled and every action records a failed delivery attempt naming the reason. That failure is the first honest report of a channel that was never working — it is not a regression. An administrator resolves it in the service configuration.
• A stored action that cannot meet the new rules was removed on upgrade, and recorded in the configuration audit log with everything needed to re-create it. It is not repaired automatically: there is no way to guess which environment a non-UUID value meant, and prepending the event type to a free-text message would produce a valid command line whose words would be read as parameters — an action that used to do nothing would start acting on the wrong thing.
• A trigger whose only remaining action was removed that way is switched off. A trigger with no enabled action finalizes as sent without attempting anything, which is exactly the state this work removes, so the trigger is disabled instead of left reporting delivery. A trigger that keeps another enabled action — an Email beside the removed OpCon Event — is left alone.

Who may save or test an OpCon Event action

Notification rights are not enough on their own. When you save an OpCon Event action, you must also hold the permission its event type requires — $JOB:KILL, for example, needs job-instances.delete. The check is scoped to the workspace of the object the event names, in the action's environment, and you also need properties.view for each property the payload names in a token. If any check fails, the save is refused with permission.denied and nothing is written.

• Only the events a save adds or changes are checked. Editing another part of a group does not fail because of an action someone else saved.
• A name built from a token — [[OI.JobName]] — cannot be looked up when you save, so the check then covers the permission alone, in any workspace.
• A test is checked as well, against the line as it renders, so the object it names is known. A test refused this way reports "Test not sent: You don't have permission to test notifications", even when you hold the notification rights — it is the event's permission you are missing.
• When the trigger fires, nobody is checked again. The event is submitted by the platform, so an action keeps running after its author loses the permission. Review OpCon Event actions when you remove someone's access.

Targeting the in-app feed

An IN_APP action reaches people by role name or by user.

• roles holds role names, not role ids. The feed matches a notification against the roles the reader actually has, by name — so a role id stored here reaches nobody. It dispatches, it is recorded as sent, and no one ever sees it. Pick roles from the picker and it stores the right thing.
• Matching ignores case. A role named Operations matches a reader whose role is operations. Role names are unique per tenant case-insensitively, so this cannot make a target ambiguous.
• A leftover role id stays visible so you can remove it. If an older configuration stored ids, the picker keeps the unrecognized value as a chip labelled with the raw text. Clearing that chip is the only way to get rid of it — selecting the right roles alongside it keeps the id, because the editor writes back everything it holds.
• Renaming a role orphans any action that named it. Names are the matching key, so a rename silently stops delivery to that role. Update the actions that named it.
• users matches a reader's id or their email address, so an email address is a valid entry despite the field name.

Inserting a property token

You can write a token by hand, or pick one. Eight action fields carry a token-insert button in the narrow column to their right; select it to open a searchable list of properties, and choosing one splices [[Scope.Name]] in at the cursor and returns you to the field. It is available wherever an action is edited in a notification group.

The eight are exactly the fields the platform resolves:

Channel	Has the picker	No picker
Email	Recipient (to), Cc, Bcc, Subject, Body	—
OpCon Event	Every text, multiline and date field the picked event type declares	Event type, Environment
In-App	Title, Body	Roles, Users

The fields without it are passed through as written, so a token typed into one arrives as literal text. The column is kept on every row either way, so the form stays aligned.

Four scopes are offered — Global, Job, Schedule and Agent. They are fixed: the list shows the same four on every one of the eight fields, and you cannot change the set.

Two things about the list itself:

• Search matches a property's name and its value. Encrypted properties and those resolved at run time can only be matched by name — there is no value to match against — and the empty state says so rather than leaving you to guess.
• The list loads up to 1000 global properties, and tells you when there are more than it fetched. It also reports a count of properties it left out because their names cannot be referenced by a token at all, or because they differ from another only in capitalization.

There is no "add a property" entry in the list. Create a global property first — see Properties and tags — then pick it here.

The picker offers more than every field can resolve

The same four scopes appear everywhere, but Job, Schedule and Agent tokens are answered from what the event captured. An Agent group captures only its own machine facts — see what an agent notification can resolve — and a Relay group captures none at all. A [[JI.$JOB NAME]] on either has nothing to resolve against.

What that costs depends on the field. In a template it degrades to its own bare name and the notification still goes out. In Recipient (to), Cc or Bcc it is not a deliverable address, so it is dropped — and if To is left with nothing deliverable, the send fails. See Addressing an email action. Prefer a Global property for a recipient field, and keep job and schedule tokens to the message itself.

Template tokens

An action's templates — subject, body, title, payload — carry property tokens, resolved when the notification is sent. Write them as [[Scope.Name]] or {{Scope.Name}}; the full grammar, scopes, date-and-time tokens and offsets are in Properties and tags.

Earlier builds substituted only six fixed names and rendered anything else as empty. That is no longer the case, and four consequences are worth knowing:

• A token that can't be resolved renders as its own bare name, and the notification still goes out. [[OI.OncallPager]] arrives as OI.OncallPager rather than as a blank space. A message about a failure must not itself fail, and a reader can act on a visibly broken token in a way they cannot act on a gap. Only the offending token degrades — every other substitution in the same field still resolves.
• A {{…}} body that was never a valid property name is no longer inert. Text like {{ops.runbook}} used to pass through exactly as written; it now renders ops.runbook, with the delimiters stripped.
• One delimiter pair per string. If a template contains [[ anywhere, only [[ … ]] resolves and every {{ … }} in it is left alone. A mixed template such as Job {{objectName}} — see [[runbook]] used to substitute and no longer does. Be consistent within a template.
• Seven names are answered by the notification itself and cannot be overridden by a property of yours: OI.OBJECTNAME, OI.OBJECTTYPE, OI.STATUS, OI.OBJECTID, OI.ENVIRONMENTID, OI.OCCURREDAT and OI.ACKURL. A tenant global genuinely named Status is unreachable from a notification template. The bare forms — {{objectName}}, {{status}} and the rest, with or without spaces inside the delimiters — keep working exactly as before.

A template may carry an expression

A template field can hold an expression — [[= ToUpper([[OI.Environment]]) ]] — computed when the notification is sent, like any other token.

The template	What happens
Holds an expression that reads	It is computed. If it fails, that field keeps the expression text — marker included, so it renders = … — and the notification still goes out.
Holds an expression that assigns — anything with a bare = between two operands	The send fails, and keeps failing until the template is edited.

A template holding any expression used to fail to send at all, because the whole [[= … ]] form was refused. Read expressions now deliver. If a notification has been silently failing since a template edit, an expression in it is worth checking first.

The assignment case is refused rather than degraded on purpose: a notification is a report, not an actor, so it registers nothing that could perform a property write — and a write that degraded quietly would be one nobody learned was attempted.

What is frozen and what is read live

The split matters because a notification can be rendered again after the event — when a delivery is retried, for example.

	Read from	Behavior
Facts about what happened — job name, status, termination text, start/end times, run time, frequency, agent, schedule name/date/id	Captured when the event fired	The message describes the state that triggered it, however much later it is delivered or re-delivered.
Configuration — your global properties (OI), thresholds (TH), resources (RU/RM)	Read when the notification is sent	Deliberately current, so an on-call number you updated an hour ago is the one that gets paged.

A Job notification can resolve the JI, MI and SI names listed in Properties and tags. A Workflow notification has no job in scope and answers the SI names plus SI.$SKD STATUS. A fact the event did not capture renders as its bare token rather than as a blank or a zero — an absent value is reported honestly instead of asserted as one that looks real.

What an agent notification can resolve

An Agent notification carries the machine facts its own event captured, so a message about an agent can name the agent:

Token	Resolves in
$MACHINE NAME	Every agent notification — Offline, Online, and all three operator marks
$MACHINE OPER STATUS	The operator-mark triggers only (Marked Offline, Marked Draining, Mark Cleared), as the display text Active, Marked Offline or Marked Draining
$MACHINE NET STATUS, $MACHINE RUNNING JOBS	Nowhere yet — nothing on the agent supplies them, so both degrade to their bare names

Write either form: [[$MACHINE NAME]] and [[MI.$MACHINE NAME]] are the same token, because a bare $MACHINE … name is routed to the MI scope.

$MACHINE OPER STATUS is deliberately absent from the connectivity triggers (Offline, Online). An operator mark is somebody's setting; whether the agent is reachable is a different fact, and a message about one must not report the other. See taking an agent out of service for the distinction.

A Relay notification resolves no machine tokens: a relay is not an agent, and a relay event carries no machine facts.

OI globals resolve for all four group types, agents and relays included. TH, RU and RM read environment state, so they resolve only for Workflow and Job notifications; in an Agent or Relay template a [[TH.…]] token degrades to its bare name, because there is genuinely no environment to read it against.

A notification resolves an encrypted property to its real value

An encrypted property token in a notification template resolves to the stored value, not to a mask. Resolution reads the real value, and the rendered subject, body, title or payload carries it — so a template naming an encrypted property puts that value in front of whoever receives the notification. The mask appears only in the platform's own record of what was substituted.

Do not name an encrypted property in a notification template unless you intend its value to be read by the recipients. This is the opposite of an event, where the same token is masked before the payload is built — the two seams behave differently, and a template moved from one to the other does not keep this property.

The OpCon Event channel is the exception, and only because its payload is stored: it refuses to submit a line carrying an encrypted value anywhere except the value of $PROPERTY:SET or $PROPERTY:ADD.

Limits

• A template is capped when you save it: 10,000 characters for a subject, a body or an event payload, and 500 for an in-app title. A severity label is capped at 20. These are refused on save rather than trimmed, so a template that is too long is an error you see immediately.
• A rendered field is capped at 10,000 characters; anything beyond that is cut.
• An in-app title is additionally cut to 512 characters once resolved — so in practice the 500-character cap on the template is the tighter of the two. That render cut was sized when substitution was a fixed six keys: a short token whose value is a long termination description can exceed it, and a clipped title still tells someone their job failed.
• Rendered output is not HTML-escaped. Email bodies are plaintext and must not be; anything rendering the in-app feed as HTML is responsible for escaping it there.

Note: the channel-form UI placeholders show ${name} style, which matches neither delimiter pair — a UI copy mismatch.

Testing a trigger

An open trigger carries a Test button beside + Action. It sends that trigger's actions now, as real notifications, so you can prove the recipients, the templates and the OpCon Event are right before a real event fires.

It sends what is in the editor. The actions currently enabled on that trigger, including edits you have not saved, and the group does not have to be saved at all — a group you are still creating can be tested. A test is validated by exactly the same rules as a save, so anything you can test you can save, and anything a save would refuse is refused here too.

Testing needs notifications.edit or notifications.create — either one, in either mode. That is deliberately not the same as saving: someone who cannot save this group may still test it. An OpCon Event action also needs the event's own permission — see Who may save or test an OpCon Event action.

Test is unavailable, and says why, when the trigger has no enabled action ("Add or enable an action to test this trigger"), when an enabled action is invalid ("Fix the invalid action before testing"), or while a test of that trigger is already running.

What you confirm first

Every test goes through a confirmation, because every test is a real send. The dialog is titled Test {trigger label} and lists one line per action on the trigger — the ones that will not be sent included, so the list is never shorter than the trigger:

Action	Reads
Email	Email to its recipients
In-App	In-App to 2 roles, 1 user — counted, so a long audience still fits a line
OpCon Event	its event type and environment
A disabled action	Not sent (disabled)

Each OpCon Event action adds a warning of its own: "This runs {event type} in {environment} for real." That channel submits the event to the platform, and a test submits it exactly as a firing would — a $JOB:RESTART in a test is a restarted job. Nothing else in the notification marks it as a test; a recipient sees the message you configured.

What resolves, and what does not

A test renders as a real firing does, with no triggering object. So:

• Global properties resolve, which is why a test is worth running at all.
• Job, Schedule and Agent tokens have nothing to resolve against. In a template one degrades to its own bare name and the message still goes; in To, Cc or Bcc it is not a deliverable address and is dropped; in an OpCon Event payload it fails that action, because that channel renders strictly. See Inserting a property token.
• The legacy fixed keys read as a test rather than as an event: {{objectName}} is the group's name — or Test Notification if you have not named it yet — {{status}} is the trigger's label, and the object type is the group's type.

What comes back

Each action is sent once, and a test is never retried — a retry would be a second real message or event. The outcome appears on a message inside the dialog: "Test sent: N of M actions succeeded". A clean run clears itself after a few seconds; anything that failed stays until you close it and offers Details, a read-only table of every action's channel, target and result.

A target is a summary, never a resolved value. Email reports a recipient count (cc and bcc included), In-App the audience it counted, and OpCon Event its configured event type. Resolved addresses are never returned or logged — a recipient assembled from an encrypted property would otherwise hand back the value the picker masks.

"Timed out — may still have been sent" means exactly that. Past its per-action ceiling the send is abandoned rather than cancelled, so the message or event may still arrive. Test again and you may send it twice.

A failed action's reason is only ever text the platform wrote itself — an email with no deliverable recipient, an in-app action with no storable target, a refused payload, a template that would not render. A transport failure gets a fixed sentence instead ("The mail server did not accept the message", "The event could not be submitted", "The in-app notification could not be stored"), because a mail server's own error text can name internal hosts and ports. The real error is in the service log under the test's correlation id.

If the request itself is refused, the message says so rather than reporting on actions: "Test not sent: You don't have permission to test notifications", or "Test not sent: Too many tests — try again in a few minutes".

Limits on a test

Limit	Value	Notes
Tests per person	30 per 15 minutes (default)	Counted per person, not per network, so one author cannot spend an office's allowance and changing networks does not reset it
Recipients per tested action	50 across To, Cc and Bcc, counted after resolution	A test-only cap — real dispatch is not bounded by it. Over it, the action fails with "A test send reaches at most 50 recipients… Trim the recipients, or save the group and let it fire." One [[Property]] token can expand to any number of addresses, which is why this is counted at send time rather than when you save
Actions per trigger	20	The same limit a save applies
Time per action	20 seconds (default, set per deployment)	Past it the send is abandoned, not cancelled, and the action reports timed out — see above

What is recorded

Nothing about a test is persisted as a notification. No notification instance and no delivery attempt row is written, so a test never appears in a group's delivery history. An in-app test does write a feed row — it has to, or nobody would see it.

The send is recorded server-side only: one log line per test naming who ran it, the tenant, the group and trigger, each action's channel and outcome, and a correlation id, flagged as a test. It carries no resolved property values and no message bodies. That log line is how an auditor tells a real message or event from a test one, since the message itself does not say.

How a status event becomes a notification

1. A transition happens — a job or workflow changes status, or an agent or relay goes offline or comes back.
2. Matching — for each enabled group of that type whose members match, for each enabled trigger the event satisfies, a notification instance is created. Configuration edits take effect within about 5 seconds, so a change you just saved may not apply to the very next event.
3. Dedupe — repeats for the same object and trigger collapse into one notification when they occur in the same 5-minute block of clock time (10:00–10:05, 10:05–10:10, and so on). The blocks are fixed, not a rolling window: events at 10:04:59 and 10:05:01 fall in different blocks and both notify, while events at 10:00:01 and 10:04:59 collapse into one.
4. Dispatch — the trigger sends its actions once, within a couple of seconds.

Instance states: PENDING → SENT, or PENDING → FAILED when every action failed. A notification with at least one action delivered is SENT, and so is one whose trigger had no enabled action.

In-app feed

A user sees a notification when one of their roles is in the action's roles, or their id or email is in its users. The feed is self-scoped — reading it needs no administrator permission, and you only ever see your own.

The Notifications panel

The bell in the main header carries a badge with your unread count, capped at 99+, and opens the panel. Two things about the bell itself:

• The count polls every 60 seconds, so a new notification appears without a reload.
• With nothing in your feed the bell is disabled, and its tooltip says No Notifications. There is no empty panel to open — and if your last notification goes while the panel is open, the panel closes.

Inside the panel:

Grouping	Two sections, Today and Older. Today means created since local midnight. Newest first within each.
Each row	The title, and a timestamp — for an Older row, MM/DD/YYYY | hh:mm AM in your local time. Unread rows are distinguished from read ones.
Read / unread	Each row toggles, both ways: Mark as read and Mark as unread. Marking read is no longer one-way.
Delete	Each row can be deleted from your feed.
Delete All	Clears your feed, read and unread, after a confirmation — This will clear all read & unread notifications. Unavailable while nothing is loaded.
Loading more	The panel pages as you scroll rather than showing a fixed number.

Everything here is per-user. One notification aimed at a role is a single record fanned out to its recipients, so deleting it — or clearing your whole feed — affects your feed only and leaves everyone else's alone. Deleting one is also final for you: reading it elsewhere does not bring it back. Delete All does not hide anything that arrives afterwards.

Acknowledgement

Notifications are not acknowledged. A notification is delivered, then read or deleted, and nothing further is needed from you.

Permissions

Object notifications (global scope):

Permission	Description
notifications.view	View notification configurations
notifications.create	Create new notification configurations
notifications.edit	Modify notification configurations
notifications.delete	Delete notification configurations
notifications.execute	Listed in the catalog, but nothing in this build checks it — granting or withholding it changes nothing

These permissions are enforced: a request without the matching permission is refused. Viewing, creating, editing and deleting groups, members, triggers and actions each require the matching notifications.<action> permission. The in-app feed is scoped to the recipient and needs no administrator permission.

Testing a trigger takes edit or create — either is enough, whether you are creating the group or editing a saved one. Testing is not saving, so someone who cannot save this particular group may still test it.

An OpCon Event action needs more than notification rights. Saving or testing one also needs the permission its event type requires, in the workspace of the object it names — see Who may save or test an OpCon Event action. See also Roles and permissions.

The interface

The Notification Manager is a list of notification groups — a table you can sort by name, type or member count, search, and page through. There is no check box column and no tab strip: the list is the only page, and a group's row menu is the way into it.

A group is created and edited in one dialog, laid out like the job editor: a fixed header over a tabbed body.

Region	What it holds	Scrolls
Header	Name, the group's Type as a 2×2 radio group, and Description. Then every alert the dialog raises — a failed save, a load that failed, no permission to save	No
Tab — Select {type}(s)	The matcher blocks for that type, each with its own heading and + {noun} button: Agents and Name Pattern for an Agent group, Relays and Name Pattern for a Relay, Workflows and Name Pattern for a Workflow, and Jobs, Name Pattern and Tags for a Job. Workflow and Job rows also carry an Environment; Agent and Relay rows do not, for a reason	Yes
Tab — Triggers & Actions	The live trigger catalog for the group's type, one row per trigger, with its actions inside	Yes
Footer	Cancel and Save	No

Only the tab body scrolls, so an alert explaining why Save is inactive stays in view however far down a long trigger list you are. Both tabs stay loaded: switching between them never discards an edit, a search you typed, or where you had scrolled to. A tab whose content is currently blocking Save carries an error dot on its label — the members tab when the group has no member.

Type is shown when you edit a group but cannot be changed, because it is immutable server-side. Seeing it is what tells you why the first tab lists jobs rather than agents.

There is no switch to turn a whole group off. To stop a group sending anything, turn off each of its triggers.

Nothing in the dialog is written until you save. Every control edits a draft, and Save sends the group, its members and its triggers together as one request — so a half-configured group is never left behind, and abandoning the dialog changes nothing.

Editing what a group already watches

A saved member is editable in place — its object, its pattern text, its tag and its environment all take a change, with the same controls a row you just added uses. Changing a typo in a pattern no longer means deleting the row and re-adding it.

• A saved row you have changed offers reset, which restores the value it was loaded with; a row you haven't changed, or one you added in this session, offers delete instead.
• Blanking a saved pattern makes the row unfinished, exactly like a half-filled new row: it is left out of the save and the rest of your membership changes still go through. It does not make the group's membership read-only.
• Membership is still replaced on save, so saving sends the whole set with your new value in it.

The trigger list

Several triggers can be open at once — opening one never collapses another. Creating a group opens with all of them collapsed; editing one opens every trigger that already has actions, so what is configured is what you see first.

Every trigger row carries its count as "N Action(s)", including "0 Action(s)". A trigger switched on with no action is a real and useless configuration, so its body warns you: "Actions must be added to correctly alert the appropriate people."

Each action inside a trigger shows its Channel and Sent To as read-only text with edit and delete beside them — the Add Action dialog owns their content — and a switch that enables or disables that action alone. Under them sit + Action and Test.

An Agent group's six triggers render as one flat list, the same as the other three types, rather than being split under connectivity and operator-mark headings. The two streams are still separate in behaviour.

An older link to a specific group still resolves: it opens the list with that group's edit dialog over it.

Troubleshooting

Symptom	Likely cause	Resolution
No notification fired	Trigger not enabled, no member match, or config change <5 s ago. A name pattern matches case-sensitively, so EOD-* misses eod-close	Confirm the trigger is on and members match — including the case of a pattern — and wait out the ~5 s snapshot (Administrator).
A notification is recorded as sent but nobody was told	The trigger fired with no enabled action	Add or enable an action on the trigger, or switch the trigger off (Administrator).
A save is refused with permission.denied after adding an OpCon Event action	You lack the permission the event type requires, in the workspace of the object it names, or properties.view for a property its payload names	Ask an administrator for that permission, or have someone who holds it save the action (Administrator).
"Sent" but nothing arrived	Email delivery is not set up for your environment, or the message was refused after it left Continuum	Contact support with the notification's time and recipients.
An OpCon Event action started reporting FAILED after an upgrade	The channel is now wired through and needs service credentials. Unconfigured, it is disabled and fails each attempt rather than reporting a delivery it never made	Configure the service credentials for the deployment. The action was not working before either (Administrator).
An OpCon Event action disappeared, or its trigger is switched off	The stored configuration could not meet the wired channel's rules and was retired on upgrade. A trigger left with no enabled action is disabled rather than left reporting delivery	Re-create the action: pick the event type and fill in its fields, then re-enable the trigger. The removed configuration is in the audit log (Administrator).
Saving a trigger is refused, naming a field on an OpCon Event action you did not edit	Every save re-checks the whole action against the event type's own rules, and the action predates those rules	Fix the field it names. The action keeps firing as stored until you do.
An OpCon Event action opens with no type picked and its saved line shown as a warning	The stored command line cannot be read into the type's fields	Choose the event again and fill in its fields. It keeps firing as saved until you do.
An OpCon Event action fails naming a field	A refusal, not a transport failure — an unresolved token, a substituted comma or ; that would shift the command line, an encrypted property where the event log would store it in the clear, or a line over the length cap	Fix the template or the property the message names. Nothing was submitted (Administrator).
An event type is rejected when you save an action	It is not a type the platform routes, or it is routed but has no handler yet, so every submission would fail	Choose one of the available types; the message says which of the two it is (Administrator).
An in-app notification is recorded as sent but nobody sees it	The action's roles holds a role id left over from an older configuration, or a role has been renamed since	Re-open the action, clear the unrecognized chip, and select the roles again (Administrator).
An Agent or Relay member is rejected on save	It names an environment, and those events aren't environment-scoped	Save the member without an environment (Administrator).
An email went to fewer people than the action names	An entry in To, Cc or Bcc resolved to something that isn't an address — an unset token, or free text — and was dropped so the rest could go out	Check the properties that field uses. The drop is recorded with the field and a count (Administrator).
An email action fails outright with no recipients	Everything in To resolved to something undeliverable. A surviving Cc deliberately doesn't rescue it	Put at least one literal address in To alongside the token (Administrator).
Save stays inactive in the group dialog	It needs a name and at least one member, and something has to have changed. A group with every trigger switched off is valid and does not block it	Fill in whichever is missing — the tab holding the problem carries an error dot, and the dialog's header names the reason (Administrator).
There is no way to turn a whole group off	There is no group-level switch	Turn off each of the group's triggers. A group with every trigger off saves and fires nothing (Administrator).
Test is unavailable on a trigger	The trigger has no enabled action, an enabled action is invalid, or a test of it is still running	The button's tooltip names which. Testing a trigger (Administrator).
A test reports "Timed out — may still have been sent"	The send passed its per-action ceiling and was abandoned rather than cancelled	Check the inbox, the feed or the event log before testing again — a second test is a second real send (Administrator).
A tested email action fails over its recipient count	A test reaches at most 50 recipients across To, Cc and Bcc, counted after resolution — a token expanded past it	Trim the recipients for the test, or save the group and let it fire for real (Administrator).
A test is refused with "Too many tests"	The per-person test limit for the window is spent	Wait out the window. The count follows the person, so changing network doesn't reset it (Administrator).
A test's OpCon Event action failed on a token	The payload used a job, schedule or agent property, and a test has no triggering object to resolve one against	Expected. Test the other channels, and prove the event by letting the group fire (Administrator).
A colleague received a real-looking alert nobody was paged about	A test is not marked in the message	Check the service log for the test record — who, group, trigger and per-action outcome (Administrator).
A token in a subject or body arrives as its own name	It could not be resolved — a typo, a property that doesn't exist, or a scope not available for that group type	Expected, and deliberate — the notification still goes out. Correct the token (Administrator).
A {{…}} token stopped substituting	The template also contains [[, which makes [[ … ]] the only pair that resolves in that string	Use one delimiter pair per template (Administrator).

Contact support when

• The OpCon Event channel reports FAILED on every action and you cannot reach whoever configures the deployment's service credentials.