Relays
- Synopsis
- Full Details
In briefA relay is the on-premises software that lets the machines OpCon Continuum cannot
reach directly — legacy LSAM machines, and OpCon MFT, OpCon RPA and EASE agents — run jobs
dispatched from the cloud. It connects outbound only, so it needs no inbound firewall change, and it
authenticates with a downloaded credentials.json.
WhereAgents › Relays lists every relay, with New Relay to register one and Download
Relay to fetch the latest release without registering anything. Each row's menu has Edit,
Rotate Credentials and Delete. The agents a relay serves are registered on the Agents
tab of the same page. The relay itself runs on your own network, where it keeps its
credentials.json and answers its local /health check.
When
- You are bringing existing LSAM machines — or an OpCon MFT, OpCon RPA or EASE agent — into Continuum, and need a relay to reach them.
- A legacy job cannot be placed, never starts, or behaves differently on one platform than on another.
- A relay's secret has been lost or exposed, or a relay has to be renamed or removed — including one that still has jobs.
A Universal Agent job never involves a relay; start with the agent instead.
How
- Register the relay on Agents › Relays with New Relay, and download its
credentials.jsonbefore you close the dialog. - Install the relay on a host in your network with the installer for its platform, copy the
downloaded
credentials.jsonover the placeholder, and start the service. On Windows the installer refuses folders a non-administrator could change, and only one relay runs on a host at a time. - Register the agents behind it on the Agents tab.
- Upgrade it with each platform release by re-running the new package's installer, which keeps the existing settings, and check its Version on the Relays tab afterwards. On Windows the installer stops and restarts the relay for you; on Linux, stop it first and start it after.
The full steps are in Manage relays.
- The client secret is shown once, when the relay is registered or its credentials are rotated. If it is lost, rotate the credentials rather than registering the relay again.
- A newly registered relay reads Offline, with a last heartbeat of Never, until the installed relay first checks in.
- Legacy agents show Unknown when their relay is stale — the relay, not the agent, is the thing to check. If the relay stays gone for an hour, they read Offline.
- Stop a relay cleanly when you can: it reads offline in about a second, rather than after about two minutes of missed heartbeats.
- A clean stop can take up to about 35 seconds while the relay finishes its shutdown.
- A relay started twice serves from one process only. The other waits as a standby, shown on the Relays tab as an instances badge, and takes over without a restart if the first stops — within seconds on a clean stop, about 40 seconds on a crash — with no offline notifications. An online relay with one process and no standby shows No Standby.
- Hand Over, on a ready standby's row in Edit Relay, moves the relay to that standby on purpose, for planned maintenance. It needs Relays Edit and cannot be cancelled once it starts.
- The relay is not upgraded for you. The Relays tab's Version column shows the build each relay
runs, so compare it with the version Download Relay offers;
—on a relay that checks in means a build from before relays reported their version. - The Relays tab needs Relays View, and each action its Relays permission. A relay's legacy agents follow the Agents permissions.
- If the LSAM Connector restarts, the relay connects it to its legacy machines again by itself.
- Mark Offline on a legacy agent has its relay close that agent's connection, and Clear Mark opens a new one, without touching the relay's other agents. An older relay keeps the connection open, and the agent's status says so.
- A Windows relay installed with a non-default
-ServiceNameneeds that-ServiceNameon every re-run of the installer; without it the installer looks for the default relay's settings instead. - Many current behaviours need a current relay — newer job types, IBM i lists, advanced exit criteria, an IBM i job's message ID, the schedule date a legacy job is sent with. An older relay can refuse a job, or quietly drop part of it.
- Upgrade the relay with the platform release, not separately: a mismatch either way stops new jobs' output from being found.
- Deleting a relay that still has jobs ends them: they fail and are not retried. The relay must have stopped at least a minute earlier, and it needs Agents Delete as well.
- When the active relay's host vanishes, the standby cannot reach a legacy agent until the agent's own session timer frees the old session. Check the machine before you rerun a Lost in Failover job: it may still have run.
See also Agents and agent pools and LSAM legacy connectors.
Task walkthrough: Manage relays. This page is the full configuration and troubleshooting reference.
A relay is the on-prem software that lets legacy LSAM machines run jobs dispatched from OpCon
Continuum in the cloud. It connects outbound only to the OpCon agent-service (no inbound firewall
changes), authenticates with a downloaded credentials.json, and bridges dispatched work to the LSAM
machines behind it. Legacy agents are reached through a relay; a relay may front more than one
legacy agent and legacy agent groups may span relays.
The relay is a pure credentials consumer — its only inputs are a credentials.json file and a few
connection environment variables. Relay support covers registration and credential download,
credential rotation, artifact distribution, installation, and a /health readiness endpoint.
Where the relay sits
The two things this picture is here to settle:
- Nothing connects inward. The relay dials out to the agent service and polls for work, so running one needs no inbound firewall change.
- A Universal Agent does not need a relay. It talks to the platform directly. The relay exists only to reach machines that cannot. If a Universal Agent job fails, the relay is not involved — and if a legacy job can't be placed, the relay is the first thing to check.
Most machines behind a relay are legacy LSAM machines, reached over the LSAM wire protocol. Three kinds are not: an OpCon MFT agent, an OpCon RPA agent and an EASE agent are each reached over their own HTTPS REST API. All three still need a relay, and each is registered under one on the Agents tab, but none of them is an LSAM — see Serving OpCon MFT agents, Serving OpCon RPA agents and Serving EASE agents, on the Agents It Serves tab.
An EASE agent is the odd one out even among those three: it is not a machine on your network at all, but Jack Henry's hosted EASE OpCon, which the relay reaches out to over the internet.
- Relay List
- Register
- Install
- Logs
- Runtime
- Agents It Serves
- Security and Updates
- Troubleshooting
Relay List
The Relays tab
Agents → Relays lists every relay in the tenant, 20 to a page, with a Name filter. Above the list, the toolbar has New Relay and Download Relay, and Delete once rows are selected.
| Column | Shows |
|---|---|
| Name | The relay's name |
| Hostname | The host name it was registered with |
| Status | Online or Offline. A new relay reads Offline until it first checks in |
| # Agents | The legacy agents registered under it |
| Version | The relay build's version, as the relay last reported it when it checked in: the release number, then + and a build identifier. — until it has reported one |
| Last Heartbeat | How long ago it last checked in; hover for the exact time. Never until it first does |
Version sorts in release order — 1.10.0 after 1.9.0, and a pre-release before its release —
and relays showing — sort last when ascending, first when descending. A relay shows — when it
has never checked in, or when the build it runs predates relays reporting their version. Because
every check-in reports it again, a relay you upgrade or downgrade shows its new build at its next
check-in, and one downgraded to a build that does not report it goes back to —.
A relay running as more than one process shows an instances badge beside its Status — for example 2 instances — when two or more of its processes have checked in within the last 90 seconds. Hover it for how many are active and how many are standing by (1 active, 1 standby). The badge turns to the warning colour, and the tooltip adds An older relay build without instance reporting is also running on this registration, when one of those processes is a build from before relays reported their instances. An Online relay running as exactly one process, with no standby, shows No Standby in the warning colour instead; hover it for No standby instance is running. If this relay stops, jobs on its agents cannot start until it restarts. It can show for a moment after a handover, until the previous active process has restarted as a standby. See More than one relay process (on the Runtime tab).
Selecting a row opens Edit Relay. Each row's menu has Edit, Rotate Credentials and Delete.
The tab needs Relays View under your roles. New Relay needs Relays Create, renaming a relay and Rotate Credentials need Relays Edit, and Delete needs Relays Delete; an action you may not take is unavailable, and its tooltip says why. A relay's legacy agents are governed by the Agents permissions. See Agents, agent pools and relays are checked.
Edit, rename and delete a relay
Edit Relay changes the relay's Name and Host Name, with Save Changes. It does not touch the relay's credentials or anything on the relay host, so there is nothing to change or restart there.
Below the form, the Instances section lists every process of this relay that has checked in during the last 24 hours, with its Host, Version, Role and Last Seen, and refreshes while the dialog is open. Role reads Active or Standby, followed by Older build for a build from before relays reported their instances, and by Stopped (it stopped cleanly) or Not responding (it stopped checking in) for a process that is no longer live. With nothing reported in that time it reads No instances have reported in the last 24 hours. When exactly one process is live and it is not a standby, the section warns No standby instance is running. If this relay stops, jobs on its agents cannot start until it restarts. The warning is not shown while a standby is taking over. A ready standby's row carries Hand Over, below.
Hand over to a standby
Hand Over moves a relay's work from its active process to a standby you choose, without waiting for a stop or a failure — before patching or rebooting the active host, for example, or to move the relay onto a host running a newer build. It is on the row of each standby that is ready to take over: a current relay build that is running and has checked in within the last 30 seconds. A relay with no ready standby shows no Hand Over. It needs Relays Edit.
- In Edit Relay, choose Hand Over on the row of the standby that should take over.
- Hand Over Relay asks first: The active instance on
<host>stops taking new work, and the standby on<host>takes over. Running jobs keep running. Once it starts, the handover cannot be cancelled. Do you want to continue? Choose Hand Over, or Cancel. - If the standby runs an older relay version than the active process, or either version is not known, Hand Over Anyway warns that handing over to an older version can undo an upgrade. Choose Hand Over to go ahead anyway, or Cancel.
The section then follows the handover, refreshing every 2 seconds while it is under way:
| The Instances section shows | Means |
|---|---|
| Handover requested. Waiting for the active instance to stop taking new work. | The request is recorded. The active process picks it up when it next renews its lease, within about 5 seconds |
| The active instance is stopping. Running jobs keep running. | The active process has begun its clean stop |
| The active instance has stopped. The standby is taking over. | The active process has given up the lease, and only the standby you chose can take it |
The standby on <host> is now active. | Done: the standby you chose is the active process. This one is a message at the top of the dialog |
| The handover did not complete. Check the instances below, then try again. | The standby did not take over. A request that is not completed lapses by itself after about a minute, and the dialog stops waiting two minutes after the request |
| A different instance took over instead of the selected standby. | Another process became active |
While a handover is under way, every Hand Over is unavailable, with A handover is in progress. There is no cancel: once the active process has begun to stop, it cannot be called back. What happens to the relay and its jobs is described under More than one relay process (on the Runtime tab): the relay and its legacy agents stay Online, and running jobs carry on.
A request can be refused. The dialog says why:
| Message | Means |
|---|---|
| Hand Over unavailable, with The active instance cannot hand over. Upgrade it, or stop it to let a standby take over. | The active process runs a relay build that cannot hand over. Upgrade it, or stop it cleanly and a standby takes over |
| This standby is not ready to take over. Wait for it to report in, then try again. | The standby has stopped or has not checked in within the last 30 seconds |
| No standby is ready to take over. | No process of the relay is ready to take over |
| This instance is already active. | The row you chose has become the active process |
| The relay has no active instance to hand over from. | No process holds the relay's lease just now |
| A handover to another standby is already in progress. | Someone else has asked for a handover to a different standby. When the request is still under way, the dialog follows it instead |
| You do not have permission to edit relays. | Your roles do not grant Relays Edit |
| The handover could not be requested. Try again. | The request did not reach the platform, or failed there |
A relay's old name stays reserved after a rename. The relay keeps the internal dispatch pool it was given under its first name, so no new relay can be registered with that name. Trying gives The relay name '…' is reserved: its dispatch pool '…' still exists (for example, kept by a relay that was renamed). Choose another name. A name another relay already has is refused with Relay with name '…' already exists.
A relay with agents cannot be deleted. Its row's Delete is disabled with Delete or move this relay's agents first., and deleting several selected relays skips the ones that still have agents and names them. The platform also refuses a delete itself, and the dialog shows its reason:
| Refusal | Means |
|---|---|
| Relay '…' has N bound agent(s); delete the agents first | Legacy agents are still registered under it. Each of them can be deleted only once it is not in use — see An agent in use cannot be deleted |
| Relay '…' has N in-flight work item(s); cannot delete | Work it dispatches is still queued, claimed or running — including a job shown as Lost in Failover. Let it finish, or kill it. On the Relays tab you are offered to end it instead, below |
A relay that still has jobs can be deleted from the Relays tab, ending them. When a delete is
refused for jobs it still has, and the relay is Offline, Delete Relay and End Jobs asks
first, with the count: Relay '<name>' still has 3 jobs: 1 not started and 2 started. Deleting it
marks all 3 as failed, and they cannot finish later. Do you want to continue? OK deletes the
relay and ends every job it was still holding, in the same step; Cancel leaves both alone.
Deleting several relays collects every busy offline relay into one confirmation.
- A job ended this way fails. A job that had started reports Relay deleted before the job finished, and none of them is retried by its job's automatic retries: the relay and its pool are gone, so there is nowhere to send it again.
- The relay has to be stopped first. A relay that is Online, or that last checked in less
than a minute ago, is not deleted with its jobs: Relay '
<name>' is online, or stopped less than a minute ago, and still has jobs. Stop the relay, wait one minute after it shows Offline, then delete it. The minute covers a stopping relay that has already reported Offline while it finishes the last request for work it had open. - It needs Agents Delete too. Ending the jobs takes Agents Delete under All workspaces as well as Relays Delete.
- Legacy agents still block it. A relay with agents registered under it cannot be deleted this way either: delete its agents first.
Deleting a relay revokes its credentials.
Download the relay without registering one
Download Relay, on the toolbar of the Relays tab beside New Relay, gets you the latest published relay build on its own — you generally want the software before you have anything to register.
The dialog shows the latest release's version and publish date, and a download button per platform. Downloading creates no relay and issues no credential.
The artifact used to be reachable only from the relay registration success screen, so getting the binary meant registering a relay you may not want — creating a record and burning a one-time client secret. Worse, reopening that dialog resets its result, so a download that failed meant registering again. Neither is necessary now.
Two things to know:
- Only the platforms the release actually publishes are offered. A button appears for a given operating system and architecture only when the release carries that artifact, so you cannot be shown a download that would then fail.
- The artifact alone cannot connect. The relay also needs its credentials and your environment's connection details, which you get by registering it. Use New Relay for both: see the Register tab.
The Download Relay action is disabled rather than hidden when there is nothing to download, and its tooltip says which of four situations you are in:
| Tooltip | Meaning |
|---|---|
| Checking for the latest relay release… | The lookup is still in flight |
| No relay release is available. | No published release could be found |
| No relay build is available for a supported platform. | A release exists but publishes nothing for a platform this page offers |
| Could not check for the latest relay release. | The lookup itself failed — this is a fault to chase, not an absent release |
Register
Register a relay
In the UI: Agents → Relays → New Relay (backend POST /api/v1/relays/register, tenant-scoped).
The dialog asks for two fields:
| Field | Rules |
|---|---|
| Name | Required. Letters, numbers, dashes, underscores and dots, up to 240 characters. Unique in the tenant, and see renaming (on the Relay List tab) for names that stay reserved. |
| Host Name | Required, up to 255 characters. The name of the host the relay runs on. |
Save registers the relay and replaces the form with its credentials and everything needed to stand it up:
| On the credentials screen | Detail |
|---|---|
| Client ID and Client Secret | Shown once, with a copy button each. The secret is masked until you show it. |
| Download credentials.json | Writes the strict { relayId, clientId, clientSecret } file the relay requires at startup. |
| I have stored this secret | Done stays disabled until you tick it, or until you choose Download credentials.json, which ticks it for you. Copying the secret does not, because a later copy can replace it on the clipboard. |
| Install the relay | The latest published release's download buttons — the same per-platform buttons as Download Relay (on the Relay List tab), so the two surfaces always offer the same builds — and a pointer to the README.md inside the extracted package. |
| Connection details | The runtime-derived AGENT_SERVICE_URL and the /agent-api path prefix for the environment, each with a copy button. |
| Run the relay | Linux and Windows command sequences, pre-filled with this environment's URL and the release's version. |
Copy the Client Secret or download credentials.json before you close the dialog. Closing it
before ticking I have stored this secret — or downloading the file, which ticks it — asks first — You won't be able to see these
credentials again. — with Keep Open and Close Anyway. If you lose the secret,
rotate the credentials rather than registering the relay again.
A new relay reads Offline until it first checks in. Registering records the relay as Offline, with a Last Heartbeat of Never. It turns Online on its first heartbeat, once it has been installed and started with these credentials — so a relay that reads Online is really running. A registered relay that is never installed simply stays Offline; it is not reported as going offline later.
If the save fails with a server error, the relay may still have been created: check the list, and if it is there, rotate its credentials to get a new secret.
credentials.json must match the exact three-key shape — a strict schema is enforced at boot, so
an extra key such as the agent-only agentPoolId causes a parse failure. The relay package ships a
shape reference at config/credentials.json.example; it is not a working credential.
Rotate credentials
If a Client Secret is lost, rotate rather than delete + re-register. UI: Rotate Credentials
on the relay's row menu on the Relays tab (on the Relay List tab); backend
POST /api/v1/relays/:id/credentials/rotate (tenant-scoped, authenticated). It generates a new
Client ID and Client Secret, rotates the active credential in place, and records who performed the
rotation and when.
The dialog warns first — Rotating credentials invalidates the current credentials immediately. The relay stays offline until it is reconfigured with the new credentials. — and Rotate stays disabled until you type the relay's name. It then shows the new pair on the same once-only credentials screen as New Relay: Client ID and Client Secret with copy buttons, Download credentials.json (which ticks the next box for you), and I have stored this secret before Done. If you lose the secret, rotate again. If the rotation reports a server error, the old pair may already be invalid: rotate again to get a working one.
Rotating does not restart anything, and the relay cannot sign in with its old file, so finish the job on the relay host straight away:
| Platform | Replace | Then |
|---|---|---|
| Linux | /etc/opcon/credentials.json | sudo systemctl restart opcon-relay |
| Windows | C:\ProgramData\OpCon\Relay\credentials.json | Restart-Service OpConRelay |
After rotation, old credentials fail OAuth: refresh-token validation verifies the token's client ID still resolves to an active credential, so a pre-rotation refresh token (valid ~24h) cannot keep minting new access tokens. Cross-tenant/missing relay → 404.
Install
Connection environment variables
| Variable | Required (Y/N) | Default | Purpose |
|---|---|---|---|
AGENT_SERVICE_URL | Y | — | Base URL of the agent-service for the environment (from the register dialog). |
AGENT_SERVICE_PATH_PREFIX | Y, for a cloud environment only | "" | Service path prefix for ALB/CloudFront routing, e.g. /agent-api. Omit for local/dedicated-origin. |
CREDENTIALS_FILE | N | ~/.opcon-relay/credentials.json | Absolute path to credentials.json (installers pre-set this). |
LSAM_CONNECTOR_PATH | Y | — | Path to the bundled LSAM connector executable. The Linux and Windows installers set it to the connector they install; a relay started without an installer will not start until it is set. |
RELAY_STATE_DIR | N | the directory holding CREDENTIALS_FILE | Where the relay keeps the state it needs to survive a restart (on the Runtime tab). The Linux installer sets /var/lib/opcon-relay. |
WORK_POLLER_WAIT_SECONDS | N | 20 (both installers pass 10) | Edge long-poll wait (seconds, max 20). The relay's own default is 20; the Linux and Windows installers set it to 10 unless you pass another value. Keep ≤10 on-prem behind CloudFront/envoy (~15s upstream timeout) to avoid 504s — so a relay started without an installer needs it set explicitly. |
HEALTH_PORT | N | 8080 | Port for the HTTP /health readiness endpoint. |
LOG_LEVEL | N | info | trace | debug | info | warn | error | fatal. |
LOG_ERRORS_TO_STDERR | N | true on Windows, false elsewhere | Also write warnings and errors to stderr, which on Windows is relay-stderr.log. Off by default elsewhere because Docker and the systemd journal merge stdout and stderr, so each line would show twice. |
SHUTDOWN_TIMEOUT_MS | N | 30000 | How long the relay allows itself to finish its shutdown, in milliseconds; a positive whole number. Neither installer sets it. The Windows installer gives the service this plus 5 seconds to stop — see How long a stop takes (on the Runtime tab). |
RELAY_INBOUND_JOURNAL_ENABLED | N | true | Whether the LSAM Connector keeps each finished-job status and each event from a legacy machine on disk until the platform has accepted it — see Job results and events survive a stop (on the Runtime tab). false turns it off: the connector then holds them in memory only. |
RELAY_JOURNAL_DRAIN_TIMEOUT_MS | N | 5000 | The longest, in milliseconds, a clean stop waits to deliver what the connector is still holding before the connector stops. 0 skips the wait; what is left is delivered at the next start. The wait also ends 12 seconds before SHUTDOWN_TIMEOUT_MS runs out. |
The relay reads its configuration entirely from environment variables. There is no
relay.yaml — the old YAML template has been removed. config/relay.env.example in the
relay package is a commented reference for the connection, path and tuning variables the
installers set.
A wrong AGENT_SERVICE_URL / missing AGENT_SERVICE_PATH_PREFIX typically surfaces as an opaque
403/404 from the cloud edge. Use the exact values from the register dialog.
Scheduling TLS variables
An agent with Use TLS for Scheduling on has its relay encrypt the connections that schedule its jobs and fetch their output, and check the agent's certificate. It needs relay 1.24.0 or later. Most relays need no extra settings: an agent certificate issued by a certificate authority the relay's host already trusts works as it is. Set these variables only to use your own certificate authority, to present a client certificate, or in a test lab. One set serves every TLS agent on the relay.
| Variable | Default | Purpose |
|---|---|---|
LSAM_TLS_CA_CERT_PATH | Unset: the host's trusted certificate authorities | A PEM file with one or more CA certificates. The agent's certificate must chain to one of them. |
LSAM_TLS_CLIENT_CERT_PATH | Unset | A PEM client certificate, for an agent that requires one. Set it with LSAM_TLS_CLIENT_KEY_PATH. |
LSAM_TLS_CLIENT_KEY_PATH | Unset | The PEM private key for that certificate (PKCS#8 or PKCS#1, not password-protected). Keep it readable only by the relay's account. |
LSAM_TLS_SKIP_VERIFY | false | Development and test only. Skips the certificate and name checks. The relay logs a warning at startup and on every TLS connection. |
- The agent's certificate must carry the agent's LSAM Host as entered, or its TLS Server Name when one is set. TLS 1.2 and TLS 1.3 only.
- At startup the relay checks each file. A missing or unreadable file, or a client certificate without its key (or the reverse), is logged as an error naming the variable. The relay still starts: agents without TLS work as before, and TLS agents stay offline until it is fixed. The relay never falls back to an unencrypted connection.
- The files are read again on every reconnect, so a renewed certificate is used from the next reconnect. A changed variable takes effect when the relay restarts.
- Every relay instance (on the Runtime tab) needs the same files at the same paths and the same variables. An instance with different files fails only its own TLS connections, with a reason in its log.
- Put the files where the relay's account can read them: in the directory you mount at
/opt/opcon-relay/configfor Docker (readable by the container userrelay, uid 1001), under/etc/opcon/lsam-tls/on Linux (readable by theopconuser, with the variables added to/etc/opcon/relay.env), or underC:\ProgramData\OpCon\Relay\lsam-tls\on Windows (with one line per variable added on the service's Environment tab withnssm edit OpConRelay; do not usenssm set … AppEnvironmentExtra, which replaces the installer's lines). Lines you add this way are kept when you re-run the installer, and the Windows uninstaller does not delete thelsam-tlsfolder.
When a TLS agent does not connect, the relay logs LSAM connection failed: TLS: … once per agent, until it connects:
| The log shows | Likely cause | Fix |
|---|---|---|
| At startup, LSAM TLS configuration error: LSAM_TLS_… cannot be read (…), or … is set but … is not | A file is missing or unreadable by the relay's account, or only one of the client certificate and key is set | Fix the path or the permissions, then restart the relay. Agents without TLS are not affected. |
TLS: certificate name mismatch (expected <name>) | The agent's certificate does not carry <name> — its LSAM Host, or its TLS Server Name | Set the agent's TLS Server Name to a name the certificate carries, or reissue the certificate. |
| TLS: certificate chain error (…) | The certificate does not chain to LSAM_TLS_CA_CERT_PATH, or to the host's trusted authorities when that is unset. With Check Certificate Revocation on, the revocation service may be unreachable | Point LSAM_TLS_CA_CERT_PATH at the issuing authority, or turn Check Certificate Revocation off. |
| TLS: handshake failed (…) | The agent does not have TLS turned on, offers only TLS 1.0 or 1.1, or requires a client certificate the relay does not present | Turn TLS on at the agent, allow TLS 1.2 or later, or set LSAM_TLS_CLIENT_CERT_PATH and LSAM_TLS_CLIENT_KEY_PATH. |
| TLS: the LSAM closed the connection after the TLS handshake (it may require a client certificate) | The agent ended the session straight after the handshake | Set the client certificate variables if the agent requires one. |
Install
There are two install paths — Linux and Windows, and both are native packages. Every artifact is
self-contained: the packages and the LSAM connector bundle their own runtimes, so no host Node
or .NET runtime is required on either platform. A Linux host must be 64-bit x86 (x86-64) with
glibc 2.28 or later — RHEL 8, Ubuntu 20.04, Debian 11 or newer — and have the ICU (libicu) and
OpenSSL 1.1.1 or later (libssl) libraries, systemd, and root (sudo). Full
per-platform steps are in the README.md at the root of the relay package.
The native installers take the connection details as arguments and set the environment for you;
only credentials.json is a file. --agent-service-url (Linux) / -AgentServiceUrl
(Windows) is required on a first install — both installers stop without it, before changing
anything. On a re-run over an existing install it is optional, like every other option: see
Re-running the installer.
Run the installer from inside the extracted package, not from the folder you downloaded into.
Both platforms' commands on the register dialog start by changing into the download folder and
extracting, because an elevated PowerShell opens in System32 and a Linux shell in your home
directory — so the first command would otherwise not find the archive.
- Linux:
sudo ./install/install.sh --agent-service-url <url>from the extractedopcon-relay-<version>/folder → app under/opt/opcon-relay, config/etc/opcon, logs/var/log/opcon-relay,opcon-relaysystemd unit. Other options (with their defaults):--agent-service-path-prefix(/agent-api; pass''for a direct origin),--work-poller-wait-seconds(10),--install-dir,--config-dir,--log-dir,--user/--group(opcon). The installer writes/etc/opcon/relay.envand systemd loads it viaEnvironmentFile— edit that file and restart to change values. Logs viajournalctl -u opcon-relay. The unit also declares a state directory,/var/lib/opcon-relay, which systemd creates and hands to the service account;/etc/opconstays read-only, as doescredentials.jsoninside it. - Windows:
powershell -ExecutionPolicy Bypass -File .\install\install.ps1 -AgentServiceUrl <url>from an Administrator PowerShell, in the extractedopcon-relay-<version>\folder. The-ExecutionPolicy Bypassis needed because a downloaded script is blocked by the default policy. →C:\Program Files\OpCon\Relay, configC:\ProgramData\OpCon\Relay(itscredentials.jsonrestricted to Administrators and SYSTEM), logsC:\ProgramData\OpCon\Relay\logsandC:\ProgramData\OpCon\Relay\dotnet; serviceOpConRelay. Other parameters:-AgentServicePathPrefix,-WorkPollerWaitSeconds,-InstallDir,-ConfigDir,-LogDir,-ConnectorLogDir,-LogRetentionDays(30, 1–3650),-ServiceName. The service is managed by the bundlednssm.exe(copied into the install dir and registered), which sets the connection settings as service environment variables and captures stdout/stderr torelay-stdout.log/relay-stderr.log. NSSM is the only supported Windows service mechanism — the old-UseNSSMswitch and thesc.exenative-service path have been removed, because the relay cannot run reliably as a bare native Windows service (it fails with error 1053). See Logs, rotation and retention (on the Logs tab) for what the two log directories hold and the rules the installer applies to them.
Two things about the installer that are easy to get wrong:
- A first install does not start the service, and it places a placeholder
credentials.json. Copy your downloadedcredentials.jsonover the placeholder first — copying onto the existing file keeps the owner and0600mode (Linux) or the restricted ACL (Windows) the installer applied — thensudo systemctl enable --now opcon-relayorStart-Service OpConRelay. A relay that starts but cannot authenticate immediately after an install is almost always a placeholder that was never replaced. An existingcredentials.jsonis never overwritten. - On Windows, do not set the connection values as machine environment variables. The service's own environment, set by the installer, takes precedence over them.
Re-running the installer
A re-run keeps the existing install's settings. Every option you do not pass keeps its current
value, so pass an option only to change it — --work-poller-wait-seconds 5, say, or
-LogRetentionDays 90. The installer prints each value with where it came from: passed, kept
from the existing install (Linux) or kept from the existing service (Windows), or default.
Linux (install.sh) | Windows (install.ps1) | |
|---|---|---|
| Where the kept values come from | The installed opcon-relay systemd unit (install folder, config folder, log folder, user and group) and relay.env in that config folder (URL, path prefix, work poller wait) | The installed service's own settings, and -LogRetentionDays from its log retention task |
| Your own additions | Lines you added to relay.env are kept. A line for a variable the unit itself sets — CREDENTIALS_FILE, LSAM_CONNECTOR_PATH, RELAY_STATE_DIR, NODE_ENV or the connector's log path — is dropped with a warning naming it, because it would silently override the folder you passed; pass the matching option instead | Environment entries you added with nssm edit are kept, and LOG_LEVEL keeps its value |
| A value it cannot use | Refused, naming the option to pass: The existing install's … cannot be read back as --…. A relay.env that is a symbolic link, or not a regular file, is refused too. The installer reads relay.env as text and never runs it | Refused, naming the option to pass: Refusing to install: the installed … service's … setting cannot be read back as -…. |
| The relay while it runs | Stop it first, re-run, then start it again: sudo systemctl stop opcon-relay, sudo ./install/install.sh, sudo systemctl start opcon-relay | Leave it running. The installer stops it and starts it again — see Upgrading a Windows relay |
In every refusal nothing was changed. A Windows relay installed with a -ServiceName other than
the default still needs that -ServiceName on every re-run, because it is how the installer finds
the service whose settings it keeps.
Upgrading a Windows relay
An upgrade is one re-run of the installer, from the new package's extracted folder, in an Administrator PowerShell, with no options:
powershell -ExecutionPolicy Bypass -File .\install\install.ps1
Do not stop or start the service yourself. The installer works in this order:
- Checks everything first — the package is complete, and every folder and setting passes what the installer checks. A package missing one of its programs is refused with Bundled … not found at …. The package is incomplete; re-download the relay artifact. Nothing was changed.
- Copies the new files next to the old ones in the install folder while the relay keeps running. If that fails, the relay was not stopped and its files were not changed.
- Stops the relay, which can take up to about 35 seconds — see How long a stop takes (on the Runtime tab).
- Swaps the new files in. If the swap fails, the previous files are put back.
- Re-creates the service with the kept settings, and starts the relay again if it was running before. A relay that was stopped stays stopped — which is how a second relay on the host stays out of the way.
If the upgrade fails after the relay stopped, the installer starts it again when its files and settings are intact and says so: The upgrade failed; started … again on the files and settings now installed. Otherwise it leaves the relay stopped and asks you to fix the error and re-run; if the service no longer exists, pass every option you used at install.
Do not interrupt an upgrade. If you stop the installer with Ctrl+C after it has stopped the relay, it starts the relay again only when the relay's program folders are whole on disk, waiting first for a stop still in progress to finish. If the files may be incomplete or from two versions, it leaves the relay stopped and says so: The upgrade did not finish and the application folders in … are incomplete, so … was not started. Re-run the installer before starting the relay. Re-run it before you start the relay. Closing the PowerShell window instead stops the installer without any of this, so re-run it in that case too.
Three things to know:
- The first upgrade from an earlier package still stops with that package's short stop limit — about 3 seconds, which can cut the relay's shutdown short. Upgrade while no jobs are running that time. Later upgrades get the full time.
- To go back to an earlier package, run
Stop-Service OpConRelayfirst — an earlier installer copies its files before it stops the service — and start it again yourself afterwards. - Connector logs left under
C:\Program Files\OpCon\Relay\dotnet\logs\by an install from before the logs moved go with the old files at the next upgrade.
The same applies to changing a setting on Windows: re-run the installer with just the option to
change, and it stops and restarts the relay for you. Alternatively, edit the service's environment
with the bundled NSSM — & "C:\Program Files\OpCon\Relay\nssm.exe" edit OpConRelay — then
Restart-Service OpConRelay.
Uninstalling a Windows relay
From the extracted folder, in an Administrator PowerShell,
powershell -ExecutionPolicy Bypass -File .\install\uninstall.ps1 removes the service, the install
folder and the log retention task, after asking for confirmation (-Force skips it). Pass
-InstallDir, -ConfigDir, -LogDir, -ConnectorLogDir or -ServiceName if you changed them at
install.
| Option | Keeps |
|---|---|
-KeepConfig | The configuration folder's files: credentials.json and the relay's state files |
-KeepLogs | Both log folders, untouched |
Without -KeepConfig, uninstall also deletes the relay's state files, relay-state.json and
connector-state.json, and then removes C:\ProgramData\OpCon\Relay once nothing is left in it. It
does not delete the relay's instance identity, relay-instance.json, so the
configuration folder of a relay that has run is kept.
In the configuration folder it deletes only files it knows by name, never the folder's whole
contents: a configuration folder that still holds anything else, or that is a junction or symbolic
link, is kept, and it says so.
Windows: what the installer checks before it changes anything
The relay service runs as SYSTEM, so the Windows installer will not use a folder or file that someone other than an administrator could change. Before changing anything, it refuses — on a fresh install and on an upgrade alike:
- a relay folder (
-InstallDir,-ConfigDir,-LogDir,-ConnectorLogDir), any of its parent folders, or a file in them that is owned by a non-administrator, or that a non-administrator can delete, rename or change the permissions of; - any of them that is a junction, symbolic link, mount point or hard link;
- a
-ConfigDir,-LogDiror-ConnectorLogDirthat overlaps-InstallDir, which uninstall deletes in full, and a log folder that is or contains-ConfigDir; - a drive root as
-ConfigDir.
An overlapping folder or a drive root is refused with a message starting Invalid, naming the folder. For an item that is a link, or that a non-administrator owns or controls, the message starts Refusing to install: and names the item and the reason, and then what to do:
| The item | What to do |
|---|---|
| A junction, symbolic link, mount point or hard link | Delete it and re-run the installer. Never icacls /reset a hard link: that changes the file it points to |
A folder another account can delete or re-permission — common for a new folder directly under C:\, which lets any signed-in user modify what is created in it | Restrict it with the icacls … /inheritance:r /grant:r … command the message gives, or choose a folder under C:\ProgramData, then re-run |
| Anything owned by a non-administrator | Delete it and re-run. To keep it, restart Windows, run takeown /F <path> /A and icacls <path> /reset, then re-run |
Unless the message says the installer has already restricted a folder, nothing was changed —
these checks run before an upgrade stops the relay, so a running relay is still running. If you
stopped the service yourself first, Start-Service OpConRelay resumes the installed version.
The configuration folder is restricted to Administrators and SYSTEM. Every install and upgrade
restricts -ConfigDir (C:\ProgramData\OpCon\Relay) to them and re-protects a credentials.json
that is not protected. Other users cannot browse the folder, though they can still open a log file
by its full path.
The install folder is restricted to Administrators and SYSTEM too. -InstallDir holds the
programs the service runs as SYSTEM, so the installer creates it restricted to them, or restricts an
existing one, before it copies anything in. Before changing anything, it also refuses:
- a drive root as
-InstallDir; - an existing
-InstallDirthat holds anything other than the relay's own files, so a shared folder such asC:\Program Files (x86)passed by mistake is never locked down for every other application in it.
A relay installed by an earlier installer to a custom folder directly under C:\ (for example
C:\OpCon\Relay) inherited permissions that let any signed-in user change it, so its next upgrade
is refused. Restrict that folder, and any parent folder the message names, with the icacls
command the message gives, then re-run the installer.
Windows: more than one relay on a host
Each relay service has its own display name. The default OpConRelay service is displayed as
OpCon LSAM Relay; a service installed with another -ServiceName is displayed as
OpCon LSAM Relay (<name>). A second relay can be installed on the same host with its own
-ServiceName, -InstallDir, -ConfigDir, -LogDir and -ConnectorLogDir, and installing it
leaves the first relay's settings as they were.
Only one relay can run on a host at a time. Every relay uses health port 8080 and LSAM
Connector port 50051, and the installer cannot change either. Stop one relay before you start the
other, and set the one not in use to start manually (Set-Service <name> -StartupType Manual) so
both do not start at boot.
A display name another service already uses is refused, before anything changes: Refusing to
install: the service … already uses the display name '…'. The service manager finds a service by
its display name, so installing would write this relay's settings to the other service. If that
service is an OpCon relay, re-run install.ps1 for it first with its own -ServiceName, which gives
it its own display name; otherwise rename it with sc.exe config <name> DisplayName= "<new name>".
A second relay installed by an earlier installer is still displayed as OpCon LSAM Relay: before
you upgrade the default relay, re-run install.ps1 for it with -ServiceName <name> and every
other option passed explicitly, because the settings stored on it may be another relay's.
Every downloaded package also contains a README.md with numbered per-platform steps and the full
parameter tables. It is the authoritative install guide for the release you downloaded.
Readiness / health
The relay runs a minimal zero-dependency /health server: 200 {"status":"ok"} when
ready, 503 {"status":"starting"} during startup, 200 {"status":"standby"} on a process
that is standing by (on the Runtime tab) behind another copy of the same relay,
503 {"status":"fenced"} on an active process that is stepping down, 404 otherwise. Readiness follows the relay's own
startup (ready after the core loops start, not-ready on connector max-restarts, closed on shutdown).
It is for local and orchestrator checks; nothing outside the host needs to reach it.
Logs
Logs, rotation and retention
A relay writes two sets of logs, because it runs two processes: the relay itself, and the LSAM Connector that speaks the legacy protocol for it.
| Log | Where | Rotation | Retention |
|---|---|---|---|
| Relay (Linux) | journalctl -u opcon-relay | systemd's journal | systemd's own journal settings |
| Relay (Windows) | relay-stdout.log (every line) and relay-stderr.log (warnings and errors only) in -LogDir (C:\ProgramData\OpCon\Relay\logs) | NSSM rotates at 10 MB while the service is running, and the retention task adds a daily boundary | Rotated relay-stdout-<timestamp>.log / relay-stderr-<timestamp>.log files are deleted after -LogRetentionDays (default 30) |
| LSAM Connector (Linux) | lsam-connector-<date>.log in --log-dir (/var/log/opcon-relay) | A new file per day | The newest 10 files are kept; the connector deletes the rest itself |
| LSAM Connector (Windows) | lsam-connector-<date>.log in -ConnectorLogDir (C:\ProgramData\OpCon\Relay\dotnet) | A new file per day | The newest 10 files are kept; the connector deletes the rest itself |
Each relay log line's time is the host's local time, with its UTC offset — for example
2026-11-05T22:00:01.123-05:00 — the same clock the LSAM Connector's log uses, so the two can be
read side by side. The Docker image writes +00:00 unless its TZ is set. Earlier builds wrote the
time as a count of milliseconds.
The relay's first log record names its build. Starting OpCon Relay Orchestrator carries
relayVersion (the same value the Relays tab shows), buildTime and
buildTarget (linux-x64, win-x64 or docker), so the log answers which build is installed
without the platform. The LSAM Connector it ships with is stamped with the same version.
The LSAM Connector log now redacts what a legacy job carries as command data. Until it did, all of the following were written to it in clear text, at levels that are on in every default install:
- the command line, parameters and prerun command of a Windows or UNIX job;
- the values of its environment variables (each variable's name is still shown);
- an Embedded Script's arguments and its whole body — the body was previously abbreviated only when it ran past 1,024 characters, so a short script was logged verbatim;
- an IBM i job's prerun and call-script commands;
- a SQL job's script statements, its environment-variable values, and its other options — which
is where a
sqlcmd -Ppassword sits; - the agent's own echo of the command line back on every job-status message.
So a password typed onto a command line, into an environment variable or into a script body was readable by anyone who could read that file on the relay host. Treat connector logs written by an earlier relay as sensitive: review them, and rotate any credential that was passed this way.
Redaction rewrites the log and nothing else. The agent still receives the real command line and environment, so no job behaves differently. Relatedly, a failure to send a frame no longer logs the frame's contents at all — only its length and message type.
On Windows, the deleting is done by a scheduled task the installer registers,
\OpCon\OpConRelay-LogRetention — daily at 03:00, as SYSTEM. It asks NSSM to rotate, then
deletes rotated relay logs past the retention period; the active relay-stdout.log and
relay-stderr.log are never deleted. Re-running the installer re-registers the task, keeping its
current -LogRetentionDays unless you pass a new one, and uninstall.ps1 removes it. If the relay's log directory has been
turned into a junction or handed to a non-administrator, the task refuses to prune rather than
deleting through it, and the refusal shows up as a non-zero Last Run Result on the task.
-LogDir and -ConnectorLogDir must be folders that hold nothing but relay logsSYSTEM creates and deletes files in both, so the installer will not accept a folder somebody else controls. It aborts on a drive root, on a junction or symbolic link, and on an existing folder containing anything that is not a relay log file — a sub-folder, the config directory, a shared log folder. Point each at its own empty folder. The installer then restricts both to Administrators and SYSTEM, with Users: read.
The same care shapes uninstall: it deletes only the relay's own log files and then the folder if
nothing is left, never a recursive delete. A folder holding anything else is kept, and says so.
-KeepLogs keeps both folders untouched.
Earlier installs wrote the LSAM Connector's log under C:\Program Files\OpCon\Relay\dotnet\logs\.
The installer now points the connector at C:\ProgramData\OpCon\Relay\dotnet instead — writing
application logs under Program Files needs privileges the service should not want. The old files
are left where they are and no longer written to; delete them when you are satisfied you don't
need them.
Runtime
Relay ↔ agent-service runtime
- The relay reports in on
POST /api/v1/relays/:id/heartbeatand fetches its members fromGET /api/v1/relays/self/agents(relay-authenticated). - Legacy agent changes made in-product hot-reconcile to the relay within about one heartbeat — no relay restart. See LSAM legacy connectors.
- An operator's Kill on a job running behind the relay also rides the heartbeat: the platform offers the pending kill on the relay's next check-in, the relay stops the process on the machine, and it acknowledges the dispatch on the beat after that. A kill that isn't acted on is re-offered every heartbeat until it is. See Killing a job on a legacy (LSAM) agent. This path exists only through a relay: a Kill never reaches a Universal Agent.
- A legacy agent's name must be 128 characters or fewer. The platform accepts names up to 255 characters, but the relay refuses to open a connection for a longer one, so that agent never comes online.
- A relay can carry any number of legacy agents. It is handed all of them, and all of them get heartbeat status applied. The relay's own views in the Agents area list all of them too, so an agent beyond the first 50 by name can be opened and edited. Earlier builds passed only the first 50 by name: beyond that, an agent was never dialed, never came online, and a job's output on it could not be retrieved. If you have a relay with more than 50 agents and the ones late in the alphabet never worked, that is what it was — nothing to reconfigure, and no need to split the relay.
- A host name is looked up on every connect. When a legacy agent's LSAM host is a host name, the relay resolves it to an address each time it connects, including every reconnect, so an address change behind the name is picked up without a restart. See A legacy agent addressed by hostname is re-resolved on every connect.
- When a relay goes stale, its legacy agents show Unknown (yellow) — relay-down, distinct from Offline (agent-down). A job aimed at one of those agents holds in Waiting: Machine until the agent is Online again, and a legacy-group job holds until the group has a member dispatch can route to. Earlier builds let a job aimed at one Unknown agent advance to Running, where no relay took it. If the relay does not come back within an hour, its Unknown agents go Offline; earlier builds left them Unknown for as long as the relay was gone. A relay that is running but has not reported one of its agents for more than five minutes holds that agent's jobs in Waiting: Machine too, even though the agent still reads Online. See Agents and agent pools and Operator response guide.
Stopping a relay
Stop the relay cleanly whenever you have the choice. The two ways a relay can stop are detected differently and leave its legacy agents in different states:
| How it stopped | Noticed after | Its legacy agents become | Queued legacy-group work |
|---|---|---|---|
| Cleanly — the service is stopped | About a second. The relay reports itself offline as the first step of its shutdown | Offline | Holds in Waiting: Machine until the relay reports in again |
| Abruptly — killed, power lost, network dropped | About 2 minutes of missed heartbeats | Unknown | Holds too, and resumes when the relay reconnects |
A clean stop also closes the relay's connection to each legacy machine before the relay exits. An OS 2200 machine accepts one connection from a relay at a time, so this is what lets the restarted relay reconnect without being refused as a second connection — see One relay connection per OS 2200 machine.
Queued legacy-group work now waits either way. An abrupt loss used to be far worse than a clean stop: the Waiting: Machine gate failed open on an Unknown feed, so the job advanced and then failed at dispatch with no online group member. An all-Unknown group now holds instead — see When a legacy group has no dispatch target.
Stopping cleanly is still the better choice, for a smaller reason: a clean stop is reflected in seconds rather than two minutes, so the status you see after a planned stop is the status. The two-minute window on an abrupt loss is deliberate — four missed check-ins — so that a brief network blip does not turn into Unknown agents and queued jobs holding through it for nothing.
This table is for a relay's last running process. When the relay also has a standby ready to take over, stopping or losing the active process changes nothing you can see: the relay and its legacy agents stay Online, no Relay Offline or Agent Offline is raised, and queued work does not hold in Waiting: Machine — see More than one relay process.
A relay's shutdown reports itself offline before it drains its work poller, so the status you see is not waiting on in-flight work. If the report cannot be delivered, shutdown continues anyway and the relay is detected by the ordinary two-minute window.
A clean stop of a relay's last process raises an Agent Offline notification for each of the relay's legacy agents, not only a Relay Offline for the relay. It used to raise the relay's event alone, so a planned relay restart sent a run of Online notifications on the way back up with no Offline to match them — which reads like a run of unexplained outages. The Offline events are raised only for agents the stop actually moved, so an agent already Offline does not produce a second one.
This applies to a clean stop. An abrupt loss moves the agents to Unknown rather than Offline and raises no per-agent event, which is deliberate: relay-down is not the same claim as agent-down, and the agents' real state is not known. When the relay returns, each agent that is still down raises one Agent Offline — including after the relay has been gone long enough for its agents to read Offline. See Agent (connectivity) triggers.
How long a stop takes
A clean stop gives the relay time to finish its shutdown — report itself offline, stop taking work
and save the state it resumes from. The relay allows itself
SHUTDOWN_TIMEOUT_MS, 30 seconds by default, and the service manager waits 5 seconds longer
than that before it ends the process:
| Platform | How long the service manager waits |
|---|---|
| Linux | 35 seconds, set in the opcon-relay unit |
| Windows | SHUTDOWN_TIMEOUT_MS plus 5 seconds — 35 seconds by default. The installer sets it on the service from the relay's own value |
So on Windows, Stop-Service OpConRelay can take that long, and until then PowerShell repeats
WARNING: Waiting for service '… (OpConRelay)' to stop.... That is expected. Earlier Windows
relays were given about 3 seconds, which could cut the shutdown short; the service picks up the new
time the next time it starts, so the stop during the first upgrade from such a package is still the
short one.
The LSAM Connector stops with the relay. It receives the same stop as the relay, so its exit during the relay's shutdown is logged as expected — LSAM Connector exited during relay shutdown — and is not treated as a crash or restarted.
Two Windows cautions:
- Restart the relay with
Restart-Service OpConRelay, nevernssm restart. The bundled NSSM gives up after a few seconds and leaves the relay stopped. - Windows gives services less time to stop when it shuts down than a normal stop allows. Before
you restart Windows, stop the relay with
Stop-Service OpConRelayif it is running jobs.
When a legacy agent stops answering without disconnecting
A legacy agent normally goes away visibly: the TCP connection closes and the connector notices at once. The hard case is an agent that stays connected and stops answering — the host was paused or frozen, or a NAT port-forward in front of it keeps the socket open after the machine behind it is gone. Nothing announces anything, and the connector's send is waiting on an acknowledgement that is never coming.
That wait used to block the whole machine. Every queued job start for that agent queues behind the outstanding acknowledgement, and so does the connector's own heartbeat to it — so jobs already running on it stayed Running with no completion, nothing new could be sent to it, and the machine went on reading as reachable. The only thing that eventually broke the deadlock was the send retry loop giving up, which takes 31 attempts at the response timeout — about two and a half hours at the 300-second default.
There is now a liveness deadline on the acknowledgement itself. If one has been outstanding
longer than LsamDefaults:AckLivenessTimeoutSeconds — 90 seconds by default — the connector
declares the agent dead on its next machine-status tick: the agent goes down, the socket is released,
and it reconnects. That is the same give-up path the retry loop always took, reached in a minute and
a half instead of two and a half hours. Setting the value to 0 or less turns the check off. The
send retry loop, the response timeout and the kill timeout are all unchanged.
The deadline is measured from when the acknowledgement first went outstanding, not from the last send, so re-sending the same frame does not push it back.
The relay re-seeds each legacy agent's state from the connector on every heartbeat, and the field it read reported the connection as up whenever it was configured up — a value that is only ever true. So a correct "agent is down" was overwritten as reachable once its event aged out, roughly three minutes later, and the agent read as available while nothing could be sent to it.
That field now reports up only when the connection is marked up, the socket really is connected, and the agent has confirmed the legacy handshake on it. Anything else is down, and a machine the connector has no connection for at all reads as unknown rather than as either.
A legacy agent addressed by hostname is re-resolved on every connect
When a legacy agent's LSAM host is a hostname, the connector resolves it to an address each time it connects — the first connect and every reconnect after it. An agent configured with an IP address is dialled as written, as before.
This is a correction, and it mattered most where hostnames matter most. The connector used to resolve the hostname once, when the agent was added, and keep the address it got. Nothing resolved it again, so an agent whose address changed under a stable hostname was still being dialled at its old one — and if another machine had since taken that address, the connector talked to that machine instead. The symptoms were a legacy agent going offline while it was running perfectly well, and, worse, a job running on the wrong host. Retrieving a job's output read the same stale address, so output came back from the wrong machine too.
Containerised agents hit this most easily, because addresses there are reassigned on an ordinary stop and start, but any environment with DHCP leases or a failover that moves an address could reach it. If you worked around it by pinning agents to IP addresses or by re-adding an agent after a restart, neither is needed now.
The character set used on the legacy wire
Text going to and from a legacy (LSAM) agent is encoded with a single-byte code page, set by
LsamDefaults:TextEncoding and defaulting to 1252 (Windows Western European). The setting takes
a code page number or name, and it applies to everything carried on that wire: command lines and
program arguments, script bodies, File Arrival paths, the event lines agents send in, and the
output files the Output tab reads from the agent.
Non-ASCII characters used to be lost. The connector encoded this traffic as plain ASCII, so
every character above U+007F — an accented letter, a currency symbol, a non-English word in a path
or a script — was replaced with ? in both directions, with nothing reported. A script body
containing é reached the machine with a ? in its place and ran, or failed, on the altered text.
Set the code page your agent hosts use and that traffic now arrives intact.
Job output was the last to follow. The output files the Output tab fetches from a legacy
agent were still read as plain ASCII after the rest of the wire moved to the code page, so an
accented letter in a job's log or report showed as ?. They are now read in the same code page. This
needs a current relay (on the Security and Updates tab); output already fetched and stored keeps its ?
characters until you use Refresh from agent.
Two constraints on what you can set, both enforced when the relay starts — an invalid value stops the relay rather than being ignored, so a misconfiguration is visible immediately:
- Single-byte and ASCII-compatible only. The legacy framing assumes one byte per character, so multi-byte and stateful code pages (932, 50220) are refused — their trail bytes can contain the very characters the framing uses to find the end of a field. EBCDIC code pages (37) are refused because they do not encode ASCII as ASCII. UTF-8, ASCII and Latin-1 are not available through this setting.
- A character the code page cannot represent becomes
?. It is never substituted with a lookalike. That matters for more than fidelity: the relay checks a value for shell-significant characters before it is encoded, so a best-fit encoder that turned a full-width$into a real$afterwards would slip one past a check that had already passed.
More than one relay process: active and standby
Only one process of a relay serves it at a time. Each running copy of a relay is an instance,
and exactly one instance is active: it holds the relay's lease, polls for work and connects to
the machines behind the relay. Any other process started with the same credentials.json — on
another host, say, or a leftover install — is a standby. A standby takes no work, connects to no
legacy agent and leaves the state files alone; it asks for the lease every 5 seconds. It logs once,
Relay instance … on … is STANDBY: instance … holds the relay lease. This process will not poll work
or connect to LSAMs. It takes over automatically if the leader stops., and its /health answers
200 {"status":"standby"}, so a container check reads it as healthy. Earlier builds let every
copy poll for work and connect to the same machines.
You can see every instance on the Relays tab (on the Relay List tab): a relay with more than one live process carries an instances badge, and Edit Relay lists each one with its host, version, role and when it last checked in.
A standby takes over without restarting. It is already running, and when it is granted the lease it starts serving in the same process, logging Relay lease granted to instance … (epoch …); promoting in-process:
- when the active process stops cleanly, it gives up the lease once it has closed its connections, and a standby takes over within about 5 seconds of that, without waiting for the lease to lapse;
- when the active process disappears — killed, or its host lost — a standby takes over within about 40 seconds, once the lease has lapsed;
- when you hand it over (on the Relay List tab), the active process learns of the request when it next renews its lease, within about 5 seconds, and logs Relay handover requested; yielding the lease. It then stops cleanly: it stops asking for work — work it was in the middle of receiving goes back to be handed out again — closes its connections to the legacy agents and delivers the job results it holds. Once every connection has closed it gives up the lease at once, without waiting for its LSAM Connector to stop. While the request stands, only the standby you chose can be granted the lease. The previous active process exits with code 75 and its service manager starts it again as a standby.
Job numbers carry across a handover. Every job a relay sends to a legacy agent carries a job number that the agent and the relay use to match later status updates to the job. The platform hands the active relay those numbers in blocks, and the sequence is held by the platform for the relay, not by the process. A standby that takes over, a relay reinstalled on a new host and a restart after a crash all continue above every number already issued, so a number is not reused while a job that holds it is still running. If the platform does not offer blocks, the relay falls back to counting locally from its saved state, as earlier builds did.
A handover is invisible to operators. While a standby is ready, the relay and its legacy agents stay Online through a stop or a crash of the active process: no Relay Offline or Agent Offline is raised, and work does not hold in Waiting: Machine. For up to 90 seconds after a takeover, while the new active process reconnects to the machines, an agent it has not reached yet keeps its status; an agent it reports up shows Online at once. A process that has begun to stop takes no new work, so a job released during the handover waits for the new active process. If no standby takes over within those 90 seconds, the relay is handled as a stopped or lost relay — see Stopping a relay.
Jobs in flight at a handover are adopted. When a standby takes over, it asks the platform which legacy jobs the relay had running and takes them over, including jobs the previous active process started. Their results are reported as they would have been: a job that finished during the gap reports its real result and releases its dependents. The new active process does not send an adopted job to its legacy agent again. Once a legacy machine reports back online, the relay waits 10 seconds by default, so the machine can deliver the results it queued, and then asks it once about each job it adopted.
If the machine answers that it does not know the job, the job is shown as Lost in Failover. It is not marked Failed, because it may have succeeded. Its dependents are held. A real result that arrives later still replaces the status. Otherwise, check the job on its machine and mark it Finished OK or Failed — see Job and workflow statuses.
When the active host vanishes — power, network or a frozen virtual machine, rather than a crash
or a clean stop — each legacy agent keeps the old session until its own inactivity timer runs out,
and the new active process cannot connect to it until then. On a Windows LSAM that timer is 310
seconds and cannot be changed; on a UNIX LSAM it is msg_timeout_in_seconds in lsam.conf, 600
seconds by default. A crash or a clean stop frees the session at once.
Lost in Failover does not prove the job did not run: a job can still start later from the old host's leftover session. A job can also start twice when its machine's session drops after the relay sent the start and before the machine acknowledged it: the start is sent again on the next session, as in Classic, and the machine may have started the job already. Before you mark such a job and run it again, check its machine's job log for that run. An active process that was paused — a suspended virtual machine, for example — no longer sends a job start it was still holding when it resumes after a standby has taken over: it sends to legacy agents only while it still holds the relay's lease. Earlier builds could start a job on a legacy machine that serves one session at a time between 2 and 20 times after the active host vanished, because the relay closed the new session it had just opened.
OpCon MFT, OpCon RPA and EASE runs are adopted too. Before it sends an MFT or RPA start or an EASE add, the relay records with the platform that it is about to, and it records the run's id as soon as the agent returns one. A standby that takes over reads those records, so it can follow a run the previous active process started, and it never sends a start or an add a second time. What happens to each run depends on how far it had got:
| When the previous process stopped | What the new active process does |
|---|---|
| The start or add had not been sent | Fails the job, which says so: Relay failed over before the RPA start was sent; nothing was started (MFT reads the same; EASE ends nothing was submitted to EASE). Nothing ran, so rerun it freely. |
| The start or add had been sent, but its id was not yet known | Shows the job as Lost in Failover, because the run may be going ahead on the agent. An MFT run is looked for on its agent first, for up to about 340 seconds after its start was sent, and is followed if it is found. |
| The run's id was known | Follows the run and reports its real result. If the agent no longer knows the run, the job is shown as Lost in Failover. |
A Kill that was requested before the handover and not yet carried out is sent again by the new active process. If the record cannot be made before the start, nothing is sent, and the job fails with MFT start not sent: the relay could not record it in agent-service; nothing was started (or the RPA or EASE equivalent). Nothing ran, so rerun the job.
An active process that loses touch steps down by itself. If it cannot renew its lease for 20
seconds while a standby is ready — one that has asked for the lease within the last 30 seconds — or is told another instance now leads, it stops taking work, closes
its connections to the legacy agents — before a standby can be granted the lease, so the standby can
connect to them — and finishes delivering the job results it already holds. A job it had just taken but not yet
sent to its machine is handed back rather than started, so the instance that leads next runs it;
earlier builds could fail such a job. Its /health answers
503 {"status":"fenced"} meanwhile. It then logs Relay instance … fenced (…); restarting as
standby, exits with code 75, and is started again as a standby. A relay running as one
process never steps down this way: cut off from the platform, it keeps its connections and carries on
when it reconnects.
Each relay keeps one instance identity. On its first start the relay writes
relay-instance.json into its state directory and reuses it on every
restart. Copy only credentials.json to another host, never this file: two processes with the same
identity look like one. A relay that cannot write the file refuses to start, naming the directory,
so make the state directory writable by the account the relay runs as.
Only one process can use a state directory. A relay holds a lock on its state directory while it runs. A second process on the same host pointed at the same directory — a second service, or a second container mounting the same folder — logs Another relay process is already running with state directory …; refusing to start and exits with code 73 before it opens any port. The lock is released when the process exits, however it exits. It does not reach across hosts, so a state directory on a network share is not supported.
Both exit codes are deliberate, and the installed services restart on them. The Linux and Windows
services the installers set up restart the relay about 10 seconds after either code: after a 75 it
comes back as a standby, and after a 73 it is refused again, and logs the same message, for as
long as the other process runs. A relay run as a container restarts only with a restart policy such
as --restart on-failure; without one it stays stopped after a 75.
A build from before relays took a lease is always treated as active, so a current build started beside it waits as a standby — it logs … is STANDBY: a relay build without lease support is active on this registration… — and the Edit Relay dialog marks a build from before relays reported their instances Older build. Such a build takes over only by restarting, and its stops and losses are handled as before. When you upgrade in place, stop the old relay cleanly and the new one becomes active at once; after an abrupt stop it waits up to 90 seconds.
Surviving a relay restart
A relay writes what it needs to pick up where it left off into its state directory —
RELAY_STATE_DIR, or the directory holding credentials.json when that is unset. Two files live
there, relay-state.json and the LSAM Connector's connector-state.json, and they have to share one
directory. The relay's own file is kept current as it works and is readable only by the account the
relay runs as.
What that state buys, when a relay restarts with work in flight:
- Legacy jobs that were running are picked up again rather than left untracked, so their completions still reach the platform.
- In-flight OpCon MFT, OpCon RPA and EASE runs are recovered and continue to be followed. An EASE run that was in flight is never re-added to the EASE schedule — see A relay restart resumes the run. A run whose start had been sent but whose id was not yet known is shown as Lost in Failover rather than Failed, because it may still be running — see Jobs in flight at a handover are adopted.
- A job the relay had just picked up is sent again. If the relay stops after claiming a job and before handing it to the machine, the job is dispatched again a few minutes later, when the work comes back round. Earlier builds left such a job claimed for good: it never ran, and it counted as in-flight work that kept the relay from being deleted. A job claimed before this change is not recovered this way.
- Job identifiers do not collide. The relay resumes its internal job counter past where it left off instead of restarting from zero and reusing a number a job still running was given.
Neither file holds a secret. Both carry only what identifies the work — job handles and identifiers, machine names, statuses and timestamps — never a credential, an API token, a property value, a command line or job output.
A relay killed part-way through a save still comes back with its work. Both files are written to
a temporary sibling and then renamed over the live one, so the file a restart reads is always a whole
one. Earlier relays wrote relay-state.json in place: a crash during a save left it truncated, the
restart could not read it, and — because an unreadable file was treated the same as no file at all —
everything above was lost at once. The job counter went back to zero and every in-flight legacy,
OpCon MFT, OpCon RPA and EASE run was dropped. There is nothing to turn on.
The LSAM Connector's file also survives a power loss. It is written all the way to disk before it replaces the live file, so an operating-system crash, a power cut or a host reset cannot leave it empty. Earlier connectors could come back from one with an empty file, and every legacy job that was running stayed running, never reporting its result.
A save that cannot be made stops an EASE or OpCon RPA run before it starts. The relay records an EASE add or an RPA start before it sends it, so that a restart can tell whether it went out. When that record cannot be saved — after a few quick retries if the file is only briefly locked — the job fails without sending anything, and says so:
- EASE add job not sent: the relay could not save its state file; nothing was submitted to EASE
- RPA process not started: the relay could not save its state file; nothing was run on the RPA Agent
Nothing ran, so rerun the job once the state directory can be written. Earlier relays sent the add or the start anyway, and after a restart reported that nothing had been sent while the job was running remotely, so rerunning it ran it twice.
Job results and events survive a stop
A legacy machine deletes a finished-job
status or an event it sends the moment the LSAM Connector acknowledges it. Earlier connectors
acknowledged it as soon as it was in memory, so a connector crash, a clean stop or a relay restart
before the platform had it lost the result for good, and the job never reported. The connector now writes
each finished-job status and each event to a journal on disk before it acknowledges it, and removes the entry
only once the platform has accepted it. After a restart or a failover the relay delivers whatever the
journal still holds, and a clean stop first tries to deliver it, for as long as
RELAY_JOURNAL_DRAIN_TIMEOUT_MS allows. It is on by default; set RELAY_INBOUND_JOURNAL_ENABLED to false
to turn it off.
- If the journal is full, the machine waits. At 50,000 entries or 64 MB the connector stops acknowledging the journaled frame, so the machine keeps it and sends it again once there is room. Nothing is acknowledged and then dropped.
- An event can arrive twice. If the connector dies after writing an event and before acknowledging it, the machine sends it again and it is applied again.
- An event the platform refuses is released. The platform refusing the contents of an event does not stall the rest: the relay retries the batch one event at a time, logs an error for the one it refuses and releases it.
- A job that finishes as the relay stops is reported in full. When a clean stop delivers the result of a legacy job the relay was following, it sends the job's whole output and the result the relay had worked out for it. Earlier relays sent only what the machine's final status carried, so such a job kept only its last line of output.
- Results are sent at most 8 at a time. A backlog delivered after the platform was unreachable no longer crowds out the requests that start new jobs.
Earlier Linux relays could not write either file: both went to /etc/opcon, which the installer's
systemd unit mounts read-only. Each save failed with a warning in the log and nothing else, so
everything above silently did not happen on a native Linux relay — a restart lost track of the jobs
that were running, and MFT and RPA runs in flight were not recovered. Docker and Windows relays were
never affected. Re-run the installer to pick up the state directory; no configuration of your own is
needed.
The same read-only sandbox stopped the LSAM Connector writing its own log file on Linux, which is
why that row appears in the log table (on the Logs tab) only now. Its output was
already reaching journalctl -u opcon-relay; the file is what was missing.
When the LSAM Connector restarts
The LSAM Connector runs as its own process under the relay, and the relay starts it again if it stops unexpectedly. The restarted connector holds no connections, so the relay connects it again to every legacy machine it serves. A machine it cannot reach at once is tried again at the relay's next heartbeat. Earlier relays did not, and every legacy machine behind the relay stayed down until the relay itself was restarted.
A dropped status stream is picked up again. The relay follows the connector's stream of machine and job updates — a machine going up or down, a job starting to run, and the feedback a job sends while it runs. If that stream breaks while the connector is still running, the relay subscribes again, after one second and then at doubling intervals of up to 30 seconds, until it succeeds or the relay stops. Earlier relays stopped receiving those updates until the connector next restarted.
Agents It Serves
Serving OpCon MFT agents
A relay reports on each check-in whether it can serve OpCon MFT agents. Three things follow from that report:
- Registering an OpCon MFT agent under a relay that does not support them is refused, with Failed to create agent. Upgrade the relay to register MFT agents.
- A relay that does not support OpCon MFT is never shown MFT agents at all, so an already- registered MFT agent behind a downgraded relay simply gets no work rather than failing oddly.
- An OpCon MFT job is refused rather than queued when the relay serving its agent pool has not reported MFT support. The message names the pool: the relay serving agent pool '…' has not reported MFT support. Upgrade the relay to run MFT jobs.
If you are adding your first OpCon MFT agent, upgrade the relay first. Everything else about the relay — registration, credentials, install, health — is unchanged: an MFT agent rides the relay you already have.
Serving OpCon RPA agents
A relay reports on each check-in whether it can serve OpCon RPA agents, on a separate flag from its OpCon MFT one. A relay that can serve MFT agents is not thereby able to serve RPA agents — each capability is reported and gated on its own. Three things follow from that report:
- Registering an OpCon RPA agent under a relay that does not support them is refused, with Failed to create agent. This relay must be upgraded to run OpCon RPA agents. Editing an existing RPA agent under such a relay is refused the same way (Failed to update agent. This relay must be upgraded to run OpCon RPA agents.), and looking up its tasks from the job editor fails with The relay serving this agent must be upgraded before it can look up RPA tasks. (you can still type a task in).
- A relay that does not support OpCon RPA is never shown RPA agents at all, so an already- registered RPA agent behind a downgraded relay simply gets no work rather than failing oddly.
- An OpCon RPA job is refused rather than queued when the relay serving its agent pool has not reported RPA support. The message names the pool: the relay serving agent pool '…' has not reported RPA support. Upgrade the relay to run OpCon RPA jobs.
If you are adding your first OpCon RPA agent, upgrade the relay first. Everything else about the relay — registration, credentials, install, health — is unchanged: an RPA agent rides the relay you already have.
Serving EASE agents
A relay reports, on each check-in, whether it can serve EASE agents — a third flag, independent of its OpCon MFT and OpCon RPA ones. The same three consequences follow:
- Registering an EASE agent under a relay that does not support them is refused, with Failed to create agent. This relay must be upgraded to run EASE agents. Editing an existing EASE agent under such a relay is refused the same way (Failed to update agent. This relay must be upgraded to run EASE agents.).
- A relay that does not support EASE is never shown EASE agents at all, so an already-registered EASE agent behind a downgraded relay simply gets no work rather than failing oddly.
- An EASE job is refused rather than queued when the relay serving its agent pool has not reported EASE support. The message names the pool: the relay serving agent pool '…' has not reported EASE support. Upgrade the relay to run EASE jobs.
If you are adding your first EASE agent, upgrade the relay first. Everything else about the relay is unchanged: an EASE agent rides the relay you already have. See EASE jobs.
Serving legacy LSAM agents
A relay also reports, on each check-in, which legacy LSAM platforms it can run jobs for. A current relay reports Windows, Unix, IBM i, MCP, SAP R/3 and CRM, z/OS, SQL, SAP BW and OS 2200 — the platforms that have job types. A relay that reports no list, because it was built before the report existed, is taken to run Windows, Unix, IBM i and SQL, so an older relay runs the same jobs it always has — and refuses SAP R/3 and CRM, SAP BW, z/OS, MCP and OS 2200 jobs until it is upgraded. A relay current enough to report a list but built before z/OS, MCP or OS 2200 jobs existed does not report that platform, and refuses those jobs the same way. Two things follow from the report:
- A legacy job is refused rather than queued when the relay serving its pool has not reported the job's platform. The job ends in Initialization Error, and the message names the relay, the pool and the platform: RELAY_LSAM_PLATFORM_UNSUPPORTED: relay '…' (agent pool '…') has not reported support for IBM i jobs. Upgrade the relay to run them.
- A least-tasked legacy-group job goes only to a member whose relay reports the job's platform. When the group's usable members are all on relays that do not, the job holds in Waiting: Machine rather than failing, and starts once such a member is available. A force start onto that group is refused — see When a legacy group has no dispatch target.
Registering a legacy agent is not checked against the report: any legacy LSAM type can be registered under any relay.
Events raised by the machines behind a relay
Traffic through a relay is not one-way. A job or process on a legacy LSAM machine can raise a Continuum event by writing a file of event syntax into the agent's MSGIN directory; the agent sends each line up the connection it already holds, and Continuum applies it. No inbound port is needed; each line carries a service-account credential, which Continuum verifies and checks against that account's roles.
Each legacy agent registration carries a Default event environment for this, and registration requires it for every agent type that raises events — every legacy LSAM type and EASE, but not OpCon MFT or OpCon RPA, which have no MSGIN directory. Until it is set, every event that machine raises is refused, and an agent registered before the field became required can still be unset — the Agents tab warns while any exist and offers one action, Assign environment…, to fix all of them. See Events raised by a legacy agent for the line format, which event families are applied, and where a rejected line goes.
Security and Updates
Security notes
- Keep
credentials.jsonat 0600 (Linux) or a restricted ACL (Windows); installers set this, and copying your downloaded file onto the placeholder preserves it — replacing the file another way may not. - Outbound only — no inbound internet ports.
8080(/health) is for local/orchestrator checks, not external exposure. - Treat the Client Secret (
clientSecretincredentials.json) like a password; never commit or share it.
Keeping a relay current
A relay is installed on your side of the boundary and is not upgraded for you, so it can be older than the platform it talks to. Several current behaviours depend on the relay carrying them, and each fails in a way that points at the wrong thing:
| What needs a current relay | What an older relay does instead |
|---|---|
| Exit criteria using Exit Criteria Result, a Range condition, or more than five conditions | Refuses the dispatch, so the job never starts. Loud, and the safe direction — but the job does not run until the relay is upgraded. |
| Retrieving output files from a UNIX or IBM i agent | Names the job the way only a Windows agent expects. On UNIX the agent answers with its own global log files; the platform now discards those before storing the listing, so the job's output comes back empty rather than wrong. On IBM i it returns nothing. IBM i took a second relay change after that, to read the OS/400 job number the agent reports — a relay current for UNIX may still return nothing for IBM i. |
| The lists on an IBM i job — message monitor, spool files, dynamic variables — and the IBM i - Operator Replay, Restricted Mode, FTP and File Arrival job types | Builds every IBM i job as a batch job. The four newer job types fail loudly, because they carry no Call — or, for File Arrival on a relay that already knows the other three, because it is refused as an unsupported IBM i job type. A job that carries lists is refused before it reaches such a relay — see the IBM i lists row below. See IBM i lists. |
| An IBM i job's message ID as its Exit Description | Does not pass the message ID on, so the job's Termination is blank when it finishes OK and holds the agent's failure text when it fails — and an Exit Description event has no message ID to compare. Quiet. |
| The Oracle, Other DB and MySQL SQL job types | Refuses the dispatch, naming the job type as unsupported, so the job never starts. Loud. |
| The SAP R/3 Job type | Refuses the dispatch with RELAY_LSAM_PLATFORM_UNSUPPORTED, because the relay has not reported that it can run SAP R/3 and CRM jobs. Loud. |
| The SAP BW Process Chain type | Refuses the dispatch with RELAY_LSAM_PLATFORM_UNSUPPORTED, because the relay has not reported that it can run SAP BW jobs. Loud. |
| The z/OS Batch Job type | Refuses the dispatch with RELAY_LSAM_PLATFORM_UNSUPPORTED, because the relay has not reported that it can run z/OS jobs. Loud. |
| The z/OS Started Task, z/OS REXX Exec and z/OS Console Command types, on a relay that already runs z/OS Batch Jobs | Fails the job at dispatch with Unsupported z/OS task type for TX1 translation, naming the job type, and nothing reaches the agent. Loud. |
| Restarting a z/OS job on a step, on a relay that already restarts other platforms' jobs on a step | Refuses the restarted job: it does not run, and is never run from the beginning in its place. Loud. |
| Viewing and editing a z/OS job's JCL | The JCL dialog says The relay serving this agent is too old for JCL access. Upgrade the relay to view or save JCL., and nothing is sent. Loud. |
| The z/OS Tracked Job and z/OS Queued Job types, on a relay that predates them | Fails the job at dispatch with Unsupported z/OS task type for TX1 translation. Loud. A pool whose relay has not reported that it can carry the agent's reference is refused with RELAY_LSAM_FEATURE_UNSUPPORTED. |
| A z/OS Batch Job's Dataset Pre-run rows, and its message, job/task, tape and REXX pre-runs, Step Control rows and Restart options | Refused: the job goes to Initialization Error with RELAY_LSAM_FEATURE_UNSUPPORTED, naming the relay, and nothing reaches the agent. Loud. |
| The six MCP types | Refuses the dispatch with RELAY_LSAM_PLATFORM_UNSUPPORTED, because the relay has not reported that it can run MCP jobs. Loud. |
| The MCP Copy, MCP Change and MCP Remove types, on a relay that runs MCP jobs but predates them | Refuses the dispatch with Unsupported MCP task type … (RC_MCP_UNSUPPORTED_TASK_TYPE). Loud. |
| An MCP job's Prerun | Refused: the job goes to Initialization Error with RELAY_LSAM_FEATURE_UNSUPPORTED. Loud. |
| An IBM i job's message monitors, spool files and dynamic variables | Refused: the job goes to Initialization Error with RELAY_LSAM_FEATURE_UNSUPPORTED. Loud. |
| The OS 2200 Job type | Refuses the dispatch with RELAY_LSAM_PLATFORM_UNSUPPORTED, because the relay has not reported that it can run OS 2200 jobs. Loud. |
| An OS 2200 Job's LSAM Resolved Dependencies | Refused: the job goes to Initialization Error with RELAY_LSAM_FEATURE_UNSUPPORTED. Loud. |
| A Windows Command job's Process Priority other than Normal | Refused: the job goes to Initialization Error with RELAY_LSAM_FEATURE_UNSUPPORTED, rather than running the program at Normal. Loud. |
| An OS 2200 Job on an agent whose Job Start Format is XML, the default | Refused: the job goes to Initialization Error with RELAY_LSAM_FEATURE_UNSUPPORTED, and the message offers setting the agent's Job Start Format to Non-XML instead. Jobs for agents set to Non-XML run on any relay. Loud. |
| An MS SQL DTExec job's Configuration File | Refused: the job goes to Initialization Error with RELAY_LSAM_FEATURE_UNSUPPORTED, rather than running the package without its configuration. Relay 1.24.0 is the first release that reports it. Loud. |
| Turning on Use TLS for Scheduling for an agent | Refused when you save the agent: Relay <name> has not reported support for TLS scheduling connections. Upgrade the relay. Relay 1.24.0 is the first release that reports it. Loud. |
| An agent's Check CRC setting | Refused when you save a changed value: Relay <name> has not reported support for the Check CRC setting. Upgrade the relay. Loud. |
| An agent's Requires XML Escape Sequences set away from its type's default, on Windows, SQL or MCP | The agent saves, but its jobs, SMAFT transfers included, go to Initialization Error with RELAY_LSAM_FEATURE_UNSUPPORTED, and the message offers resetting the setting to its default. Agents left at their defaults run on any relay. Loud. |
| Mark Offline stopping communication with a legacy agent | The mark is accepted and new work is held, but the relay keeps the agent's connection open. The agent's status shows Communication is not stopped. Upgrade the relay to stop communication with this agent. Every running instance of the relay must be current. |
| A legacy job's Pre-run Active status while the agent runs the job's prerun | Does not report the prerun phase, so the job reads Running through its prerun. Quiet. |
| Reading a legacy job's output files in the relay's code page (on the Runtime tab) | Reads them as plain ASCII, so every accented or other non-ASCII character in the output shows as ?. Quiet. |
| Sending an IBM i job's values as you typed them | Converts quotes, ampersands and angle brackets to XML entities, so a quoted value in a Call or an FTP Remote System reaches the machine as entity text, and an IBM i - File Arrival job refuses those characters in its File Name. |
| A Windows - File Arrival file name wrapped in double quotes | Refuses the dispatch on any double quote in the file name, so the job never starts. Loud. |
| Submitting an IBM i batch job | The name the relay sends contained a character an IBM i job name cannot hold, so the agent refused the submission and the job failed immediately with Initialization Error. |
| Sending a legacy job its workflow's schedule date | Sends the relay's own UTC date instead. West of UTC an evening dispatch therefore carries tomorrow's date, so a File Arrival window watches the wrong day and a path built from %SMA_MSLSAM_SCHEDULE_DATE% names the wrong day's file. Quiet, and wrong only some of the time, which is what makes it worth checking for. |
| Detecting a legacy agent that stops answering without disconnecting (on the Runtime tab) | Waits for the send retry loop to give up — about two and a half hours at the default response timeout — during which the agent's running jobs sit in Running with no completion and it still reads as reachable. |
| Restarting a job on a step | Collects no steps from the agent, so the Restart dialog offers none. A job restarted on a step is refused rather than run from the beginning: it goes to Initialization Error with RELAY_LSAM_FEATURE_UNSUPPORTED, naming the relay. Loud. |
| A legacy job's agent feedback and the LSAM Feedback events that watch it | Passes none of it on, so the job's Agent Feedback section stays empty and an LSAM Feedback event never fires. Quiet. |
| Listing a legacy job's output while it is still running | Reports the job's LSAM job number only when the job ends, so the Output tab has no file list to show until then. Quiet. |
The platform checks the settings a relay would otherwise drop. A relay reports which of them it
can send: restarting on a step, tracked and queued z/OS jobs, IBM i lists, MCP preruns, z/OS step
control, pre-runs and restart options, OS 2200 preruns and file dependencies, the XML job start
format for OS 2200 agents, Windows process priorities, DTExec configuration files, and an agent's
Requires XML Escape Sequences set away from its default. A job that
needs one its relay has not reported is refused with RELAY_LSAM_FEATURE_UNSUPPORTED: relay '…'
(agent pool '…') has not reported support for … Upgrade the relay. — or, where there is another way
to run the job now, Upgrade the relay, or … naming it — never run without it. Relays older than this
release report only the first two, so upgrade your relays before jobs use the others. A job on a
legacy agent group goes to a member whose relay has reported what the job needs, when there is one.
The platform does not check the remaining rows for you: nothing
distinguishes an out-of-date relay from a job with no output. When legacy behaviour looks wrong in a way only some platforms show,
check the relay build first. The Relays tab's Version column (on the Relay List tab) shows the build each relay is
running; compare it with the version Download Relay offers. A relay that has checked in and still
shows — runs a build from before relays reported their version. Both Download Relay and the register
dialog's download links always serve the latest published release, and Download Relay gets you it
without registering anything.
A legacy job's output files are named by the agent from the job name the relay sent at dispatch, and the request that lists them is built independently by the platform at read time. The two must agree exactly, so the character rule they both apply changed on both sides at once. Upgrade the relay when you take the platform release, not separately — a skew in either direction stops new jobs' output from being found, and there is no safe side to be on.
A current relay removes the need to match from here on. It reports the exact name it sent with each job's completion, and the platform keeps that name and uses it to ask for the job's output rather than working it out again. For jobs a current relay dispatched, the two can no longer disagree. Jobs dispatched before, or by an older relay, still rely on the platform working the name out.
Two consequences for jobs that ran before the upgrade:
- Their output files keep the old names on the agent's disk, permanently. Output that was already viewed stays available, because it is held after the first read. Output that was never viewed comes back empty, and no upgrade order changes that.
- Refresh from agent on such a job clears what is held and re-reads, so it turns an old cached listing into an empty one. That is deliberate: a listing the platform cannot vouch for is dropped rather than kept, which is what removed the wrong-output exposure below.
Running jobs is unaffected either way — this is about retrieving output, not dispatching work.
An output listing is now checked against the job that asked for it
The platform discards returned files whose names do not belong to the requesting job. Before this, a request that named the job in a form the agent did not recognise still got an answer: a UNIX agent appends its own global log and error files to a listing that matched nothing, and that came back as a successful, non-empty result and was stored and served as the job's output. An operator could read another job's output believing it was their own.
The check applies to UNIX listings, where the request carries the job's name and the agent names the output files from it. IBM i listings are not filtered: an IBM i listing is a set of spooled-file descriptions rather than file names, and its request carries the job number the agent reported. Windows listings are not filtered either: the file names there carry no such prefix. Files that do not match are dropped rather than the whole request failing, so a job that legitimately produced no output of its own still reports an honest empty listing.
The platform repeats the check where it stores the listing. The relay filters the listing as it receives it, and the platform now applies the same check again before it keeps the result — so a relay that does not filter, because it is older, cannot get the agent's global log stored as a job's output. A UNIX job whose listing matched nothing shows no output instead.
Troubleshooting
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
A UNIX or IBM i job's output files are empty, or hold the agent's own logfile/errfile | The relay predates per-platform output requests. The platform now discards a UNIX agent's global log before storing a listing, so new reads come back empty; a wrong list stored before that stays until refreshed | Upgrade the relay, then use Refresh from agent on any job whose wrong list was already stored. See Keeping a relay current (on the Security and Updates tab). |
| A UNIX or IBM i job that ran before a relay upgrade now lists no output files | Its files were written under the old naming rule and no longer match what is asked for | Expected, and not repairable — the names on the agent's disk do not change. Output viewed before the upgrade is still held. Re-run the job to get output under the current rule. |
| An IBM i job fails immediately with Initialization Error and the agent reports a submission failure | The relay predates the IBM i job-name character fix | Upgrade the relay — see IBM i LSAM. |
| A Windows - File Arrival job whose file name is wrapped in double quotes fails at dispatch, refused for the double quote in its file name | The relay predates accepting quoted file names | Upgrade the relay — see Quotes in the file name. |
| An Oracle, Other DB or MySQL job never starts, and its dispatch is refused naming the job type as unsupported | The relay predates those SQL job types | Upgrade the relay — see Keeping a relay current (on the Security and Updates tab). |
| An IBM i job's Termination stays blank when it finishes OK, and an Exit Description event never matches its message ID | The relay predates carrying the IBM i message ID through to the job | Upgrade the relay — see The message ID is the job's Exit Description. |
| A legacy job fails at dispatch with RELAY_LSAM_PLATFORM_UNSUPPORTED: relay '…' (agent pool '…') has not reported support for … jobs | The relay serving the job's pool has not reported that platform | Upgrade the relay — see Serving legacy LSAM agents (on the Agents It Serves tab) (Administrator). |
| A legacy job fails at dispatch with LSAM_AGENT_TYPE_MISMATCH: this job needs an agent of type … | The job's machine name points at an agent of another type behind the relay | Point the job at an agent of the type the message names — see A legacy job aimed at an agent of another type (Builder). |
| A legacy job never starts and its dispatch is refused on exit criteria | The relay predates Exit Criteria Result, range conditions, or tables over five rows | Upgrade the relay, or have the table rewritten as up to five plain conditions with Exit Criteria Result Fail. |
| Boot error: "Unrecognized key" / "relayId must be a valid UUID" / "clientId is required" | Edited/old credentials.json with wrong or missing keys | Use the file exactly as downloaded: { relayId, clientId, clientSecret }. If you no longer have it, rotate the credentials (on the Register tab) for a new one. |
Opaque 403/404 from the cloud on startup | Wrong AGENT_SERVICE_URL / missing AGENT_SERVICE_PATH_PREFIX | Use the exact connection values from the register dialog. |
Install stops: --agent-service-url is required on a first install / -AgentServiceUrl is required on a first install | A first install run without the connection URL, or a re-run that found no installed relay with one — on Windows, often a relay installed with another -ServiceName re-run without it | Pass --agent-service-url (Linux) / -AgentServiceUrl (Windows) with the value from the register dialog, or the relay's own -ServiceName. Nothing was changed. |
| A re-run is refused: … cannot be read back as --… (Linux) or Refusing to install: the installed … service's … setting cannot be read back as -… (Windows) | A setting on the existing install is not in the form the installer can keep | Pass the option the message names — see Re-running the installer (on the Install tab). Nothing was changed. |
| A Linux re-run warns Dropped relay.env settings the unit sets | relay.env held a line for a variable the systemd unit sets itself | Pass the matching option (--install-dir, --config-dir or --log-dir) instead of setting it in relay.env. |
| Windows install fails: Bundled … not found at …. The package is incomplete | Incomplete/partial relay artifact | Re-download the relay artifact. Nothing was changed, and a running relay was not stopped. |
A Windows Stop-Service OpConRelay repeats Waiting for service … to stop | The relay is finishing its shutdown, which can take up to about 35 seconds | Expected; wait for it — see How long a stop takes (on the Runtime tab). |
A Windows relay is stopped after nssm restart | The bundled NSSM gives up on a restart after a few seconds | Start it with Start-Service OpConRelay, and restart it with Restart-Service OpConRelay from then on. |
| Windows install stops with Refusing to install: … is not safe for the relay to use: … | A relay folder, a parent folder or a file in them is owned or controllable by a non-administrator, or is a link | Follow the fix the message gives — see What the installer checks (on the Install tab). |
| Windows install stops with Refusing to install: the service … already uses the display name '…' | Another service, often a second relay from an earlier installer, has the display name this install would set | Re-run install.ps1 for that relay first, with its own -ServiceName and every other option passed explicitly — see More than one relay on a host (on the Install tab). |
| Windows install aborts: the log folder "must be a dedicated relay log folder" | -LogDir or -ConnectorLogDir points at a drive root, a junction, or a folder holding something else | Point it at its own empty folder — see Logs, rotation and retention (on the Logs tab). |
| Rotated relay logs keep accumulating past the retention period | The \OpCon\OpConRelay-LogRetention task is missing, disabled, or refusing to prune | Check the task's Last Run Result. A non-zero result means it refused the folder (a junction, or an owner that is not Administrators/SYSTEM) or could not delete a file. Re-running the installer re-registers the task. |
| Old credentials still work after a suspected leak | Credentials were never rotated | Rotate credentials (invalidates the old pair, incl. refresh tokens). |
| A relay's Delete is disabled, or the delete is refused | It still has legacy agents, or work it dispatches is still in flight | Delete the relay's agents on the Agents tab, let its work finish or kill it, then delete it. If the relay is Offline, you can instead confirm Delete Relay and End Jobs — see Edit, rename and delete a relay (on the Relay List tab). |
| Deleting a busy relay says … is online, or stopped less than a minute ago, and still has jobs | Its jobs can be ended only once the relay has stopped | Stop the relay, wait one minute after it shows Offline, then delete it again (Administrator). |
| A job failed with Relay deleted before the job finished | Its relay was deleted while the job was still with it, ending the job | Run the job again on an agent behind another relay (Operator/Administrator). |
| A relay action or tab is unavailable, with You do not have permission to … | Your roles do not grant that Relays permission | Grant it on the role's Global tab — see Agents, agent pools and relays are checked (Administrator). |
| Every legacy machine behind a relay went down after the LSAM Connector restarted, and stayed down until the relay was restarted | The relay predates re-connecting machines after a connector restart | Upgrade the relay — see When the LSAM Connector restarts (on the Runtime tab). |
| A new relay stays Offline, with Last Heartbeat Never | It has not checked in since it was registered: it is not installed or started, or it cannot authenticate or reach the platform | Install and start it, and check /health and the relay logs — see Install (on the Install tab) and the boot-error rows above (Administrator). |
| A new relay's name is refused as reserved | A relay that was renamed away from that name still holds it | Choose another name. |
| Legacy agents show Unknown | The relay is stale/down | Check the relay (health, logs, connectivity) — not the individual agents. |
| A legacy job works through one relay and not through another | The relays run different builds | Compare their Version on the Relays tab (on the Relay List tab) with the version Download Relay offers, and upgrade the older one — see Keeping a relay current (on the Security and Updates tab). |
A relay that checks in shows Version — | It runs a build from before relays reported their version | Upgrade the relay. |
A relay process is healthy but runs no jobs, and logs … is STANDBY: instance … holds the relay lease; /health reads standby | Another process of the same relay, started with the same credentials.json, is active | Expected. Check the relay's instances in Edit Relay, and stop the process you did not mean to run; if it was the active one, the standby takes over. See More than one relay process (on the Runtime tab). |
| A relay exits with code 73 about every 10 seconds, logging Another relay process is already running with state directory …; refusing to start | A second service or container on the same host uses the same state directory | Stop the duplicate. See More than one relay process (on the Runtime tab). |
| A relay restarts with exit code 75, after logging Relay instance … fenced (…); restarting as standby or Relay handover requested; yielding the lease | The active process stepped down: another instance leads, it could not renew its lease while a standby was ready, or it was handed over (on the Relay List tab) to a standby. An older relay build also exits 75 when its role changes | Expected. If it runs as a container, give the container a restart policy such as --restart on-failure, or it stays stopped. |
| A relay shows No Standby on the Relays tab | It runs as one process, so nothing takes over if that process stops | Start a second copy of the relay on another host with the same credentials.json (never relay-instance.json) — see More than one relay process (on the Runtime tab). If it appears just after a handover, wait for the previous active process to restart as a standby (Administrator). |
| Hand Over is unavailable, with The active instance cannot hand over. Upgrade it, or stop it to let a standby take over. | The active process runs a relay build that cannot hand over | Upgrade the relay, or stop the active process cleanly and let the standby take over — see Hand over to a standby (on the Relay List tab) (Administrator). |
| A relay's standby has no Hand Over | The standby is not ready: it has stopped, has not checked in within the last 30 seconds, or is an older build | Check the standby's Role and Last Seen in Edit Relay, and start or upgrade it (Administrator). |
| The handover did not complete. Check the instances below, then try again. | The chosen standby did not take over before the request lapsed | Check that the standby is still running and reporting in, then hand over again — see Hand over to a standby (on the Relay List tab) (Administrator). |
A relay refuses to start, naming its state directory, because it cannot write relay-instance.json | The account the relay runs as cannot write to its state directory | Make the state directory writable by that account. |
| A relay's instances badge is in the warning colour | An older relay build, from before relays reported their instances, is running on the same registration alongside a current one | Find it in Edit Relay (Older build) and stop or upgrade it. |