Linux Agent Recovery¶

Use this guide to localize a Linux swarm-agent failure without exposing host credentials, customer details, raw logs, or control-plane payloads. Use the checks below for a standard installer-created service. For Windows, use the Windows service guide.

Default installation paths¶

swarm-agent.service runs /opt/swarm-agent/swarm-agent, reads the environment file, and defaults to https://swarmhosts.com. Its service permissions include CAP_NET_ADMIN, CAP_NET_RAW, CAP_CHOWN, CAP_DAC_OVERRIDE, and CAP_FOWNER. These support relay networking, public-connection checks, and file operations in game-owned folders without changing their contents recursively. Restart=always retries after a three-second delay. The installer also provisions ip, iptables, sysctl, and wg, which the agent preflights before advertising symmetric managed-relay routing.

When Docker is the selected installer-managed runtime, the installer enables docker.service for boot startup whether Docker was newly installed or already present. The agent rechecks that enablement after Docker becomes ready and repairs it when the service account has the required root privileges.

Automatic agent updates preserve the configured identity, data paths, environment, and service unit. A full installer rerun replaces the binary, environment file, and unit with generated content. It does not read your old installation options from that environment file: use the original server, name, identity, installation, data-root, and port-range options, and privately retain and reapply custom environment settings. Do not rerun it with defaults on a customized installation. There is no supported Linux rollback.

Repair file-access capability drift¶

Automatic binary updates preserve the installed systemd unit. Older units may therefore still omit CAP_DAC_OVERRIDE and CAP_FOWNER, even after the agent binary updates successfully. The host page shows Game-owned files → Service capability update required and the agent emits a startup warning with the missing permissions. The gateway can remain reachable while edits to game-owned directories fail.

The rootful agent already controls the container runtime. Its supported service uses CAP_CHOWN, CAP_DAC_OVERRIDE and CAP_FOWNER to preserve game ownership and permissions during atomic file publication. These capabilities belong to the agent service; they are not added to game containers. A non-root/manual agent reports owner_only and can edit only files its OS user can access.

After reviewing the host's pending operations, an owner can inspect the additive repair without downloading an agent or registering again:

curl -fsSL https://swarmhosts.com/agent/install/linux.sh -o /tmp/swarmhosts-install.sh
sudo bash /tmp/swarmhosts-install.sh --repair-file-access --dry-run

Apply that reviewed change, then restart only the agent:

sudo bash /tmp/swarmhosts-install.sh --repair-file-access
sudo systemctl restart swarm-agent.service

For a custom service name, set SWARM_AGENT_SERVICE_NAME explicitly when invoking the installer. Repair requires an existing root Swarm Hosts service. It adds a service override for the two missing file permissions and reloads systemd. It preserves the original unit, all other drop-ins, binary, environment, identity, and every game file. A conflicting existing repair file is refused. A later operator override that still excludes the capabilities produces an error instead of claiming success. The repair does not restart either the agent or container runtime automatically.

Verify Game-owned files becomes Ready on the host page after a fresh heartbeat and perform a create/read/conditional-save check in a uniquely owned disposable file. Confirm the game container ID, start time, restart count, and world permissions remain unchanged. New text files and new editor directories inherit their parent owner; replacements preserve the previous file's owner and ordinary permission mode. Files with setuid, setgid, or sticky bits are refused by the file editor. Delete only the disposable test file after the check. Capability readiness does not prove a filesystem is writable, mounted, healthy, or permitted by SELinux/AppArmor; those failures remain separate diagnostics.

Managed storage recovery¶

The agent boot-validates the managed selection and keeps a mode-0600 storage-locations.json.previous copy beside it. If the newest selection is invalid or unavailable, the heartbeat reports prior_known_good while the agent continues on that validated copy. If neither managed file is usable, it reports legacy_fallback and uses --data-root, SWARM_PRIMARY_BACKUP_DIR, and SWARM_REPLICA_DIR.

Check metadata without displaying either JSON file:

sudo stat -c 'storage-config type=%F owner=%U group=%G mode=%a bytes=%s' /var/lib/swarm-agent/storage-locations.json
sudo stat -c 'storage-previous type=%F owner=%U group=%G mode=%a bytes=%s' /var/lib/swarm-agent/storage-locations.json.previous

Do not hand-edit, copy between hosts, or delete these files as a first recovery step. Confirm the effective source and error in the Swarm Host view, restore the selected local mount, and restart the agent. A location change is allowed only when the existing and selected game/backup locations contain no managed data; this feature never migrates or adopts data.

First Registration¶

Keep the identity in a directory owned by the agent account or root, with no group or other write permissions. Saving an identity never changes permissions on an existing parent directory.

Generate a one-use code from Profile → Connect a host and use the hidden installer prompt. Codes expire after 15 minutes. For unattended installation, SWARM_USER_TOKEN is the root-only environment fallback. The agent first writes a private pending-enrollment.json beside the identity file. Successful registration publishes the confirmed identity; never print or copy either JSON file.

The installer removes the code only after reading a confirmed enrolled identity, then restarts the service. A pending identity alone never triggers removal. If enrollment is still incomplete after 60 seconds, the code remains in the mode-0600 environment file for retries. Keep the pending file when restarting: a response lost after enrollment can resume even after code expiry. If the code expired before enrollment, generate a fresh code and rerun the installer. A rejected existing identity requires deliberate recovery; it never silently re-enrolls. Check presence only:

sudo awk -F= '$1 == "SWARM_USER_TOKEN" { found=1 } END { print found ? "token-key-present" : "token-key-absent" }' /etc/swarm-agent/swarm-agent.env

Find the failure with read-only checks¶

Stop at the first layer that disagrees with the expected state. Every command returns fixed fields, fixed-path metadata, or bounded categories.

1. Unit and process:

   bash sudo systemctl show swarm-agent.service --no-pager --property=LoadState,ActiveState,SubState,UnitFileState,MainPID,ExecMainCode,ExecMainStatus,NRestarts

   Expect loaded, active, and running. A changing PID, nonzero status, or rising restart count indicates startup failure.

2. Identity presence and metadata only:

   bash sudo stat -c 'identity type=%F owner=%U group=%G mode=%a bytes=%s' /var/lib/swarm-agent/identity.json

   A missing or empty identity means registration has not completed. If present, check Swarm hosts in the app. Confirm you are signed in to the account that enrolled the host. An old heartbeat means the saved identity alone does not prove the host is currently connected.

3. Control-plane reachability:

   bash curl --silent --show-error --output /dev/null --connect-timeout 5 --max-time 10 --write-out 'control-plane http=%{http_code} seconds=%{time_total}\n' https://swarmhosts.com/

   HTTP 000 indicates DNS, TLS, routing, or egress trouble. Do not query agent API endpoints or capture request/response payloads.

4. Selected runtime readiness. Run only the runtime selected in the UI:

   ```bash sudo docker info --format 'runtime=docker version={{.ServerVersion}} driver={{.Driver}} containers={{.Containers}}'

   Or:¶

   sudo podman info --format 'runtime=podman version={{.Version.Version}} graph={{.Store.GraphDriverName}}' ```

   Failure explains runtime-blocked tasks. Never dump container configuration or environments.

   For Docker, also confirm reboot persistence without changing it:

   bash sudo systemctl is-enabled docker.service

   Expect enabled. If it is disabled, sudo systemctl enable docker.service restores startup at boot without reinstalling the agent. Check why it was disabled before changing a host you manage with other tools.

5. Managed-relay readiness. If the UI says the host does not support symmetric WireGuard routing, verify the fixed prerequisite fields without displaying route or peer data:

   bash for tool in ip iptables sysctl wg; do command -v "$tool" >/dev/null && printf '%s=present\n' "$tool" || printf '%s=missing\n' "$tool"; done sudo systemctl show swarm-agent.service --no-pager --property=AmbientCapabilities,CapabilityBoundingSet

   Expect every tool to be present and both capability fields to include cap_net_admin. A missing package or service permission needs repair. If rerunning the installer, follow the custom-setting precautions above; use the file-access repair only for its documented warning.

6. Task polling symptoms. Compare only UI state: latest heartbeat, runtime readiness, and task state. A recent heartbeat with pending_deploy suggests dispatch/capacity; dispatched without change suggests polling/execution. Never copy raw task or API payloads.

7. Data-root capacity and permission metadata:

   bash sudo stat -c 'data-root type=%F owner=%U group=%G mode=%a' /var/swarm/deployments df --output=pcent,avail /var/swarm/deployments | tail -n 1 | awk '{print "data-root used=" $1 " available-kib=" $2}'

   The output omits filesystem and device names.

8. Published port metadata only:

   ```bash sudo docker ps --format '{{.Ports}}' | head -n 50

   Or:¶

   sudo podman ps --format '{{.Ports}}' | head -n 50 ```

   Compare protocols and assigned ports with the UI. Do not collect names, IDs, labels, mounts, paths, or environments.

9. Direct versus relay. Record from the UI only endpoint type (direct or relay), reachability state, protocol, and port—not backend addresses. Direct failure with relay success suggests firewall/NAT routing. Both failing with a healthy agent suggests the listener, protocol, or upstream network.

Collect safe support details¶

Copy only outputs from steps 1–8 and this fixed event-category summary. The journal input is capped at 200 records and reduced locally; no line is emitted:

sudo journalctl --unit=swarm-agent.service --lines=200 --no-pager --output=cat |
  awk 'BEGIN{IGNORECASE=1} /register/{r++} /heartbeat/{h++} /task|poll/{t++} /runtime|docker|podman/{c++} /error|failed|exception/{e++} END{printf "events registration=%d heartbeat=%d task=%d runtime=%d error=%d\n",r,h,t,c,e}'

Before sharing, confirm the package contains no environment contents, token values, identity JSON, container environments, raw task/API payloads, journal lines, customer names or paths, or backend addresses. Use generic labels.

Before making recovery changes¶

The decision-tree commands inspect state; the earlier repair commands change service permissions and restart the agent. Save the relevant diagnostic output and review pending operations before making a recovery change. If you are helping with someone else's host, agree on the change with its owner first. Preserve identities and worlds, and change one thing at a time so you can verify the result.

Private LAN replica recovery¶

If a Swarm Zone reports degraded replication, read the current private address and LAN backup gateway port from the owner-only Swarm Host Overview page. Confirm that LAN backup replication says Ready and LAN replication capability says Reported. On the source host, confirm that the exact reported port is listening; do not infer it from the assigned game port range:

sudo ss -ltn | awk -v port=8085 '$4 ~ ":" port "$" {print "gateway=listen"}'
curl --fail --max-time 5 http://192.168.1.20:8085/healthz >/dev/null && printf 'peer-gateway=reachable\n'

Replace the example private address and port with the reported values. Permit that TCP port only between zone peers in the host firewall. Both peers need a fresh heartbeat that reports LAN backup replication as ready. The default archive ceiling is 64 GiB, and a stalled network read times out after 120 seconds. Cross-filesystem or restricted filesystems may fall back from a hard-link to a full local copy, increasing disk and write-I/O requirements. When both peers support incremental backup transfer, only missing backup chunks cross the LAN. Otherwise, the encrypted full archive crosses the LAN. Zone backup data does not use the public game relay.