Linux Agent Recovery¶

Use this guide to localize a Linux swarm-agent failure without exposing host credentials, customer details, raw logs, or control-plane payloads. These checks match the checked-in installer and agent runtime. For Windows, use the Windows service guide; its service contract is different.

Installed Contract¶

Item	Default	Ownership and mode
Agent	/opt/swarm-agent/swarm-agent	root:root, 0755
Environment	/etc/swarm-agent/swarm-agent.env	root:root, 0600
Identity	/var/lib/swarm-agent/identity.json	state directory root:root, 0700; never display identity contents
Deployment data	/var/swarm/deployments	root:root, 0755 when created
Unit	/etc/systemd/system/swarm-agent.service	root:root, 0644

swarm-agent.service runs /opt/swarm-agent/swarm-agent, reads the environment file, and defaults to https://swarmhosts.com. Its allowlisted capabilities are CAP_NET_ADMIN, CAP_NET_RAW, and CAP_CHOWN; Restart=always retries after a three-second delay. The installer also provisions ip, iptables, sysctl, and wg, which the agent preflights before advertising symmetric managed-relay routing.

Reinstalling or updating preserves the configured identity and deployment data paths. It replaces the downloaded binary, environment file, and systemd unit with installer-generated content. There is no supported Linux rollback.

First Registration¶

The preferred path is the hidden interactive token prompt. For unattended installation, SWARM_USER_TOKEN is the non-interactive, root-only environment fallback. Successful registration creates the identity file; never print or copy its JSON.

After detecting identity creation, the installer removes only the token line and restarts the service. If no identity appears after 60 seconds, it leaves the token in the mode-0600 environment file for retries. Check presence only:

sudo awk -F= '$1 == "SWARM_USER_TOKEN" { found=1 } END { print found ? "token-key-present" : "token-key-absent" }' /etc/swarm-agent/swarm-agent.env

Read-Only Decision Tree¶

Stop at the first layer that disagrees with the expected state. Every command returns fixed fields, fixed-path metadata, or bounded categories.

1. Unit and process:

   bash sudo systemctl show swarm-agent.service --no-pager --property=LoadState,ActiveState,SubState,UnitFileState,MainPID,ExecMainCode,ExecMainStatus,NRestarts

   Expect loaded, active, and running. A changing PID, nonzero status, or rising restart count indicates startup failure.

2. Identity presence and metadata only:

   bash sudo stat -c 'identity type=%F owner=%U group=%G mode=%a bytes=%s' /var/lib/swarm-agent/identity.json

   Missing or empty means registration has not completed. If present, check Swarm hosts in the UI. No host suggests registration visibility; an old heartbeat suggests heartbeat delivery.

3. Control-plane reachability:

   bash curl --silent --show-error --output /dev/null --connect-timeout 5 --max-time 10 --write-out 'control-plane http=%{http_code} seconds=%{time_total}\n' https://swarmhosts.com/

   HTTP 000 indicates DNS, TLS, routing, or egress trouble. Do not query agent API endpoints or capture request/response payloads.

4. Selected runtime readiness. Run only the runtime selected in the UI:

   ```bash sudo docker info --format 'runtime=docker version={{.ServerVersion}} driver={{.Driver}} containers={{.Containers}}'

   Or:¶

   sudo podman info --format 'runtime=podman version={{.Version.Version}} graph={{.Store.GraphDriverName}}' ```

   Failure explains runtime-blocked tasks. Never dump container configuration or environments.

5. Managed-relay readiness. If the UI says the host does not support symmetric WireGuard routing, verify the fixed prerequisite fields without displaying route or peer data:

   bash for tool in ip iptables sysctl wg; do command -v "$tool" >/dev/null && printf '%s=present\n' "$tool" || printf '%s=missing\n' "$tool"; done sudo systemctl show swarm-agent.service --no-pager --property=AmbientCapabilities,CapabilityBoundingSet

   Expect every tool to be present and both capability fields to include cap_net_admin. Rerunning the current installer repairs packages and the generated unit while preserving the host identity and deployment data.

6. Task polling symptoms. Compare only UI state: latest heartbeat, runtime readiness, and task state. A recent heartbeat with pending_deploy suggests dispatch/capacity; dispatched without change suggests polling/execution. Never copy raw task or API payloads.

7. Data-root capacity and permission metadata:

   bash sudo stat -c 'data-root type=%F owner=%U group=%G mode=%a' /var/swarm/deployments df --output=pcent,avail /var/swarm/deployments | tail -n 1 | awk '{print "data-root used=" $1 " available-kib=" $2}'

   The output omits filesystem and device names.

8. Published port metadata only:

   ```bash sudo docker ps --format '{{.Ports}}' | head -n 50

   Or:¶

   sudo podman ps --format '{{.Ports}}' | head -n 50 ```

   Compare protocols and assigned ports with the UI. Do not collect names, IDs, labels, mounts, paths, or environments.

9. Direct versus relay. Record from the UI only endpoint type (direct or relay), reachability state, protocol, and port—not backend addresses. Direct failure with relay success suggests firewall/NAT routing. Both failing with a healthy agent suggests the listener, protocol, or upstream network.

Bounded Support Package¶

Copy only outputs from steps 1–8 and this fixed event-category summary. The journal input is capped at 200 records and reduced locally; no line is emitted:

sudo journalctl --unit=swarm-agent.service --lines=200 --no-pager --output=cat |
  awk 'BEGIN{IGNORECASE=1} /register/{r++} /heartbeat/{h++} /task|poll/{t++} /runtime|docker|podman/{c++} /error|failed|exception/{e++} END{printf "events registration=%d heartbeat=%d task=%d runtime=%d error=%d\n",r,h,t,c,e}'

Before sharing, confirm the package contains no environment contents, token values, identity JSON, container environments, raw task/API payloads, journal lines, customer names or paths, or backend addresses. Use generic labels.

Approval-Required Mutations¶

Everything above is read-only. Obtain host-owner approval before installer reruns; environment edits; identity deletion or replacement; forced registration; token rotation; service lifecycle actions; runtime installation, restart, or prune; network changes; data-root ownership or permission changes; or deployments. Preserve evidence first and make one approved change at a time.