Maintenance

Post-maintenance node re-entry sweep checklist (Slurm NHC)

Read-only checklist for returning a compute node to service after maintenance (MLSys 2026 Guard, arXiv 2605.17879: sweep before re-entry). Resolves hostname (FQDN or short) to its leaf uplink port via gpu_nodes (fallback: last 24 h of gpu_telemetry) and evaluates: uplink — risk-engine risk_score / severity and telemetry silence of the uplink optic; los — OpenConfig transceiver v1.0.0 RX LOS/LOL reset-safe increase over the last 1 h (only when the columns exist); gpu — latest DCGM health per GPU (detachment, XID, ECC DBE, DCGM health, thermal, roster); nccl — recent NCCL Inspector / RAS events (a non-failing ncclras -D diagnostic — OK/PASS or SKIP, which share ras_severity='info' — is counted as nccl.diag_info_checks ("OK or skipped", not a pass count), never as a RAS event or a low-bandwidth collective; a diagnostic FAIL still counts as RAS critical). Each check is pass | warn | fail | unknown; recommended is hold when any check fails (or warns with strict=true), else sweep_pass. sections=uplink is the cheap mode used by scripts/nhc/check_optical_uplink.nhc (one PostgreSQL lookup + two ClickHouse point reads): the NHC check dies when uplink.risk_score >= risk_hold or uplink.silent is true. Sections degrade to unknown (never 500) when a backend or column is missing. 404 only when the hostname is genuinely neither registered in gpu_nodes nor seen in gpu_telemetry; a lookup that could not be answered because a store is down returns 503 (lookup_backend_error) so the client's fail-mode — not its unknown-node policy — decides. Tenant-scoped keys (non-default) only resolve nodes whose switch belongs to that tenant, and every optical read is filtered by tenant_id.

get/api/maintenance/node-sweep-checklist

Request

  • Base URL: https://api.fiber.ac (the document declares no server; this is the origin it was published from)
  • URL: https://api.fiber.ac/api/maintenance/node-sweep-checklist
  • Auth: none declared

Query parameters

hostnamestring required

Compute node hostname (FQDN or short; matched against gpu_nodes.hostname)

sectionsstring

all or comma list of uplink,los,gpu,nccl — uplink alone is the cheap NHC mode

risk_holdnumber

uplink risk_score at/above which the node is held (default 60 = warning+critical)

silence_secinteger

uplink telemetry older than this is silent (default 180 s = collector watchdog floor)

nccl_minutesinteger

NCCL / RAS look-back window in minutes

strictboolean

Escalate warn checks (thermal, partial roster, RAS, LOS warn, risk warning) to hold

Response

Per-check verdicts, uplink/los/gpu/nccl detail blocks and recommended sweep_pass | hold

unknown required

Changes

No recorded changes to this operation across all 1 revision of this API.