NCCL

Ingest NCCL RAS (Reliability/Availability/Serviceability) events

Accept NCCL RAS subsystem events (NCCL 2.24+) — rank_hang, op_timeout, init_failure, collective_mismatch, comm_abort, watchdog_timeout. Events are stored alongside Inspector telemetry and become available to /api/nccl/correlation for L1 (optical) ↔ L2 (IB) ↔ application-health (RAS) cross-layer root-cause evidence — FIBER's 3-layer fault chain. Payload: {"events":[{"ts","comm","rank","node","ras_event","ras_severity","hang_duration_sec"}, ...]}.

NCCL 2.31.2 RAS diagnostics: add an optional diagnostics field — either the raw text of ncclras -D / NCCL_RUN_RAS_DIAGNOSTICS=1 output (lines NCCL DIAG [OK] <check> / NCCL DIAG [FAIL] <check>: <detail>; the NCCL log prefix host:pid:tid [dev] is understood and sets the node) or a list of {check, status: OK|FAIL|WARN|SKIP, detail, node?}. Each check becomes one nccl_events row with ras_event_type=diag_<check> (gpu_inventory, cuda_driver, volatile_ecc, nvlink_state, nccl_env_consistency, ...), ras_severity critical for FAIL / warning for WARN / info for OK, ras_detail = the reason. Top-level node, ts, job are defaults for the diagnostics. events may be empty when only diagnostics are posted.

post/api/nccl/ras

Request

  • Base URL: https://api.fiber.ac (the document declares no server; this is the origin it was published from)
  • URL: https://api.fiber.ac/api/nccl/ras
  • Auth: none declared

Request body

object required

Response

Insert + breakdown counts (RAS critical / warning, ib errors, diag_checks / diag_failed)

unknown required

Changes

No recorded changes to this operation across all 1 revision of this API.