Active Health Checks with CHC

Cluster Health Checks (CHC) active tests place GPUs and network connections under load to identify hardware faults and performance issues.

Types of Active Tests

Active tests consume GPU, host, or network resources. Select a test based on the issue you want to investigate, and confirm that it supports your GPU shape.


Test category	Examples and test names	When to use it
GPU diagnostics and stress	NVIDIA DCGM (dcgm; Kubernetes suite example: dcgm:1), GPU Fryer (gpu_fryer), GPU burn (gpuburn), silent-data-corruption checks (gpu_sdc_checker), AMD RVS (rvs), AMD Field Health Check (agfhc) and MiniHPL (minihpl), and host stress scenarios (sunvts).	Investigate GPU stability, compute correctness, or faults that appear under sustained load.
Data movement	PCIe transfers (gpu_bandwidth) and single-node or multi-node NVBandwidth (nvbandwidth_single, nvbandwidth_multi).	Validate transfers between the host and GPUs or among GPUs.
RDMA and network performance	RDMA write bandwidth/latency (ibwrite), network throughput (iperf), and Mellanox port and cable inspection (mlxlink_check).	Investigate connectivity or throughput between selected nodes.
Collective communication	NVIDIA NCCL within a node (nccl_single), across nodes over RDMA (nccl_multi_node), or over rack-local NVLink (nccl_multi); AMD RCCL (rccl_single , rccl_multi). Pre-flight examples: nccl_multi_node_fast in Slurm and nccl_multi_node_smoke in Kubernetes.	Validate the communication paths used by distributed jobs. Short presets provide a pre-flight check; broader runs provide deeper evidence.
Training workload validation	TorchTitan training (ml-training-job in Dr. HPC v2 and Kubernetes; ml_training_job in the Slurm oci-chc command).	Exercise a training workflow after the basic device and communication checks pass. Additional images and scheduler resources are required.

Test Catalog and Enablement Matrix

To learn about the health checks available for your GPU shape, review the test catalog and test enablement matrix. These documents are included with the container image and are the definitive reference for the checks provided by that specific release. See Find Tests Supported by Your Configuration.

Before Running Active Tests

Identify the exact nodes to test and record their shape, image, driver, and CHC release.
Reserve suitable capacity or select idle nodes according to your cluster's scheduling policy. Before starting an on-demand test, confirm that the selected nodes have enough available resources. Longer stress or training tests can disrupt other workloads that share those resources.
Ensure the nodes can access the required container images, GPU and remote direct memory access (RDMA) devices, and result destinations. Configure the documented IAM permissions for result upload and diagnosis. A Diagnosis Store associates the cluster's diagnostic results with its Object Storage bucket.
For Slurm, confirm that your controller account can allocate the selected GPU nodes and has passwordless SSH access to them. The controller requires Bash, curl, jq, Python 3, the OCI CLI, and the Slurm CLI. GPU workers require compatible drivers, Enroot, and Pyxis. Use a writable directory mounted at the same absolute path on the controller and workers. Check access to the add-on and image registry and OCI result services.
For Kubernetes, verify the active-test CRD, controller, selected template ConfigMaps, and a compatible MPI Operator for multi-node tests. Confirm idle capacity before submitting a suite. Stress tests alongside customer workloads require explicit approval. Configure the worker identity and result resources before enabling active tests.
For Kubernetes MPIJob tests and the DR-HPC-managed ML workflow, verify the qualified MPI Operator and its namespace access. An outer Job template doesn't necessarily remove this dependency. Ordinary single-node Jobs don't require MPI Operator.
Have your administrator verify the IAM policies for both the controller provisioning identity and the GPU workers' runtime identity. Public image download access doesn't grant permission to create or use your private result resources. Review the resources that oci-chc setup might create before running it.

How to Run Active Tests

You can run active tests through Slurm, through Kubernetes, or by invoking Dr. HPC v2 directly on a supported GPU host. Select the path supported by your cluster and the test you need.


Option	Set up and run	Check the result
Slurm	Install or verify the CHC add-on on the controller. Use `oci-chc test` to submit a test to selected nodes through Slurm. See Run Active Health Checks on Slurm.	Check the application result and the artifact paths printed by `oci-chc`. A completed Slurm job alone isn't a passing test.
Kubernetes	Install or verify the CHC Helm chart and active-test components. Create an active test suite that selects eligible nodes. Kubernetes schedules the workload. See Run Active Health Checks on Kubernetes.	Check the suite result, workload and pod status, logs, and diagnostic JSON. A completed workload alone isn't a passing test.
Direct Dr. HPC v2	On a supported GPU host with the oci-dr-hpc-v2 binary available, run an applicable test, for example `oci-dr-hpc-v2 active-checks --test dcgm --output-dir ./drhpc-dcgm`.	Check the active result's final status, individual checks, remediation output, and the verbose.log file in the output directory.

The direct-engine command is a separate workflow from the Slurm and Kubernetes integrations. Some direct tests require additional tools or a cluster environment. Multi-host tests also require --nodes-list. See "Direct Engine Invocation and Result Files" in /usr/share/doc/oci-dr-hpc-v2/test_catalog.md.

On OKE, GPU host repair is a separate, optional integration. Running an active test doesn't enable repair. See Repairing GPU Worker Nodes with OKE and Repairing GPU Hosts.

Interpret Results and Select the Next Action


Result	What to do
PASS	The executed check met its configured expectation. Confirm which tests and nodes ran, and review warnings or skipped checks. For Slurm, check the application's result. A completed scheduler job alone isn't a pass.
WARN	Review the reported condition and decide whether deeper validation is needed.
FAIL	Inspect the affected device, measurement, result JSON, logs, and available recommendation. Follow your isolation or repair procedure.
TEST_ERROR	The test couldn't produce trustworthy evidence. Resolve execution, dependency, permission, or parsing problems before drawing a health conclusion.
SKIP or SKIPPED	The test didn't run. It provides no evidence that this part of the node is healthy.
SUBMITTED	An asynchronous Slurm job was accepted. Inspect its final application result and artifacts after the job finishes.

For both Slurm and Kubernetes, check the job state and the diagnostic result separately. A completed Slurm job or Kubernetes workload means the scheduled work finished. It doesn't establish that Dr. HPC completed the requested tests or that those tests passed. Review the diagnostic output to confirm which tests ran and what each test reported.

Check each requested test for a result such as PASS, FAIL, or TEST_ERROR, and review any warnings or skipped checks. A skipped test provides no evidence of health, and an execution error can prevent a test from producing a diagnostic result. Result fields and file locations vary by integration, so use the logs and artifact paths reported by your Slurm or Kubernetes run. If uploading results or requesting remote diagnosis fails, preserve the local diagnostic output and investigate that failure separately from the test result.

Keep the results and logs with the test time, selected nodes, software and image versions, job or workload identifier, and any recommended actions. After a repair or configuration change, collect fresh passive-check results and rerun the relevant active tests to verify the current node state.