Using Cluster Health Check Components

To get started, let's identify where OCI health checks can help:

• Is this node reporting a health problem? Use OCA passive checks to examine the node's current state and obtain fault information and recommended actions.
• Is this cluster's GPU and network healthy before I run my next workload? Use CHC active checks on selected nodes before a large job, after a configuration change, or when investigating a failure.
• Can OKE recover a node without an operator performing every step? Configure OKE GPU host repair for supported signals and actions, with controls for workload disruption.

For a new cluster, use the CHC integration in the applicable OCI HPC stack and review which components that stack enables. For an existing cluster, update or enable OCA independently and install the scheduler-specific CHC add-on when you need active tests. You don't need to rebuild your cluster to add active validation.

The components you select work together to help you monitor node health, investigate issues, and verify recovery through the following workflow.

1. Observe node health. Oracle Cloud Agent (OCA) runs passive checks and reports detected conditions. The checks compare the observed hardware and software state with expectations for the shape.
2. Validate selected capacity. You submit active tests through Slurm or Kubernetes. Cluster Health Checks (CHC) tests return results, diagnostic evidence, and recommendations where available.
3. Decide what action to take. Review the affected node, the failed check, and the recommended action. A configuration issue, a recoverable instance fault, and hardware that needs OCI repair require different responses.
4. Recover and verify. Your operator workflow or an enabled Oracle Kubernetes Engine (OKE) repair policy performs the supported action. Review a fresh passive result and, when appropriate, rerun an active test before returning the node to normal workload use.

Passive checks and active checks provide different evidence. A passive pass doesn't establish performance under load. An active pass applies to the selected test, nodes, and time of the run. A failed active test doesn't, by itself, authorize OKE to reboot a node.

To learn about the health checks available for your GPU shape, review the test catalog and test enablement matrix. These documents are included with the container image and are the definitive reference for the checks provided by that specific release.

• See /usr/share/doc/oci-dr-hpc-v2/test_catalog.md for the names and purpose of individual engine checks.

  The full Dr. HPC v2 catalog is broader than the tests packaged for any one Slurm or Kubernetes release. The test catalog describes the passive and active tests, their configuration options, how they run, and the tools they require.

  For example, the catalog identifies nccl_multi_node_fast and nccl_multi_node_quick as presets of nccl_multi_node, rather than separate engine tests. The Kubernetes nccl_multi_node_smoke example is a scheduler-specific suite name. Use the scheduler-specific support matrix before selecting a test.

• See /usr/share/doc/oci-dr-hpc-v2/test_enablement_matrix.md to view the supported GPU shapes for active tests.

  Your Oracle Cloud Agent (OCA) integration and node shape determine which active checks are available and enabled.