Oracle Cloud Infrastructure Documentation

HeatWave Cluster Data Recovery

HeatWave cluster automatically recovers the data stored in the memory when there is a node failure, cluster restart, or DB system restart.

HeatWave cluster performs the data recovery process when:
  • One or more HeatWave nodes have failed and restarted.
    Note

    HeatWave monitors HeatWave nodes status regularly and if there is no response from a node after 60 seconds, HeatWave automatically attempts to bring the node online.
  • The HeatWave cluster is restarted.
  • The DB system is restarted.
  • High availability is enabled or disabled on the DB system.
  • A switchover or failover has occurred in the DB system with high availability enabled.
HeatWave data can be recovered from the HeatWave Storage Layer, or reloaded from the DB system and Lakehouse Object Store.
  • Recover from the HeatWave storage layer
    HeatWave first tries to recover data from the HeatWave storage layer, which is created when you enable the HeatWave cluster for the first time. To facilitate recovery, data is persisted to Object Storage when data is loaded into the HeatWave cluster and when data changes is propagated from the DB system to the HeatWave cluster. Loading data from the HeatWave storage layer is faster because the data does not need to be converted to the HeatWave storage format, as is required when loading data from the DB system or Lakehouse Object Store.
    Note

    You can monitor the status of the HeatWave cluster by checking the HeatWave cluster metric named HeatWave cluster health status. See HeatWave Cluster Metrics. The status shows 1 for RECOVERING and the status changed to 0 for HEALTHY when the recovery has completed.
    Note

    Automatic loaded tables are not recovered when the DB system has restarted, automatic. loaded tables are recovered when the DB system remains running without a restart.
  • Reload from the DB system or Lakehouse Object Store
    The data cannot be recovered from the HeatWave storage layer in these situations and all tables must be reloaded from the DB system or Lakehouse Object Store:
    • Recovery from the HeatWave storage layer has failed due to Object Storage outage or network outage.
    • The binary logs required to perform change propagation of the data restored from the HeatWave storage layer has been purged.
    • The DB system is upgraded and the data stored in the HeatWave storage layer is not compatible between the old and new versions.
    • High availability is enabled or disabled on the DB system.
    • A standalone DB system with version lower than 9.2.0 is restarted.
    • A restart, switchover, or failover has occurred in the DB system with high availability enabled.
      Note

      As of version 9.2.1, InnoDB tables can be recovered from the HeatWave storage layer. However, Lakehouse tables need to be reloaded from the Lakehouse Object Store.
    Note

    Automatic loaded tables are not reloaded.
    After the DB system restarts, these tables cannot be recovered from the HeatWave storage layer and need to be reloaded from the DB system or Lakehouse Object Store:
    • Tables with dictionary encoded columns.
    • Stale tables.
    • Tables which were not captured in periodic metadata checkpoints.
    • Tables where partition load or unload operations were in progress when the DB system had stopped abruptly.
    Note

    You can monitor the reload progress with the HeatWave cluster metric named HeatWave cluster data load progress. See HeatWave Cluster Metrics. The reload process takes time, especially if the data is large. Don't try to restart again as each restart will need to reload all the data from the beginning again.

During recovery, HeatWave cluster automatically reloads the data. However, if the MySQL Server is in the SUPER_READ_ONLY mode, you cannot load data into HeatWave cluster, and the HeatWave recovery fails. Disable the SUPER_READ_ONLY mode to load data. See Resolving SUPER_READ_ONLY and OFFLINE_MODE Issue Caused by Low Free Storage Space.

When you unload a table, the data is removed from the HeatWave cluster, and in a background operation, it is removed from the HeatWave Storage Layer in the Object Storage too.