Collecting Crash Dumps Using Kdump Utility

If you are using an Oracle Linux platform image, Kdump is installed and either fully configured or partially configured. You can change the memory amount that is reserved on the kernel to save the crash dump, also called a crashkernel memory reservation. In Oracle Linux 8, and earlier, the default memory reservation is set to adjust automatically: GRUB_CMDLINE_LINUX="crashkernel=auto". However, crashkernel=auto is not supported for Oracle Linux 9, so you must set a specific amount of reserved memory using the crashkernel parameter.

To set the memory reservation for a crash dump:

1. From a command line, use your administrative privileges and connect to the instance using SSH.
2. Edit the /etc/default/grub file to set the reserved memory. For example:
   • Set the memory reserve to 64 MB, for example:

     GRUB_CMDLINE_LINUX="crashkernel=64MB"

   • Set the amount of reserved memory as a variable using the crashkernel=<range1>:<size1>,<range2>:<size2> syntax. For example:

     GRUB_CMDLINE_LINUX="crashkernel=512M-2G:64M,2G-:128M"

   • Define an offset value for the reserved memory. Because the crashkernel reservation occurs early in the boot process, some systems require that you reserve memory with a certain fixed offset. When a fixed offset is specified, the reserved memory begins at that point. For example, to reserve 128 MB of memory, starting at 16 MB:

     GRUB_CMDLINE_LINUX="crashkernel=128M@16M"

3. Save the changes and refresh the grub configuration:

   sudo grub2-mkconfig -o /boot/grub2/grub.cfg

4. Reboot the instance to apply changes.

Using the /etc/kdump.conf, you can change the location in which the crash dump files are saved, transfer them via SSH or export them to a network share.

1. From a command line, use your administrative privileges and connect to the instance using SSH.
2. Edit the configuration file at /etc/kdump.conf file and remove the # comment character at the beginning of each line that you want to enable.
   • Change the default directory (/var/oled/crash) for the crash dump files, for example:

     path /usr/local/cores

   • Transfer crash dump files over a secure shell connection, for example:

     ssh <user@example.com>
     sshkey /root/.ssh/<mykey>

   • Set the crash dump files to be exported to a compatible network share, for example:

     nfs <example.com>:/<output>

   See the kdump.conf.5 file in the /usr/share/man/man5/kdump.conf.5.gz archive for more information.

3. When you have finished, save the changes and restart the kdump service.

   sudo systemctl restart kdump.service 

4. Reboot the instance

By default, if Kdump fails to send its result to the configured output locations, it reboots the server. This action deletes any data that has been collected for the dump. To prevent this outcome, change the Kdump configuration.

1. From a command line, use your administrative privileges and connect to the instance using SSH.
2. Edit /etc/kdump.conf to uncomment and change the default value in the file as follows:

   default dump_to_rootfs

   The dump_to_rootfs option tries to save the result to a local directory, which can be useful if a network share is unreachable. You can use shell instead to copy the data manually from the command line.

   Note
   
   

   The poweroff, restart, and halt options are also valid for the default kdump failure state. However, performing these actions causes you to lose the collected data if those actions are performed. See the kdump.conf.5 file in the /usr/share/man/man5/kdump.conf.5.gz archive for more information.

3. When you have finished, save the changes and restart the kdump service.

   sudo systemctl restart kdump.service

4. Reboot the instance.

Test the Kdump configuration by crashing the kernel which triggers the service to collect a crash dump. Then, review the crash dump.

1. From the command line, connect to the instance using SSH
2. Make sure Kdump is running:

   systemctl is-active kdump

3. Initiate the crash from the console or command line:

   echo 1 > /proc/sys/kernel/sysrq
   echo c > /proc/sysrq-trigger

   This forces the kernel to crash and the dump files are copied into the /var/oled/crash/<ip-address>-<YYYY-MM-DD>-<HH:MM:SS> directory, by default, or to the location you have selected in the configuration.
4. Reboot the instance.
5. Review the crash dump files in the /var/oled/crash/<ip-address>-<YYYY-MM-DD>-<HH:MM:SS>:
   • The kernel message buffer includes the most essential information about the system crash and it is always dumped first in to the vmcore-dmesg.txt file. This is useful when an attempt to get the full vmcore file fails, for example because of lack of space on the target location.
   • As the kexec tool boots into the second kernel and captures the contents of the crashed kernel’s memory, it also writes to the kexec-dmes.log file so you can trace the process. For example, at the end of the file you can see the crash dump save process:

     ...
     Feb 07 16:28:19 linux9 systemd[1]: Starting Kdump Vmcore Save Service...
     Feb 07 16:28:19 linux9 kdump[504]: Kdump is using the default log level(3).
     Feb 07 16:28:19 linux9 kdump[541]: saving to /kdumproot/var/oled/crash/127.0.0.1-2025-02-07-16:28:19/
     Feb 07 16:28:19 linux9 kdump[546]: saving vmcore-dmesg.txt to /kdumproot/var/oled/crash/127.0.0.1-2025-02-07-16:28:19/
     Feb 07 16:28:19 linux9 kdump[552]: saving vmcore-dmesg.txt complete
     Feb 07 16:28:19 linux9 kdump[554]: saving vmcore
     Feb 07 16:28:21 linux9 kdump.sh[555]: 
     Checking for memory holes                         : [  0.0 %] /                  
     ...
     Copying data                                      : [100.0 %] \           eta: 0s
     Feb 07 16:28:21 linux9 kdump.sh[555]: The dumpfile is saved to /kdumproot/var/oled/crash/127.0.0.1-2025-02-07-16:28:19//vmcore-incomplete.
     Feb 07 16:28:21 linux9 kdump.sh[555]: makedumpfile Completed.
     Feb 07 16:28:21 linux9 kdump[559]: saving vmcore complete
     Feb 07 16:28:21 linux9 kdump[561]: saving the /run/initramfs/kexec-dmesg.log to /kdumproot/var/oled/crash/127.0.0.1-2025-02-07-16:28:19//

   • The vmcore file contains the crash dump information. To analyze the crash dump, you need a utility that can read the vmcore file format. See Analyzing Crash Dumps for information on using the crash utility.

You can use the crash utility to analyze the crash dumps collected by Kdump. In Oracle Linux platform images, crash is installed by default. For other Linux instances, use the command line to install it: sudo dnf install crash.