When a kernel crash or kernel panic occurs then the running kernel runs ‘exec(dump kernel)‘ and it loads dump kernel from reserve memory and then contents of RAM and Swap is copied to more file either on local disk or on remote disk and finally reboot the box.
[root@cloud ~]# vi /etc/kdump.conf path /var/crash core_collector makedumpfile -c default reboot
[root@cloud ~]# systemctl start kdump.service [root@cloud ~]# systemctl enable kdump.service [root@cloud ~]#
[root@cloud crash]# systemctl is-active kdump.service [root@cloud crash]# service kdump status
[root@cloud ~]# echo 1 > /proc/sys/kernel/sysrq ; echo c > /proc/sysrq-trigger
This will create a crash dump file (vmcore ) under ‘/var/crash‘ file system.
[root@cloud ~]# ls -lR /var/crash /var/crash: total 0 drwxr-xr-x. 2 root root 42 Mar 4 03:02 127.0.0.1-2016-03-04-03:02:17 /var/crash/127.0.0.1-2016-03-04-03:02:17: total 135924 -rw-------. 1 root root 139147524 Mar 4 03:02 vmcore -rw-r--r--. 1 root root 35640 Mar 4 03:02 vmcore-dmesg.txt [root@cloud ~]#
To use the crash, make sure two packages are installed : ‘crash & kernel-debuginfo‘
Type ‘ps‘ command to list the Process which were running when the system got crashed.
crash> files PID: 5577 TASK: ffff88007b44f300 CPU: 0 COMMAND: "bash" ROOT: / CWD: /root FD FILE DENTRY INODE TYPE PATH 0 ffff880036b85000 ffff8800796fa540 ffff88007966f4d0 CHR /dev/pts/0 1 ffff880036b73900 ffff880068c409c0 ffff8800794a8d10 REG /proc/sysrq-trigger 2 ffff880036b85000 ffff8800796fa540 ffff88007966f4d0 CHR /dev/pts/0 10 ffff880036b85000 ffff8800796fa540 ffff88007966f4d0 CHR /dev/pts/0 255 ffff880036b85000 ffff8800796fa540 ffff88007966f4d0 CHR /dev/pts/0 crash>
Type ‘sys’ command to list the system info when it got crashed.
crash> sys KERNEL: /usr/lib/debug/lib/modules/3.10.0-327.10.1.el7.x86_64/vmlinux DUMPFILE: /var/crash/127.0.0.1-2016-03-04-14:20:06/vmcore CPUS: 1 DATE: Fri Mar 4 14:20:01 2016 UPTIME: 00:02:00 LOAD AVERAGE: 0.75, 0.48, 0.19 TASKS: 115 NODENAME: cloud.linuxtechi.com RELEASE: 3.10.0-327.10.1.el7.x86_64 VERSION: #1 SMP Tue Feb 16 17:03:50 UTC 2016 MACHINE: x86_64 (2388 Mhz) MEMORY: 2 GB PANIC: "SysRq : Trigger a crash" crash>
To get help of any command on crash prompt , type ‘help <command>‘ , example is shown below.