Analyzing Regression Cores

This document explains how to analyze core-dumps obtained from regression machines, with examples.

Download the core-tarball and extract it.
cd into directory where the tarball is extracted.

[sh]# pwd
/home/user/Downloads
[sh]# ls
build  build-install-20150625_05_42_39.tar.bz2  lib64  usr

Determine the core file you need to examine. There can be more than one core file. You can list them from './build/install/cores' directory.

[sh]# ls build/install/cores/
core.9341  liblist.txt  liblist.txt.tmp

In case you are unsure which binary generated the core-file, executing 'file' command on it will help.

[sh]# file ./build/install/cores/core.9341 
./build/install/cores/core.9341: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy'

As seen, the core file was generated by glusterfsd binary, and path to it is provided (/build/install/sbin/glusterfsd).

Now, run the following command on the core:

gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.xxx' <target, say ./build/install/sbin/glusterd>
In this case,
gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.9341' ./build/install/sbin/glusterfsd

You can cross check if all shared libraries are available and loaded by using 'info sharedlibrary' command from inside gdb.
Once verified, usual gdb commands based on requirement can be used to debug the core. bt or backtrace from gdb of core used in examples:

Core was generated by `/build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f512a54e625 in raise () from ./lib64/libc.so.6
(gdb) bt
#0  0x00007f512a54e625 in raise () from ./lib64/libc.so.6
#1  0x00007f512a54fe05 in abort () from ./lib64/libc.so.6
#2  0x00007f512a54774e in __assert_fail_base () from ./lib64/libc.so.6
#3  0x00007f512a547810 in __assert_fail () from ./lib64/libc.so.6
#4  0x00007f512b9fc434 in __gf_free (free_ptr=0x7f50f4000e50) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/mem-pool.c:304
#5  0x00007f512b9b6657 in loc_wipe (loc=0x7f510c20d1a0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/xlator.c:685
#6  0x00007f511cb8201d in mq_start_quota_txn_v2 (this=0x7f5118019b60, loc=0x7f510c20d2b8, ctx=0x7f50f4000bf0, contri=0x7f50f4000d60)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:2921
#7  0x00007f511cb82c55 in mq_initiate_quota_task (opaque=0x7f510c20d2b0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:3199
#8  0x00007f511cb81820 in mq_synctask (this=0x7f5118019b60, task=0x7f511cb829fa <mq_initiate_quota_task>, spawn=_gf_false, loc=0x7f510c20d430, dict=0x0, buf=0x0, contri=0)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:2789
#9  0x00007f511cb82f82 in mq_initiate_quota_blocking_txn (this=0x7f5118019b60, loc=0x7f510c20d430) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:3230
#10 0x00007f511cb82844 in mq_reduce_parent_size_task (opaque=0x7f510c000df0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:3117
#11 0x00007f512ba0f9dc in synctask_wrap (old_task=0x7f510c0053e0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/syncop.c:370
#12 0x00007f512a55f8f0 in ?? () from ./lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
(gdb)