fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
sscox has joined #systemtap
hpt has joined #systemtap
yogananth has quit [Ping timeout: 265 seconds]
yogananth has joined #systemtap
orivej has quit [Ping timeout: 268 seconds]
orivej has joined #systemtap
orivej has quit [Ping timeout: 268 seconds]
hpt has quit [Ping timeout: 260 seconds]
hpt has joined #systemtap
sapatel has quit [Remote host closed the connection]
hpt has quit [Ping timeout: 260 seconds]
hpt has joined #systemtap
hpt has quit [Ping timeout: 260 seconds]
hpt has joined #systemtap
khaled has joined #systemtap
hpt has quit [Ping timeout: 240 seconds]
orivej has joined #systemtap
khaled has quit [Remote host closed the connection]
khaled has joined #systemtap
sscox has quit [Ping timeout: 240 seconds]
tromey has joined #systemtap
sscox has joined #systemtap
yogananth has quit [Remote host closed the connection]
magrawal_ has joined #systemtap
<magrawal_> @all I need one small help on systemtap
<magrawal_> can anyone help me?
<magrawal_> I am trying to probe mutex in systemtap, i am getting dwarf error even glibc-devel && debug packages are installed
<fche> hi
<fche> magrawal_, what's the stap script and the error?
<fche> (and distro/etc. version)
<magrawal_> fche, i am probe mutex like below
<magrawal_> probe process("/usr/lib64/libglusterfs.so.0").library("/usr/lib64/libpthread-2.17.so").function("pthread_mutex_lock").return {
<magrawal_> // Note: this is lock.return so at this point the current thread is holding
<magrawal_> // the lock. No other thread can modify mutex_waits[$mutex] till this thread
<magrawal_> // calls unlock.
<magrawal_> if (target() == pid()) {
<magrawal_> contended = thread_contended[tid()];
<magrawal_> if (contended) {
<magrawal_> delete thread_contended[tid()]
<magrawal_> mutex_waits[$mutex] <<< (gettimeofday_us() - contended);
<magrawal_> if (@count(mutex_waits[$mutex]) == 1) { // First contention of this mutex
<magrawal_> printf ("mutex %p: contended\n", $mutex)
<magrawal_> // We need to print the stack trace here, not in end() to get a useful trace.
<magrawal_> mutex_stacks[$mutex] = sprint_ubacktrace()
<magrawal_> }
<magrawal_> }
<magrawal_> }
<magrawal_> }
<magrawal_> glusterfs.so is our product library
<magrawal_> semantic error: while processing probe process("/usr/lib64/libglusterfs.so.0.0.1").library("/usr/lib64/libpthread-2.17.so").function("pthread_mutex_lock") from: process("/usr/lib64/libglusterfs.so.0").library("/usr/lib64/libpthread-2.17.so").function("pthread_mutex_lock")
<magrawal_> i am getting above error
<magrawal_> semantic error: debuginfo scope not found for module '/usr/lib64/libpthread-2.17.so', cannot resolve context variable [man error::dwarf]: identifier '$mutex' at :39:23
<magrawal_> source: if (++mutex_locks[$mutex] > 1) // Contended
<fche> ok, nice
<magrawal_> glibc-debuginfo(devel|headers) all packages are installed
<fche> what's your distro?
<magrawal_> CentOS Linux release 7.7.1908
<magrawal_> systemtap-devel-4.0-11.el7.x86_64
<fche> what's your glibc version? (rpm -qa glibc'*')
<fche> interestingly, if you could get hold of newer elfutils, systemtap could download centos7 glibc debuginfo from our server
<fche> if that were missing
<magrawal_> glibc-debuginfo-common-2.17-292.el7.x86_64
<magrawal_> glibc-headers-2.17-292.el7.x86_64
<magrawal_> glibc-2.17-292.el7.x86_64
<magrawal_> glibc-common-2.17-292.el7.x86_64
<magrawal_> glibc-debuginfo-2.17-292.el7.x86_64
<magrawal_> glibc-devel-2.17-292.el7.x86_64
<fche> but you seem to have the right info
<magrawal_> yes
<fche> stap -L 'process("/usr/lib64/libpthread-2.17.so").function("pthread_mutex_lock")' <<< does that work for you?
<magrawal_> stap -L 'process("/usr/lib64/libpthread-2.17.so").function("pthread_mutex_lock")'
<magrawal_> process("/usr/lib64/libpthread-2.17.so").function("pthread_mutex_lock")
<magrawal_> not showing any variable names like mutex
<fche> process("/usr/lib64/libpthread-2.17.so").function("__pthread_mutex_lock@../nptl/pthread_mutex_lock.c:61") $mutex:pthread_mutex_t*
<fche> (that's here, on the same version of glibc on a rhel7 box)
<fche> your output lacks even the @../nptl ... source part
<magrawal_> yes
<fche> it's as though the debuginfo is just not correct somehow
<magrawal_> what could the reason of this
<fche> or not being used
<fche> do you have some wacky STAP_DEBUGINFO_PATH or something ?
<magrawal_> No
<magrawal_> do i need to update something in $PATH
<magrawal_> current PATH variable is /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
<fche> strace -eopen,openat stap -L 'process("/usr/lib64/libpthread-2.17.so").function("pthread_mutex_lock")' 2>&1 | grep debug
<fche> to see if it finds your glibc-debuginfo file
<magrawal_> open("/usr/lib/debug/.build-id/8b/33f7f8c86f8d544c63c5541a8e42b3ddfef8b1.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
<magrawal_> open("/var/cache/abrt-di/usr/lib/debug/.build-id/8b/33f7f8c86f8d544c63c5541a8e42b3ddfef8b1.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
<magrawal_> open("/usr/lib64/.debug/libpthread-2.17.so.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
<magrawal_> open("/usr/lib64/libpthread-2.17.so.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
<magrawal_> open("/usr/lib64/libpthread-2.17.so.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
<magrawal_> open("/usr/lib/debug/usr/lib64/libpthread-2.17.so.debug", O_RDONLY) = 3
<magrawal_> open("/var/cache/abrt-di/usr/lib/debug/usr/lib64/libpthread-2.17.so.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
<magrawal_> open("/var/cache/abrt-di/usr/lib/debug/lib64/libpthread-2.17.so.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
<magrawal_> open("/var/cache/abrt-di/usr/lib/debug/libpthread-2.17.so.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
<fche> that first open should have worked
<fche> can you do an rpm -V glibc-debuginfo or rpm -qf glibc-debuginfo | grep 33f7f8
<fche> might you have a mixture of rhel and centos or homegrown binaries ?
<fche> btw I show that particular file as present in glibc-debuginfo-2.17-292.el7.x86_64.rpm
<fche> so yeah our debuginfod server could give it to your copy of systemtap on demand :)
<fche> https://copr.fedorainfracloud.org/coprs/fche/elfutils/ <<< includes a rhel7 build of apprx. git-master elfutils
<magrawal_> i am using rhel-7
<fche> that's a centos buildid
<fche> epel7 works on centos7 too
<magrawal_> yes
<magrawal_> it shows centos buildid
<magrawal_> for rhel-7 also
<fche> maybe you have a rhel glibc-debuginfo
<magrawal_> rpm -qf glibc-debuginfo-2.17-292.el7.x86_64 no output
<magrawal_> yes
<fche> yeah, that would not work
<fche> you need a matching set
<fche> sorry I meant rpm -ql glibc-debuginfo
<fche> but anyway rpm -qi glibc and rpm -qi glibc-debuginfo should show matching build hosts, vendors etc.
<fche> if you have rhel glibc-debuginfo mixed with a centos glibc .... yeah not going to work (buildids mismatch)
<magrawal_> ok
<magrawal_> thanks for helping , i will reinstall centos glibc-debuginfo
<fche> (again, consider installing that new version elfutils; then systemtap can download these sorts of files from our debuginfod.systemtap.org server)
<magrawal_> thanks it is working fine
<magrawal_> fche++
<fche> very good
magrawal_ has quit [Remote host closed the connection]
<fche> will add a blurb to the error::dwarf2 man page to cover this scenario
irker201 has joined #systemtap
<irker201> systemtap: fche systemtap.git:refs/heads/master * release-4.2-40-g91ffb97ad / man/error::dwarf.7stap man/warning::debuginfo.7stap: man pages: tweak dwarf warnings, mention mismatches and debuginfod http://tinyurl.com/salxq8h
sapatel has joined #systemtap
gromero has joined #systemtap
sapatel has quit [Quit: Leaving]
sapatel has joined #systemtap
irker201 has quit [Quit: transmission timeout]
sapatel_ has joined #systemtap
sapatel has quit [Ping timeout: 268 seconds]
_whitelogger has joined #systemtap
CME_ has joined #systemtap
CME has quit [Ping timeout: 260 seconds]
CME_ is now known as CME
tromey has quit [Quit: ERC (IRC client for Emacs 28.0.50)]
sscox has quit [Ping timeout: 268 seconds]
khaled has quit [Quit: Konversation terminated!]
mjw has quit [Quit: Leaving]