10:10 <roboman> Hello. I was trying to add a "probe procfs("test").read {" and I seem to be getting an error message about being unable to dereference a pointer of incomplete type in the function "_stp_mkdir_proc_module"(that is inside systemtap/runtime/procfs.c). Is there any header file I should be including? I'm on systemtap version 1.7.

10:16 <invano> Hello everyone. I'm debugging a weird behavior and I just reached stapiu_mmap_found and the mmap_callback. At this point I can go to have a look at task_finder.c and task_finder2.c. May I ask you guys what is the difference between the two?

12:38 <fche> hi invano

12:38 <invano> hi fche

12:39 <fche> task_finder2.c is used on relatively modern kernels (3.5+ IIRC)

12:39 <fche> task_finder.c is for older ones

12:39 <fche> see the top part of task_finder.c for the selection algorithm

12:39 <invano> yes at the end I started looking at task_finder2

12:42 <invano> my problem is that uprobes in 1 lib are not found in any process in stapiu_probe_prehandler. apparently it's added in target->processes stapiu_change_plus and maybe removed somehow immediately after..and that's why my probe_handler is not triggered. so now I was going up to mmap and now task_finder2

12:43 <fche> maybe the shlib file name is not a match? or is of a different version than the buildid found at stap translate time?

12:43 <fche> I'd be surprised if a bug of such a fundamental nature existed in the runtime

12:44 <invano> filename is a match. even better, if I do a staprun without -c (so system-wide) I can see the probes working perfectly over systemd.

12:45 <invano> and apparently the probes are working only there

12:46 <invano> they reach UPROBE_HANDLER_REMOVE in the probe_prehandler in other execs

12:46 <fche> what is your stap script, and your invocation command line?

12:48 <invano> very simple

12:49 <invano> probe process("/opt/glibc/lib/libc-2.19.so").mark("memcmp") { /*a simple print over $$name*/}

12:50 <invano> invoked as 'staprun -v mod.ko -c /a.out AAAAA'

12:50 <invano> in the same stp file I also have 'probe process("/opt/glibc/lib/ld-2.19.so").mark("memcmp") { /*a simple print over $$name*/}'

12:51 <fche> (isn't that the same thing?)

12:51 <invano> ld and libc have basically the same memcmp function but the probes are two

12:51 <fche> ah libc vs ld

12:51 <invano> yes but one over ld and the other over libc

12:51 <fche> ok, so systemwide it works, but -c ... doesn't ?

12:52 <invano> debugging the kernel I see that with '-c' probe for ld is found correctly whereas probe for libc gets UPROBE_HANDLER_REMOVE *even if* it goes through stapiu_change_plus in a first moment

12:52 <invano> if I go systemwide I see it working correctly over the systemd process

12:53 <fche> though systemd shouldn't be loading /opt/glibc/... I'd assume

12:53 <invano> sorry it's systemd-journal

12:54 <fche> can you strace ./a.out for mmap etc. to see exactly which libraries it loads (& unloads!)

12:54 <fche> sure, but I wouldn't expect to see /opt/glibc/ non-system copies of libc to be used by system copies of systemd/+journal/anything

12:54 <invano> a.out just has a memcmp. already did strace and gdb work and it correctly resolve the right shlib

12:55 <invano> (yeah that's because I was doing someexperiments and I put this custom libc systemwide)

12:56 <invano> from gdb I can also see the nop instr converted to break. and to me, at first impact, it seems that kernel/events/uprobes.c and arch/mips/kernel/uprobes.c is handling the stuff correctly

12:57 <invano> I can put somewhere some printks taken during the execution of the staprun

12:57 <invano> maybe it's easier to read the printks than explain here

13:02 <fche> yeah probably - there's a little conditional-debug tracing widget in the runtime too

13:09 <invano> ok here you have a staprun run with kernel logging, stap kmodule included https://pastebin.com/QeQFqvEz

13:14 <invano> you can see the target process for the probe on ld is always found .e.g line 1152, but never found for the probe on libc e.g. ~line 1447

16:34 <irker763> systemtap: dsmith systemtap.git:refs/heads/master * release-3.1-86-gda23dba / httpd/client.py httpd/main.cxx httpd/server.cxx httpd/server.h: The httpd server now support downloading the built module. http://tinyurl.com/y9tdkemk

18:28 <irker763> systemtap: dsmith systemtap.git:refs/heads/master * release-3.1-87-g4fb34c4 / httpd/Makefile.am httpd/Makefile.in httpd/api.cxx httpd/api.h httpd/main.cxx httpd/server.cxx httpd/server.h: Performed some code rearranging in the httpd server code. http://tinyurl.com/y73oz8bo

20:19 <irker763> systemtap: dsmith systemtap.git:refs/heads/master * release-3.1-88-ge34298d / httpd/README: Add httpd/README describing the web api. http://tinyurl.com/ybkwuy7q