16:51 <misha354_> hi, was wondering if anyone here has used perf_events for tasks similar to systemtap. I'm on a systemtap that doesn't have systemtap, trying to figure out why an application is blocking. I've generated an off-cpu kernel flame graph, but the stack traces end and glibc low level thread lock functions, without links to the actual application code

16:51 <misha354_> *on a system*

16:52 <misha354_> https://svgshare.com/i/AhU.svg

16:52 <fche> can you get deeper backtraces?

16:53 <misha354_> I'm not sure how do that

16:53 <misha354_> perf record -e sched:sched_stat_sleep -e sched:sched_switch -e sched:sched_process_exit -a -g --call-graph dwarf -o perf.data.raw --pid $(pgrep named | head -n 1) -- sleep 10

16:53 <misha354_> here is what I am using to generate the backtraces

16:54 <fche> might need to bump up the stack-size that's being recorded, or might try the --call-graph=lbr mode

16:55 <fche> (btw that svg gives me just a single stop icon kind of thing)

16:55 <misha354_> https://gist.github.com/misha354/c4edbf2b04818fbc89ac93cb58a6e9e4#file-offcpuflamegraph-svg

16:55 <misha354_> how about this?

16:55 <misha354_> it's surprisingly hard to find free hosting for svg images

16:58 <fche> it's just a file :) put it anywhere

17:01 <fche> yeah not sure why you're seeing only that little info. what tool created the svg?

17:02 <misha354_> flamegraph.pl from https://github.com/brendangregg/FlameGraph

17:02 <misha354_> https://gist.github.com/misha354/1565f597c8f6d0ef376b3561fb937979

17:03 <misha354_> here is my script that generated it

17:04 <fche> so I wonder if that stack-collapse widget may be omitting certain levels

17:04 <fche> am not familiar with this widgetry TBH - too many parts.

17:04 <fche> Hm, I'm surprised we haven't created a stap script that generates such svg directly. (I understand that wouldn't help you on that particular machine.)

17:07 <misha354_> Thanks for your help!

17:08 <misha354_> It appears to be a perf issue. Here is a sample stack trace before the widget

17:08 <misha354_> https://gist.github.com/misha354/cc3bba9190a404195022b0d81aee5ace

17:08 <misha354_> I'm going to explore raising the stack depth of the perf record as you suggested

17:09 <fche> yeah, or using the lbr data

17:09 <fche> it is also possible that the userspace app that calls into libpthread has been stripped of its unwinding data, so perf couldn't go much farther back

17:09 <fche> worth a look

17:10 <misha354_> the main limitation to using systemtrace on that machine is that I can't install debug symbols for the kernel

17:10 <misha354_> is there a way around that limitation?

17:22 <misha354_> *to using systemtap*

17:23 <misha354_> Also, FWIW, other stack traces from the userland application are complete. It's only these libpthread that don't go down into userland

17:23 <misha354_> Anyway, I appreciate your time! Thank you

17:29 <fche> kernel debuginfo is not required just to collect userspace backtraces

17:29 <fche> or to do many other systemtappy things

17:29 <fche> and indeed you don't have to process debuginfo on the same machine as where you want to run the stap script

17:29 <fche> (see [man stap] -p4, stap --remote=HOST, etc.)

17:35 <misha354_> Thank you, I will look into those options, and probably be back

17:35 <misha354_> :)

17:36 <fche> okie dokie