20:54 <gilfoyle> hi folks, I'm learning systemtap and playing with the probe python.function.entry

20:54 <fche> yo

20:54 <gilfoyle> I can't seem to see any activity for my multi-thread testing script :)

20:54 <gilfoyle> hi fche :)

20:55 <gilfoyle> I will pastebin my script + stap

20:55 <fche> sure

20:55 <fche> what OS ?

20:56 <gilfoyle> Linux 3.10

20:57 <fche> I mean which distro

20:57 <gilfoyle> CentOS 7.2

20:59 <gilfoyle> http://paste.ubuntu.com/23134310/

20:59 <gilfoyle> this results in no output

21:00 <fche> how about single-threaded?

21:03 <fche> interesting, I think I'm seeing the same thing on fedora24.

21:04 <fche> single-threaded things work fine

21:04 <fche> it's as though the multithreaded libpython doesn't use the code paths where the sys/sdt.h macro calls were inserted

21:13 <gilfoyle> sorry, I had to step away from the computer

21:16 <gilfoyle> I haven't tried single-threaded, but using the pyfuntop.stp I could see it picking up on other python processes, including printing stuff out of the threading.py, except for this test I posted. Essentially, I was at a dtrace talk last night and at the end I was talking to the speaker and told him I believed this could be done with systemtap, so I was going to see how easy it was

22:05 <gilfoyle> fche: I tried re-writing that program in ruby and similarly, I don't see much activity from stap

22:07 <gilfoyle> this one using the ruby.cmethod.entry, http://paste.ubuntu.com/23134580/

22:18 <fche> the reasons for similar behavior between ruby & python are likely to be accidental

22:20 <fche> (multithreaded programs in general, at the kernel level, are perfectly fine)

22:20 <gilfoyle> I suspected :)

22:21 <gilfoyle> so out of curiosity, what's the standard that allows both dtrace and systemtap to share probes?

22:21 <fche> they don't actually share probes (at the object code level)

22:21 <fche> it's only at the source level - the {STAP,DTRACE}_PROBEN(...) macros look the same

22:21 <fche> their implementation is completely different

22:22 <gilfoyle> so same interface in other words?

22:22 <fche> same API to the program that #includes <sys/sdt.h>

22:30 <gilfoyle> oh, I see how it works now.

22:46 <gilfoyle> fche: is there a way for me to tell if the binary has been built with the system tap probes?

22:52 <fche> readelf -n $BINARY

22:53 <fche> and look for NT_STAPSDT notes

22:53 <fche> for something like python, may need the /usr/lib64/libpython*so rather than /usr/bin/python

23:07 <gilfoyle> awesome, thanks!

23:13 <gilfoyle> sorry.. I'm a bit curious, what's the reason for the elf notes?

23:13 <fche> those notes are what systemtap (and gdb e.g.) look for in order to locate the macro call sites, and their parameters

23:19 <gilfoyle> thank you! I'd like to understand this a bit better. I'll grab the sources

23:20 <gilfoyle> out of curiosity, is there an official tree on github?

23:20 <fche> git://sourceware.org/git/systemtap.git

23:22 <fche> interesting trace from

23:22 <fche> # sudo stap -e 'probe process("/usr/lib64/libpython2.7.so").function("*").call { if(uid()==500) println(pp()) }' # substitute your uid

23:22 <fche> while your multithreaded python script is running

23:23 <fche> I see no PyEval_EvalFrameEx once the threads are running

23:24 <fche> http://paste.fedoraproject.org/422122/31482147/ <-- printing the tid() and ppfunc()

23:25 <fche> it's as though there's an error during internal libpython evaluation process

23:31 <gilfoyle> interesting :)

23:31 <fche> methinks something is wrong with your test program (& multithreaded python in general)

23:32 <fche> if you replace that time.sleep(2) with print ("yo") you get stap-side results just fine

23:32 <gilfoyle> oh :)

23:32 <fche> # stap -e 'probe python.function.entry { if (uid()==500) println(filename, " ", funcname, " ", lineno) }' # again change 500

23:32 <fche> (or drop it, for systemwide probing)

23:32 <gilfoyle> maybe worth diving deep into how time.sleep is implemented

23:33 <gilfoyle> I think strace suggested it was select() based

23:33 <fche> from a stack traceback experiment, it looked as though the error occurred during module-time loading

23:33 <fche> ie didn't even get as far as trying to run the sleep function

23:33 <gilfoyle> Oh, I see.

23:33 <fche> # stap -e 'probe process("/usr/lib64/libpython2.7.so").function("*").call { if(uid()!=500) next; println(tid(), " ", ppfunc()) if (ppfunc() =~ "PyErr") print_ubacktrace() }'

23:34 <gilfoyle> let me try these

23:34 <fche> that last one was for getting a backtrace in the PyErr* type functions

23:38 <gilfoyle> the more I use systemtap the more amazed I am at how awesome this is and how little people know about it.

23:38 <fche> hey help spread the word

23:39 <fche> it's most credible from users :)

23:39 <gilfoyle> true!

23:39 <gilfoyle> :)

23:42 <gilfoyle> fche: did you run the code above with -d /usr/lib64/python2.7/lib-dynload/timemodule.so ?

23:42 <gilfoyle> it appears to provide the PyEval_*

23:42 <fche> nope, but you should

23:42 <gilfoyle> s/provide/print

23:42 <fche> it may be important that I'm running on fedora 24 here, not rhel7

23:43 <fche> just had similar symptoms as you at the beginning

23:43 <gilfoyle> true :)

23:44 <gilfoyle> this reminds me, I should reopen my FAS account

23:44 <fche> if you like; you won't need that for anything stap related

23:44 <gilfoyle> I was going off on a tangent :-)