fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
pwithnall_____ has quit [Ping timeout: 240 seconds]
_whitelogger has joined #systemtap
scox has joined #systemtap
gromero has quit [Ping timeout: 255 seconds]
hpt has joined #systemtap
RustJason has joined #systemtap
<introom> how is ebpf support in systemtap? and is there any performance difference between current ebpf (bcc) and systemtap ?
RustJason has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
RustJason has joined #systemtap
RustJason has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
RustJason has joined #systemtap
sanoj has joined #systemtap
RustJason has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
pwithnall_ has joined #systemtap
pwithnall_ has quit [Quit: pwithnall_]
pwithnall_____ has joined #systemtap
RustJason has joined #systemtap
slowfranklin has joined #systemtap
nkambo_ has joined #systemtap
nkambo has quit [Ping timeout: 255 seconds]
<slowfranklin> Hi folks!
<slowfranklin> Anyone around that might be able to tell why something like this would fail:
<slowfranklin> stap -v -e 'probe process(14093).library("*").function("gencache_stabilize") {printf("func\n")}'
<slowfranklin> I get:
<slowfranklin> semantic error: while resolving probe point: identifier 'process' at <input>:1:7
<slowfranklin> source: probe process(14093).library("*").function("gencache_stabilize") {printf("func\n")}
<slowfranklin> $ ps auxwww | grep 14093 | grep -v grep
<slowfranklin> root 14093 0.0 0.4 359212 16488 ? S 10:21 0:00 /home/slow/git/samba/scratch/bin/default/source3/smbd/smbd -D
<slowfranklin> If I use the path instead of the pid it works
<slowfranklin> Systemtap translator/driver (version 3.1/0.169, rpm 3.1-5.fc25) this is
RustJason has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
orivej has joined #systemtap
mjw has joined #systemtap
hpt has quit [Quit: Lost terminal]
RustJason has joined #systemtap
RustJason has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
sanoj has quit [Quit: Leaving]
pwithnall_ has joined #systemtap
pwithnall_ has quit [Quit: pwithnall_]
RustJason has joined #systemtap
gromero has joined #systemtap
wcohen has quit [Ping timeout: 240 seconds]
drsmith_away is now known as drsmith
mbenitez has joined #systemtap
mbenitez has joined #systemtap
mbenitez has quit [Changing host]
gromero has quit [Ping timeout: 240 seconds]
<fche> hm maybe the .library("*") part of the logic doesn't know how to handle pid#'s
RustJason has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
gromero has joined #systemtap
wcohen has joined #systemtap
<slowfranklin> hm
<slowfranklin> I'm using the processname and just pass -x pid for now
<slowfranklin> thanks
<fche> righto
hpt has joined #systemtap
RustJason has joined #systemtap
brolley has joined #systemtap
gila_ has quit [Ping timeout: 246 seconds]
gila has joined #systemtap
drsmith is now known as drsmith_away
drsmith_away is now known as drsmith
plhu has joined #systemtap
RustJason has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
orivej has quit [Ping timeout: 240 seconds]
drsmith is now known as drsmith_away
drsmith_away is now known as drsmith
<plhu> Hello, I try to understand the arguments of setsockopt syscall but I have no idea how to understand its argument values. Is there any place e.g. in kernel docs/code to look at?
<fche> setsockopt is kind of like ioctl - it is not one call but a family of functions hidden in one
irker352 has joined #systemtap
<irker352> systemtap: scox systemtap.git:refs/heads/master * release-3.1-116-g2dd0c8a / client-http.cxx client-http.h: Add preliminary http client support. http://tinyurl.com/yb7q3otr
<irker352> systemtap: scox systemtap.git:refs/heads/master * release-3.1-118-g88f7bbe / : Merge branch 'master' of ssh://sourceware.org/git/systemtap http://tinyurl.com/ybpq9g26
<fche> Have you seen [man setsockopt] and want to look at the implementation in the kernel?
<plhu> fche: I read the setsockopt man page and found a reference to socket(7) and I remember somehow how this works, but I have no idea where to find the mapping of socket option names SO_* to the binary flags.
<fche> /usr/include/asm-generic/socket.h from glibc ?
<plhu> fche: thanks. Did you already know this or is there any doc that have not come to my mind?
hpt has quit [Ping timeout: 260 seconds]
<plhu> I mean, looking for socket.* or in glibc ogre grepping its header files to SO_ should come to my mind but if there would be some other way, I would like to know for the future.
<fche> all hail the power of grep :)
<plhu> ah, ok
<fche> should also be in the kernel uapi/ headers
orivej has joined #systemtap
orivej has quit [Ping timeout: 240 seconds]
<irker352> systemtap: dsmith systemtap.git:refs/heads/master * release-3.1-119-g073f2c2 / runtime/stack.c: Disable page faults in _stp_valid_pc_addr(). http://tinyurl.com/y8cug48h
gila has quit [Quit: My Mac Pro has gone to sleep. ZZZzzz…]
orivej has joined #systemtap
plhu has quit [Ping timeout: 240 seconds]
<slowfranklin> gna
<slowfranklin> I have a script with working userspace dwarf function probes on fedora 25 with stap 3.1
<fche> ok
<slowfranklin> On rhel 7.2 with kernel 3.10.0-327.el7.x86_64 it chokes on the $return and function $parameter access
<slowfranklin> eg
<slowfranklin> probe process("winbindd").function("wb_ping_send").return {
<slowfranklin> running["wb_ping_send", $return] = gettimeofday_us()
<slowfranklin> }
<fche> you'll need userspace process debuginfo to get at params
<slowfranklin> Sure, I have them on both systems.
<slowfranklin> On rhel 7.2 I get:
<slowfranklin> semantic error: unresolved target-symbol expression: identifier '$return' at winbindd.stp:335:44
<slowfranklin> source: running["winbindd_lookupsid_send", $return] = gettimeofday_us()
orivej has quit [Ping timeout: 260 seconds]
<fche> would this be a void function ? tried without $return ?
<slowfranklin> on fedora25 the build is with -g -O0, on rhel it's probably -g -O2
<slowfranklin> Can that make a difference?
<fche> a void function doesn't have a $return ... try $$return (stringified to empty on void functions)
<slowfranklin> no. As said, the exact same script works on another system...
<slowfranklin> It returns a pointer to a struct
<fche> ok.
<slowfranklin> struct tevent_req * that is
<fche> is there any message other than that semantic error?
<slowfranklin> Full invocation:
<slowfranklin> # stap winbindd.stp
<slowfranklin> semantic error: unresolved target-symbol expression: identifier '$return' at winbindd.stp:335:44
<slowfranklin> source: running["winbindd_lookupsid_send", $return] = gettimeofday_us()
<slowfranklin> ^
<slowfranklin> Pass 2: analysis failed. [man error::pass2]
<slowfranklin> Can it be the -O2 ?
<slowfranklin> Or the older stap or kernel?
<fche> shouldn't be; stap is comfortable with optimized code. should not be kernel - this is not kernel-side code.
<slowfranklin> hm
<fche> shouldn't be older stap either (though it's a year+ old probably), this is not new functionality
<fche> tried advice in man error::pass2? like rerun with a little more stap --vp 01 pass-2 verbosity?
<slowfranklin> Other probes in the same script (earlier up the file) don't give an error:
<slowfranklin> probe process("winbindd").function("_wbint_PingDc") {
<slowfranklin> running["_wbint_PingDc", tid()] = gettimeofday_us()
<slowfranklin> }
<slowfranklin> So other dwarf sym function probes
<slowfranklin> That compiles fine
<fche> depending on your binary, this may not actually require dwarf
<fche> is it possible that the winbindd binary on rhel7 is stripped?
<slowfranklin> yup
<slowfranklin> # file /usr/lpp/mmfs/bin/winbindd
<fche> readelf -S .../binary
<slowfranklin> /usr/lpp/mmfs/bin/winbindd: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=dc0216d881b68229b4d34b95029b748da15492ac, stripped
<fche> being 'stripped' means debuginfo has been removed
<slowfranklin> But debug sym package is installed
<slowfranklin> yes, seperate package
<fche> ok, so a stap --vp 02 or more would let us see whether stap can find that debuginfo
<slowfranklin> ok
<fche> --vp 02 is not the same as --vp 2 :)
<fche> it's verbosity 2 for pass 2
<slowfranklin> Damn :)
<fche> (that's transcribed correctly in [man error::pass2]
<fche> it doesn't seem like it found the debuginfo
<slowfranklin> Hm. But it looks very similar to the working system.
<slowfranklin> ah, no
<slowfranklin> I see the difference
<slowfranklin> Bad: probe _wbint_PingDc@:-1 process=/usr/lpp/mmfs/bin/winbindd reloc=.dynamic pc=0x713f0
<slowfranklin> Good: probe _wbint_PingDc@../source3/winbindd/winbindd_dual_srv.c:754 process=/home/slow/git/samba/scratch/bin/default/source3/winbindd/winbindd reloc=.dynamic pc=0x675fe
<fche> yup
<slowfranklin> Damn. I'm not really familiar with the magic of stripping debug symbols from binaries and how they're put into seperate RPMs and how the tools put it all together again
<fche> do you have a rpm version/release list, to ensure there's no mismatch there?
<slowfranklin> On the Fedora system I'm just directly working with binaries from from a build of a git checkout
<slowfranklin> On the RHEL system it's all from rpmbuild with seperate debug sym RPM
<fche> that should all Just Work (tm); stap (elfutils) knows about such conventions
<slowfranklin> Yeah
<slowfranklin> Afair it works for eg gdb, but lemme check
<slowfranklin> ahhhhhhh, looks like someone messed up the packages...
<slowfranklin> This is from attaching with gdb to a process:
<slowfranklin> Reading symbols from /usr/lpp/mmfs/lib64/samba/vfs/shadow_copy2.so...
<slowfranklin> warning: the debug information found in "/usr/lib/debug//usr/lpp/mmfs/lib64/samba/vfs/shadow_copy2.so.debug" does not match "/usr/lpp/mmfs/lib64/samba/vfs/shadow_copy2.so" (CRC mismatch).
<slowfranklin> warning: the debug information found in "/usr/lib/debug/usr/lpp/mmfs/lib64/samba/vfs/shadow_copy2.so.debug" does not match "/usr/lpp/mmfs/lib64/samba/vfs/shadow_copy2.so" (CRC mismatch).
<slowfranklin> Reading symbols from /usr/lpp/mmfs/lib64/samba/vfs/shadow_copy2.so...(no debugging symbols found)...done.
<slowfranklin> (no debugging symbols found)...done.
* fche cries quietly in the rain
* slowfranklin head hits table
<slowfranklin> Thanks a *lot* for pointing me in the right direction!
<fche> no problemo
<fche> we really should get the tool to explain these situations better
<fche> (and at the same time not drown people in info, like that --vp 02 does)
<slowfranklin> I guess a big warning that debug symbols can't be found *whenever* a user uses those probes might make sense?
<fche> well, except that stap can do -some- stuff without, like place all those .function probes
<fche> it's the $vars that triggered the dwarf need
<slowfranklin> yup
<slowfranklin> Fwiw, I'm currently adding probes to Samba as well as write several script for instrumentation
<fche> excellent
<slowfranklin> I really really like this stuff!
<mjw> fche, do you have an idea of what/when stap should say to make this easier for users?
<mjw> slowfranklin, or you. At what point would you have (with hindsight) expected stap to say something that would have saved you some time (and what would that have been).
<slowfranklin> mjw: hi
<slowfranklin> From reading man stapprobes I was under the impression that process().function() probes require dwarf debug symbols
<slowfranklin> That doesn't seem to be the case
<mjw> So when stap said:
<mjw> semantic error: unresolved target-symbol expression: identifier '$return' at winbindd.stp:335:44
<slowfranklin> So an additional error at the point where accessing $return failed would have been helpful
<mjw> It should have also have said something like:
<mjw> at address xxx in module foobar (no debuginfo)
<slowfranklin> yes
<fche> slowfranklin, see SYMBOL-TABLE mention of process.function also
<mjw> Lets see, the other probe worked because it could figure out the address for the function "_wbint_PingDc" in winbindd without debuginfo?
<fche> yes.
<slowfranklin> yes
<mjw> OK, that should be doable.
<mjw> except I don't have time right now... sigh.
<fche> :)
<slowfranklin> :)
<slowfranklin> Nevertheless, thanks for your time! ;) And for your help!
<fche> slowfranklin, in the mean time, any idea about the nature of the debuginfo mismatch/missingness?
<slowfranklin> Some sort of packaging madness, someone must have messed up the cluster
<fche> do you guys have wet noodle executions over there?
<slowfranklin> Wtf?
pwithnall_____ has quit [Quit: pwithnall_____]
<mjw> what?
<fche> oh maybe it's a toronto thing
<slowfranklin> wet noodle executions? Reminds of: dinner time....
<fche> someone who does something particularly sneaky gets a bowl of wet noodles on their desk
<mjw> yes! dinner!
<slowfranklin> hehe, I must remember that
<fche> and they must cereomonially consume and/or dispose of it, while meditating on their wrongdoing
<slowfranklin> :D
<slowfranklin> Well, I'm still so excited that I could implement full instrumentation of winbindd in 30 mins thanks to Systemtap, I don't bother... :)
<slowfranklin> I don't even have to add probes to the code, just function names and a few params
<slowfranklin> Just freaking awesome!
<fche> heh, yeah, that's the promise .. sure frustrating when things go wrong
<slowfranklin> Well, at a certain complexitiy level you just have to have a certain frustration level -- or get another job
<fche> it's only right
<slowfranklin> But maybe I'm just too used to the insanity that lies in the Samba code and the Windows protocols. 8)
<fche> hey can I quote you, next time we get a usability bug report? :)
<slowfranklin> You bet you can! ;)
slowfranklin has quit [Quit: slowfranklin]
slowfranklin has joined #systemtap
slowfranklin has quit [Client Quit]
<irker352> systemtap: scox systemtap.git:refs/heads/master * release-3.1-120-g8256fa7 / client-http.cxx client-http.h: Revert "Add preliminary http client support." while configure support is added. http://tinyurl.com/ybbwbbjw
pwithnall_ has joined #systemtap
gila has joined #systemtap
orivej has joined #systemtap
pwithnall_ has quit [Quit: pwithnall_]
drsmith is now known as drsmith_away
slowfranklin has joined #systemtap
gromero has quit [Ping timeout: 255 seconds]
mjw has quit [Quit: Leaving]
pwithnall_ has joined #systemtap
slowfranklin has quit [Quit: slowfranklin]
wcohen has quit [Ping timeout: 260 seconds]
mbenitez has quit [Quit: Leaving]
pwithnall_ has quit [Quit: pwithnall_]
brolley has left #systemtap [#systemtap]
irker352 has quit [Quit: transmission timeout]
wcohen has joined #systemtap
gila has quit [Quit: My Mac Pro has gone to sleep. ZZZzzz…]