13:34 <irker390> systemtap: dsmith systemtap.git:refs/heads/master * release-3.0-116-g6329df7 / runtime/dyninst/addr-map.c runtime/linux/addr-map.c runtime/linux/loc2c-runtime.h: Partial 20236 fix: Add access_ok() to lookup_bad_addr(). http://tinyurl.com/z42xvqz

14:50 <lukas`> Hello to the community. I am quite new in systemtap and I go through some tutorial trying to trace a statement from a source file. It works if the statement comes from the code but I need it from dynamic library and it tend to finish on semantic error: 'semantic error: while resolving probe point: identifier 'process' at <input>:1:7'. The only difference seems to be that the source file comes from the library. Must

14:50 <lukas`> the library file be mentioned in the stap line?

14:50 <lukas`> Or is there another condition to be fulfilled?

15:08 <lukas`> Ok, I have my answer now, thanks

15:09 <fche> hi lukas`

15:10 <fche> what was your solution?

15:12 <lukas`> fche: Frank?

15:12 <fche> today, my secret name is Franck

15:12 <fche> don't tell anyone

15:12 <lukas`> fche: I think I read one of your messages in some related thread ... anyway the solution was to mention the libc in the process argument.

15:13 <lukas`> fche: sure :-)

15:13 <lukas`> fche: anyway, thanks for previous help with redis PMDA

15:13 <fche> yup, probes are relative to an executable or shared library

15:13 <fche> np

15:14 <fche> can you think of a place in docs or in the diagnostics where we could make this clearer?

15:14 <lukas`> fche: well, I try to write a script that would trace *alloc and free to detect some annoying memory leak which usual tools are hard to find.

15:15 <lukas`> fche: this is hard to tell, for me the SystemTap was uneasy to understand and een trey on Gentoo and before I personally spoke to Lukas Berk, I even did not believe that I could use it.

15:15 <lukas`> fche: it was harder than experimenting with dtrace

15:16 <fche> aha. we'll be putting in some serious effort into easing this stuff more, soon

15:16 <fche> can you enumerate briefly the problems?

15:16 <fche> (thanks lberk for helping the fellow lukas!)

15:16 <lberk> np :)

15:18 <lukas`> fche: well, lberk just had a presentation on a Red Hat conference and I just asked him - "I tried the SystemTap few times and I found it fairly complicated to be used, maybe convenient for expertes on kernel. Should it be so? Ans he said, no, it should be fairly easy ..."

15:19 <lberk> devconf last year?

15:19 <lukas`> lberk: yes

15:21 <lukas`> fche: okay, imagine a server connected to a load generator. Processing the request leads to further requests to other servers, several protocol stacks are involved. Now, ocasionally, the memry utilization of the process in a load/stability test starts to grow. The tricky thing is that it may or may not start to grow. Usual tools like gperftools or valgrind tools did not show any real leaks, so the memory is probably

15:21 <lukas`> just not being freed under some special conditions.

15:22 <lukas`> fche: now, tricky things are - it happens only sometimes, e.g. on three of four machines loaded with the same throughput and we do not know the starter.

15:23 <lukas`> fche: we know it is heap ... I checked the pmap output.

15:24 <lukas`> now, I've been thinking to trace ?alloc and free with userspace stack and match it and compare the newly allocated amount of memory with RSS increases.

15:25 <fche> righto

15:25 <fche> glibc has some sys/sdt.h markers in it that may help trace the major events

15:25 <fche> stap -L 'process("/lib64/libc.so.6").mark("memory*")'

15:26 <fche> (assuming your distro compiled those bits in)

15:26 <lukas`> that gives me nothing on RHEL6.8

15:26 <fche> http://paste.fedoraproject.org/379046/91799714/ <-- fedora 22

15:26 <fche> yeah, probably too old :( rhel7 has some

15:28 <fche> anyway without them, use normal process("...libc...").function("...") type probes

15:28 <fche> http://developers.redhat.com/blog/2015/01/06/malloc-systemtap-probes-an-example/

15:29 <fche> (again that's probably not going to work as is on rhel6.8)

15:29 <lukas`> Here is where I lack some knowledge and have weak points - malloc de facto does not allocate the memory, just reserves it. The memory increases the RSS at the time the page (or group of pages) starts to be used. Therefore I am not even sure with my approach.

15:30 <fche> you'd see the effects of posix level malloc/free traffic

15:30 <fche> new arenas being allocated (via sbrk or mmap) over time

15:31 <lukas`> fche: I already have this post open ... try to understand it

15:32 <lukas`> fche: but I guess that since the problem appears just sometimes a longer run will be necessary and therefore some kind of aggregation - that is yet for me to understand.

15:33 <fche> my guess (!) is that if you monitor arena creation long after the program has initialized and reached steady state of operation,

15:33 <fche> then any -new- arena allocations are suspicious

15:34 <fche> (btw have you tried running things under valgrind?)

15:34 <lukas`> You asked what you should added to the docs - I think the main problem with systemtap. I think that this tool could be tremendously useful for many people. The problem is that you cannot learn it without examples. This is not like a language where you can add some debugger or trace lines as not so many people know internals and can judge what do they see.

15:35 <fche> we're thinking hard about that problem right now, and could use any & all advice about how we could help such learning

15:35 <lukas`> fche: sure, valgrind was our second choice (as the problem happen only when the system is loaded to a certain level - another tricky part of the problem)

15:35 <fche> yup, been there done that

15:38 <lukas`> fche: Well, as the systemtap requires some effort for the user, I would start with something highly motivating like showing examples of medium to hard bugs like e.g. leaks or, crashes where looking to the code is not enough. Show developers or even testers or admins a way how they can get details about the problem or focus on the right area in code and they will be okay to spend some time (e.g. imagine performance t

15:38 <lukas`> esters that wish to get rid of bottlenecks that their developers cannot cope with)

15:39 <fche> yeah, that's a good description of the motivation

15:39 <fche> one needs to leap to a technical implementation though :)

15:39 <lukas`> ... then, advance to the next level - describe some simple behaviour of kernel so that the SystemTap users could study it - e.g. memory allocator

15:40 <lukas`> fche: well, start with a problem, demonstrate it on a software that a lot of people use and is easily configurable, or share it in buggy configuration via Docker.

15:40 <lukas`> buggy configuration or buggy implementation ... simply as and example of a bug that can be traced

15:41 <lukas`> show it on Apache, Bind, Samba, some FTP server or so

15:41 <lukas`> ... then you may show something cool like tracing KVM (wow a lof cloud users would love to know why their machine sometimes behave as it does)

15:44 <lukas`> fche: thanks for hints, I have to leave now. But I guess I will be back soon. BTW, I would be grateful for some code review and notes about my Redis PMDA and if anyone would be interested, I work on NutCracker PMDA and more detailed Bind server PMDAs now.

15:44 <fche> sure, pls consider sending a (reminder?) email over to the mailing list; others are also keen to review new folks' code

17:50 <irker390> systemtap: flu systemtap.git:refs/heads/master * release-3.0-117-ge7540f5 / elaborate.cxx parse.cxx staptree.cxx staptree.h translate.cxx: Fix pausing of probes in monitor mode http://tinyurl.com/zjr7jds

19:09 <irker390> systemtap: rth systemtap.git:refs/heads/rth/bpf * release-3.0-118-g3a8a8ae / : Initial commit of stap bpf support http://tinyurl.com/gmapc77

19:17 <jistone> rth, that's quite a low-key commit message for such big functionality

19:17 <fche> we'll make a big fuss in the next release notes

19:18 <jistone> starting planning now what you will wear in the parade

19:18 <fche> there better be a chorus line

19:25 <lukas`> what goodie is coming?

19:26 <jistone> lukas`, rth will send an email announcement, but that commit is a big hint

22:03 <irker390> systemtap: fche systemtap.git:refs/heads/rth/bpf * release-3.0-119-g87d6733 / bpf-translate.cxx: bpf-translate: tolerate script with no kprobes http://tinyurl.com/hg974a4

22:14 <irker390> systemtap: fche systemtap.git:refs/heads/rth/bpf * release-3.0-120-g516ee2b / bpf-translate.cxx: bpf-translate: same as above, for {begin,end}_probes http://tinyurl.com/h8lsalq

22:32 <irker390> systemtap: fche systemtap.git:refs/heads/rth/bpf * release-3.0-121-g217ee8c / bpf-translate.cxx staptree.cxx: bpf processing: pass token*'s with thrown exceptions http://tinyurl.com/j29x9fd