15:15 <gregwork> are there any useful tap's for troubleshooting hadoop performance ?

15:16 <fche> hi

15:16 <fche> I am not aware of any scripts targeted to hadoop per se, but stap has helped with various system- and sometimes jvm-level problems

15:17 <gregwork> can you run multiple stap scripts at the same time ?

15:17 <fche> certainly

15:18 <fche> so what kinds of problems are you seeing, and what brings you to stap ?

15:20 <gregwork> oh im involved with a task force here to investigate hadoop perf issues in prod. We have got a static workload we can run over and over on the data science admins looking at the app, I was going to look at the view from the OS perspective. Observe where the kernel thinks the app is spending its time, what syscalls/io etc

15:20 <gregwork> im hoping to compare notes as we run through the workload

15:21 <fche> aha. yeah, syscalls analysis is a pretty basic use of the tool. also good if one needs to go deeper - device drivers, file systems, etc. internals

15:22 <gregwork> i know that strace can be run with -c

15:22 <gregwork> to count the syscalls used by a process

15:22 <gregwork> is there a stap/deeper equiv

15:23 <fche> sure, stap works easily systemwide rather than per-process, and uses a much smaller interference interface to gather the syscall stuff

15:23 <fche> t

15:23 <fche> there is a strace workalike script in there that you could try just to get the feel

15:23 <gregwork> which one is that

15:23 <fche> (but note, your kernel version needs to be <= 4.16 for now)

15:24 <fche> % stap --example strace.stp should find it

15:24 <fche> https://sourceware.org/systemtap/examples/ <-- see also

15:24 <gregwork> this is rhel7

15:24 <gregwork> so its going to be much older than 4.16

15:24 <fche> right, things work fine there

15:25 <fche> but --example doesn't exist yet, till 7.6

15:25 <gregwork> i believe this is 7.5

15:25 <fche> so check out /usr/share/systemtap/examples

15:25 <fche> or % stap -i

15:26 <fche> stap> sample syscall

15:27 <gregwork> what are your thoughts on oprofile

15:27 <gregwork> in comparison to systemtap

15:27 <fche> very different feel; well suited to its type of problem

15:28 <fche> more similar to perf

15:50 <gregwork> is there any integration of systemtap with prometheus or equiv

15:50 <fche> get out of my head :)

15:50 <fche> git systemtap just gained an exporter glue process that lets stap scripts export data to prometheus (or compatible consumers such as pcp)

16:10 <gregwork> is that capability new new or would the rhel7 version have it

16:10 <fche> very new

16:10 <fche> maybe future rhel7 will

16:10 <fche> but really there isn't that much to it - a small python script that's invoked by systemd

16:11 <fche> does not represent any change to core systemtap binaries at all

16:11 <fche> if you're a customer, opening a bugzilla.redhat.com RFE to backport that function into future rhel7 would be very helpful

16:11 <gregwork> i am, but in the mean time i am curious as to what would be the most reasonable way to get this feeding into prom

16:12 <gregwork> redhat rfe process is not quick

16:13 <fche> for experiment, build your own stap out of git

18:54 <fche> agentzh, some more verbosity and a dyninst version would help with PR23605

19:45 <agentzh> fche: replied there :)

20:33 <fche> hm, do I understand correctly that this is intermittent?

20:34 <agentzh> right

20:34 <fche> is it 'just' an OOM situation perhaps?

20:35 <fche> (don't see how a pure userspace independent dyninst job should be affected by anything else, except in terms of resource exhaustion)

20:35 <agentzh> i just got a different error with the same test case.

20:36 <agentzh> updated the PR ticket.

20:36 <agentzh> this time it is an assertion failure.

20:36 <agentzh> the VM has plenty of RAM.

20:36 <agentzh> 12G in total.

20:36 <agentzh> and the only thing running is 8 stap processes.

20:37 <fche> right, but depending on what those 8 stap processes are doing, you might run out of ram in the vm

20:37 <agentzh> all very trivial scripts like that one.

20:37 <agentzh> no real payload at all.

20:37 <agentzh> just unit tests.

20:37 <agentzh> no big ones.

20:37 <agentzh> unlike stap's official test suite.

20:38 <agentzh> (which has load tests)

20:38 <agentzh> fche: is -vvv enough for debugging that dyninst problem?

20:38 <agentzh> or should i do -vvvv?

20:39 <fche> the more the merrier generally, as per [man error::reporting]

20:39 <fche> w.r.t. dyninst, if the problem is reasonably reproducible, an strace of the process would be helpful

21:01 <agentzh> fche: strace on the target process or on the stapdyn process?

21:03 <fche> stapdyn mainly

21:03 <fche> if you strace the target, dyninst (via ptrace) will fail anyway

21:04 <agentzh> seems like i should strace the full stap process tree then.

21:04 <fche> the stapdyn one alone would be fine

21:04 <fche> though it's not easy, if it's intermittent, so you can't just run stapdyn FOO.so etc.

21:06 <agentzh> right.

21:08 <agentzh> okay, just got one -vvvv output sample for the assertion failure in dyninst.

21:09 <agentzh> will paste it to the PR.

21:09 <fche> ok

21:14 <agentzh> added the attachment to the PR.

21:14 <agentzh> i'll add the strace thing

21:14 <agentzh> strace -f stap ... should be enough?

21:16 <fche> ideally stap --runtime=dyninst -p4 SOMETHING

21:16 <fche> then strace -f stapdun FOO.so in a loop until it crashes

21:21 <agentzh> hmm, seems like i should use mozilla rr to record it instead.

21:21 <agentzh> then we could have the whole history.

21:21 <agentzh> about everything.

21:21 <fche> if you like, you're welcome to try

21:22 <fche> still, a live strace (or similar) of the dyninst target executable will cause stapdyn to fail, so must not do that

21:22 <agentzh> yeah, i'll try once i have a chance. currently in the middle of something else.

22:35 <agentzh> fche: i added a new test module in this patch: https://sourceware.org/ml/systemtap/2018-q3/msg00156.html

22:36 <agentzh> do you like it?

22:36 <agentzh> the good news is that my test generator itself is also simpler :)