fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
Guest87320 has quit [Quit: quit]
thibaultcha has joined #systemtap
thibaultcha is now known as Guest90308
Guest90308 has quit [Client Quit]
<fche>
hi modem
<fche>
yeah couple of things going on there; tail-calls to functions may not be caught by the *retprobe mechanism the expected manner
khaled has quit [Quit: Konversation terminated!]
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
thibaultcha has joined #systemtap
thibaultcha is now known as Guest31990
hpt has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
Guest31990 has quit [Quit: quit]
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
orivej has quit [Ping timeout: 240 seconds]
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
_whitelogger has joined #systemtap
derek0883 has quit [Remote host closed the connection]
khaled has joined #systemtap
orivej has joined #systemtap
thibaultcha has joined #systemtap
thibaultcha is now known as Guest71081
orivej has quit [Ping timeout: 240 seconds]
Guest71081 has quit [Quit: quit]
hpt has quit [Ping timeout: 272 seconds]
orivej has joined #systemtap
thibaultcha has joined #systemtap
mjw has joined #systemtap
thibaultcha is now known as Guest7257
Guest7257 is now known as thibaultcha
thibaultcha has quit [Changing host]
thibaultcha has joined #systemtap
thibaultcha has quit [Quit: quit]
thibaultcha has joined #systemtap
thibaultcha is now known as Guest61279
Guest61279 is now known as thibaultcha
thibaultcha has quit [Changing host]
thibaultcha has joined #systemtap
thibaultcha has quit [Quit: quit]
thibaultcha has joined #systemtap
thibaultcha has quit [Changing host]
thibaultcha has joined #systemtap
_whitelogger has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
amerey has joined #systemtap
sscox has quit [Ping timeout: 240 seconds]
tromey has joined #systemtap
sscox has joined #systemtap
derek0883 has joined #systemtap
orivej has joined #systemtap
<kerneltoast>
hi fche, i have a question about the comment for __stp_utrace_attach_match_tsk
<kerneltoast>
i'm not quite sure what the difference between path_tsk and match_tsk is
<kerneltoast>
specifically for the case when they are not equal, i'm not sure why they must be different
orivej has quit [Ping timeout: 240 seconds]
<fche>
kerneltoast, sorry, not sure either. have you identified uses where the two are different?
<fche>
(maybe, a multithreaded app runnign exec() from a non-main thread, thus changing its effective pid?)
<kerneltoast>
fche, i noticed that __stp_utrace_attach_match_tsk doesn't attach to _stp_target when path_tsk != match_tsk and _stp_target == path_tsk->pid
amerey has quit [Remote host closed the connection]
<fche>
hm, could be related to stap -c /bin/cmd
<agentzh>
fche: it's not -c CMD.
<agentzh>
the target processes are always running, before staprun starts.
<agentzh>
they are daemons and never stop.
<agentzh>
so exec() is not involved here.
<agentzh>
we use staprun -x PID
<agentzh>
and the target process is already there when staprun runs.
<agentzh>
otherwise we cannot know its pid.
<agentzh>
kerneltoast: so the pid indeed matches?
<agentzh>
but the struct task pointers don't?
<kerneltoast>
agentzh, the pid matches path_tsk, but not match_tsk. the struct task pointers are different
<agentzh>
will you check the contents of those two task struct?
<agentzh>
to see how they are different.
<agentzh>
if the pids are the same.
<kerneltoast>
they are both nginx processes, but with different pids
<agentzh>
oh so they are both legit.
<kerneltoast>
yeah
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
orivej has joined #systemtap
derek0883 has quit [Ping timeout: 240 seconds]
derek0883 has joined #systemtap
agentzh has quit [Quit: leaving]
agentzh has joined #systemtap
<agentzh>
fche: okay, indeed you are right, it's the nginx process calling exec() when running an external shell cmd...
<fche>
hey when you figure something like that out,
<fche>
please feel encouraged to add a comment to the relevant piece of code to record your insight
<agentzh>
yeah, that's a good idea.
<agentzh>
we are still fighting with several stap bugs in fuzz testing atm. all of them are related to the task_finder/utrace/process.begin probe.
<fche>
interesting
<agentzh>
leading to Dl hanging stapio processes, CPU softlock and panic.
<agentzh>
the Sl+ stapio one is the most gentle thing. and it seems that it's due to exec shell cmd as mentioned above.
<agentzh>
we're still digging.
<agentzh>
and those issues can be reproduced in a wide variety of kernel versions.
<agentzh>
and quite easy to reproduce
<agentzh>
but tough to troubleshott.
<agentzh>
since significant load is required.
<agentzh>
and we have to use 32 or even 64 cpu cores to reproduce.
<fche>
lockdep unhelpful?
<agentzh>
small boxes are fine.
<agentzh>
not helpful.
<agentzh>
the issue looks like missing locks
<agentzh>
instead of wrong use of locks.
<fche>
aha
<agentzh>
like memory corruptions and use after free. etc.
<agentzh>
we'll try kasan.
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
<fche>
ah neat
derek0883 has quit [Ping timeout: 258 seconds]
derek0883 has joined #systemtap
amerey has joined #systemtap
tromey has quit [Ping timeout: 240 seconds]
irker765 has joined #systemtap
<irker765>
systemtap: scox systemtap.git:scox/tls * release-4.3-78-gb5752f1c7 / tapset/linux/tls.stp testsuite/systemtap.base/tls.exp: Support tls on ppcle
<agentzh>
fche: we've confirmed that the stapio Sl+ hang is due to the target pid being the process in the middle of fork() and exec().
<agentzh>
in this case, we should at least print out a warning if we do detect it?
amerey has quit [Remote host closed the connection]
<fche>
is this an error condition, or just one that the code doesn't happen to handle correctly?
<agentzh>
the condition could be _stp_target != 0 && path_tsk != match_tsk && _stp_target == path_tsk->pid
amerey has joined #systemtap
amerey has quit [Remote host closed the connection]
amerey has joined #systemtap
<agentzh>
fche: the behavior is not intended. the target pid is the parent process, but the utrace callback is run on the child process which is just forked.
<agentzh>
in __stp_utrace_attach_match_tsk, path_tsk is the parent process, while the match_tsk points to the forked process.
<agentzh>
_stp_target == path_tsk->pid
<agentzh>
and the utrace callback is never fired on the parent process.
<agentzh>
but only on the child process which fails the pid test.
<agentzh>
i don't think this behavior is intended, is it?
<agentzh>
kerneltoast can correct me if i miss anything here.
<agentzh>
fche: do you know how utrace attach works for processes without clone/exec syscalls?
<fche>
well, no, not off the top of my head
<fche>
but there are interrupt type calls available
<agentzh>
k
<agentzh>
we'll keep digging.
<agentzh>
this task finder thing is really convoluted :)
<agentzh>
hopefully kerneltoast is not too pissed off by it *grin*
<fche>
yeah
<fche>
part of the complication is that task_finder was written for the old rhel5/rhel6 utrace in kernel
<fche>
so that was the target api
<fche>
with rhel7 era kernel (3.10ish) bringing uprobes/etc. fully in kernel, we decided for whatever reason to keep task_finder more or less as is, and instead emulate old utrace on top of new tracepoints etc.
<agentzh>
okay, that explains the complexity.
<agentzh>
i also saw ptrace is involved in some cases.
<agentzh>
is it still used by modern kernels?
<agentzh>
it says working around utrace bugs...
<fche>
umm ptrace is a user space API/ABI for debuggers, so it is definitely still around
<fche>
utrace bugs .... ummm doesn't ring bells and would be kind of weird now that the 'utrace' we have in the runtime is the emulation of the old thing
<agentzh>
i just tried myself and it seems ptrace is not called on my kernels anyway.
<agentzh>
okay, WORKAROUND_BZ467568 is always defined in staprun.
<agentzh>
that's why.
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #systemtap
<fche>
sometimes old bug tracking systems are a curse :) so much archaeology to remind us of everything we've forgotten
<agentzh>
heh
<agentzh>
fche: okay, it seems like the __stp_tf_clone_worker callback runs on the wrong process, and then __stp_utrace_task_finder_target_quiesce never has a chance to run.
<agentzh>
the former runs upon the child process just forked.
<agentzh>
is there any flag which controls duplicate runs of the utrace callback?
<agentzh>
i guess maybe there's such a flag which suppresses the later callback run.
<fche>
dunno
<fche>
on the bright side, I cannot say that I forgot :)
<fche>
because this part of the code was written by other folks entirely :)
<agentzh>
k
<agentzh>
thanks anyway :)
<agentzh>
are those folks still around here?
<fche>
well I'm here as a shoulder to cry on
<fche>
hmmm not really, and I expect their memories are at least as clouded as mine
<fche>
so we probably have to reverse-engineer this thing just the way y'all are doing
<agentzh>
okay...
* kerneltoast
cries
<kerneltoast>
my brain melted digging in task_finder2 all week
* fche
offers professional but compassionate comfort :)
<fche>
kerneltoast, tried adding more dbug* calls?
<kerneltoast>
fche, i littered it with printks
<kerneltoast>
and just now executed some syscalls inside the module to get more info on a process
djellemah has joined #systemtap
derek0883 has quit [Remote host closed the connection]