fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 256 seconds]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
wielaard has quit [Quit: Leaving]
orivej has quit [Ping timeout: 260 seconds]
hpt has joined #systemtap
_whitelogger has joined #systemtap
derek0883 has quit [Ping timeout: 246 seconds]
derek0883 has joined #systemtap
lindi- has quit [Quit: No Ping reply in 180 seconds.]
lindi- has joined #systemtap
_whitelogger has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 240 seconds]
amerey_ has joined #systemtap
derek0883 has joined #systemtap
amerey has quit [Ping timeout: 246 seconds]
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
przemoc has quit [Ping timeout: 260 seconds]
derek0883 has quit [Ping timeout: 272 seconds]
orivej has joined #systemtap
orivej has quit [Ping timeout: 240 seconds]
khaled has joined #systemtap
linus2 has quit [Ping timeout: 260 seconds]
linus2 has joined #systemtap
hpt has quit [Ping timeout: 260 seconds]
mjw has joined #systemtap
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
khaled has quit [Ping timeout: 272 seconds]
khaled has joined #systemtap
wcohen has quit [Remote host closed the connection]
amerey_ has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
xlei has joined #systemtap
<tux3>
So, I have a maybe dump question regarding missed probes. I saw on the wiki for skipped probes "This should not happen, except perhaps in extreme low-kernel-memory conditions"
<tux3>
But I also noticed that if I have a mix of regular kretprobe that are ftrace-optimizable, and also kprobes that are ftrace-optimizable, I get misses in kprobe_ftrace_handler()
<tux3>
So I added a dump_trace() and it seems to me that the regular non-ftrace probe gets preempted while running, hits an ftrace probe, and now I have kprobe_running() inside ftrace(), causing a miss...
<tux3>
So, essentially, does anyone know if this is a known issue? Is there maybe a reason we don't disable preemption everywhere until hitting a `__this_cpu_write(current_kprobe, NULL);` :/ ?
<fche>
the kernel ftrace / optimized-k*probes facility has had problems on & off
<fche>
current stap disables as much as it can due to these bugs
<tux3>
I'm running with optimized kprobes off. I guess should try to disable ftrace opt?
<tux3>
I can't even see where that IRQ is coming from, every call to `__this_cpu_write(current_kprobe, NULL);` I see is properly in a preempt disabled context...
derek0883 has quit [Remote host closed the connection]
<fche>
yeah, sorry, not sure; mhiramat still works on this area of the kernel every now & then
<tux3>
no worries, thank you =]
derek0883 has joined #systemtap
amerey has joined #systemtap
amerey has quit [Remote host closed the connection]
amerey has joined #systemtap
orivej has joined #systemtap
amerey has quit [Quit: Leaving]
amerey has joined #systemtap
<tux3>
Oh I think I found the fix (or at least the problem): kprobe_busy_end() in trampoline_handler() is not protected by kretprobe_hash_unlock(), so we happily take an interrupt with kprobe_running()
<tux3>
Clearly blindly changing the locking order without having any idea what I'm doing is a safe and sane thing to do, so let's try that =)
amerey has quit [Remote host closed the connection]
<fche>
RIGHT
amerey has joined #systemtap
mjw has quit [Quit: Leaving]
irker760 has quit [Quit: transmission timeout]
<tux3>
I had to do some gross irqsaving, but happy to report my fix "works" \o/
<tux3>
(And just in case anyone reading the scrollback has the same problem with missed probes in ftrace, enjoy a dirty patch: https://paste.debian.net/1165329/)
<fche>
ship it :)
* fche
both loves and hates that systemtap has been used to trigger kernel bugs for, well, over a decade
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 240 seconds]
khaled has quit [Quit: Konversation terminated!]
khaled has joined #systemtap
derek0883 has joined #systemtap
tromey has quit [Quit: ERC (IRC client for Emacs 27.1)]
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]