fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
hpt has joined #systemtap
khaled has quit [Quit: Konversation terminated!]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 260 seconds]
derek0883 has joined #systemtap
orivej has quit [Ping timeout: 246 seconds]
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek088_ has joined #systemtap
derek0883 has quit [Ping timeout: 272 seconds]
derek0883 has joined #systemtap
derek088_ has quit [Read error: Connection reset by peer]
derek0883 has quit [Ping timeout: 272 seconds]
hpt has quit [Ping timeout: 246 seconds]
hpt has joined #systemtap
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 260 seconds]
lijunlong has quit [Ping timeout: 260 seconds]
lijunlong has joined #systemtap
khaled has joined #systemtap
orivej has joined #systemtap
mjw has joined #systemtap
orivej has quit [Ping timeout: 260 seconds]
hpt has quit [Ping timeout: 264 seconds]
orivej has joined #systemtap
orivej has quit [Ping timeout: 264 seconds]
wcohen has quit [Remote host closed the connection]
wcohen has joined #systemtap
orivej has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
tromey has joined #systemtap
orivej has joined #systemtap
orivej has quit [Ping timeout: 260 seconds]
amerey has joined #systemtap
orivej has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
<kerneltoast>
fche, i'm surprised you think the patch is plausible and not an atrocity :P
<kerneltoast>
add_timer_on is used safely but it needs to go anyway, since it doesn't exist on 2.6.32
<fche>
yeah, just surprised add_timer_on is Safe in all our contexts when add_work or whatnot isn't
<kerneltoast>
add_timer_on doesn't have a circular dependency on the runqueue lock
<kerneltoast>
which is lucky but also dirty :)
<fche>
yes but it might have a dependency on something else held while e.g. timer-related tracepoints or kprobes fire
<kerneltoast>
fche, excellent point. you've just made this harder to backport
<kerneltoast>
the !timer_pending(&log->timer) check on add_timer_on prevents a similar deadlock
<kerneltoast>
i guess i can just use a schedule_work_on directly then and do the same kind of check
<kerneltoast>
no i can't
<kerneltoast>
so, older kernels don't have add_timer_on
<kerneltoast>
the timer_pending check isn't safe without add_timer_on and TIMER_IRQSAFE | TIMER_PINNED
<kerneltoast>
i used timer_pending while IRQs are disabled on the local CPU and while it is guaranteed that the timer will only run on the specified CPU (thanks to add_timer_on and TIMER_PINNED), and it will run to completion without being interrupted (thanks to TIMER_IRQSAFE)
<kerneltoast>
so my timer_pending check actually implies that the timer is done running when it is not pending
<fche>
yes but consider the reentrancy possibility - fired from a kprobe within the timer subsystem
<kerneltoast>
yep i am
<kerneltoast>
timer_pending will block it
<kerneltoast>
if the timer is queued but not executed
<kerneltoast>
timer_pending doesn't use any locks
<kerneltoast>
oh but
<kerneltoast>
it could get queued for the first time inside a kprobe inside the timer subsystem
<kerneltoast>
😊🔫
<fche>
don't ship it
<kerneltoast>
ok so WAT DO
<kerneltoast>
i'm outta ideas
<kerneltoast>
we could poll
<kerneltoast>
that's the nuclear option
<fche>
can you avoid doing the wakeup, and hope that some other probe point comes along with enough wakeup moxy to get it done?
<kerneltoast>
can't
<kerneltoast>
i tried that
<kerneltoast>
our lean test suite got angery
<fche>
angery is bangery
amerey has quit [Remote host closed the connection]
amerey has joined #systemtap
orivej has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
<kerneltoast>
agentzh, what kind of garbled data were you seeing without the inode lock? If the only source of concurrency writing to that inode is from stap, then we can just replace the inode lock with our own spinlock
<kerneltoast>
if there was data passing through the inode from elsewhere or multiple different stap modules then we need the inode lock
<agentzh>
kerneltoast: it is only stap.
<agentzh>
but the other end (staprun/stapio) uses the read syscall to read data.
<agentzh>
if we can avoid that read syscall, then we are good.
<agentzh>
that read syscall uses the inode mutex.
<kerneltoast>
agentzh, we can replace that with a custom sysfs file to read the log from
<kerneltoast>
fche, does that sound feasible?
<kerneltoast>
I'm not sure why an inode is abused to extract logs from stap modules; do you know why that is, instead of just using sysfs?
<agentzh>
it's the standard relayfs API, i think.
<fche>
custom transport i.e., replacing relayfs ......... that's a big change
<fche>
it's not the inode only per se
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
<kerneltoast>
bigger than the print patch i've shown you?
<fche>
yeah, getting rid of relayfs's transport would be a big deal
tromey has quit [Quit: ERC (IRC client for Emacs 27.1.50)]
<kerneltoast>
darn
<kerneltoast>
hrtimers have tracepoints inside locks as well
<kerneltoast>
fche, i've got something less confusing to ask you about as well
<fche>
shoot
<kerneltoast>
could you take a look at the CANCEL_IF_WORK and FREE_IF_WORK macros?
<kerneltoast>
FREE_IF_WORK always prints a message when the task_work_cancel fails
<kerneltoast>
CANCEL_IF_WORK only prints such a message when STP_TF_DEBUG is enabled
<kerneltoast>
should FREE_IF_WORK be brought to parity with CANCEL_IF_WORK?
fche has quit [Remote host closed the connection]
fche has joined #systemtap
fche has quit [Changing host]
fche has joined #systemtap
<fche>
eww, that's confusing
<fche>
and FREE_IF_WORK just apes task_work_cancel()'s message so you can't even tell which one did it
<kerneltoast>
we saw the message from FREE_IF_WORK (for utrace_resume) and i suspect it was caused by a failure to add the utrace_resume work
<kerneltoast>
since >resume_work_added can race in stp_task_notify_resume() when stp_task_work_add() fails
<kerneltoast>
so FREE_IF_WORK could think there's a task work queued up when there never was one
<fche>
it'd be nice to make the STP_TF_DEBUG conditionality work by other than duplicating the macro body with and without the prink
<kerneltoast>
do you see any way that the printk could indicate a fatal issue?
<kerneltoast>
i checked and i think it's fine, assuming my theory behind the cause of the issue is correct
<fche>
sorry, not familiar in the least
<kerneltoast>
excellent
khaled has quit [Remote host closed the connection]
khaled has joined #systemtap
derek0883 has quit [Remote host closed the connection]