00:01
derek0883 has quit [Remote host closed the connection]
00:37
mjw has quit [Quit: Leaving]
00:41
orivej has quit [Ping timeout: 246 seconds]
01:06
lijunlong has quit [Read error: Connection reset by peer]
01:10
derek0883 has joined #systemtap
01:11
lijunlong has joined #systemtap
01:22
derek0883 has quit [Remote host closed the connection]
01:22
derek0883 has joined #systemtap
01:23
<
irker300 >
systemtap: fche systemtap.git:master * release-4.4-1-g1f608d213 / runtime/procfs.c runtime/transport/procfs.c: PR26665: relayfs-on-procfs megapatch, rhel6 tweaks
01:23
<
irker300 >
systemtap: fche systemtap.git:master * release-4.4-2-g931e0870a / po/cs.po po/en.po po/fr.po po/pl.po po/systemtap.pot: releng: update-po
01:28
hpt has joined #systemtap
01:43
orivej has joined #systemtap
02:32
orivej has quit [Ping timeout: 240 seconds]
03:14
<
kerneltoast >
fche, hiya
03:14
<
fche >
going to bed
03:14
<
kerneltoast >
hey it was a quick thing
03:14
<
fche >
NOT WANTING TO HEAR OF ANOTHER BUG
03:15
<
kerneltoast >
*is a quick thing
03:15
<
kerneltoast >
turns out checking preempt_count() was nonsense because the centos 7 kernel doesn't have CONFIG_PREEMPT_COUNT enabled
03:15
<
kerneltoast >
so disregard any of my findings involving preempt_count()
03:15
<
irker300 >
systemtap: fche systemtap.git:master * release-4.4-3-g34e62f15d / runtime/stp_utrace.c: RHBZ1892179: handle exhausted stp_task_work structs
03:16
<
kerneltoast >
and lend me a shoulder to cry on
03:16
<
kerneltoast >
as i start debugging this back from step 1
03:16
<
kerneltoast >
:))))))))))))
03:18
<
fche >
haha shpi it
03:19
<
kerneltoast >
kILL mE
03:19
<
fche >
ummmm no thanks
03:19
<
kerneltoast >
sigkill me?
03:19
<
kerneltoast >
how about sigsegv
03:20
<
kerneltoast >
anyway feel free to resume your canadian things
03:24
<
fche >
those things consist of ... well, sleeping.
04:22
khaled__ has quit [Quit: Konversation terminated!]
04:26
derek0883 has quit [Remote host closed the connection]
04:35
<
agentzh >
fche: lockdep kernel found some old utrace deadlock bug which kerneltoast has prepared a patch. we're currently testing it. will share it here for your review. just FYI. sleep well :)
04:38
derek0883 has joined #systemtap
05:02
<
kerneltoast >
mutex_trylock used inside an interrupt
05:02
<
kerneltoast >
* This function must not be used in interrupt context. The
05:02
<
kerneltoast >
* mutex must be released by the same task that acquired it.
05:25
_whitelogger has joined #systemtap
05:51
orivej has joined #systemtap
05:53
derek0883 has quit [Remote host closed the connection]
05:59
<
agentzh >
kerneltoast fche: okay, it was introduced by one of my recent patches, like a few months ago?
05:59
<
agentzh >
i didn't know mutex_trylock() is unsafe in interrupt contexts.
05:59
<
agentzh >
according to kerneltoast, maybe we can simply disable preempts around that lock?
06:00
<
agentzh >
otherwise we would have to avoid print flushing in all those interrupt contexts.
06:00
<
kerneltoast >
disable interrupts, you mean
06:00
<
agentzh >
i'm fine with either approach. the 2nd approach would require much larger changeset. we currently have no reliable way to know if we are in an interrupt context.
06:01
<
agentzh >
the probe handler caller has to pass down such info.
06:01
<
agentzh >
oh yeah, disable interrupts. sorry.
06:01
beauty1 has quit [Ping timeout: 260 seconds]
06:02
<
agentzh >
this is a fundamental problem we have to address cleanly.
06:05
derek0883 has joined #systemtap
06:06
derek0883 has quit [Remote host closed the connection]
06:06
derek0883 has joined #systemtap
06:12
derek0883 has quit [Ping timeout: 244 seconds]
06:15
beauty1 has joined #systemtap
06:40
orivej has quit [Ping timeout: 246 seconds]
06:43
<
kerneltoast >
the .c file for the faulting stap module is in that gist as well (scroll down)
06:52
<
agentzh >
kerneltoast: seems like the first error is "BUG: unable to handle kernel paging request at ffffe8ffff621000"?
06:53
<
agentzh >
instead of the soft lockup?
06:53
orivej has joined #systemtap
06:54
<
agentzh >
it's in kretprobes, it seems.
06:54
<
agentzh >
and also in stp_lock_probe().
09:22
hpt has quit [Ping timeout: 265 seconds]
10:45
khaled__ has joined #systemtap
10:52
mjw has joined #systemtap
11:20
<
fche >
agentzh, kerneltoast, 'morning
11:21
<
fche >
both those tracebacks appear to relate to printing
11:22
<
fche >
yeah that inode trylock stuff sounds agentzh-ish familiar, doesn't it
11:22
<
fche >
and yeah we should be strictly atomic down in this part of the code, non-mutexy
11:28
orivej has quit [Ping timeout: 265 seconds]
11:52
<
fche >
agentzh, kerneltoast, guys, I feel so let down with you not being here all night with mne
11:52
<
fche >
it's just not the same
13:46
tromey has joined #systemtap
14:21
derek0883 has joined #systemtap
14:26
derek0883 has quit [Ping timeout: 260 seconds]
15:19
orivej has joined #systemtap
15:29
xlei has joined #systemtap
16:31
<
kerneltoast >
fche, as much as I'd love to debug stap at 3am, i do enjoy sleeping :)
17:09
derek0883 has joined #systemtap
17:12
<
fche >
it's a tossup for me
17:12
<
fche >
sleep or debugging
17:12
<
fche >
tossing a coin every night
17:30
<
kerneltoast >
true, I've made major breakthroughs at 4am
17:31
<
kerneltoast >
and then when i read my code the next morning and realize i pushed it, i start feeling a knot in my stomach
17:31
<
kerneltoast >
I have a no-push rule after midnight
17:34
<
fche >
every moment is after -some- midnight tho
17:35
<
kerneltoast >
oh boy, does that mean i've been pushing garbage this whole time?
17:35
<
fche >
has always been . gif
17:35
<
kerneltoast >
i thought that was a static meme
17:35
<
kerneltoast >
not a gif
17:36
<
fche >
close enough!
17:49
* fche
is old enough to remember gifs used as static images
17:50
<
fche >
the Before Times of <1992 when JPEG came on the scene
17:56
derek0883 has quit [Remote host closed the connection]
17:57
derek0883 has joined #systemtap
18:09
derek0883 has quit [Remote host closed the connection]
18:11
derek0883 has joined #systemtap
18:13
<
kerneltoast >
wow, before i was born
18:14
<
kerneltoast >
that must've been right after task finder was written
18:14
<
fche >
no, no, task finder comes from the paleozolic era
18:15
<
kerneltoast >
ah, of course. i meant task finder2
18:15
<
fche >
now now anyway it's fine code for its day and its context, and glad we're making it better
18:17
<
kerneltoast >
i wonder how often the original stap authors thought about giving up
18:17
<
fche >
never give up, never surrender
18:18
<
kerneltoast >
there's just so much duct tape
18:18
<
kerneltoast >
and then duct tape to reinforce the duct tape
18:18
<
kerneltoast >
and then a kernel update breaks one layer of duct tape
18:18
<
kerneltoast >
so you gotta duct tape that
18:19
<
fche >
well, such is life with the (lack of) constraints of the kernel api evolution
18:19
<
fche >
but that's ok
18:19
<
fche >
did y'all have any good news re. the bugs being chased down?
18:20
<
kerneltoast >
the crash happened here because `str` was invalid:
18:20
<
kerneltoast >
for (i = 0; i < len && str <= end; ++i)
18:20
<
kerneltoast >
*str++ = *ptr++;
18:20
<
kerneltoast >
else/* %s format */
18:21
<
kerneltoast >
and that snippet is from _stp_vsprint_memory, which was called from stp_printf_2
18:21
<
kerneltoast >
stp_printf_2 generates `str` as follows:
18:21
<
kerneltoast >
str = (char*)_stp_reserve_bytes(num_bytes);
18:21
<
kerneltoast >
so the pointer from _stp_reserve_bytes was poop
18:22
<
fche >
well that's a poopy development
18:22
<
fche >
what was the ptr?
18:22
<
kerneltoast >
BUG: unable to handle kernel paging request at ffffe8ffff621000
18:22
<
kerneltoast >
must be use-after-free i guess
18:23
<
fche >
weird, wonder where that range of values points to
18:23
<
kerneltoast >
you mean that address?
18:25
<
kerneltoast >
well, dmesg says the faulting process' task struct address is ffff88001f4f0000, and this is a little bit after that
18:25
<
kerneltoast >
so probably just kernel heap
18:26
<
kerneltoast >
(since task structs are allocated from the kernel heap)
18:27
<
kerneltoast >
i would try and find out when the values from _stp_reserve_bytes aren't safe to use
18:28
<
fche >
maybe back up on that inode mutex thingamabob from earlier on, and switch the mindset to "this must be atomic code"
18:28
<
kerneltoast >
not, "this must be atomic code", because it is atomic
18:29
<
kerneltoast >
but a mutex cannot be owned by interrupt context because it's not backed by a task struct
18:29
<
kerneltoast >
so the mutex owner would be nonsense, and that's what lockdep was warning about
18:31
irker300 has quit [Quit: transmission timeout]
18:33
<
kerneltoast >
okay, it's an out of bounds issue
18:33
<
kerneltoast >
crash> kmem ffffe8ffff621000
18:33
<
kerneltoast >
ffffe8ffff621000: kernel virtual address not found in mem map
18:33
<
kerneltoast >
crash> kmem ffffe8ffff620fff
18:33
<
kerneltoast >
kmem: WARNING: cannot make virtual-to-physical translation: ffffe8ffff621000
18:33
<
kerneltoast >
VMAP_AREA VM_STRUCT ADDRESS RANGE SIZE
18:33
<
kerneltoast >
ffff880174af3800 ffff880174aea700 ffffe8fffde00000 - ffffe8ffffe00000 33554432
18:33
<
kerneltoast >
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
18:33
<
kerneltoast >
ffffea0005704a40 15c129000 0 ffff880174ae9e00 1 2fffff00000000
18:33
<
kerneltoast >
crash> kmem ffffe8ffff621000
18:33
<
kerneltoast >
kmem: WARNING: cannot make virtual-to-physical translation: ffffe8ffff621000
18:33
<
kerneltoast >
ffffe8ffff621000: kernel virtual address not found in mem map
18:34
<
kerneltoast >
crash>
18:34
<
kerneltoast >
oops i pasted too much
18:34
<
kerneltoast >
ffffe8ffff621000 is the faulting address, ffffe8ffff620fff is 1 byte before the faulting address
18:34
<
kerneltoast >
ffffe8ffff620fff is valid, ffffe8ffff621000 is not
18:35
<
kerneltoast >
so da loopdy loop went too far
18:40
<
fche >
hey, getting used to crash(1), good job
18:40
<
fche >
can you tell whether the loop has in fact overflown the thing, vs. started off at the bad address?
18:41
<
kerneltoast >
lemme see which register has `i` in it...
18:42
<
kerneltoast >
back into ghidra we go
18:47
<
kerneltoast >
`i` got optimized out, nice
18:48
<
fche >
step up out of the function and look there?
18:52
<
kerneltoast >
ok, the `end` pointer is in %r13, which is ffffe8ffff621004
18:52
<
kerneltoast >
so it started off at a bad address
18:53
<
kerneltoast >
_stp_reserve_bytes didn't reserve enough bytez
18:56
<
kerneltoast >
_stp_reserve_bytes relies on _stp_print_flush to free up space
18:56
<
kerneltoast >
but _stp_print_flush can return prematurely due to lock contention
18:57
<
kerneltoast >
in which case pb->len will not be adjusted
18:57
<
kerneltoast >
err no, pb->len is just set to 0
18:57
<
kerneltoast >
in stp_print_flush
18:58
<
kerneltoast >
before anything else is done
18:58
<
kerneltoast >
i suspect the issue is that the prints can be called from inside an interrupt, which ties back to the other bug
18:59
<
kerneltoast >
so pb->len is unreliable. we need to disable interrupts when reading pb->len inside _stp_reserve_bytes
19:00
<
fche >
lock contention -> dropped data is an acceptable price
19:01
<
fche >
but yeah when I say "atomic" it means "callable from any context, incl. interrupts etc."
19:01
<
kerneltoast >
supermegaatomic
19:01
<
kerneltoast >
so there are a couple things we can do
19:02
<
kerneltoast >
we could forbid printing inside interrupt probes at the translation layer
19:02
<
kerneltoast >
which would take care of the mutex bug as well
19:03
<
kerneltoast >
but i dunno how to catch every probe that's in an interrupt
19:03
<
kerneltoast >
so that might not be feasible
19:05
<
kerneltoast >
we can still do printing inside an interrupt, but we can only add those messages to the buffer. flushing isn't allowed
19:05
<
kerneltoast >
in that case, we'd just need to sprinkle in some local_irq_save into _stp_reserve_bytes and _stp_unreserve_bytes
19:06
<
kerneltoast >
and then add some machinery to forbid _stp_print_flush from running in an interrupt
19:14
<
kerneltoast >
this is a pain to fix :)
19:15
<
fche >
suppressing the flush is probably a manageable thing
19:15
<
fche >
heck it's probably a job for another task_work kind of callback maybe
19:15
<
kerneltoast >
no easy way to check if we're in an interrupt though
19:16
<
kerneltoast >
yeah we can dump it onto a regular worker but then we'll have no way to clear up the buffer from _stp_reserve_bytes
19:16
<
fche >
ok, like for the task-finder, delegating things to a task_work by default sounds fine to me
19:16
<
kerneltoast >
so we're gonna be dropping messages like crazy
19:17
<
fche >
well let's hope not
19:17
<
kerneltoast >
this check inside _stp_reserve_bytes will be converted to a message drop:
19:17
<
kerneltoast >
if (unlikely(numbytes > size))
19:17
<
kerneltoast >
_stp_print_flush();
19:18
<
fche >
(there is a 'dropped' counter)
19:18
<
fche >
and we could use this opportuhnity to bump up default buffer sizes if indeed this is hit much more
19:18
<
kerneltoast >
i don't see any dropped counter being used
19:19
<
kerneltoast >
there's no such thing in the parent function calling _stp_reserve_bytes
19:19
<
fche >
hmmmmmmmmmmmmmmm
19:19
<
fche >
atomic_inc(&_stp_relay_data.dropped); <<< there's that but I thought we had another
19:19
<
fche >
(that's relay_v2.c)
19:20
<
kerneltoast >
if str is null it just silently drops the message
19:21
<
fche >
let's add an atomic_t counter of those events happening and then figure out when/how to best print them
19:21
<
fche >
probably near sthutdown an stp_warn() thing
19:22
<
fche >
my my I'm surprised at a call to stp_print_flush from deep down atomic code like that, ... does seem fishy!
19:23
<
kerneltoast >
the atomic counter would need to be added inside translate.cxx and i think my brain will explode if i try doing it
19:24
<
kerneltoast >
unless you want it straight inside _stp_reserve_bytes
19:24
<
fche >
in whatever file has that function, yes
19:29
<
kerneltoast >
time to rewrite runtime/linux/print.c
19:30
<
fche >
on the bright side it's not big
19:30
<
fche >
but rewrite? er probably unnecessary.
19:31
<
kerneltoast >
the problem goes much deeper than _stp_reserve_bytes
19:31
<
kerneltoast >
there's also _stp_print and _stp_print_char
19:32
<
kerneltoast >
and _stp_vlog in runtime/linux/io.c
19:32
<
kerneltoast >
which uses a pointer allocated inside print.c (Stp_lbuf)
19:33
<
kerneltoast >
and also calls print_flush
19:34
<
kerneltoast >
though at least it's not relying on _stp_print_flush to free up space in the buffer
19:34
<
fche >
yeah all this code should be atomic, so print_flush needs to become atomic (and not assumed capable of freeing up space)
19:34
<
kerneltoast >
yep and that's why it's gonna look like a rewrite
19:36
<
fche >
hm there we have that _stp_transport_failure atomic in print_flush.c
19:37
<
kerneltoast >
that should be moved into print.c i guess
19:38
<
fche >
yeah probably
19:39
<
fche >
wdyt, would have it done by tomorrow morning, say? who needs to sleep??!?!?!
19:40
<
kerneltoast >
does your irc client support utf?
19:41
<
fche >
dunno if that's a steak or texas on the right
19:42
<
kerneltoast >
squint harder
19:44
<
fche >
oh god .... "watergun" emoji .... is that the PC version of "gun" or is that a separate thing?
19:46
<
fche >
Unicode Character 'PISTOL' (U+1F52B) sigh
19:49
<
kerneltoast >
you can thank apple for initially ruining it
19:49
<
kerneltoast >
then everyone else followed suit
19:50
<
fche >
thanks apple for initially ruining it etc. etc. etc.
19:50
<
fche >
ok so thanks for digging into this aread
19:50
<
kerneltoast >
i'd like to thank my friend ghidra for making this possible
19:51
<
kerneltoast >
along with objdump and crash(1)
20:01
orivej has quit [Ping timeout: 246 seconds]
20:12
<
kerneltoast >
i'm waiting
20:12
<
fche >
I'm not giving you a trophy or something, and this is not a stage
20:13
<
fche >
there is no fancy-dressup party
20:13
<
fche >
so save your "thanks to your agent and your publicist" for later :)
20:13
<
kerneltoast >
can i thank my agentzh?
20:43
khaled__ has quit [Quit: Konversation terminated!]
20:51
tromey has quit [Quit: ERC (IRC client for Emacs 27.1.50)]
21:06
zodbot has quit [Read error: Connection reset by peer]
21:15
derek0883 has quit [Remote host closed the connection]
21:17
<
agentzh >
seems like we need more kernel geeks in EU to cover 24x7 :D
21:18
<
agentzh >
oh, aisa too.
21:18
<
agentzh >
fche kerneltoast: i think we need a way to pass down the info about whether it is an interrupt context probe.
21:18
<
agentzh >
we lack such info atm.
21:18
<
agentzh >
fed by the probe handler caller.
21:19
<
agentzh >
in_atomic() and in_interrupt() are unreliable as we've seen.
21:19
<
agentzh >
fche: you mentioned you were working on the new transport impl. how was that going?
21:19
<
kerneltoast >
agentzh, that seems more difficult to orchestrate
21:19
<
agentzh >
do we need to coordinate?
21:19
<
fche >
it's not that new or big
21:20
<
fche >
agentzh, can't really pass in ... we can' tgenerally know
21:20
<
fche >
so assume YES
21:20
<
agentzh >
so we just assume the worst?
21:20
<
agentzh >
then we would never do print_flush() in probe handler?
21:21
<
agentzh >
that's...sad...
21:21
<
agentzh >
we may flood the work queue?
21:21
derek0883 has joined #systemtap
21:21
<
kerneltoast >
flooding a workqueue is not a problem
21:22
<
agentzh >
kerneltoast: you mentioned that we could simply disable interrupts around the mutex trylock thing?
21:22
<
agentzh >
that's already off the table now?
21:22
<
kerneltoast >
agentzh, yeah that's not correct
21:22
<
kerneltoast >
i was mistaken
21:22
<
agentzh >
then it'll be a relatively large patch.
21:23
<
kerneltoast >
we can't acquire the mutex at all inside an interrupt, as we discussed
21:23
<
kerneltoast >
yep big patch
21:23
<
agentzh >
okay, i saw the messages now.
21:23
<
kerneltoast >
the print_flush will be delegated to schedule_work_on()
21:23
<
kerneltoast >
and we might need to increase the buffer size if we end up dropping a lot of messages
21:23
<
agentzh >
so many fundamental changes lately :)
21:23
<
kerneltoast >
yep, no rest for me
21:24
<
kerneltoast >
fix one bug and another comes crawling out :)
21:24
<
agentzh >
i'm worried about large buffers.
21:24
<
agentzh >
since we need to run it in embedded systems....
21:24
<
agentzh >
where ram is very limited.
21:24
<
kerneltoast >
stap is already quite liberal with its memory usage though
21:25
<
agentzh >
just don't want it to be a lot worse :)
21:25
<
kerneltoast >
how small of a system are you thinking of?
21:25
<
agentzh >
1g at most...
21:25
<
agentzh >
some servers at also in this range.
21:26
<
agentzh >
like the cheapest cloud servers.
21:26
<
kerneltoast >
stap currently works fine on these systems?
21:26
<
agentzh >
quirky on 512m, fine on 1g.
21:26
<
agentzh >
we have a lot of 1g servers ourselves in our mini CDN network.
21:26
<
agentzh >
it's been fine.
21:27
<
agentzh >
also we may have to run stap to debug large memory usage when the system is already swapping and etc...
21:27
<
agentzh >
so the smaller the better...
21:28
<
agentzh >
oh well fortunately we only output stuff in probe end {} and probe timer.s() ourselves.
21:28
<
agentzh >
not very frequently.
21:28
<
agentzh >
it'll be crazy to print() inside probe timer.profile.
21:29
zodbot has joined #systemtap
21:29
<
agentzh >
some stap tests do that for the sake of testing.
21:29
<
agentzh >
but that's it.
21:29
<
kerneltoast >
and that makes it all explode. i hope that's the cause for the lockup, and we'll get all three of these issues knocked out in one patch
21:30
<
agentzh >
the kernel space programming is a horror world due to all those interrupts, preempts, and locks.
21:30
<
agentzh >
i had a lot of sleepless nights due to those things.
21:31
<
kerneltoast >
hopefully you're sleeping better now :)
21:31
<
agentzh >
lockdep makes my life easier.
21:31
<
kerneltoast >
though maybe not while i'm running stuff on your machines
21:31
<
agentzh >
yeah, thanks to kerneltoast :)
21:31
<
agentzh >
i sleep a lot better now :)
21:31
<
agentzh >
don't worry about machines, we now shut the doors of the bedroom :)
21:32
<
agentzh >
so all yours.
21:32
<
kerneltoast >
awesome
21:33
derek0883 has quit [Remote host closed the connection]
21:33
<
agentzh >
yeah, fche doesn't mind big patches.
21:34
<
agentzh >
as long as you run his giant test suite :)
21:34
<
fche >
run it twice for good luck ;)
21:34
<
agentzh >
or even more times...
21:35
* agentzh
has already offered his big machines for kerneltoast to burn.
21:35
* agentzh
has solar panels on his roof fortunately.
21:37
wcohen has quit [*.net *.split]
21:37
CME has quit [*.net *.split]
21:37
wcohen has joined #systemtap
21:37
CME has joined #systemtap
21:39
mjw has quit [Quit: Leaving]
21:40
fLiPr3VeRsE has quit [*.net *.split]
21:40
modem has quit [*.net *.split]
21:40
xlei has quit [*.net *.split]
21:40
sscox has quit [*.net *.split]
21:40
eichiro has quit [*.net *.split]
21:41
eichiro has joined #systemtap
21:41
xlei has joined #systemtap
21:41
sscox has joined #systemtap
21:41
zodbot has quit [*.net *.split]
21:41
jistone has quit [*.net *.split]
21:41
modem has joined #systemtap
21:41
fLiPr3VeRsE has joined #systemtap
21:41
DTEIT has quit [*.net *.split]
21:42
DTEIT has joined #systemtap
21:42
zodbot has joined #systemtap
21:42
jistone has joined #systemtap
21:46
DUKENUKEM has quit [*.net *.split]
21:46
darvon has quit [*.net *.split]
21:46
serhei has quit [*.net *.split]
21:46
tux3 has quit [*.net *.split]
21:46
thibaultcha has quit [*.net *.split]
21:47
serhei has joined #systemtap
21:47
darvon has joined #systemtap
21:47
DUKENUKEM has joined #systemtap
21:49
* fche
has bad news about the coming season
21:50
thibaultcha has joined #systemtap
21:50
tux3 has joined #systemtap
21:51
ema has quit [*.net *.split]
21:51
zamba has quit [*.net *.split]
21:53
zamba has joined #systemtap
21:53
ema has joined #systemtap
21:53
gavinguo___ has quit [*.net *.split]
21:53
gavinguo___ has joined #systemtap
21:53
lijunlong has quit [*.net *.split]
21:53
pviktori has quit [*.net *.split]
21:53
przemoc has quit [*.net *.split]
21:53
kerneltoast has quit [*.net *.split]
21:53
pviktori has joined #systemtap
21:53
przemoc has joined #systemtap
21:53
lijunlong has joined #systemtap
21:53
kerneltoast has joined #systemtap
21:54
ggherdov has quit [*.net *.split]
21:54
xar- has quit [*.net *.split]
21:54
lindi- has quit [*.net *.split]
21:54
tonyj has quit [*.net *.split]
21:54
agentzh has quit [*.net *.split]
21:54
fche has quit [*.net *.split]
21:54
xar- has joined #systemtap
21:54
ggherdov has joined #systemtap
21:54
lindi- has joined #systemtap
21:55
tonyj has joined #systemtap
21:55
agentzh has joined #systemtap
21:55
fche has joined #systemtap
21:56
ggherdov has quit [Ping timeout: 254 seconds]
21:58
derek0883 has joined #systemtap
22:04
khaled__ has joined #systemtap
22:06
ggherdov has joined #systemtap
23:00
derek0883 has quit [Ping timeout: 246 seconds]
23:03
derek0883 has joined #systemtap
23:33
orivej has joined #systemtap