fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
przemoc has joined #systemtap
hpt has joined #systemtap
mjw has quit [Quit: Leaving]
orivej has joined #systemtap
_whitelogger has joined #systemtap
Amy1 has quit [Ping timeout: 246 seconds]
Amy1 has joined #systemtap
khaled has joined #systemtap
sscox has quit [Ping timeout: 252 seconds]
sscox has joined #systemtap
sscox has quit [Ping timeout: 252 seconds]
sscox has joined #systemtap
hpt has quit [Ping timeout: 240 seconds]
pviktori has quit [Remote host closed the connection]
pviktori has joined #systemtap
mjw has joined #systemtap
sapatel has joined #systemtap
tromey has joined #systemtap
amerey has joined #systemtap
khaled has quit [Remote host closed the connection]
<Amy1>
how to run stap xx.stp yy.bin and run yy.bin in another path?
<Amy1>
how to run ”stap xx.stp yy.bin“ and run ”yy.bin“ in another path?
<fche>
you mean stap -c yy.bin &; cp yy.bin zz.bin; ./zz.bin ?
<Amy1>
I mean that how to run ”stap xx.stp yy.bin“ and run ”yy.bin“ in another path?
<fche>
right now, process probes are path-match based
<fche>
so probe process("./a.out") ... and the running a.out from another path - even the same file - will not match
<Amy1>
if there are no pattern about path1 and path2, how to write the glob?
<fche>
{/path1,/path2} maybe
<fche>
or
<fche>
one probe per path
<fche>
keep trying stuff
yogananth has quit [Read error: Connection reset by peer]
<Amy1>
semantic error: no match
<Amy1>
if I use process("/home/amy/*/MD") and run MD in /home/amy/dir2
<Amy1>
the result is wrong
<Amy1>
all values are 0
<fche>
should work ... stap -t to trace counts of probe hits
<Amy1>
WARNING: missing unwind/symbol data for module
<Amy1>
all prited values are 0
<Amy1>
stap traced values are all error
<Amy1>
fche: ?
<agentzh>
Amy1: maybe you can try stap -vvvv to see more details on your side.
<agentzh>
i've never used globs in probes myself though.
<agentzh>
fche: i see stap currently uses vmalloc for maps and ctxs. but the mempool and task finder still use kmalloc for memory blocks potentially larger than the page size. shall we switch the latter two to vmalloc?
<agentzh>
i also see kmalloc is used for potentially large blocks in stap runtime's addr-map. is it worth migrating too?
<agentzh>
we often run into ko loading failures due to the mempool requires large physically contiguous memory blocks in th mempool when allocating ctl buffers for the channels.
<fche>
we've flipped this way and that w.r.t. kmalloc and vmalloc and stuff over the years
<fche>
how large memory are we talking about ?
<agentzh>
78KB for a single block.
<agentzh>
at maximum for one of our stp scripts.
<agentzh>
i traced all the kmalloc allocations for the .ko module.
<Amy1>
agentzh: thanks
<agentzh>
any specific reasons for using kmalloc for mempool and task finder? do they involve DMA in any way?
<agentzh>
right now we have to flush page cache or to do a memory compatction to allow the .ko to load successfully.
<agentzh>
it's especially common on machines with small memory or long running servers.
<fche>
we can't take kernel-side soft page faults from probe context, that's one complication
<agentzh>
seems like they were not invoked by probe handlers, but init_module handlers?
<fche>
at reference time I mean
<fche>
ISTR some types of kernel allocated memory still uses an MMU based mapping that's filled via kernel-side page faults
<agentzh>
hmm, but stap maps are already using vmalloc?
<agentzh>
stap maps are surely referenced from within probe handlers already.
<fche>
yeah
<fche>
so which allocation proper are you looking at ?
<agentzh>
vmalloc avoids memory blocks from being swapped out.
<agentzh>
so there won't be any page faults anyway?
<fche>
(kernel can't swap itself anyway)
<agentzh>
a sec
<agentzh>
runtime/mempool.c:60 m = (struct _stp_mem_buffer *)_stp_kmalloc(alloc_size);
<agentzh>
m = (struct _stp_mem_buffer *)_stp_kmalloc(alloc_size);
<agentzh>
and also: runtime/linux/task_finder2.c:1197 vma_cache = _stp_kmalloc(sizeof(struct vma_cache_t) * file_based_vmas);
<agentzh>
btw what happens when a soft page fault is triggered in a probe handler? panic? freezes?
<fche>
we disable interrupts routinely while running probe handlers; the kernel is/was unhapy about taking faults at such times
<agentzh>
i see the stap code always uses kzalloc() so the vmalloc'd memory pages are already in the kernel master page tables?
<agentzh>
so no page faults will be possible?
<fche>
my memory is not fresh on the rationale
<fche>
feel free to experiment & report findings !
<agentzh>
okay, i'll do some experiments.
<agentzh>
so what do you mean by "unhappy". we'll just get 0 when a soft page fault is needed?
<agentzh>
sorry, let me rephrase. when the interruptes are disabled, what is the behavior when reading a page requiring a page fault?
<agentzh>
we get an EFAULT erorr code?
* fche
remembers crashes
<agentzh>
ah
<agentzh>
fche: another quick queston, i found that when using kprobes to probe on a kernel function in the .ko module generated by stap, print_backtrace() fails to provide the full kernel backtrace. but calling the kernel function dump_stack() works (though with "?" frame prefixes). is it a known problem?
<agentzh>
wondering if we can simply borrow the code from dump_stack()/
<agentzh>
?
<agentzh>
the tracee module was compiled with debuginfo and -fno-omit-frame-pointer already.
<fche>
stap --all-modules ...
<fche>
or stap -d tracee-module
<agentzh>
already added both, same thing.
<agentzh>
tested on both fedora 28 and 30 (x86_64).
<fche>
it oughta work in that case
<fche>
(we don't need frame pointers; we use dwarf unwinding)
<agentzh>
i'll add some debugging code inside print_backtrace() then.
<fche>
there is some -D already
<agentzh>
okay
<agentzh>
another thing i've noted is that it seems like when tracee modue is loaded *after* the tracer module, symbols won't get used for probefunc() or print_backtrace().
<agentzh>
but when the tracee is loaded before the tracer, symbols look fine.
<agentzh>
do we need something similar to task finder for dynmaically loaded kernel modules?
<fche>
hm, there is a module_notifier chain that is/was there that we used for this
<agentzh>
interesting. i'll dig it up to see why the symbols are not used.
<agentzh>
thanks
<agentzh>
fche: i've checked the userland memory page causing a read fault in stap scripts, it has read permissions in /proc/PID/maps.
<agentzh>
it's really strange.
<agentzh>
/proc/PID/pagemap also shows that page is present and not swapped and has a page frame number.
<agentzh>
i traced the user_long_error() tapset function being executed, it seems EFAULT was thrown by the kernel's __get_user() API.
<agentzh>
any suggestions on how to trace this further?
<agentzh>
many thanks
<agentzh>
and it happens on bare metal.
<agentzh>
100% reproducible for that particular user process and userland address.
<fche>
works on other processes? other addresses?
<agentzh>
i've been scratching my head for days on this issue.
<agentzh>
yeah, working on other processes.
<agentzh>
*works
<agentzh>
that memory is not shared and is exclusively mapped for the target process.
<agentzh>
*that memory page
<agentzh>
it's swapbacked.
<agentzh>
but not in swapcache nor is swaped out.
<agentzh>
could this be a hardware issue?
<agentzh>
like a faulty ram stick or DIMM slot?
<fche>
i doubt that'd show this way
<agentzh>
okay
<agentzh>
that box is using 4.15 kernel, but recent.
<agentzh>
*quite recent
<agentzh>
4.15.18, to be exact.
Amy1 has quit [Killed (Sigyn (Stay safe off irc))]
tromey has quit [Quit: ERC (IRC client for Emacs 28.0.50)]