fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
wmealing has joined #systemtap
scox has joined #systemtap
hpt has joined #systemtap
hkshaw has joined #systemtap
hkshaw has quit [Ping timeout: 245 seconds]
srikar_away is now known as srikar
hkshaw has joined #systemtap
atomlin has joined #systemtap
atomlin has left #systemtap [#systemtap]
srikar is now known as srikar_away
irker602 has quit [Quit: transmission timeout]
srikar_away is now known as srikar
wmealing has quit [Quit: Leaving.]
wmealing has joined #systemtap
wmealing1 has joined #systemtap
wmealing has quit [Read error: Connection reset by peer]
naveen has joined #systemtap
ego has joined #systemtap
ananth has joined #systemtap
ananth has quit [Ping timeout: 276 seconds]
ananth has joined #systemtap
hkshaw has quit [Ping timeout: 245 seconds]
hkshaw has joined #systemtap
wmealing1 has quit [Quit: Leaving.]
hkshaw has quit [Quit: Leaving.]
hkshaw has joined #systemtap
wakatana2 has joined #systemtap
wakatana2 has quit [Max SendQ exceeded]
wakatana2 has joined #systemtap
wakatana2 has quit [Ping timeout: 272 seconds]
hkshaw1 has joined #systemtap
wmealing has joined #systemtap
hkshaw has quit [Ping timeout: 272 seconds]
hpt has quit [Ping timeout: 265 seconds]
atomlin has joined #systemtap
atomlin has left #systemtap [#systemtap]
naveen has quit [Ping timeout: 260 seconds]
wmealing has quit [Quit: Leaving.]
wmealing has joined #systemtap
naveen has joined #systemtap
<jvv>
hi, how can I convert systemcall number that is stored in $syscall variable to name of this syscall ?
wmealing has quit [Quit: Leaving.]
wmealing has joined #systemtap
mjw has joined #systemtap
wmealing has quit [Quit: Leaving.]
ego has quit [Ping timeout: 265 seconds]
scox has quit [Ping timeout: 260 seconds]
srikar is now known as srikar_away
ananth has quit [Quit: Leaving]
srikar_away is now known as srikar
srikar is now known as srikar_away
srikar_away is now known as srikar
<lberk>
jvv: I believe the syscall it's name is stored in the 'name' convenience var
<lberk>
s/syscall it's/syscall's/
<jvv>
lberk: I use process("PATH").syscall probe… so I dont think I have access to that… tried name already, but got Warning that variable is not assigned
wmealing has joined #systemtap
<lberk>
ah, sorry, I'm not sure of a way to grab the name of the syscall in that case (perhaps somebody else will know)
ego has joined #systemtap
<jvv>
I dont understand why syscall probe and process().syscall probe have different sets of variables…
<lberk>
jvv: the syscall probes have the 'name' var explicitly defined in their tapsets
<lberk>
wmealing: wow, with that badge it looks like you propelled fche into the top 500 fedora contributors too haha ( https://badges.fedoraproject.org/badge/top-500 under 'latest activity' )
<jvv>
lberk: hmmm… yeah I see that… for every syscall there is a method correlated and there is name assignation… like name = "chmod"… interesting
<lberk>
yes, what I was looking for earlier was a stap function to look up and compare the syscall number to the list on that kernel/arch, couldn't find such a function
<lberk>
might just have to be done manually
wmealing has quit [Quit: Leaving.]
mbenitez has joined #systemtap
mbenitez has joined #systemtap
wmealing has joined #systemtap
<wmealing>
sweeet
<wmealing>
fche, power overwheeelming
<jvv>
and indeed it is… so it seems that I would have to pipe output of my stap script through some other script to convert numbers to names
<jvv>
hmmm… I dont get… so syscall probe has its own tapset and process probe doesnt ?
<lberk>
jvv: syscall probe points are probes on specific (known in advance) kernel functions, where convenience variables such as the name can be provided, general process("") probe points that can't always be provided, simply because we have no knowledge of the binary you'll potentially probing
* lberk
goes to grab his morning coffee
<jvv>
i'm drinking my 4th ;)
<fche>
hi folks
<fche>
re syscall numbers, see PR12748
<jvv>
fche: PR12748 ?
<lberk>
sourceware.org/PR12748
srikar is now known as srikar_away
<jvv>
oh, so I understand that there were such ideas for easy conversion… but its not there, right ?
<fche>
someone (tm) just needs to create the tables
<fche>
it's not difficult, just painstaking :)
<fche>
probably we could steal a syscall# <-> syscall-name table from somewhere else
<jvv>
hehe :)
<jvv>
ok, I was stubborn and wanted to avoid it but I came to conclusion that probe syscall.* { if (execname() == "meh") … } is better way to go :)
<jvv>
but either way I've learned something so thanks
<fche>
see also the strace.stp script
scox has joined #systemtap
<jvv>
I saw, but too much is happening there for me
zodbot has quit [Disconnected by services]
telnoratti has quit [Ping timeout: 260 seconds]
wcohen has quit [Ping timeout: 265 seconds]
serhei has joined #systemtap
Meths_ has joined #systemtap
jistone_ has joined #systemtap
Humble has quit [*.net *.split]
jistone has quit [*.net *.split]
Meths has quit [*.net *.split]
serhei_ has quit [*.net *.split]
jistone_ is now known as jistone
zodbot has joined #systemtap
Humble has joined #systemtap
drsmith has joined #systemtap
hpt has joined #systemtap
<fche>
jvv, sure, it's a pretty complex script, but it's because it tries to be so configurable
wmealing has quit [Quit: Leaving.]
wcohen has joined #systemtap
hpt has quit [Ping timeout: 240 seconds]
brolley has joined #systemtap
hkshaw1 has quit [Quit: Leaving.]
tromey has joined #systemtap
nkambo has quit [Ping timeout: 256 seconds]
naveen has quit [Quit: Leaving.]
irker940 has joined #systemtap
<irker940>
systemtap: dsmith systemtap.git:refs/heads/master * release-2.9-219-g90fde66 / testsuite/systemtap.base/at_var.exp testsuite/systemtap.base/at_var.stp: Get systemtap.base/at_var.exp to pass on s390x. http://tinyurl.com/hxr4qfs
nkambo has joined #systemtap
ego has quit [Quit: Leaving]
sjas has joined #systemtap
hkshaw has joined #systemtap
scox_ has joined #systemtap
scox has quit [*.net *.split]
mbenitez has quit [*.net *.split]
chris2 has quit [*.net *.split]
chris2 has joined #systemtap
hkshaw has quit [Ping timeout: 256 seconds]
mbenitez has joined #systemtap
mbenitez has quit [Changing host]
mbenitez has joined #systemtap
hkshaw has joined #systemtap
Humble has quit [Changing host]
Humble has joined #systemtap
wcohen has quit [Changing host]
wcohen has joined #systemtap
scox_ has quit [Changing host]
scox_ has joined #systemtap
<jvv>
hmmm… what does $$params mean and why there are 2 dollars there ? (I am looking at para-callgraph.stp example)
<fche>
see man stap
<fche>
$params would mean the variable 'params' in the context of the probed program
<fche>
$$params is a synthetic variable to mean 'a dump of all variables' in scope in that context
<fche>
or rather all parameters of a function-entry type probe
<jvv>
hmmm… in man stap I cant see anything about $$params
<fche>
that's where probe-specific context variables are documented
<jvv>
also my mistake… its not params but parms :)
<fche>
ISTR at one point we supported both; maybe still do, just one's undocumented
<jvv>
I think I have in general problem with moving around all of this… there is man stap, man stapprobes, man -k function::, man -k tapset::, man -k probe::, man -L 'kernel.function("")', etc
<jistone>
could be we need a librarian's help
<jvv>
hehe ;)
<fche>
well, there's a lot of stuff to document
<fche>
there are many cross references
<fche>
the $var -> stapprobes one would be a good link to elaborate upon in man stap though
<fche>
or maybe we need a 'guide to other documentation' section high up in man stap
<fche>
that explains function::* etc.
<fche>
would anyone like to take that little task on?
<fche>
(filing a PR so we don't forget)
nkambo has quit [Ping timeout: 264 seconds]
<josch_>
so... maybe somebody give me a general hint on what what i'm doing wrong or whether what i want really doesn't exist. What I want is to be able to reliably get all calls to systemcalls like execve, open and write (and their return values) of a process and all its future children and grand children. Auditd seems to not be able to do that because it cannot be told to monitor all future grandchildren of a process (only direct future children). ...
<josch_>
... Ptrace is out because processes can break out of being ptraced and software that tries to use ptrace itself will not work when being ptraced. So I looked at systemtap but it seems that it is easy for probes to get skipped. For example when I try out EXAMPLES/process/strace.stp like this:
<josch_>
now i could increase values like TRYLOCKDELAY, MAXTRYLOCK, KRETACTIVE and the like but I'd only know whether I increased them enough *after* probes got skipped and it doesn't seem right to set these values to millions "just to be sure"
<josch_>
maybe systemtap isn't for me and I need to look at yet another solution?
<josch_>
thoughts?
<fche>
hey josch_
<fche>
it's a tricky question, there's no way to guarantee tracing non-intrusively without risking some sort of buffer from being overwhelmed
<fche>
can try stap -g --suppress-time-limits ... also e.g.
<fche>
or use the kernel.trace("sys_enter"), kernel.trace("sys_exit") low level tracepoints; those are lighter weight than the usual kprobes
<fche>
but lack the fancier argument-decoding technology
<fche>
did stap -t indicate why the probes were skipped?
<josch_>
yeah, unfortunately I don't only want to know which syscall was called but also what the arguments were - which of course is yet another problem but nut such a fundamental one as the possibility to miss calls
<fche>
probably the solution is to have a more specific script that doesn't try to trap all syscalls
<josch_>
so even with --suppress-time-limits 28 probes got skiped due to "missed kretprobe/1"
<fche>
just the ones you care about
<josch_>
fche: i have that and made the same observations there
<josch_>
fche: i used the strace.stp as an example to avoid copypasting code around and requiring you to get familiar with my stp file
<josch_>
and just having fewer probes will not remove the underlying fundamental problem
<josch_>
in cases where lots of calls are done at the same time the problem would resurface, no?
<fche>
if there is a lot of activity -on those syscalls- then probably yes
<fche>
e.g. are you sure you need return values ? skipping .return probes could make it more robust
<fche>
write syscalls occur all the time though, incl. by systemtap's internal ops, so there'll be some background traffic that needs to be filtered out in any case
<irker940>
systemtap: dsmith systemtap.git:refs/heads/master * release-2.9-220-g3f04097 / buildrun.cxx runtime/linux/alloc.c runtime/linux/autoconf-module_layout.c runtime/linux/print.c runtime/transport/symbols.c tapset/linux/context.stp: Fix PR9497 by updating the runtime to handle linux 4.4 commit 7523e4dc50. http://tinyurl.com/hbk7gvf
<fche>
(confirming that using the kernel.trace("sys_enter") tracepoint-based probes, none skipped here)
<josch_>
as you can see i'm not an expert in this area so the general question remains: there exists no tracing mechanism that does the reliable tracing i want and that processes can not escape from?
<fche>
I think you only have various degrees of reliability; I can think of reasons for <100% in apprx. all of them, if they're relatively nonintrusive
<fche>
(I mean beyond stap too)
<josch_>
okay
mjw has quit [Quit: Leaving]
<jistone>
tangentially, I was just looking at available syscalls tables for PR12748, and AFAICT perf/ftrace just drop x86-compat syscalls
<jistone>
(rather than trying to maintain two tables)
<fche>
can try perf's widgetry too (though note that too can exhaust buffers etc.); ISTR they have -some- syscall decoding capability
<fche>
thanks drsmith for the quick fix!
<drsmith>
you are welcome
<jistone>
(see ARCH_TRACE_IGNORE_COMPAT_SYSCALLS)
<jistone>
josch_, if you're really worried about processes escaping notice, perhaps the audit system is for you?
<fche>
jistone, he mentioned auditd (and its difficulty following children)
<fche>
(and I'm not sure it's 100% reliable to the point of holding up the target thread(s) until userspace auditd writes to disk)
<jistone>
hmm, ok I scanned the scrollback too quickly, sorry
wcohen has quit [Ping timeout: 250 seconds]
<fche>
josch_, so FWIW the way I'd pursue this quest in stap-land would be to use the kernel tracepoints and hack up some basic argument-decoding based upon the tracepoint arguments
<josch_>
fche: right, thank you for your advice, i'll look into it a bit more tomorrow
<josch_>
my goal is to be able to analyze software build logs (specifically debian package build logs) and unfortunately lots of test-suits for example do really weird or stressing stuff (that's also why i cannot use ptrace)
<fche>
ah interesting
<fche>
(don't forget openat(2) and similar syscall families)
<josch_>
so i'd like to collect information about: which program is writing what to the build log, which processes are being spawned (including very short-lived processes) and which was touched and written to by whom
<josch_>
yeah it becomes very easy to forget system calls especially as the linux kernel evolves and as multiple different architectures are involved
<josch_>
i might just start using ptrace first for a proof of concept and just ignore packages that use ptrace themselves during their build
<josch_>
at least ptrace, as far as i know, doesn't skip system calls (except if the process being monitored actively tries to escape being ptraced)
<fche>
it's quite intrusive though and can hide (maybe also cause?) race conditions
<josch_>
the good news is that the "reproducible builds project" (bit by bit reproducible binaries) will show if the tracing influenced the build output in any way (in which case a package would be unfit to be ptrac-ed)
<jistone>
you can just trace sys_enter to get them all, and decide how to process them later
<jistone>
that's no different than PTRACE_SYSCALL
<jistone>
(but less overhead, surely)
<josch_>
jistone: while i understand that systemtap's sys_enter is very light weight it might still suffer from probes being skipped when lots of syscalls happen, no?
<jistone>
hmm, maybe some buffering issues, but not like KRETACTIVE which is very easy to exceed
<jistone>
josch_, you said you were mostly hitting kprobe/kretprobe skips, which won't be an issue with tracepoints
<josch_>
okay
drsmith has left #systemtap [#systemtap]
<josch_>
is there an example of how a kernel.trace("sys_enter") based probe can be used while at the same time gathering all the function arguments? I imagine the tricky bit here would be to write the function specfic code that deals with the different argument types per syscall
<jistone>
we don't have an example of that afaik
mbenitez has quit [Quit: Leaving]
<josch_>
yes, i cannot find one in the git at least
<jistone>
perf does have hard-coded decoding for syscall tracepoints -- maybe that would suit you better after all?
<jistone>
even just: perf trace $command
<josch_>
i stopped reading about perf when i read it was for "statistical profiling" which meant to me that it only captures a subset of all calls made
<jistone>
that's its origin, an oprofile sort of operation
<fche>
it does have non-sampling modes
<josch_>
okay, thanks then i'll put that on my list of tools to evaluate :)
<fche>
re. decoding sys_enter ... $id has the syscall#, $regs->$REGFIELD to get access to the abi-level registers
<fche>
it's quite low-level
<josch_>
whoops, yeah i wasn't planning to mess around at that level
<fche>
it's something a stap tapset could do ... but it'd be a big undertaking (like the nd_syscalls* was)
<jistone>
ugh
<fche>
(probably it could be commonized)
<jistone>
if we thought about doing that *again*, maybe it'd be possible to generate syscalls/nd_syscalls/trace_syscalls from some common base
<jistone>
(yeah that)
<fche>
(or have them be based on macros to do the common stuff)
tromey has quit [Quit: ERC (IRC client for Emacs 25.0.50.3)]
zodbot has quit [Quit: puiterwijk]
brolley has left #systemtap [#systemtap]
zodbot has joined #systemtap
cbzx has joined #systemtap
zodbot has quit [Write error: Connection reset by peer]