fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
<agentzh>
it does not have to be in mainstream kernel.
<agentzh>
it could just be an out-of-tree kernel module.
<agentzh>
right now stap always inserts a new kernel module and uploads it, which is a bit wasteful.
<agentzh>
the kernel module could be a vm which keeps running in the kernel space.
<agentzh>
though an ad-hoc kernel module exposes more optimization opportunities for specific stap scripts.
<agentzh>
we are also thinking about rolling out one since we're also VERY unhappy about the ebpf's kernel side limitations, which is inhuman after years of stap usage.
<agentzh>
stap is so powerful and innovative in many ways, even as compared to dtrace.
<agentzh>
bcc is still like stone age things.
<fche>
we'd really appreciate an outsider like yourself writing up such observations
<fche>
it'd help prioritize our work and maybe be leverage w.r.t. the kernel folks
<agentzh>
fche: sure, will do.
<agentzh>
i already have a talk in nginx conf 2018 which will talk about stap, bcc, gdb and etc.
<agentzh>
it will be in Oct.
<agentzh>
*have proposed a talk
<agentzh>
will definitely write more blog posts too.
<fche>
looking forward to them
<agentzh>
mozilla rr is also on our radar.
<agentzh>
thanks
pokk11 has joined #systemtap
<pokk11>
This blog is essentially an ad for the Handshake ICO scam with a one-line "denial" of involvement mixed in there. It's obviously very unethical of Christel to not mention her own involvement in the scam which the blog post promotes.
<pokk11>
Consider Andrew Lee's involvement, Andrew Lee is Christel's boss at London Trust Media and he also controls the majority of freenode voting rights. Andrew Lee also heads the handshake ICO scam. Coincidence?
<pokk11>
Don't support freenode and their ICO scam, switch to a network that hasn't been co-opted by corporate interests. OFTC or efnet might be a good choice. Perhaps even https://matrix.org/
pokk11 has quit [Remote host closed the connection]
<GorillaWarfare28>
This blog is essentially an ad for the Handshake ICO scam with a one-line "denial" of involvement mixed in there. It's obviously very unethical of Christel to not mention her own involvement in the scam which the blog post promotes.
<GorillaWarfare28>
Consider Andrew Lee's involvement, Andrew Lee is Christel's boss at London Trust Media and he also controls the majority of freenode voting rights. Andrew Lee also heads the handshake ICO scam. Coincidence?
<GorillaWarfare28>
Don't support freenode and their ICO scam, switch to a network that hasn't been co-opted by corporate interests. OFTC or efnet might be a good choice. Perhaps even https://matrix.org/
GorillaWarfare28 has quit [Killed (Sigyn (Spam is off topic on freenode.))]
<agentzh>
fche: seems like my stap test suite run never ends on my federa 26 (kernel 4.16.11). it locks up the CPU infinitely: "kernel:watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stapio:37300]"
<agentzh>
now i cannot ssh to that vm. and all the CPU core is 100%.
<agentzh>
any hints on debugging this would be very appreciated.
<fche>
dmesg should at least identify the last script
<agentzh>
so i should kill that vm now?
<fche>
or check its console out
<fche>
suggest netconsole or console-to-serial-tty-to-host-file for development vm's
<agentzh>
oh, yeah, i should have configured that.
<agentzh>
thanks for the tip.
<fche>
4.6.11 is old ... it might be afflicted by a series of kernel bugs introduced post-spectre
<fche>
that nuke kprobes and some related facilities
<agentzh>
okay, i've finally found and installed the kernel 4.16.6 packages for fedora 27 on Koji.
<wmealing>
noice.
<wmealing>
say i wanted to bail out of a function early, in a probe.. ie, entirely out of the function.. be gone with the evil.. is guru mode the only way ?
<agentzh>
wmealing: because stap does not support the latest kernel 4.17 packages on fedora 27.
<wmealing>
agentzh: doesn't support how ?
<agentzh>
the new kernel 4.17 breaks stap's syscall layer and the stap team is still working on it.
<wmealing>
oh damn, i didnt know.. thanks for that.
<agentzh>
sure
<wmealing>
is this some kind of kprobes breakage ?
<agentzh>
oops, sorry, i mean installed kernel 4.16.16 package, not 4.16.6. the latter is too old.
* wmealing
looks at his kernel version.
<wmealing>
3.10.0-862.11.3.el7.x86_64
<agentzh>
hah, still RHEL/CentOS 7 :)
<agentzh>
pretty safe then.
<wmealing>
aint never gunna give me up
<agentzh>
wmealing: regarding to your question, i'm about to proposing a patch for that.
<agentzh>
my patch originally changed the exit() tapset function, but fche doesn't like changing the current behavior. so maybe abort() or quit() then.
<wmealing>
agentzh: is it -in- systemtap or something that i can abuse easily ?
<agentzh>
it will be a stap core patch.
* wmealing
nods
<agentzh>
i can paste my current patch now if you'd like to give it a try.
<wmealing>
sadly, i'm problem solving as we speak under a deadline.
<wmealing>
i dont want to go to kprobes if i can help it.
<wmealing>
i see a lot of the example scripts are about "monitoring" where as i need to change behavior.
<wmealing>
agentzh: can you execute guru code inside a probe ?
<agentzh>
wmealing: you can use error("") + try/catch to do that.
* wmealing
will try, ive not used either of those
<wmealing>
well, that didnt work how i thought it would.
<wmealing>
non guru mode crashed.
<wmealing>
wait, it abused $return, its guru mode.
<wmealing>
maybe nto crashed, but its not responding. thats cute
<wmealing>
ok,i know what it was
<wmealing>
returning the positive error value
<wmealing>
not the correct negative error value
<wmealing>
ie, not -EACCESS
<wmealing>
this is a bit frustrating.. i think stap is eliding an expression which it thinks is side affect free, but it needs it later on.
<wmealing>
and complaisn it can't find local
* wmealing
thinks
<wmealing>
forgot $ is source local
orivej has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
<invano>
fche: sorry but after leaving I didn't have time to join again yesterday
<invano>
oh wow I was sure to have CONFIG_DEBUG_INFO but I didn't see the _REDUCED. Let me recompile but I assume the problem is gone
<invano>
I need time, a lot, to remember all those compilation flags..still far to be a gcc/dwarf expert :)
wmealing has quit [Remote host closed the connection]
<invano>
yes, thanks fche
orivej has joined #systemtap
mjw has joined #systemtap
<invano>
it's always me with my problems. I was using non-dwarf kprobes before. 1y ago and more I was without debuginfo in kernel. I do have dbginfo now, so I tried to switch to dw_syscalls.*. Also, translation of the stap script to C is much faster w/ dwarf.
<invano>
With dw_syscall.* I'm encountering this warning repeated quite a lot: "WARNING: instance of overloaded function will never be reached: identifier '_stp_syscall_nr' at /usr/local/share/systemtap/tapset/linux/aux_syscalls.stp:119:10 source: function _stp_syscall_nr:long ()"
<invano>
never mind..it was me mixing up tapsets from systemtap3.3 and systemtap git -.-
<fche>
hey no problem, just am glad we're figuring this out
<invano>
yes yes many thanks
<invano>
I'm seeing the same @cast issue on MIPS now, even if I have CONFIG_DEBUG_INFO_REDUCED not set. Just some time to finish some stuff with arm and then I switch to the mips and I investigate that
<invano>
btw it's more than one year now that I have some patches to fix(&add) MIPS o32 support for uprobes. I should share them
orivej has quit [Ping timeout: 240 seconds]
<fche>
sure
<fche>
with the arm trace, the eu-readelf -w of the typequery*ko file was what provided the hint I needed
<fche>
the compiler flags at the top
<invano>
Yes took note, including the explanation you wrote yesterday
<invano>
Anyway, are there any known big issues at the moment with systemtap git version?
<fche>
syscall probe aliases on kernel 4.17+ generally broken
<invano>
ah ok 4.17
<invano>
I did a git pull && install for fun and it breaks on compilation
<fche>
shouldn't do that
<invano>
warnings on various @define in syscalls + compilation error on a type cast with CONTEXT->sregs
<invano>
compilation of the stap script sorry
<fche>
would be curious about those errors, but that part is a work-in-progress
<invano>
I can send you a log, or we just wait for the wip to finish and then check again
<fche>
fpaste-ing it would be fine
<invano>
sure
<invano>
ok https://pastebin.com/rapwUNTe - I used double -v combo otherwise it gets too big for pastebin. I can put more v tho if you need it. same command I pasted yesterday for cross-compiling with arm. You trigger it with -e "probe syscall.* {..}"
<invano>
warnings start at line 2061. int-to-pointer-cast error on line 2560
pviktori has quit [Quit: No Ping reply in 180 seconds.]
<fche>
looks like a possible mixture of work-in-progress-ness and maybe you didn't run a make install lately?
tonyj has quit [Ping timeout: 244 seconds]
<invano>
I ran make uninstall on 3.3rel and make install on git
<fche>
so looking at that last pastebin, I wouldn't worry too much about the warnings
<fche>
but the cast error is more interesting
<invano>
yup it's what blocks everything
jhg_ has quit [Read error: Connection reset by peer]
jhg_ has joined #systemtap
<fche>
yeah, ok, so it's probably a simple enough explanation
<fche>
we're getting 64-bit ints via the context / function parameters; we're casting down through a void* into a struct pt_regs*
<fche>
and on arm that's 32 measly bits die
<fche>
wide
<fche>
for some reason we're not seeing that diagnostic on other platforms ... not sure why
<fche>
but will add a more paranoid cast sequence
<fche>
ok try git master now
<invano>
let me see
<invano>
great
<fche>
SHIP IT
<invano>
warnings still there but compilation works fine
<fche>
ok
<invano>
oh wow you got a patch for mips o32 in January this year cool! Exactly same stuff I did with my patches in the past. nice!
<fche>
I believe many of those patches came from the community years ago; we just finally pushed them in
<invano>
I only see a mismatch in tapset/mips/registers.stp
<fche>
pls send your patch if you see a problem
<invano>
_reg_offsets["zero"] for CONFIG_32 is 24. If I'm not wrong, that should be because struct pt_regs had a unsigned long pad0[6] before. Starting from kernel 3.16.3 pad0[6] became pad0[8] so all _reg_offsets for CONFIG_32BIT should be realigned accordingly with a 8B shift
<invano>
sure!
<fche>
I don't think we have any mips hardware here to test on.
orivej has joined #systemtap
brolley has joined #systemtap
aryehw has joined #systemtap
tromey has joined #systemtap
<invano>
yes I can test it
<invano>
In the meanwhile, on mips, after a probe syscall.*: semantic error: No cfa_ops supplied, but needed by DW_OP_call_frame_cfa: identifier '$ringid' at /usr/local/share/systemtap/tapset/linux/sysc_add_key.stp:35:19
<fche>
not sure whether that represents a problem in systemtap or gcc or the kernel; I remember x86 assembler in the kernel needing some custom dwarf-y assembly to make it complete
<invano>
not sure either..I guess more the combo toolchain/gcc/kernel
<invano>
I hit a similar problem on glibc e.g. extracting vars from process("libc").function("x") with DW_OP_GNU_entry_value
<fche>
that one's a systemtap shortcoming (not enough dwarf5 knowledge iirc)
<invano>
yeah I was reading it on a couple of bug reports
<invano>
but is there something to probe <alias>.* but a specific one? like probe syscall.* except syscallX or probe process(Y).function("*") except function Y. For syscall I found a set declaration for doing syscall.{x,y,z}. However, I'm not finding any kind of not operator
<agentzh>
it would be great if you can have a look.
<agentzh>
the release 3.3's test suite has far fewer failures than the master branch on my side, though it also runs into CPU stuck after 1 hour's run.
<fche>
agentzh, thanks for filing
<fche>
stap-report would be useful
<fche>
as would the dmesg|tail
<agentzh>
fche: the box has rebooted and i'm afraid dmesg is not very helpful? i pasted the /var/log/messages output to the PR ticket already.
<fche>
ah, the errrors just before the stuckage would also be good
<fche>
but yow, memory allocation failed, well that's abnormal; we -might- not handle it quite right, though we have tests for it
<fche>
a bunch of sigsegvs afterwards, yow
<agentzh>
fche: already included all the lines a few minutes right before the stuck in the ticket :)
<agentzh>
earlier messages are not errors and are not relevant.
<fche>
ok, so abnormality started at the mempool_init failure? ok
<agentzh>
fche: just added an attachment to the PR for the stap-report output.
<fche>
how much ram on the box, btw? I don't think stap-report collects that but should
<fche>
ok, no excuse there. I mean stap in some contexts insists on using completely free memory for allocation, and that could fail, but still obviously that should not be likely, nor bring the machine down.
<agentzh>
that's all ram i have in the physical box (which is a mid-2015 MBP)
<fche>
ah, was the crash in a smaller vm?
<agentzh>
all the results are from the VM itself.
<fche>
hm, so the vm memory limit is the same as the host physical memory? interesting
<agentzh>
fche: oops, sorry, my bad, the host machine has 16G of ram.
<agentzh>
the VM has 8G.
<agentzh>
i was running make -j8.
<agentzh>
so it might be even more hungery about memory.
<fche>
ok
<agentzh>
so to work around it, i should try enlarging the vm's memory assignment and using smaller job count?
<agentzh>
but ideally we could get it fixed :)
<agentzh>
quick and normal test failures would be much better than CPU stuck.
<fche>
of course
<agentzh>
sometimes i was hoping that we could use mozilla rr to record such failures and replay back and forth as we wish :)
<agentzh>
fche: please let me know if you need any more info. appreciate your help!
<fche>
yeah, not sure how fast progress is likely there; just would start reading alloc code in the runtime
<agentzh>
gotcha.
<agentzh>
fortunately it seems to be quite easy to reproduce on my side, on 2 different physical boxes.
<agentzh>
one was using kernel 4.16.11 against stap master on a federa 26 vm atop Intel NUC, the other was using kernel 4.16.16 against stap 3.3 release tag on a fedora 27 vm atop MBP.
<agentzh>
just let the test suite run like 1 hour or so and boom, all CPU cores are stuck.
<fche>
something's odd there
<fche>
we have a bunch of bots routinely building master & running the full testsuite
tromey has quit [Read error: Connection reset by peer]
<agentzh>
the master branch has WAY more test failures than the 3.3 release tag on my side.
<agentzh>
so i have to give up fighting the master.