fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
derek0883 has quit [Ping timeout: 265 seconds]
orivej has joined #systemtap
orivej_ has quit [Read error: Connection reset by peer]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 246 seconds]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 256 seconds]
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
hpt has joined #systemtap
orivej_ has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
derek0883 has joined #systemtap
derek088_ has joined #systemtap
orivej_ has quit [Ping timeout: 260 seconds]
orivej has joined #systemtap
derek0883 has quit [Ping timeout: 256 seconds]
derek088_ has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 256 seconds]
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
irker909 has quit [Quit: transmission timeout]
orivej has quit [Ping timeout: 258 seconds]
orivej has joined #systemtap
orivej_ has joined #systemtap
orivej has quit [Ping timeout: 264 seconds]
orivej has joined #systemtap
orivej_ has quit [Ping timeout: 246 seconds]
orivej has quit [Ping timeout: 240 seconds]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek has joined #systemtap
derek is now known as Guest68924
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
gromero has quit [Ping timeout: 272 seconds]
gromero has joined #systemtap
derek0883 has joined #systemtap
Guest68924 has quit [Ping timeout: 260 seconds]
derek0883 has quit [Ping timeout: 260 seconds]
derek0883 has joined #systemtap
khaled_ has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 256 seconds]
wmealing has quit [Remote host closed the connection]
orivej has joined #systemtap
orivej has quit [Ping timeout: 264 seconds]
orivej has joined #systemtap
orivej has quit [Ping timeout: 260 seconds]
orivej has joined #systemtap
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
mjw has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
orivej_ has joined #systemtap
orivej_ has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
hpt has quit [Ping timeout: 264 seconds]
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #systemtap
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #systemtap
orivej_ has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
orivej_ has quit [Ping timeout: 260 seconds]
orivej has joined #systemtap
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
tromey has joined #systemtap
orivej_ has joined #systemtap
orivej has quit [Read error: Connection reset by peer]
orivej_ has quit [Ping timeout: 260 seconds]
orivej has joined #systemtap
orivej has quit [Read error: Connection reset by peer]
orivej has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #systemtap
orivej_ has joined #systemtap
orivej has quit [Read error: Connection reset by peer]
irker726 has joined #systemtap
<irker726> systemtap: wcohen systemtap.git:master * release-4.3-4-gcc7b62017 / testsuite/systemtap.examples/memory/overcommit.meta: Add overcommit.stp to examples that can run in bpf.
agentzh has joined #systemtap
agentzh has quit [Remote host closed the connection]
agentzh has quit [Changing host]
agentzh has joined #systemtap
sscox has quit [Ping timeout: 246 seconds]
orivej_ has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
sapatel has joined #systemtap
orivej_ has joined #systemtap
orivej has quit [Read error: Connection reset by peer]
orivej has joined #systemtap
orivej_ has quit [Ping timeout: 264 seconds]
orivej has quit [Ping timeout: 260 seconds]
orivej has joined #systemtap
sscox has joined #systemtap
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
orivej has quit [Read error: Connection reset by peer]
orivej has joined #systemtap
derek0883 has quit [Ping timeout: 260 seconds]
orivej has quit [Ping timeout: 246 seconds]
orivej has joined #systemtap
khaled_ has quit [Quit: Konversation terminated!]
<agentzh> fche: i digged deeper into the garbaled stap output issue using the simplest .stp script here: https://gist.github.com/agentzh/b44384ad9f3804d85ddfbf771094a718
derek0883 has joined #systemtap
<agentzh> i found that the buggy case is a race condition
<agentzh> basically the staprun receives STP_REQUEST_EXIT and sends back STP_STOP before it reads any data in its reader_thread.
khaled has joined #systemtap
<agentzh> and the ko side receices STP_STOP and calls _stp_transport_data_fs_stop() which in turn calls relay_flush(). relay_flush() forces switching to the next subbuf. and thus staprun then first receives 21 byte of good data, and then the full subbuf size of garbaled data (in this case all \0 bytes).
<agentzh> that .stp script is easy to reproduce when repeatedly running staprun test.ko in 20 threads. on my side it is hit every 24s ~ 3min.
<agentzh> the staprun -vvv output for the buggy case is like this: https://gist.github.com/agentzh/283a15f617f751cf9faa786632b1e516
<agentzh> the buggy case's stap output (via -o a.txt) is like this: https://gist.github.com/agentzh/5967e0a9ce90e494ee0c88cb4d9ecef6
<agentzh> removing the relay_flush() call from the _stp_transport_data_fs_stop func seems to fix this example in particular but i'm not sure about it. will you shed some light here? it looks quite compliciated on the ko side regarding the transport layer.
orivej has quit [Ping timeout: 246 seconds]
<agentzh> btw, i ran this on kernel 5.0.16 of fedora 28. but we also saw this on the 3.10 kernel of centos 7.
<agentzh> it appears that staprun would read a full subbuf amount of garbled data when it tries to read between the relay_flush() and relay_close() calls on the ko side.
<agentzh> once i removed the relay_flush() call from _stp_transport_data_fs_stop(), i can no longer reproduce the issue using the same 20-thread load test for 8+ hours of continous running.
<agentzh> the default read buf size in the reader_thread of staprun is smaller than the default subbuf size in the stap runtime (128KB < 256KB) so the former is always reading the full reader buf size. i tried increased the former size to be larger than the latter, and then it always reads the full subbuf size of garbaled data.
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
<agentzh> fche: btw, i created this PR to record the vma leak issue as suggested: https://sourceware.org/bugzilla/show_bug.cgi?id=26123
<agentzh> will reference it in my patch.
<fche> thanks
<agentzh> sure
<fche> trying to follow along with your relay analysis bit
<fche> not sure where you perceive a race condition -- between what and what ?
<agentzh> staprun reading relay data file betweeen ko calls relay_flush() and relay_close().
<agentzh> removing he relay_flush() call makes the issue disappear.
<agentzh> normally staprun reads the data before the relay_flush() call (and also before receiving the STP_REQUEST_EXIT control cmd).
<agentzh> i'm surprised that relay_flush() would turn on a garbled new subbuf for the reader's read() syscall to read.
<agentzh> or switch on
<agentzh> my gut feeling is that removing relay_flush() is not the right fix. but i'm not sure what the right one is. that's where i need help :)
<agentzh> i stumbled upon the overwrite mode used in the relay api. but it is still reproducible even after i disabled the overwrite mode in the stap runtime completely.
<agentzh> so it's not the culprit.
<agentzh> it'll be great if there's any suggestions on debugging this any further.
<fche> this is very old code, so no fresh memories etc.
<agentzh> yeah, dating back to 2009 :)
<agentzh> not sure if it's compatibility issues with modern kernels.
<agentzh> i saw it copies and pastes a lot of kernel code into the stap runtime.
<irker726> systemtap: yichun systemtap.git:master * release-4.3-5-gfb59e8c75 / runtime/vma.c: PR26123: fixed memory leaks in vma map for kernels without CONFIG_UTRACE.
<fche> agentzh, I think you're right that relay_flush() is not disposable in this context --- that is how any pent-up messages from a probe-end / probe-error would end up being brought to staprun userspace attention
<fche> the question becomes where the bad data comes from
<fche> whether it's corrupt at the time of the relay
<fche> or in transit
<fche> so ... I'd probably instrument this part of the code somewhat. dump the rchan vitals
<fche> maybe look for \0 chars in there
<fche> would be good transcribe your investigation into another bugzilla entry
<agentzh> yeah, sure.
hassan64 has joined #systemtap
<hassan64> This error was reported a few days ago
<fche> aha. the #if/#endif guard in the ubuntu bug# should help this
<fche> will need a few variants of it for the various affected macros
<hassan64> fche: is it a systemtap issue. What's solution ?
<fche> https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1883343 << added a comment there, if it's the same problem
<hassan64> yes same bug
<hassan64> any thing needs to be done at kernel settings ?
<fche> I don't think so
<fche> a handful of #if/#endif's inserted in a systemtap runtime .h file should fix this
<hassan64> so it is then systemtap issue rather. isn't it ?
<fche> so it seems
<hassan64> There are 3 files under "runtime" directory 1) runtime_context.h 2) runtime_defines.h 3) runtime.h
<hassan64> runtime.h
<hassan64> "#if/#endif", i think already present. isn't it?
<fche> the ubuntu issue https://paste.centos.org/view/540ccea5 << has this patch
<fche> that's the file you'd probably have to hack on
<hassan64> fche: I think I have updated "compat_unistd.h"
<fche> is that enough to make it work for you?
<hassan64> no, I am getting error as I reported
<fche> I believe that's the file
<fche> but I don't see any modifications in your gist version of it relative to git/master
<hassan64> why systemtap developers are not able to fix such issues. I am using 4.3/0.180 systemtap
derek0883 has quit [Ping timeout: 264 seconds]
<fche> we don't have access to every type of hardware / distro / version.
<fche> and time is limited
<hassan64> so systemtap is compatible to which architectures. Can you please name those
<hassan64> I have machines like *AARCH*, *ARMv7*, *MIPSEL*, *MIPSEB*, *MIPS64EL*, *MIPS64EB*, "POWERPC", "i386",
<fche> a range, including arm64, but not the entire matrix of architecture x kernel-version x distro etc. is testable here
derek0883 has joined #systemtap
<hassan64> "staprun" and its associated binaries are all compiled to target machines. But when I do translation (*.stp -> *.ko) while sittinh at host (CROSS COMPILATION), it causes such errors, different errors with different architecture
<hassan64> fche: Any reason for that ?
<fche> yes, some porting may be required for different architecture / version combinations. this is normal.
<hassan64> I have done all things what I can do at Kernel level settings of these architetcure like UPROBES, KPROBES, TRACERS, DEBUGFS etc
<hassan64> But when I do translation, error occurs. Let take aarch example
<hassan64> I need to get strace output, but it fails.
<hassan64> while considering that const_inst.h is updated file.
<hassan64> Can you please solve my issue
<fche> are you able to give a fix a try yourself? the ubuntu bug I pointed to includes the beginning a possible fix
<fche> would you like help to finish that one?
<hassan64> is it an issue with my ubuntu machine?
<hassan64> yes sure I will help you
<hassan64> But I am unable to know where to fix that issue.
<fche> the systemtap source file identified in the patch needs to be updated
<fche> then systemtap reinstalled
orivej has joined #systemtap
<hassan64> But I am using systemtap master 4.3 ver. It should have been fixed instead I fix it
<fche> in the open source community, this is how things sometimes work. We work together to improve the software.
<hassan64> Btw, the contents in https://paste.centos.org/view/raw/540ccea5 are not found in my "const_unistd"
<fche> runtime/linux/compat_unistd.h
<fche> that is installed under .../usr/share/systemtap/runtime/ eventually
orivej has quit [Quit: No Ping reply in 180 seconds.]
<hassan64> yes that's the path : "/usr/share/systemtap/runtime/linux/compat_unistd.h"
orivej has joined #systemtap
derek0883 has quit [Ping timeout: 260 seconds]
orivej has quit [Quit: No Ping reply in 180 seconds.]
<irker726> systemtap: sapatel systemtap.git:master * release-4.3-6-g5b8f6c174 / bpf-translate.cxx: remove comment
orivej has joined #systemtap
<hassan64> #define __NR_compat_clone __NR_ia32_clone
<hassan64> It says that "On older kernels (like RHEL5), we have to define our own 32-bit syscalls number "
<hassan64> that's why "ia32"
<hassan64> ?
<hassan64> But in the post https://paste.centos.org/view/raw/540ccea5 there is no such "ia32"
<hassan64> =L
<irker726> systemtap: wcohen systemtap.git:master * release-4.3-7-g0a281a96d / testsuite/systemtap.examples/general/sizeof.meta testsuite/systemtap.examples/general/sizeof.stp: Make sizeof.stp runnable with the bpf backend.
derek0883 has joined #systemtap
<hassan64> I am afraid I cannot fix this at this hour
orivej_ has joined #systemtap
orivej has quit [Read error: Connection reset by peer]
orivej_ has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
orivej has quit [Ping timeout: 264 seconds]
orivej has joined #systemtap
tromey has quit [Quit: ERC (IRC client for Emacs 28.0.50)]
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #systemtap
<hassan64> checked out the latest ver of systemtap 4.4, but still getting the error
<hassan64> unable to patch here in "/usr/local/share/systemtap/runtime/linux/compat_unistd.h" to fix error: "__NR_compat_clock_getres" .
orivej_ has joined #systemtap
orivej has quit [Ping timeout: 264 seconds]
orivej_ has quit [Ping timeout: 256 seconds]
sscox has quit [Ping timeout: 260 seconds]
<agentzh> fche: i've added more printk() to the relay_v2.c file and noted that the data is actually consumed *during* the relay_flush() call.
<agentzh> I got this for the bad case: [174091.684393] relay after flush: subbufs produced: 1, subbufs consumed: 1, bytes consumed: 0, offset: 0
<agentzh> while for the good cases, it should be [174090.680330] relay after flush: subbufs produced: 1, subbufs consumed: 0, bytes consumed: 21, offset: 0
<agentzh> and right before the relay_flush() call, both good and bad cases are the same: [174091.684336] relay before flush: subbufs produced: 0, subbufs consumed: 0, bytes consumed: 21, offset: 21
<agentzh> the patch i added for such debugging is here: https://gist.github.com/agentzh/51ffa8f2ca895264fab0f3706f187439
<agentzh> any suggestions to check any further?
<fche> could run a stap script in parallel to monitor the operation of relayfs - relay_flush() in particular
<fche> like varwatch.stp
<agentzh> okay, will check it out. thanks
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek088_ has joined #systemtap
derek088_ has quit [Remote host closed the connection]
derek0883 has quit [Read error: Connection reset by peer]
derek0883 has joined #systemtap
hassan64 has quit [Remote host closed the connection]
derek0883 has quit [Remote host closed the connection]
mjw has quit [Quit: Leaving]
derek0883 has joined #systemtap
derek0883 has quit [*.net *.split]
CME has quit [*.net *.split]
serhei has quit [*.net *.split]
jhg_ has quit [*.net *.split]
khaled has quit [*.net *.split]
agentzh has quit [*.net *.split]
ggherdov has quit [*.net *.split]
zodbot has quit [*.net *.split]
jistone has quit [*.net *.split]
sapatel has quit [*.net *.split]
wcohen has quit [*.net *.split]
tonyj has quit [*.net *.split]
pviktori has quit [*.net *.split]
xar- has quit [*.net *.split]
ema has quit [*.net *.split]
przemoc has quit [*.net *.split]
DUKENUKEM has quit [*.net *.split]
fche has quit [*.net *.split]
gavinguo_ has quit [*.net *.split]
gregwork has quit [*.net *.split]
ChanServ has quit [*.net *.split]
Amy1 has quit [*.net *.split]
higgins has quit [*.net *.split]
drsmith has quit [*.net *.split]
darvon has quit [*.net *.split]
fLiPr3VeRsE has quit [*.net *.split]
irker726 has quit [*.net *.split]
eichiro has quit [*.net *.split]
drsmith has joined #systemtap
jistone has joined #systemtap
higgins has joined #systemtap
derek0883 has joined #systemtap
zodbot has joined #systemtap
Amy1 has joined #systemtap
DUKENUKEM has joined #systemtap
przemoc has joined #systemtap
ema has joined #systemtap
pviktori has joined #systemtap
tonyj has joined #systemtap
jhg_ has joined #systemtap
wcohen has joined #systemtap
sapatel has joined #systemtap
xar- has joined #systemtap
agentzh has joined #systemtap
ChanServ has joined #systemtap
irker726 has joined #systemtap
CME has joined #systemtap
eichiro has joined #systemtap
khaled has joined #systemtap
serhei has joined #systemtap
_whitelogger has joined #systemtap
ggherdov has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
sapatel has quit [Quit: Leaving]
khaled has quit [Quit: Konversation terminated!]
<agentzh> fche: seems like relay_flush() has inherient race conditions with relay_file_read()...
<agentzh> or the relay_switch_subbuf() func which relay_flush() calls.
<agentzh> the new subbuf relay_switch_subbuf() switches to somehow got consumed by relay_file_read() in the middle.
sscox has joined #systemtap
<agentzh> i've added some data of a particular pattern into the next subbuf and they got read out by staprun from the very beginning to the very end without any loss.
<agentzh> so it doesn't have to be \0 but whatever left in the next subbuf.