14:54 <irker295> systemtap: mcermak systemtap.git:refs/heads/mcermak/PR20333 * release-3.0-792-g0d9cac8 / : Merge syscalls with identical 'name' to common files. http://tinyurl.com/hf84dn7

15:28 <irker295> systemtap: fche systemtap.git:refs/heads/master * release-3.0-402-ga226065 / main.cxx: tweak unused $n/@n warning for argv tapset users http://tinyurl.com/jluub9v

15:28 <irker295> systemtap: fche systemtap.git:refs/heads/master * release-3.0-403-g344478f / : testsuite infrastructure: change stap_run api http://tinyurl.com/z6ufysl

15:30 <irker295> systemtap: dsmith systemtap.git:refs/heads/master * release-3.0-404-g0d3c21b / python/Makefile.am python/Makefile.in: Fix the python helper module compiliation. http://tinyurl.com/zwtd5rr

15:30 <irker295> systemtap: dsmith systemtap.git:refs/heads/master * release-3.0-405-gd5eac54 / NEWS: Announce python probe support in NEWS. http://tinyurl.com/h8lpjq3

15:33 <fche> woohoo

15:34 <fche> drsmith, no easy way to automate invocation of the -c python with -m HelperSDT ?

15:38 <fche> for packaging, wouldn't the stap-resolve-module-function.py belong in the systemtap-devel subrpm rather than -runtime ?

18:28 <irker295> systemtap: fche systemtap.git:refs/heads/master * release-3.0-406-gc8cef43 / testsuite/systemtap.base/set_user.exp testsuite/systemtap.base/simple.exp: testsuite infrastructure: change stap_run api http://tinyurl.com/jo4gu7j

18:28 <irker295> systemtap: fche systemtap.git:refs/heads/master * release-3.0-407-g8065f5d / systemtap.spec: tweak python helper installation paths http://tinyurl.com/heml5w8

18:54 <irker295> systemtap: dsmith systemtap.git:refs/heads/master * release-3.0-408-gca67717 / systemtap.spec: Move and update comment in systemtap.spec. http://tinyurl.com/j3xv2ck

19:17 <fche> an thanks david

20:00 <bendlas> Hi, stap says to me /tmp/stapXRUH8r/stap_2defeccfa7cfafeb9eebe99059ea78f1_1276_src.c:10:29: fatal error: runtime_defines.h: No such file or directory

20:00 <bendlas> It looks like the Makefile doesn't pass EXTRA_CFLAGS to the compiler ...

20:01 <bendlas> any ideas?

20:09 <fche> hi bendlas, what stap version, built/installed?

20:10 <bendlas> fche: reproducable on 2.9, 3.0, current git; as well as with gnumake 3.8 and 4.2; as well as with gcc 4.9 and 5.4

20:11 <fche> did you 'make install' ?

20:12 <bendlas> it's installed

20:12 <bendlas> here is an example makefile: https://hastebin.com/nejijihege.mk

20:12 <bendlas> apparently, it misses a `-I .../runtime/linux` flag?

20:13 <fche> shouldn't be needed

20:13 <fche> the .../runtime_defines.h does a nested include

20:13 <fche> so line 124 on your paste should have been enough

20:13 <fche> does /nix/store/8b11aa5zz0cal71cww6mkyaz6rsn17nj-systemtap-2017-01-16/share/systemtap/runtime exist, and is readable to your stap user?

20:14 <fche> like if you run % sudo stap the root user may not have read permission.. just speculating

20:15 <bendlas> yes, everything readable

20:16 <fche> hm ... try strace -eopen -f stap -p4 ....script...

20:16 <bendlas> it also contains a `runtime_defines.h`, which includes the `linux/runtime_defines.h`, as you said

20:18 <bendlas> => /nix/store/8b11aa5zz0cal71cww6mkyaz6rsn17nj-systemtap-2017-01-16/bin/stap: invalid option -- 'f'

20:18 <bendlas> erm ...

20:18 <fche> it's an strace option, not an stap option

20:21 <bendlas> fche: thats a 11M log

20:21 <fche> there you go. so I'd look at those lines that relate to runtime_defines.h

20:21 <fche> and see what's up

20:21 <fche> how do you invoke systemtap normally? running as root? stapdev-group? # sudo ?

20:23 <bendlas> regular user or sudo, it fails at the module build

20:23 <fche> ok, if you can't figure out what's what in that log file, send it over pls

20:25 <bendlas> it doesn't seem to be looking for runtime_defines.h in the runtime dir: https://hastebin.com/exovawocec.log

20:25 <bendlas> this are all the lines mentioning `runtime_defines.h`

20:25 <bendlas> fche: seems like make doesn't pass `EXTRA_CFLAGS`, somehow

20:27 <fche> try % stap --poison-cache --vp 0002 -p4 -e 'probe begin{}'

20:27 <fche> that should show you the make command being run

20:27 <fche> % stap -k .....

20:27 <fche> then you can cd to that directory, and run the same env .... command by hand

20:28 <fche> env -uARCH -uKBUILD_EXTMOD -uCROSS_COMPILE -uKBUILD_IMAGE -uKCONFIG_CONFIG -uINSTALL_PATH -uLD_LIBRARY_PATH PATH=/usr/bin:/bin:/home/fche/bin:/home/fche/bin:/home/fche/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/home/fche/bin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/home/fche/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/sbin:/sbin:/usr/sbin:/sbin make -C /lib/modules/4.8.15-200.fc24.x86_64/build M=/var/tmp/stapozrw9U

20:28 <fche> modules CONFIG_DEBUG_INFO= CONFIG_STACK_VALIDATION= ARCH=x86_64 --no-print-directory -j9 <-- on my f24 box

20:32 <bendlas> fche: https://hastebin.com/cokudapuqi.log

20:34 <fche> ok nice, so you can play with it without stap in the middle

20:34 <fche> try make V=1 stap_14852_src.o

20:35 <bendlas> error stays exactly the same as without `V=1` (last invokation in previous log)

20:35 <fche> so am looking for the actual flags to the compiler

20:36 <fche> ie. whether the earlier ones like -freorder-blocks made it in

20:36 <fche> hmmmmm

20:37 <fche> from that configuration (trying to copy it here), I'm getting this error too

20:37 <fche> so when I wrote 'make V=1 ...' I didn't quite mean make V=1

20:38 <fche> I meant the whole env .... make -C .... M=... command line, with V=1 added

20:39 <bendlas> can I override, which cc make calls?

20:39 <fche> are you dealing with a cross-compilation scenario?

20:40 <fche> how are/were you trying to override?

20:40 <bendlas> no, but a cc wrapper

20:40 <bendlas> `make CC=$(unwrapped gcc)` fails the same way ...

20:41 <fche> ok ... well your wrapper script would have to ensure all of its parameters passed through verbatim

20:42 <bendlas> it does, but better to use the unwrapped cc, stap links its own runtime anyway ..

20:43 <fche> yes. so can you paste the "env ... make -C ... V=1 ... " command line's output?

20:43 <fche> maybe your wrapper is interfering anyway

20:44 <fche> could try taking away the double-quotes from that EXTRA_CFLAGS += -I"/...../" line in Makefile

20:45 <bendlas> regarding -C, that shouldn't be necessary, the makefile is generated from `stap -r ...`, see `-I/nix/store/wlbrbgwc8kn54b7sdxdfi2j42s923j2a-kbuild-4.9.2-merged` in the log

20:45 <fche> the -C is needed because out-of-tree kernel module builds are run out of the kernel build tre

20:48 <bendlas> fche: btw, putting `CFLAGS := $(EXTRA_CFLAGS)` fixes this error

20:48 <bendlas> putting * at the end of `Makefile`

20:49 <fche> that shouldn't be necessary though - would like to know why your particular site needs it

20:50 <bendlas> fche: so somehow, the builtin implicit rule for *.o files doesn't pick up `EXTRA_CFLAGS`

20:50 <fche> yeah so the question is what's special about your site that this happens (only?) there

20:51 <bendlas> I'll read up on `EXTRA_CFLAGS` ...

20:52 <fche> is your kernel tree one that has had a corresponding kernel been built from ? so it has .config etc. ?

20:56 <bendlas> I don't see how to get at the env command for my machine ..

20:57 <fche> the stap --vp 0002 -p4 command line showed it

21:02 <bendlas> i don't see it, see last log ..

21:02 <bendlas> however .. you already guessed right: removing the quotes from `-I"..."` did the trick: https://hastebin.com/ufirogitux.log

21:03 <fche> well that's interesting

21:03 <fche> so something in between the makefile and the gcc command invocation appears to be messing this up

21:05 <bendlas> the way I'm calling it, there is just a plain gnumake4.2 in between my shell and a plain gcc4.9

21:05 <bendlas> without the quotes it works, with them it fails

21:06 <fche> any odd $LANG ?

21:08 <bendlas> well, maybe it's the kernel's make file, I'm on 4.9.2, also pretty vanilla. the kbuild dir is directly constructed from our kernel: https://github.com/NixOS/nixpkgs/blob/bb8ded9a78b61f2a23b2e4a0e516533ae0c0d1d4/pkgs/development/tools/profiling/systemtap/default.nix#L24

21:08 <bendlas> `$LANG` is `en_US.UTF-8`

21:11 <bendlas> i had the linked version already working at some point, so unless one of our updates to gcc or gnumake broke it, it was likely a kernel upgrade. I could go back to 4.4 or something, for testing ...

21:11 <fche> I'd suspect something other than kernel version

21:11 <fche> this quotation etc. business is working on kernels ancient and super fresh, distros old and new

21:12 <fche> something is peculiar about your site, just not sure what yet

21:12 <fche> want to try another strace, this time focusing on # strace -eexecve -f env .... make .... ?

21:13 <bendlas> without the fixed quotes?

21:13 <fche> yeah. or even with

21:13 <fche> I'm still thinking there's a weird wrapper in there

21:13 <fche> maybe like vanilla ice

21:15 <bendlas> ugh, strace -eexecve butchers my pathes and arguments b/c too long

21:18 <bendlas> there is something peculiar ..

21:20 <bendlas> fche: notice the messed up `-I` flag for the runtime: https://hastebin.com/qadefesege.log

21:21 <fche> yeah, it's been pooched by then

21:21 <fche> though assemblers don't normally look at c header files

21:23 <bendlas> gcc invokations above have the same SNAFU

21:23 <bendlas> the `.stap_..._src.o.cmd` still is correct, though

21:35 <bendlas> fche: thanks for the help so far, I think I can figure it out from here ...

21:35 <fche> righto

21:35 <fche> have a theory yet as to the " signs?

21:38 <bendlas> yeah, probably a wrapper sneaking in somewhere ... I'll just start with an empty path and work my way up from there

21:38 <fche> righto

21:38 <fche> yeah, it's a PITA to quote properly from e.g. a shell script wrapper

21:41 <bendlas> if you ever figure out the why and wherefore of `"$@"`, call me

21:41 <fche> exactly

22:16 <bendlas> fche: what generates the `.stap_....o.cmd` files? the .cmd file already has `-I/nix/store/b7bv8p46yy7imf3jskkv4jqdz5z0n5k0-linux-4.9.2-dev/lib/modules/4.9.2/source/"/nix/store/8b11aa5zz0cal71cww6mkyaz6rsn17nj-systemtap-2017-01-16/share/systemtap/runtime"`

22:20 <fche> the kbuild makefile creates those

23:08 <bendlas> fche: could you check, that in your makefile, the runtime directory also is passed quoted?

23:09 <bendlas> I don't see how that mangled build flag could be produced by anything other than the kbuild makefile

23:19 <bendlas> make -n even produces the shell command to print the faulty .cmd file

23:19 <bendlas> * print -> produce

23:49 <bendlas> I can verify now, that my build works against kernel 4.4, but not against 4.9.2

23:49 <bendlas> trying against 4.9.4, now