00:55
orivej has quit [Ping timeout: 246 seconds]
00:55
orivej has joined #systemtap
01:52
hpt has joined #systemtap
01:59
LW has quit [Quit: Leaving]
02:06
khaled has quit [Quit: Konversation terminated!]
02:21
LW has joined #systemtap
02:27
<
LW >
root@CyOS:/data/work/stp# stap -v -e 'probe kernel.function("netif_receive_skb").return { print_backtrace(); }'
02:27
<
LW >
Pass 1: parsed user script and 476 library scripts using 122964virt/86708res/6172shr/80920data kb, in 260usr/60sys/316real ms.
02:27
<
LW >
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 180644virt/145408res/7176shr/138600data kb, in 830usr/70sys/901real ms.
02:27
<
LW >
Pass 3: translated to C into "/tmp/stap3xWbP9/stap_5282cd7c6d3e3598f3cca611c1b241a5_1125_src.c" using 180644virt/145956res/7496shr/138600data kb, in 410usr/30sys/450real ms.
02:27
<
LW >
Pass 4: compiled C into "stap_5282cd7c6d3e3598f3cca611c1b241a5_1125.ko" in 9070usr/1920sys/9653real ms.
02:27
<
LW >
Pass 5: starting run.
02:27
<
LW >
WARNING: /usr/local/bin/staprun exited with signal: 9 (Killed)
02:27
<
LW >
Pass 5: run completed in 0usr/20sys/14real ms.
02:27
<
LW >
Pass 5: run failed. [man error::pass5]
02:27
<
LW >
root@CyOS:/data/work/stp#
02:27
<
fche >
does dmesg explain ?
02:28
<
LW >
why "print_backtrace" can not be work?
02:28
<
fche >
I don't think the problem is with backtrace.
02:28
<
fche >
staprun being killed with -9 is something else
02:28
<
LW >
stap -v -e 'probe kernel.function("netif_receive_skb").return { printf("hello\n"); }' can work well
02:29
<
LW >
just instead printf("hello") of backtrace.
02:29
<
fche >
hm shouldn't be happening
02:29
<
fche >
does dmesg offer any information?
02:30
<
fche >
what OS/kernel/arch?
02:31
<
LW >
root@CyOS:/data/work/stp# dmesg -c
02:31
<
LW >
[295633.847450] PGD 0 P4D 0
02:31
<
LW >
[295633.847444] BUG: unable to handle kernel NULL pointer dereference at 0000000000000100
02:31
<
LW >
[295633.847456] Oops: 0000 [#11] PREEMPT SMP PTI
02:31
<
LW >
[295633.847461] CPU: 0 PID: 1143 Comm: staprun Tainted: G D O 4.19.163 #1
02:31
<
LW >
[295633.847465] Hardware name: Default string Default string/SKYBAY, BIOS 5.12 07/26/2018
02:31
<
LW >
[295633.847474] RIP: 0010:init_module+0x679/0xd70 [stap_5282cd7c6d3e3598f3cca611c1b241a5_1143]
02:31
<
LW >
[295633.847479] Code: 48 e8 eb 17 31 cb 4c 39 f3 74 7a e8 e1 17 31 cb 4c 8b 63 f8 4d 85 e4 74 e4 e8 d3 17 31 cb b9 13 00 00 00 49 8d 37 49 8d 3c 24 <f3> a6 0f 97 c0 1c 00 84 c0 74 b9 e8 b7 17 31 cb b9 0a 00 00 00 48
02:31
<
LW >
[295633.847483] RSP: 0018:ffff925481f6fc28 EFLAGS: 00010293
02:31
<
LW >
[295633.847488] RAX: ffff8dfb5a28a7c0 RBX: ffff8dfb141d1900 RCX: 0000000000000013
02:31
<
LW >
[295633.847491] RDX: 0000000000000000 RSI: ffffffffc2a552fe RDI: 0000000000000100
02:31
<
LW >
[295633.847495] RBP: ffff925481f6fc68 R08: ffffffff8ee0ad40 R09: ffffb2547fd886f0
02:31
<
LW >
[295633.847499] R10: 0000000000000200 R11: 000000000089532f R12: 0000000000000100
02:31
<
LW >
[295633.847502] R13: ffff8dfb141d1870 R14: ffff8dfb141d1e10 R15: ffffffffc2a552fe
02:31
<
LW >
[295633.847507] FS: 00007fcc62215740(0000) GS:ffff8dfb5ea00000(0000) knlGS:0000000000000000
02:31
<
LW >
[295633.847510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:31
<
LW >
[295633.847514] CR2: 0000000000000100 CR3: 000000043eb72001 CR4: 00000000003606f0
02:31
<
LW >
[295633.847518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
02:31
<
LW >
[295633.847521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
02:31
<
LW >
[295633.847524] Call Trace:
02:31
<
LW >
[295633.847531] ? kvfree+0x44/0x60
02:31
<
LW >
[295633.847539] ? _stp_ctl_send+0x580/0x580 [stap_5282cd7c6d3e3598f3cca611c1b241a5_1143]
02:31
<
LW >
[295633.847544] do_one_initcall+0x5f/0x1f0
02:31
<
LW >
[295633.847549] ? vfree+0x51/0xa0
02:31
<
LW >
[295633.847554] ? kvfree+0x5b/0x60
02:31
<
LW >
[295633.847560] ? kmem_cache_alloc+0x26/0x140
02:31
<
LW >
[295633.847566] do_init_module+0x55/0x1e3
02:31
<
LW >
[295633.847572] load_module+0x22ce/0x26d0
02:31
<
LW >
[295633.847577] ? __symbol_put+0x90/0x90
02:32
<
LW >
[295633.847584] ? apic_timer_interrupt+0xa/0x20
02:32
<
LW >
[295633.847592] ? _copy_from_user+0x50/0xa0
02:32
<
LW >
[295633.847598] __do_sys_init_module+0x161/0x1c0
02:32
<
fche >
hm you should definitely try stap 4.5 then
02:32
<
LW >
[295633.847606] __x64_sys_init_module+0x15/0x20
02:32
<
LW >
[295633.847610] do_syscall_64+0x5d/0x170
02:32
<
LW >
[295633.847616] entry_SYSCALL_64_after_hwframe+0x44/0xa9
02:32
<
LW >
[295633.847620] RIP: 0033:0x7fcc611df13a
02:32
<
LW >
[295633.847626] Code: 48 8b 0d 51 8d 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 af 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1e 8d 2c 00 f7 d8 64 89 01 48
02:32
<
LW >
[295633.847629] RSP: 002b:00007ffcb26fc008 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
02:32
<
fche >
there were some transport related changes
02:32
<
LW >
[295633.847634] RAX: ffffffffffffffda RBX: 000055a6aef71040 RCX: 00007fcc611df13a
02:32
<
LW >
[295633.847638] RDX: 000055a6aef71040 RSI: 0000000000856b30 RDI: 00007fcc60649010
02:32
<
LW >
[295633.847641] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000005
02:32
<
LW >
[295633.847645] R10: 0000000000000005 R11: 0000000000000246 R12: 00007fcc60649010
02:32
<
LW >
[295633.847648] R13: 00007fcc62215698 R14: 00007ffcb26fc0c0 R15: 0000000000000000
02:32
<
LW >
[295633.847653] Modules linked in: stap_5282cd7c6d3e3598f3cca611c1b241a5_1143(O+) stap_5282cd7c6d3e3598f3cca611c1b241a5_1097(O+) stap_bf098df4a1984de92061e2e5af89368_31461(O+) stap_0de1b5124c07f3ded3b7bdec34c8f98_21269(O+) stap_bc52dab39665f5d7e09472152877239_20692(O+) stap_44dc5b88023ae4b507460cc896c377e_20121(O+) stap_d2de03e29e44ad29157b884e7389b8f_19551(O+) stap_d2de03e29e44ad29157b884e7389b8f_18949(O+) stap_bf098df4a1984de92061e2e5af89368_17795(O+)
02:32
<
LW >
stap_b6f7f46bceca0d3c5932e2504cfc2d9_17213(O+) stap_b6f7f46bceca0d3c5932e2504cfc2d9_17205(O+) bridge stp llc fuse binfmt_misc intel_rapl x86_pkg_temp_thermal intel_powerclamp cdc_ether coretemp usbnet r8152 i915 mei_me mei video sch_fq_codel parport_pc lp parport ip_tables x_tables igb [last unloaded: stap_4179390984d246879daadbce4da9fc77_414]
02:32
<
LW >
[295633.847685] CR2: 0000000000000100
02:32
<
LW >
[295633.847690] ---[ end trace f2d2061321ef5ade ]---
02:32
<
LW >
[295633.847698] RIP: 0010:init_module+0x679/0xd70 [stap_b6f7f46bceca0d3c5932e2504cfc2d9_17205]
02:32
<
LW >
[295633.847703] Code: 48 e8 6b 47 98 cd 4c 39 f3 74 7a e8 61 47 98 cd 4c 8b 63 f8 4d 85 e4 74 e4 e8 53 47 98 cd b9 13 00 00 00 49 8d 37 49 8d 3c 24 <f3> a6 0f 97 c0 1c 00 84 c0 74 b9 e8 37 47 98 cd b9 0a 00 00 00 48
02:32
<
LW >
[295633.847706] RSP: 0018:ffff925483d1fc28 EFLAGS: 00010293
02:32
<
LW >
[295633.847711] RAX: ffff8dfb2e918d40 RBX: ffff8dfb5ac9d100 RCX: 0000000000000013
02:32
<
LW >
[295633.847714] RDX: 0000000000000000 RSI: ffffffffc03e22fe RDI: 0000000000000100
02:32
<
LW >
[295633.847718] RBP: ffff925483d1fc68 R08: 0000000000006500 R09: ffffb2547fd86500
02:33
<
LW >
[295633.847722] R10: 0000000000001c48 R11: ffffbfd110f12188 R12: 0000000000000100
02:33
<
LW >
[295633.847725] R13: ffff8dfb5ac9d070 R14: ffff8dfb5ac9d610 R15: ffffffffc03e22fe
02:33
<
LW >
[295633.847730] FS: 00007fcc62215740(0000) GS:ffff8dfb5ea00000(0000) knlGS:0000000000000000
02:33
<
LW >
[295633.847733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:33
<
LW >
[295633.847737] CR2: 0000000000000100 CR3: 000000043eb72001 CR4: 00000000003606f0
02:33
<
LW >
[295633.847741] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
02:33
<
LW >
[295633.847744] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
02:33
<
LW >
root@CyOS:/data/work/stp#
02:40
<
LW >
thanks, the stap 4.5 work well
02:54
<
fche >
ah good to know
02:56
<
LW >
I am newbie for stap, how to monitor/printf a local var in kernel function?
02:56
<
fche >
you put a .statement() probe into the proper place in the interior of that function, then just print it as any other $context variable
02:56
<
LW >
static int ip_rcv_finish_core(struct net *net, struct sock *sk,
02:56
<
LW >
struct sk_buff *skb, struct net_device *dev)
02:56
<
LW >
const struct iphdr *iph = ip_hdr(skb);
02:56
<
LW >
int (*edemux)(struct sk_buff *skb);
02:56
<
LW >
struct rtable *rt;
02:56
<
fche >
stap -L 'kernel.statement("ip_rcv_finish_core@*:*")'
02:57
<
fche >
to list the source file/line ranges & variables available for probing
02:57
<
LW >
how to printf the "int err" var?
02:57
<
fche >
println($err)
02:58
<
LW >
root@CyOS:/data/work/stp# stap -L 'kernel.function("ip_rcv_finish_core")'
02:58
<
LW >
kernel.function("ip_rcv_finish_core@net/ipv4/ip_input.c:309") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
02:58
<
fche >
.function probes at function entry
02:58
<
fche >
you want a .statement probe in the interior of a function
03:00
<
LW >
root@CyOS:/data/work/stp# stap -L 'kernel.statement("ip_rcv_finish_core@*:*")'
03:00
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:309") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:00
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:318") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:00
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:317") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:00
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:319") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:00
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:320") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:00
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:322") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:324") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:325") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:326") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:327") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const* $err:int
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:339") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:340") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:341") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const* $err:int
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:346") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const* $err:int
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:347") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:348") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:349") $idx:u32 $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:350") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:351") $idx:u32 $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:352") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:356") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:360") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:361") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:362") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:01
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:363") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:02
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:364") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:02
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:383") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:02
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:384") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:02
<
fche >
oh no oh no make it stop
03:02
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:388") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:02
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:391") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const* $err:int
03:02
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:392") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const*
03:02
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:395") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const* $err:int
03:02
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:396") $net:struct net* $skb:struct sk_buff* $dev:struct net_device* $iph:struct iphdr const* $err:int
03:02
<
LW >
kernel.statement("ip_rcv_finish_core@net/ipv4/ip_input.c:398") $net:struct net* $skb:struct sk_buff* $dev:struct net_device*
03:02
<
fche >
so anyway you can see which line#'s correspond to statements where the $err variable survived optimization
03:02
<
LW >
root@CyOS:/data/work/stp#
03:02
<
fche >
you can pick the ones where it works and which by inspection of the source code works best
03:03
<
fche >
.... or you can use a prepackaged little script like
03:03
<
fche >
stap --example varwatch.stp 'kernel.statement("ip_rcv_finish_core@*:*")' '$err'
03:03
<
fche >
in which case it'll tell you when that variable changes value
03:05
<
LW >
why the local var (struct rtable *rt) is not listed in the stap -L
03:06
<
LW >
Can't all variables be observed?
03:06
<
fche >
it may just not have survived optimization
03:06
<
fche >
gcc performs heroics but it's still not as observable as a -g compiled binary
03:07
<
fche >
your kernel is young enough to be affected by an lkml error of judgement, see the release notes re. "-fno-var-tracking-assignments"
03:07
<
LW >
if i use -o0, all the var should be observed?
03:08
<
fche >
probably, but then the kernel might not be compilable at all
03:08
<
fche >
it's a tough balance
03:09
<
fche >
if your kernel build turns on -fno-var-tracking-assignments, that should be removed and a normal build attempted
03:10
<
LW >
how to check "-fno-var-tracking-assignments" for my kernel config?
03:10
<
fche >
check your kernel Makefile
03:10
<
LW >
is not controlled by the .config?
03:10
<
fche >
not really, it was hard-coded, unfortunately
03:14
<
fche >
anyway, zZzZz time here, good luck!
03:15
<
LW >
thank you very much.
03:15
<
fche >
np, it's good to work with someone sharp & responsive!
03:17
<
LW >
thanks, have a good night.
03:44
khaled has joined #systemtap
04:19
LW has quit [Remote host closed the connection]
04:20
LW has joined #systemtap
05:19
orivej has quit [Ping timeout: 240 seconds]
05:25
orivej has joined #systemtap
05:36
LW has quit [Quit: Leaving]
07:12
fdalleau_away is now known as fdalleau
09:41
hpt has quit [Ping timeout: 265 seconds]
10:49
orivej has quit [Ping timeout: 265 seconds]
11:39
fdalleau has quit [Ping timeout: 246 seconds]
11:58
fdalleau has joined #systemtap
12:40
fdalleau is now known as fdalleau_away
12:47
fdalleau_away is now known as fdalleau
13:47
tromey has joined #systemtap
14:02
amerey has joined #systemtap
14:14
orivej has joined #systemtap
18:45
sscox has quit [Quit: sscox]
18:51
sscox has joined #systemtap
19:01
fdalleau is now known as fdalleau_away
19:58
fedeztk has joined #systemtap
19:59
fedeztk has quit [Client Quit]
20:52
tromey has quit [Quit: ERC (IRC client for Emacs 27.1)]
22:41
amerey has quit [Quit: Leaving]
23:05
<
kerneltoast >
fche, you see that email from sscox about TLS
23:05
<
kerneltoast >
"runtime: fix symbol lookups when the first section isn't executable" is back with a vengeance
23:06
<
kerneltoast >
the problem is still in _stp_vma_mmap_cb()
23:06
<
kerneltoast >
getting rid of that extra if condition for the huge-page case fixes it