12:59 <__positron> I have built and installed a custom kernel based on v4.8.4 on my Ubuntu 14.04. I would like to inspect skb inside the function __netif_receive_skb_core. I probed this function during call and printed out $skb$$. The fields are filled with the address of $skb. I have also tried by printing it with $params$$. Do you have any suggestions on what could have gone wrong?

13:02 <fche> __positron, could you fpaste.org your script and its output ?

13:03 <fche> and also stap -V ?

13:07 <__positron> fche: version 3.2/0.158, commit release-3.1-59-g35c6bf4e9242

13:07 <fche> ok, brand new

13:12 <__positron> fche: https://paste.fedoraproject.org/paste/1QF39FL27JZeGmG1QgMrJV5M1UNdIGYhyRLivL9gydE=

13:22 <fche> ok, so that seems to be working appropriately

13:22 <fche> pointers inside these structures won't be followed for pretty-printing purposes, only nested structs

13:23 <fche> which part of $skb are you interested in?

13:40 <__positron> Ah. I see. If that's the case, printing with $skb->member should work. Right? I am interested in skb->protocol for now. I am debugging a custom network driver in which the packets are dropped during reception. So, I am more or less interested in all the interesting fields that would shed light on why the packet might get dropped.

13:41 <__positron> I have tried it with $skb->len and $skb->protocol. I don't see any sane values. https://paste.fedoraproject.org/paste/U9hmEEFXmONpPKIHtwlLV15M1UNdIGYhyRLivL9gydE=

13:42 <fche> that does look like bad data

13:42 <fche> just checking: is that structure supposed to be initialized at function call entry time?

13:45 <fche> yeah, must be

13:45 <fche> see also kernel tracepoint ways of getting at closely related events

13:46 <__positron> I guess it should be. To cross check I tried using the same probe function on a different machine with 4.4.0-75-generic and version 2.9/0.165, Debian version 2.9-2ubuntu2 (xenial). It gives me sane result.

13:46 <fche> probe kernel.trace("netif_rx") { println ($skb->len) } e.g.

13:46 <fche> % stap -L 'kernel.trace("netif_*")'

13:46 <fche> kernel.trace("net:netif_rx_entry") $skb:struct sk_buff const*

13:46 <fche> kernel.trace("net:netif_rx") $skb:struct sk_buff*

13:46 <fche> kernel.trace("net:netif_receive_skb_entry") $skb:struct sk_buff const*

13:46 <fche> kernel.trace("net:netif_receive_skb") $skb:struct sk_buff*

13:46 <fche> kernel.trace("net:netif_rx_ni_entry") $skb:struct sk_buff const*

13:53 <__positron> Surprisingly, probing the tracepoint doesn't print any result on v4.8.4 kernel. I checked with -v flag, it compiles and runs fine. But no result.

13:55 <fche> which tracepoint did you try?

13:55 <fche> I just named a random one to give the idea

13:55 <__positron> stap -v -e 'probe kernel.trace("net:netif_rx") { println ($skb->len) }'

13:56 <__positron> I tried without net: prefix too.

13:58 <fche> oh hold on a bit

13:58 <fche> you're using git systemtap

13:58 <fche> we're tracking down a regression related to the merge of the bpf backend prototype

13:58 <fche> could you go back to the 3.1 release and try again?

14:02 <__positron> :facepalm: I shouldn't have tried it with the latest master. building it now.

14:04 <fche> no I mean -not- the latest master

14:04 <fche> we know there are some problems in the latest master. I'm suggesting going back to the previous release

14:05 <__positron> yeah. I agree. latest master is not always a good starting point. I switched to v3.1 and it works fine. Thank you so much.

14:16 <fche> thanks. it usually is a fine version to use

14:16 <fche> this particular regression is very unusual and should be fixed in a few days

20:56 <irker262> systemtap: dsmith systemtap.git:refs/heads/master * release-3.1-60-g5c5a0bf / Makefile.am Makefile.in config.in configure configure.ac httpd/Makefile.am httpd/Makefile.in httpd/main.cxx httpd/server.cxx httpd/server.h: Added initial http web service server code. http://tinyurl.com/mbsx7n4