#systemtap on 2016-02-05 — irc logs at freenode.irclog.whitequark.org

2015-11-12 23:18 fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged

00:58 hpt has joined #systemtap

01:12 modem_ has quit [Read error: Connection reset by peer]

01:13 pfalleno1 has quit [Ping timeout: 245 seconds]

01:19 khaled has joined #systemtap

01:23 scox has joined #systemtap

01:25 pfallenop has joined #systemtap

01:30 khaled has quit [Quit: Leaving]

01:31 pfallenop has quit [Ping timeout: 250 seconds]

01:51 pfallenop has joined #systemtap

02:03 pfallenop has quit [Ping timeout: 250 seconds]

02:04 pfallenop has joined #systemtap

02:20 pfallenop has quit [Remote host closed the connection]

02:21 pfallenop has joined #systemtap

03:01 irker895 has quit [Quit: transmission timeout]

03:47 ego_ has joined #systemtap

03:56 ego_ has quit [Ping timeout: 240 seconds]

03:57 ego_ has joined #systemtap

04:10 ego_ has quit [Ping timeout: 276 seconds]

04:11 ego_ has joined #systemtap

04:31 hkshaw has joined #systemtap

04:46 ego_ has quit [Ping timeout: 260 seconds]

04:49 ego_ has joined #systemtap

04:57 srikar_away is now known as srikar

05:00 naveen has joined #systemtap

05:02 ego_ has quit [Ping timeout: 260 seconds]

05:55 hkshaw has quit [Quit: Leaving.]

06:04 pfallenop has quit [Ping timeout: 248 seconds]

06:10 hkshaw has joined #systemtap

06:11 hkshaw has quit [Client Quit]

06:11 hkshaw has joined #systemtap

06:11 ananth has joined #systemtap

06:17 pfallenop has joined #systemtap

06:26 pfallenop has quit [Ping timeout: 256 seconds]

06:28 pfallenop has joined #systemtap

06:44 pfallenop has quit [Remote host closed the connection]

06:44 pfallenop has joined #systemtap

07:09 prerna has joined #systemtap

08:16 prerna has quit [Quit: This computer has gone to sleep]

08:35 ego has joined #systemtap

08:49 ego has quit [Ping timeout: 264 seconds]

08:51 ego has joined #systemtap

09:01 prerna has joined #systemtap

09:06 hpt has quit [Ping timeout: 272 seconds]

09:42 mjw has quit [Quit: Leaving]

10:01 mjw has joined #systemtap

10:23 ego has quit [Ping timeout: 256 seconds]

10:26 nkambo has quit [Ping timeout: 245 seconds]

10:31 modem_ has joined #systemtap

10:32 ego has joined #systemtap

10:51 nkambo has joined #systemtap

10:53 naveen has quit [Quit: Leaving.]

10:56 nkambo has quit [Ping timeout: 256 seconds]

11:00 scox has quit [Ping timeout: 240 seconds]

11:08 zodbot has quit [Disconnected by services]

11:12 hkshaw has quit [Ping timeout: 264 seconds]

11:12 modem has quit [Ping timeout: 264 seconds]

11:12 pmuldoon has quit [Ping timeout: 264 seconds]

11:20 zodbot has joined #systemtap

11:24 pmuldoon has joined #systemtap

11:25 modem has joined #systemtap

11:25 hkshaw has joined #systemtap

11:26 hkshaw has quit [Client Quit]

11:32 ananth has quit [Quit: Leaving]

11:46 nkambo has joined #systemtap

11:58 nkambo1 has joined #systemtap

11:59 nkambo has quit [Ping timeout: 248 seconds]

12:13 nkambo2 has joined #systemtap

12:16 nkambo1 has quit [Ping timeout: 252 seconds]

12:19 nkambo2 has quit [Ping timeout: 256 seconds]

12:19 hkshaw has joined #systemtap

12:21 nkambo has joined #systemtap

12:32 nkambo has quit [Ping timeout: 245 seconds]

12:34 nkambo has joined #systemtap

12:39 hkshaw has quit [Ping timeout: 240 seconds]

12:39 flu_ has joined #systemtap

12:40 nkambo has quit [Ping timeout: 252 seconds]

12:40 nkambo has joined #systemtap

12:46 nkambo has quit [Ping timeout: 240 seconds]

12:46 nkambo has joined #systemtap

12:50 nkambo1 has joined #systemtap

12:52 nkambo has quit [Ping timeout: 256 seconds]

12:54 hkshaw has joined #systemtap

12:55 nkambo2 has joined #systemtap

12:56 nkambo1 has quit [Ping timeout: 276 seconds]

13:02 prerna has quit [Quit: This computer has gone to sleep]

13:17 wcohen has quit [Ping timeout: 245 seconds]

13:20 hkshaw has quit [Quit: Leaving.]

13:43 scox has joined #systemtap

13:49 nkambo2 has quit [Ping timeout: 240 seconds]

13:49 nkambo2 has joined #systemtap

14:01 mbenitez has joined #systemtap

14:01 mbenitez has quit [Changing host]

14:01 mbenitez has joined #systemtap

14:15 ego has quit [Ping timeout: 248 seconds]

14:45 brolley has joined #systemtap

14:51 tromey has joined #systemtap

15:01 srikar is now known as srikar_away

15:20 srikar_away is now known as srikar

15:30 wcohen has joined #systemtap

15:39 nkambo1 has joined #systemtap

15:42 nkambo2 has quit [Ping timeout: 240 seconds]

16:06 sfink has quit [Ping timeout: 245 seconds]

16:09 nkambo1 has quit [Ping timeout: 260 seconds]

16:12 nkambo has joined #systemtap

16:27 jameshyde_ has joined #systemtap

16:28 <jameshyde_> is it possible to use systemtap to see activities for pgfree/s in sar?

16:33 eichiro has joined #systemtap

16:46 scox has quit [Ping timeout: 264 seconds]

16:46 <fche> jameshyde_, most likely

16:46 <fche> just need to reverse engineer what /proc value sar is pulling out for that field, and what kernel-side event that corresponds to

16:47 <fche> there is kernel.trace("kmem:mm_page_free") and kernel.trace("kmem:mm_page_free_batched") e.g.

16:48 <jameshyde_> ok thanks

17:00 <jameshyde_> fche: when trying this in RHEL6, got this:

17:00 <jameshyde_> source: probe kernel.trace("kmem:mm_page_free") {

17:00 <jameshyde_> semantic error: no match (similar tracepoints: kfree, mm_pagevec_free, kmem_cache_free, mm_page_alloc, module_free)

17:00 <fche> ah yes, tracepoints do vary between kernels

17:00 <fche> am just hunting down some info for you now, stand by

17:02 <fche> (sar pgfree/s comes from /proc/vmstat line pgfree)

17:02 <jameshyde_> ok

17:04 <jameshyde_> we noticed large number of pgfree/s, so wondering which proc is really doing that. systemtap might be able to pinpoint this..

17:04 <fche> should be able to, definitely

17:06 zorachus has left #systemtap [#systemtap]

17:08 <fche> ogling linux mm/vmstat.c , include/linux/vmstat.h, include/linux/vm_event_item.h

17:09 <fche> aaaand the winner is __count_vm_event(PGFREE);

17:10 <fche> inconveniently, I don't see kernel tracepoints in the vicinity, in the upstream git kernel

17:15 <fche> jameshyde_, just wondering for testing purposes, what version of rhel6 are you running?

17:21 <fche> jameshyde_, ok, have something for you to try:

17:22 <jameshyde_> rhel6.6 here

17:22 <fche> http://paste.fedoraproject.org/319067/

17:22 <fche> these are approximations only - may overcount the pgfree/s metric a little bit

17:23 <fche> they work by tracing the entry to both functions in mm/* that can cause incrementing of the PGFREE counter

17:25 <fche> now if you want to track -rates-, for example rates in excess of N pages per second, we can do that

17:29 <jameshyde_> thanks fche, this is very helpful! would love to have that if possible ;)

17:29 <fche> ok, will do a v2 moohahha

17:36 <fche> http://paste.fedoraproject.org/319071/ <- v2

17:36 <fche> this one periodically reports accumulated pgfree counts on a per-thread basis

17:38 naveen has joined #systemtap

17:39 <jameshyde_> awesome! this is great, thanks a lot.

17:39 prerna has joined #systemtap

17:39 <fche> ok. can you work from that, or shall we try generalizing it further?

17:39 <fche> we already have the eventcount.stp script that could be used like ...

17:40 <fche> stap eventcount.stp 'kernel.function("free_hot_cold_page")' 'kernel.function("__free_pages_ok")'

17:40 <fche> except that can't account for $order (power-of-two sets of pages)

17:54 prerna has quit [Quit: This computer has gone to sleep]

18:04 <jameshyde_> fche: would love to have it generalized a bit further ;)

18:04 <fche> what would you like ideally?

18:10 srikar is now known as srikar_away

18:11 naveen has quit [Quit: Leaving.]

18:15 <fche> jameshyde_, ^^ ?

18:18 <jameshyde_> fche: is it possible to seperate kernel proc from user proc?

18:18 <jameshyde_> ppid=2 would be kernel proc

18:19 <jameshyde_> this is pretty good already ;)

18:20 <fche> thing is some of these paging decisions may be performed -by- kernel threads, but on behalf of userspace actions

18:20 sfink has joined #systemtap

18:20 <fche> (in a sense they are all userspace actions)

18:20 <fche> it's almost as we'd like to know whether the target page was mapped into a userspace process or not

18:21 <jameshyde_> ok

18:26 <fche> There are $page->flags one could look at, but it's getting pretty far from my expertise

18:26 <fche> if you know that you're getting high numbers (rates) from some particular process,

18:26 <fche> a version of the second script could do something like

18:26 <fche> function freed(n) { history ... <<< n; if (@sum(history[...]) > NUMBER) { print_backtrace() } ' kind of thing

18:27 <fche> ie look back up the callstack to see why

18:27 <fche> print_ubacktrace() too possibly

18:27 <jameshyde_> this might be exactly what we need, thanks!

18:27 <fche> it's an iterative process to discover what's going on; can't give you an all-singing all-question-answering script

18:37 dancancode has joined #systemtap

19:03 scox has joined #systemtap

19:23 modem_ has quit [Ping timeout: 240 seconds]

19:25 modem_ has joined #systemtap

20:23 flu_ has quit [Quit: flu_]

21:15 nkambo1 has joined #systemtap

21:16 nkambo has quit [Ping timeout: 260 seconds]

21:19 nkambo2 has joined #systemtap

21:21 nkambo1 has quit [Ping timeout: 252 seconds]

21:23 nkambo1 has joined #systemtap

21:25 nkambo2 has quit [Ping timeout: 240 seconds]

21:29 nkambo1 has quit [Ping timeout: 272 seconds]

21:31 nkambo has joined #systemtap

21:36 nkambo1 has joined #systemtap

21:39 nkambo has quit [Ping timeout: 245 seconds]

21:42 nkambo1 has quit [Ping timeout: 240 seconds]

21:49 nkambo1 has joined #systemtap

21:59 nkambo2 has joined #systemtap

22:01 nkambo1 has quit [Ping timeout: 240 seconds]

22:03 mbenitez has quit [Quit: Leaving]

22:10 tromey has quit [Quit: ERC (IRC client for Emacs 25.0.90.1)]

22:21 nkambo1 has joined #systemtap

22:24 nkambo2 has quit [Ping timeout: 272 seconds]

22:32 nkambo2 has joined #systemtap

22:34 nkambo1 has quit [Ping timeout: 245 seconds]

22:35 nkambo1 has joined #systemtap

22:37 nkambo2 has quit [Ping timeout: 260 seconds]

22:41 nkambo2 has joined #systemtap

22:44 nkambo1 has quit [Ping timeout: 276 seconds]

22:44 nkambo has joined #systemtap

22:45 nkambo2 has quit [Ping timeout: 245 seconds]

22:47 nkambo has quit [Read error: Connection reset by peer]

22:47 wcohen has quit [Ping timeout: 240 seconds]

22:50 nkambo has joined #systemtap

22:54 sfink has quit [Ping timeout: 240 seconds]

23:10 brolley has left #systemtap [#systemtap]

23:12 nkambo has quit [Ping timeout: 240 seconds]

23:12 nkambo1 has joined #systemtap

23:23 nkambo1 has quit [Ping timeout: 272 seconds]

23:37 wcohen has joined #systemtap

23:43 mjw has quit [Quit: Leaving]