13:09 <jacksgt> Hi everyone, are there any other resources on SystemTap

13:10 <jacksgt> * on how SystemTap internals work except the source code itself?

14:05 <fche> jacksgt, there is the INTERNALS source file

14:05 <fche> there is the old architecture paper

14:05 <fche> there is the [man stap] page

14:06 <fche> there is stap -k -vv .... which lets you look at the generated artifacts

14:06 <fche> would be glad to answer a more specific question

14:08 <jacksgt> fche: Thanks for your answer. I already read the INTERNALS file as well as the manpage, but I couldn't really get a gist on how SystemTap probes instrument the code without changing the source.

14:09 <jacksgt> fche: To put it in simpler terms: how does SystemTap the application jumps to function xyz _now_?

14:09 <fche> for the normal linux kernel module based backend, we make calls to kernel facilities such as tracepoints, uprobes, kprobes, to hook into the target software

14:10 <fche> stap doesn't -cause- jumps within the program. It sort of listens for the app jumping within itself, tripping across a breakpoint.

14:14 <jacksgt> Are we talking about hardware breakpoints?

14:14 <fche> usually software breakpoints (int3 on x86)

14:15 <fche> which (in this context) the kernel puts into itself or userspace apps

14:25 <jacksgt> So basically this (from the GDB internals)?

14:25 <jacksgt> Software breakpoints require GDB to do somewhat more work. The basic theory is that GDB will replace a program instruction with a trap, illegal divide, or some other instruction that will cause an exception, and then when it’s encountered, GDB will take the exception and stop the program. When the user says to continue, GDB will restore the original instruction, single-step, re-insert the trap, and continue on.

14:25 <jacksgt> https://sourceware.org/gdb/wiki/Internals/Breakpoint%20Handling

14:27 <fche> kind of, except that in the case of systemtap, the kernel is doing this

14:27 <fche> (for the traditional lkm runtime. for stap --runtime=dyninst, it's all different again; binary rewriting in userspace using the dyninst library)

14:29 <jacksgt> Ok, but I thought the code/text segment is marked as read-only. I could see how SystemTap (or any debugger for that matter) could modify the segment before the application launches, but how is this possible while the program is already running?

14:33 <fche> the kernel is All Powerful

14:34 <fche> and even debuggers can modify other processes' read-only-mapped page segments via ptrace(2), which cause the kernel to create a private modified copy

14:40 <jacksgt> Amazing.

14:41 <jacksgt> fche: Do you also happen to know how the DynInst runtime works?

14:42 <fche> not deeply

14:42 <fche> dyninst is a ginormous (tm) project with its own research organization behind it

14:43 <fche> it disassembles and snips and tucks and cuts & pastes binary pieces together

14:43 <fche> it's true magic

14:43 <fche> (but imperfect magic)

15:02 <jacksgt> Indeed! :D

15:02 <jacksgt> fche: Thanks for your answers :-)

15:03 <fche> np