11:52 <gromero> fche: Hi. I would like to have the floating-point registers available on Power for probes in userspace. Basically CONTEXT->uregs (which rely on struct pt_regs, i.e. thread_struct->regs) is used to get the register values, however floating-point values resides in thread_struct->fp_state and not in thread_struct->regs. Do you know how what's the correct way to implement it on systemtap?

11:55 <fche> gromero, hi

11:55 <gromero> btw, the fp registers are also available in userspace in the signal handler through ucontext_t->uc_mcontext

11:55 <gromero> fche: hi!

12:00 <fche> gromero, new tapset functions, or extended ones like register("name") could be one way

12:00 <fche> is there a dwarf register# for them, when userspace code passes fp values that way?

12:05 <gromero> fche: I believe in register("name") would be nice, that's what I'm looking for.

12:05 <gromero> ABI says the following about register #s:

12:05 <gromero> DWARF | Register Number | Register Name | Width (Bytes)

12:05 <gromero> Reg | 32 - 63 | f0 - f31 | 8

12:06 <fche> ok, then it should be possible to have both register() and @uregister(#) work for those

12:07 <fche> see powerpc/registers.stp -- it does assume pt_regs, e.g. in _stp_get_register_by_offset

12:08 <gromero> exactly

12:08 <gromero> but pt_regs has not information about fp

12:08 <gromero> afaics

12:09 <fche> yes. so one'd have to extend some of those tapset functions to understand two register banks

12:09 <fche> register(name) could stay untouched

12:10 <fche> _reg_offsets could contain some magic value (bitmask field?) to indicate a non-pt-regs location for the register

12:10 <gromero> but regs is used to get the values, by regs = (CONTEXT->user_mode_p ? CONTEXT->uregs : CONTEXT->kregs);

12:11 <gromero> uregs and kregs are actually taken from thread_struct->regs, i.e. from pt_regs

12:11 <fche> yes, hat function (_stp_get_register_by_offset) would have to look for the distinguished offset parameter, and if present, search in the other thread info struct

12:11 <fche> and reject on !CONTEXT->user_mode_p etc.

12:12 <fche> so, the suggestion is: put fp registers into _reg_offsets[] but with magic offset numbers (dunno, add 0x10000)

12:12 <gromero> but that would require to extend the context members in ./runtime/common_probe_context.h ?

12:12 <fche> then have _stp_get_register_by_offset(..) recognize those offset offsets

12:13 <fche> extend the context-> to take a copy of the userspace fp_state? probably not, but can do if really needed. chances are the stap tapset function (which is embedded-C in this case) can access current->fp_state probably.

12:16 <gromero> extend the context-> to take a copy of the userspace fp_state? correct, that what I was thinking of. I was basically unsure if I'm able to access current->fp_state directly, but I see, if current->fp_state can be used directly in _stp_get_register_by_offset() it looks straightforward.

12:16 <gromero> I'll give it a try :)

12:16 <fche> yeah. just be careful to add any runtime tests - dunno for example if current->fp_state is always guaranteed to actually be set

12:19 <gromero> on power fp_state will only be set if the process when off the CPU at least once (for a context switch, syscall, etc, for instance) and if FP facility is being used, i.e. if the process executed at least an instruction using a FP register. Would the be a problem?

12:19 <gromero> *the process went off the CPU at least once

12:19 <fche> not being very familiar with powerpc behaviour here, I'd say -- add checks to the tapset function to look for validity

12:20 <gromero> ah, ok I see

12:20 <gromero> where should I add the runtime tests ?

12:20 <fche> because embedded-C is unprotected C, and you don't want someone crashing your kernel just because they ran a probe on an integer-only process that hasn't scheduled off the cpu at least once

12:21 <gromero> I see. I'll do that.

12:22 <fche> looking forward to the patch!

12:28 <gromero> fche: sure! ;)

14:09 <gromero> fche: does stap relies on ptrace syscall to get the pt_regs for uregs and kregs?

14:09 <gromero> I'm not quite sure peeking current->fp_state.fpr member is the correct thing to do for reading the fp register for uprobes...

14:10 <gromero> on ptrace() it's possible to pass the register name for PT_KEEPUSER and get the fp values

14:11 <fche> ptrace syscall is totally irrelevant to systemtap

14:11 <gromero> kernel will take care to determine if it must read from regs (pt_regs) or from fp_state based on that 'addr' passed to ptrace()

14:11 <gromero> quite similar to what I'm trying to do

14:12 <gromero> oh I see

14:12 <fche> we're talking about kernel-side runtime, so we may have to replicate that effort in our runtime code (unless it happens to be packaged in such a way that we can call into it from within the kernel)

14:12 <fche> (that's unlikely)

14:12 <gromero> I see

16:47 <irker340> systemtap: sapatel systemtap.git:refs/heads/sapatel/pr23285 * release-4.1-42-ga9282dc / bpf-translate.cxx: partially working thread and named pipe mechanism http://tinyurl.com/y3wlmqls

16:48 <gromero> fche: well, still no luck on getting fpr values correctly. it looks like that current->thread.fp_state.fpr[32][2] member are all zeroed when probe code is executed, i.e. after uprobe_notify_resume, returning from the trap exception

16:49 <gromero> so much that even using printk() in _stp_get_register_by_offset I can get them, using something like:

16:49 <gromero> + for (i1 = 0; i1 < 32; i1++)

16:49 <gromero> + for (i2 = 0; i2 < 2; i2++)

16:49 <gromero> + printk("fpr[%d][%d] = %llx\n", i1, i2, current->thread.fp_state.fpr[i1][i2]);

16:49 <gromero> odd

16:50 <gromero> since, gpr, nip and all pt_regs members are correctly set at the point .. .

16:51 <gromero> lower fpr fpr[x][0] is supposed to be zero tho, since fp is only 64-bit and a represented and 128-bit entities for convenience as the can refer to to vsx0-31 (which are 128-bit)

16:53 <gromero> *in _stp_get_register_by_offset I *can't* get them

21:51 <gromero> what's the root cause usually for "registration error (rc -22)" ?

21:52 <gromero> probe fails to attach on a kernel function, but I would like to know if there is a simply way to find out why exactly. ..

22:05 <gromero> ah, it's blacklisted : (

22:12 <fche> should add some such possibilities to the error::pass5 man page

22:13 <fche> gromero, re. the fp regs, you're sure this is for a target program that uses those fp regs ?

22:14 <fche> would be worth checking out the powerpc arch bits in the kernel. it's possible (likely?) that the kernel doesn't even bother save those fpregs into that struct unless necessary (about to context-switch to some other fp-busy program), and otherwise just leave them in the cpu regs

22:15 <gromero> fche: yes, I guess it uses fp regs, I crafted it with some inline asm and gdb tells me it's effective

22:15 <fche> and where did you set the probe to interrupt that task ?

22:16 <gromero> yes, your hypothesis is pretty good and I'm actually trying to confirm that, but I'm having a hard time ...

22:16 <fche> setting up some stap probes inside those parts of the kernel that deal with the fp regs may help

22:16 <fche> i.e., use stap to trace the fp reg manipulations

22:16 <gromero> for the probe I have:

22:16 <gromero> probe process("/home/gromero/x").function("a_func").return {

22:16 <gromero> printf("%0lx\n", register("r3"));

22:16 <gromero> printf("%0lx\n", register("f0"));

22:16 <gromero> }

22:17 <gromero> f0 will hit the crafted code in _stp_get_register_by_offset that uses current->thread.fp_state.fpr[i1][i2]

22:17 <gromero> x.c is:

22:17 <gromero> void a_func(int i)

22:17 <gromero> {

22:17 <gromero> printf("%d\n", i);

22:17 <gromero> asm volatile ("li 3, 0xbee;"

22:17 <gromero> "mtvsrd 0, 3;"

22:17 <gromero> "mtvsrd 32,3;"

22:17 <gromero> "fctid 1, 0;");

22:17 <gromero> }

22:17 <gromero> int main()

22:17 <gromero> {

22:18 <gromero> a_func(0xf);

22:18 <gromero> }

22:18 <gromero> vs0 is actually f0

22:18 <gromero> but I also tested vs32 and other values

22:18 <gromero> > setting up some stap probes inside those parts of the kernel that deal with the fp regs may help

22:19 <gromero> that's where I am now

22:19 <gromero> but it's not being easy to find a function I can trust for that purpose

22:20 <gromero> what if trap exception won't save fp in the fp_state struct I'm wondering ...

22:21 <gromero> I'm inclined to say fp are not saved on the exception alone, only if a context switch happens

22:22 <gromero> let me try to add some loops after messing with vs0 to see if I get a context-switch

22:24 <fche> would check for those parts of arch/powerpc/* that mention MSR_FP

22:28 <gromero> okay

22:28 <fche> save_fpu() relevant symbol

22:29 <fche> flush_all_to_thread() <-- our pal

22:30 <gromero> I tried giveup_all already btw

22:31 <fche> called how?

22:32 <gromero> wait, I found fpr struct set after returning from save_fpu() o/

22:33 <gromero> interesting !

22:33 <gromero> set with `0xbee` as expected

22:33 <gromero> let me see the trace

22:34 <gromero> fp_state.fpr[0]=0x0000000000000bee0000000000000000

22:34 <gromero> vr_state.vr[0] =0x00000000000000000000000000000000

22:34 <gromero> Returning from: 0xc00000000000f4f8 : save_fpu+0x0/0x2ac [kernel]

22:34 <gromero> 0xc000000000020b64 : giveup_all+0xe4/0x110 [kernel]

22:34 <gromero> Returning to : 0xc000000000020468 : __giveup_fpu+0x28/0x80 [kernel]

22:34 <gromero> 0xc000000000020b64 : giveup_all+0xe4/0x110 [kernel] (inexact)

22:34 <gromero> 0xc000000000021aa8 : __switch_to+0x108/0x460 [kernel] (inexact)

22:34 <gromero> 0xc000000000144664 : do_task_dead+0x64/0x70 [kernel] (inexact)

22:34 <gromero> 0xc000000000107848 : do_exit+0x818/0xd50 [kernel] (inexact)

22:35 <gromero> 0xc000000000107e50 : do_group_exit+0x60/0x110 [kernel] (inexact)

22:35 <gromero> 0xc000000000107f24 : sys_exit_group+0x24/0x30 [kernel] (inexact)

22:35 <gromero> 0xc00000000000bde4 : system_call+0x5c/0x70 [kernel] (inexact)

22:38 <gromero> but on the same process "x", after that hit, other hits to save_fpu() return 0 in fpr[0]

22:39 <gromero> fche: like: https://pastebin.com/raw/ThN4kQbE

22:40 <gromero> that's unexpected, since I would expect that fp_state would be in sync to what's found in the register in userspace ...

22:40 <gromero> until the end of the process f0=0xbee

22:40 <fche> right but the kernel might not save the register state back to the task_struct every time

22:41 <fche> s/might/will/

22:41 <fche> unless that giveup_all() or equivalent is done every time you want to read the ->Fp_regs

22:42 <gromero> okay, I'll confirm that with the kernel folks tomorrow, but if it holds true, how should we proceed ?

22:42 <gromero> we basically can't trust fp_state context at will : /

22:42 <fche> we need to arrange to call giveup_all() or equivalent when/if the fpregs are being sought

22:43 <fche> this may be possible from within the stap tapset function that realizes the user is asking for fpregs

22:43 <gromero> I see. I think it's feasible and won't harm the process state too

22:44 <gromero> stap probe will always be executed when leaving kernel space to userspace

22:44 <gromero> from ret_from_except_lite()

22:44 <gromero> calling do_notify_resume()

22:44 <gromero> that will check if uprobe is in use IIRC

22:45 <gromero> okay, I'll give it a try

22:45 <gromero> I'm going off for today

22:45 <gromero> thanks a lot for all the help!

22:45 <fche> np