01:23 <agentzh> fche: cool

01:26 <agentzh> thanks

02:03 <agentzh> fche: will you shed some light on this issue? https://sourceware.org/bugzilla/show_bug.cgi?id=26537

02:03 <agentzh> it happens from time to time, especially in our stress test.

02:23 <agentzh> it looks like a deadlock to me.

10:05 <fche> agentzh, well, wouldn't call that part a deadlock; the loop is intentional

10:06 <fche> but something among the writers of that stp_task_work_callbacks atomic is not getting cleared - i.e., one of the related callbacks seems stuck

10:06 <fche> a systemwide stack traceback may help explain

10:48 <zhuizhuhaomeng> hello @fche I found an error 'ERROR: probe process("/usr/local/openresty/nginx/sbin/nginx").begin registration error (rc -114)' which is -EALREADY. I lookup the code and found it maybe return by __stp_utrace_attach with __STP_TASK_FINDER_EVENTS UTRACE_RESUME args.

10:49 <zhuizhuhaomeng> is it a real error or we can ignore?

10:50 <fche> like the error# agentzh was talking about yesterday, I suspect this relates to very short-lived processes

10:57 <zhuizhuhaomeng> how can i debug this problem?

11:29 <fche> zhuizhuhaomeng, not sure, sorry. there are no debugging prints in stp_utrace.c; maybe adding some is a start

17:04 <agentzh> fche: okay, something like 'foreach bt' in the crash session?

17:05 <fche> think so

17:05 <fche> don't remember the kgdb rune but yeah

17:05 <fche> if you have access to /proc/$$/stack or whatever you'd used in the bz, then yeah that but for other threads in the system

17:06 <agentzh> okay, will try.

17:11 <agentzh> fche: found that we can reproduce kernel panic relatively consistently in __stp_tf_clone_worker() on a 32c/64t system using kernel 4.14.

17:12 <fche> any idea about the cause/mechanism ?

17:12 <agentzh> the crash bt is like this: https://gist.github.com/agentzh/934ff6090bed25b3c1383d2837fbed18

17:13 <agentzh> just added more info to this gist.

17:13 <agentzh> it seems like this callback is called with NULL 'work' argument.

17:14 <agentzh> the dmesg buffer contains this (using the crash cmd 'log'): https://gist.github.com/agentzh/7cd62681ee7c304afde80bfe16da193b

17:15 <fche> ok, don't have a theory as to why work would be ull

17:15 <agentzh> the crashing source line is /usr/local/openresty-stap/share/systemtap/runtime/linux/task_finder2.c:956

17:15 <agentzh> struct utrace_engine_ops *ops = \

17:15 <agentzh> (struct utrace_engine_ops *)tf_work->data;

17:15 <agentzh> this line

17:15 <fche> but could add some defensive coding into the clone_worker gadget to skip such things

17:16 <fche> is this git/master systemtap?

17:16 <agentzh> it's not master, but should be recent enough.

17:17 <agentzh> yeah, the strange thing that it seems impossible to feed a NULL work arg.

17:18 <fche> is the work arg itself null or some content of the struct is? (as though it was freed/memset(0)'d ?)

17:20 <agentzh> i checked the assembly, the work arg should be the rdi reg which is 0.

17:20 <agentzh> and tf_work->data is (char*) work - 0x8 after the gcc optimization.

17:20 <agentzh> which matches the offending addr fffffffffffffff8

17:21 <agentzh> i looked at the kernel func task_work_run() and it does not seem possible to call work->func(work) with a NULL work...

17:21 <agentzh> not sure if there's any other places which would call this callback func.

17:22 <agentzh> brb

17:23 <fche> try collecting thread start/stop timestamp data, maybe someone else is clearing the work queue behind our backs

18:36 <agentzh> hmm, but task_work_run() has proper task level locks so it shouldn't happen in the first place?

18:37 <agentzh> we also got panics when loading the 3.10 kernel of centos 7.

18:37 <agentzh> but the panic call stack is kinda different.

18:38 <agentzh> for centos 7's 3.10 kernel, the backtrace is like this: https://gist.github.com/agentzh/acf3fee9c03eeb390418d26a91341a1c

18:40 <agentzh> fche: it's the work arg pointer itself being NULL in the kdump for the 4.14 kernel.

18:41 <agentzh> the thread start/stop timestamps would be huge, since we are doing stress testing here and lots of processes in the system coming and going.

18:42 <agentzh> i've checked the current process running the fatal task_work job, which is a bash process which is still running (task->state == 0).

18:43 <agentzh> and its task->task_works pointer is not NULL: task_works = 0xffff8a948cd77a60, and i've checked it still has 3 callback_head entries in the list whose ->func all point to utrace_resume.

18:44 <agentzh> though those 3 callbacks do belong to other concurrent stap modules, not the current one.