fche changed the topic of #systemtap to: http://sourceware.org/systemtap; email systemtap@sourceware.org if answers here not timely, conversations may be logged
lijunlong has quit [Ping timeout: 240 seconds]
lijunlong has joined #systemtap
khaled has quit [Quit: Konversation terminated!]
hpt has joined #systemtap
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
lijunlong has quit [Ping timeout: 260 seconds]
lijunlong has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
khaled has joined #systemtap
derek0883 has quit [Remote host closed the connection]
_whitelogger has joined #systemtap
<ema> > Systemtap now supports extracting 64-bit floating point
<ema> \o/
hpt has quit [Ping timeout: 240 seconds]
mjw has joined #systemtap
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 264 seconds]
<fche> ema, suggestions welcome as to what other arithmetic etc. processing would be helpful
amerey has joined #systemtap
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #systemtap
derek0883 has joined #systemtap
derek0883 has quit [Remote host closed the connection]
derek0883 has joined #systemtap
<kerneltoast> but i need to make another change to my patch in order to prevent printing inside an NMI tracepoint from causing panics
<kerneltoast> if an NMI arrives on a CPU that's in the process of printing something, we'll have no choice but to drop all of the NMI's prints
<fche> results look okay, curious about that process_by_pid bit
* fche is not consistently here this week tho, so may delay comms
<kerneltoast> are ya gonna be around today?
<fche> yeah, probably 90 mins or so
<fche> 90 mins hence
<kerneltoast> ah just for the next 90 min?
<fche> gone for 90 mins
<fche> back later
derek0883 has quit [Ping timeout: 264 seconds]
<kerneltoast> o
<kerneltoast> cool, i'll add the NMI bits to my patch and give you a fresh testsuite diff
derek0883 has joined #systemtap
<kerneltoast> fche, this version has NMI protection: https://gist.github.com/kerneltoast/3d4878dcb68b0c959637fa0243b122bd
<kerneltoast> running the testsuite now
derek0883 has quit [Remote host closed the connection]
mjw has quit [Quit: Leaving]
orivej has quit [Ping timeout: 272 seconds]
derek0883 has joined #systemtap
derek0883 has quit [Ping timeout: 264 seconds]
derek0883 has joined #systemtap
<fche> dude
<fche> hm that was a little longer (teaching one of my boys to drive :-O)
<fche> would that nmi_lock be more simply named something like an anti-reentrancy lock?
<kerneltoast> it's literally just for NMIs though because IRQs are disabled
derek0883 has quit [Remote host closed the connection]
<kerneltoast> protip when learning how to drive: dedicate one foot to each pedal to react faster
<fche> good tip, will be sure to shred it
<kerneltoast> fche, if there were reentrancy other than from NMIs then the print driver would have exploded by now
<kerneltoast> just like your car will after your son drives it alone for the first time
<kerneltoast> source: my dad's '93 volvo died while i was driving it on the highway
derek0883 has joined #systemtap
<kerneltoast> i had been ignoring every light being lit up on the dashboard but the car still drove so my dad suggested i take it to the mechanic
<kerneltoast> turns out the car didn't last to the mechanic
<fche> errr, forget I mentioned it .... I need to sleep peacefully tonight :)
<kerneltoast> and i couldn't turn on hazards either because the battery died
<fche> wellthen you just do a flip & roll & crash, to make the hazard obvious
<fche> hth
<kerneltoast> of course, how could i have been so naive
<kerneltoast> so in summary, i think the name should be kept as is
<fche> hm
<fche> is the print_lock reentrant enough w.r.t. nmis?
<kerneltoast> yep read the comment above the read trylock
<fche> aha
<kerneltoast> the testsuite exposed this NMI issue
<fche> there are few sicknesses for which _trylock is not a cure
<fche> it's like a magic potion
<kerneltoast> cures everything
<kerneltoast> anti deadlock
<kerneltoast> can't cure mutex in IRQ though
<fche> ok, let us know how it likes the testsuite
<kerneltoast> I'll let ya know in an hour
<kerneltoast> It's still chug-a-lug lugging along
derek0883 has quit [Ping timeout: 265 seconds]
<fche> oh good
derek0883 has joined #systemtap
<kerneltoast> fche, new diff with the NMI stuff is more interesting: https://gist.github.com/kerneltoast/3d4878dcb68b0c959637fa0243b122bd#file-systemtap-sum-diff
<fche> a diff of the .log files would be good there
<kerneltoast> unless there's some reentrancy
<kerneltoast> maybe it's reentrancy
<kerneltoast> probably is i guess
<kerneltoast> yeah it is
<kerneltoast> and i know where it is
<kerneltoast> poo
<kerneltoast> in_nmi() isn't reliable
<fche> <bane> of course </bane>
<kerneltoast> well i dunno what to do about them NMIs
<kerneltoast> they just come outta nowhere and wreck stuff
<fche> do you have a dmesg or something so I can play along at home?
<kerneltoast> play along how
<kerneltoast> i have a log with a backtrace of one NMI wreaking havoc
<fche> yeah that
<kerneltoast> #0 [ffff88017a4c8ad8] _raw_read_lock at ffffffff81788414
<kerneltoast> #1 [ffff88017a4c8ae8] probe_6338 at ffffffffc067bfea [stap_b0645462cf706434a0a94992f03a9cf_17052]
<kerneltoast> #4 [ffff88017a4c8b58] __perf_event_overflow at ffffffff811a90e7
<kerneltoast> #2 [ffff88017a4c8b00] handle_perf_probe at ffffffffc067b4d0 [stap_b0645462cf706434a0a94992f03a9cf_17052]
<kerneltoast> #3 [ffff88017a4c8b48] enter_perf_probe_0 at ffffffffc067b703 [stap_b0645462cf706434a0a94992f03a9cf_17052]
<kerneltoast> #5 [ffff88017a4c8b90] perf_event_overflow at ffffffff811b28e4
<kerneltoast> #6 [ffff88017a4c8ba0] handle_pmi_common at ffffffff8100a9a0
<kerneltoast> #7 [ffff88017a4c8de0] intel_pmu_handle_irq at ffffffff8100ac7f
<kerneltoast> #8 [ffff88017a4c8e38] perf_event_nmi_handler at ffffffff81789031
<kerneltoast> #9 [ffff88017a4c8e58] nmi_handle at ffffffff8178a93c
<kerneltoast> #10 [ffff88017a4c8eb0] do_nmi at ffffffff8178ab5d
<kerneltoast> #11 [ffff88017a4c8ef0] end_repeat_nmi at ffffffff81789d9c
<fche> ok so what's breaking?
<kerneltoast> probes can be called from NMI context. if you have a print() inside such a probe, and the NMI arrives while there is already a print() in progress on the current CPU, bad things happen
<fche> by what mechanism
<kerneltoast> whaddya mean
<fche> as in ... what bad things happen, how?
<kerneltoast> the explanation for that is right above
<kerneltoast> in addition to that, there could be no panic and instead the print buffer would just get mangled
<fche> ok, I need to figure it out with small words, I'm on vacation officially so let's pretend I'm not very smart
<fche> (p.s. not pretending :-)
<fche> so
<fche> still trying to figure out the precise point at which something goes *wrong*
<kerneltoast> let's say i have this here code:
<kerneltoast> log->len += 10;
<kerneltoast> memcpy(&log->buf[log->len - 10], "1234567890", 10);
<kerneltoast> if (log->len + 10 > MAX_LOG_LEN) flush_da_buffer();
<kerneltoast> at the start of this code, log->len is MAX_LOG_LEN - 10, so flush_da_buffer() isn't called
<kerneltoast> now after that if-statement gets executed, an NMI comes flying out of the sky
<kerneltoast> log->len += 1337;
<kerneltoast> and the NMI decides to do this:
<fche> ok so my question there is why the nested nmi can't tell that the print subsytem is 'locked' already
<fche> so it can drop the output or whatever
<kerneltoast> because we lock it with local_irq_save()
<fche> well that get_context_() ditty used an explicit counter to detect reentrancy
<kerneltoast> there is some necessary print reentrancy that would be difficult to get rid of
<kerneltoast> we want to allow reentrancy on the local CPU
<kerneltoast> unless it's an NMI
<fche> "we want to allow reentrancy" .... are you quite sure?
<kerneltoast> yeah having a bunch of nested local_irq_saves should be allowed
<fche> yes that code allows it
<fche> but "want to allow" --- not necessarily.
<kerneltoast> NMIs just ruin our days
<fche> well why? I'd be perfectly happy with prevention of reentrancy, as that has been our architected model
<fche> we have those skipped* counters for this reason e.g.
<kerneltoast> it's just going to be a bit of work to get rid of the reentrancy i guess
<fche> detect & give-up is just fine
amerey has quit [Quit: Leaving]
<kerneltoast> testsuite disagrees
<kerneltoast> this breaks the stack printers
<kerneltoast> because they're implemented using bubblegum and duct tape
<kerneltoast> and parts from my dad's '93 volvo
<fche> well, think about it. I don't remember ever wanting or preferring reentrancy with respect to probes/etc.
<fche> I mean lots of things break that way, like the one-per-cpu context* structure we use
<kerneltoast> i'll need to make some _nolock functions
derek088_ has joined #systemtap
derek0883 has quit [Ping timeout: 264 seconds]
derek088_ has quit [Ping timeout: 264 seconds]
orivej has joined #systemtap
derek0883 has joined #systemtap
<kerneltoast> fche, i can just whitelist the stack functions from the reentrancy checks because they already have a context pinned
<kerneltoast> that might still be messy though...
<kerneltoast> blasted NMIs
<kerneltoast> i guess it'd work
<fche> not sure why the stack functions should be exceptional
derek0883 has quit [Remote host closed the connection]