<xiangfu>
kyak, (jow said he was going to talk to the committer ) great.
<wpwrak>
kristianpaul: just catching up with some really old stuff ...  216.239.32.2: i can ping it but not connect to port 80. neither from home nor via a server in the US.
<kristianpaul>
he, okay :)
<kristianpaul>
i need to remenber was was the porpuse of my question, but first finish breadfast ;)
<wpwrak>
kristianpaul: and the .2 doesn't work because it's .21. can reach .21 without problems from home :)
<wpwrak>
DocScrutinizer: CH1 shows a line that's normally pulled high with 10 kOhm and bypassed with 220 pF to ground. could be pulled low by an open collector, but we have reason to believe this isn't happening in this case.
<wpwrak>
DocScrutinizer: CH2 shows a quite unrelated line driven by a push-pull output. one thing they have in common is that they're on adjacent balls of the FPGA.
<wpwrak>
DocScrutinizer: when measuring the resistance between the two, we get about 10 kOhm in one direction, about 120 kOhm in the other. the 10 kOhm seems low and if i model this, i get roughly the right curve with similar values: http://downloads.qi-hardware.com/people/werner/m1/tmp/pin34.ps
<wpwrak>
DocScrutinizer: however, what kind of failure would really explain this ? (this is something we've only observed on two out of ~80 boards so far, and the other one may have a higher resistance between the two signals)
<DocScrutinizer>
sounds odd
<wpwrak>
DocScrutinizer: to say the least ;-)
<DocScrutinizer>
you considered clamping diodes when you did your measuring?
<wpwrak>
i thought of them but can't quite picture a credible failure scenario that involves them. they could explain any sneak current when measuring in-circuit and with power off, though.
<DocScrutinizer>
yes, sure they do, and it comes out as 10k on one meter, and as 307k on another, and yet another may tell it's 600R
<wpwrak>
;-)
<wpwrak>
but then, when the system is powered, they shouldn't get in the way
<DocScrutinizer>
never probe outputs (or inputs) for resistance against ground or against another pin without proper voltage on clamping diodes
<DocScrutinizer>
yes
<DocScrutinizer>
exactly
<wpwrak>
i wonder if a diode test would yield more useful results. kinda tricky, though. some meters go up to relatively high voltages and i'm not sure adam has a good one.
<wpwrak>
if the system is powered, we can't really measure much resistance anyway. so it's an attempt to make do with what little we have
<DocScrutinizer>
unless your probing voltage is of wrong polarity (ground clamp diode kicks in, 0V7) or of higher voltage than VDD (VDD clamp diode kicks in, result random)
<DocScrutinizer>
another way to probe such things is with really good meter that has probing voltage <0.3V
<DocScrutinizer>
no non-linear component should spoil your results then - usually
<DocScrutinizer>
yet another way to probe is comparative probing between known good and DUT
<DocScrutinizer>
needs some experience to read the results of all those probings
<DocScrutinizer>
basically on a device without VDD applied, each pin is connected to each other pin via 2 diodes in series, no matter which is + and which -
<DocScrutinizer>
at least initially, until your meter has charged the PSU buffer capacitors ;-P
<wpwrak>
yeah ;-)
<wpwrak>
on DUTs, we got about 60/120 kOhm, compared to the 10/120 kOhm on the suspicious device
<wpwrak>
s/DUTs/good DUTs/
<DocScrutinizer>
I think we called this reverse alien powering a circuit
<wpwrak>
;-))
<DocScrutinizer>
well, 60/120 vs 10/120 sounds like chip defect
<wpwrak>
interesting .. my fluke 8845A goes up to 10 V when measuring resistance (10 MOhm). the picotest m3500a goes up to 5 V. my handheld UT60G -230 mV
<DocScrutinizer>
or a short to some other pin you haven't considered
<DocScrutinizer>
yeah voltage on R test, a way underrated specifications detail of meters
<wpwrak>
there are no other paths that really seem to make sense. already this one (the adjacent but otherwise unrelated balls) was a lucky guess
<DocScrutinizer>
worst I've seen was some 40V, though on a electron valve heathkit device built for probing up to 100GR
<DocScrutinizer>
or 10GR, can't recall
<DocScrutinizer>
wpwrak: a soldering short rarely acts as diode
<wpwrak>
wow, 40 V not bad. 10 GOhm is even better, though :)
<DocScrutinizer>
so your reading had to be 10/10 rather than 10/120
<wpwrak>
yes, accidental creation of a PN barrier somehow seems improbable :)
<DocScrutinizer>
possible but unlikely, yes
<DocScrutinizer>
esp on TWO devices
<DocScrutinizer>
that's when I came up with silicon defect
<wpwrak>
of course, if we can do that just out of thin air, perhaps we should try our luck with cold fusion next ;-)
<wpwrak>
the boards had some light rework (addition of a 0402 component, replacement of another 0402 component, some wires) in the vicinity of those pads, including on traces leading to them
<DocScrutinizer>
OTOH on-chip defects that create shorts to a random PN structure nearby aren't uncommon
<DocScrutinizer>
often triggered by ESD
<wpwrak>
if a defect was caused by this rework, what are our chances to see anything useful on x-ray ? likewise, in case of ESD ?
<DocScrutinizer>
zero
<wpwrak>
sigh
<DocScrutinizer>
I'd guess
<DocScrutinizer>
I'd not bet a penny on it
<wpwrak>
on xray or on xray being useless ?
<DocScrutinizer>
you don't see silicon defects of that class in usual optical examination
<DocScrutinizer>
before sth shows up in xray even microscopy you need a severe burnout in the die
<wolfspraul>
the xray is an smt xray, not a foundry xray :-)
<wpwrak>
i was wondering is there could perhaps be a zone that looks abnormal, like lighter/darker than the rest or such
<DocScrutinizer>
a electromigration-alike defect is usually invisible unless you use electron microscopy I think
<wpwrak>
hmm
<wpwrak>
maybe we should just fire a few neutrinos at it and check how they get deflected :)
<DocScrutinizer>
let me put it this way: do your se the structures of a single transistor/diode on your xray? then you got chances to see the defect as well
<wpwrak>
anyone got the number of CERN ? it's about time they do something useful with their LHC :)
<DocScrutinizer>
hehe
<wpwrak>
(see transistors) i wouldn't count on that ;-) probably a few orders of magnitude too low-res
<DocScrutinizer>
yes, that's what I meant
<wpwrak>
are there other symptoms we could probe for that could reveal this sort of damage ? besides the "resistance"
<DocScrutinizer>
1st approach: unsolder the chip and then probe again, without adjacent circuitry
<wolfspraul>
ma-tek could do it, but no way we will go in that direction http://www.ma-tek.com/
<wpwrak>
phew. that a ~450 balls FPGA.
<wolfspraul>
just replace the chip, and/or write off the board
<DocScrutinizer>
:shrug:
<DocScrutinizer>
anyway rework next to those pins smells like ESD damage
<DocScrutinizer>
symptoms are matching
<DocScrutinizer>
I've seen similar symptoms before
<wpwrak>
wolfspraul: let's see if we get the same on the other board. what troubles me is that 0x3c didn't show anything suspicious on the resistance test.
<wpwrak>
wolfspraul: if he have this sort of failures, we need at least a way to detect them. even if it means the boards in question can't be salvaged.
<wpwrak>
s/he/we/
<wolfspraul>
I think Adam can smell them from 10m distance now.
<wolfspraul>
normally the decisions are made much faster and more radical, just replace/discard, all the way to happiness :-)
<wpwrak>
DocScrutinizer: yes, it all points in that direction. there are a few more things on those nets, but i can't really come up with a good theory how they would cause such effects. not without some very long failure chains. of course, if you assume triple- or quadruple failures even at different points, then anything is possible ;-)
<wolfspraul>
but of course it's good that we study a bit, a little knowledge never hurts. And Adam can learn a lot too I think.
<wolfspraul>
unless I feel there is something to be learnt for other boards, I feel pretty confident even I know how to 'fix' those boards :-) just throw away and replace with another one :-)
<wpwrak>
wolfspraul: naw, he can't smell these yet. he does occasionally see the boards go crazy, but even that comes and goes. maybe he catches all the afternoon boards but the morning boards are a bit colder and some escape. it's really weird.
<wpwrak>
wolfspraul: yes, if it's a defect in the chip, your fix is the only thing that's even remotely reliable ;-)
<wolfspraul>
I feel we've learnt 98% of what we can learn.
<wolfspraul>
I wouldn't want to take the chips to ma-tek, pay 20,000 USD, and finally have the final ultimate proof that it's an ESD damage because Adam touched the board in a certain way during some rework.
<wolfspraul>
that would be horrible waste
<DocScrutinizer>
I managed to "repair" such chips with a second OV treatment
<wpwrak>
wolfspraul: on 0x77, it seems we're pretty much exhausted our possibilities, yes
<DocScrutinizer>
;-D
<wpwrak>
wolfspraul: 0x3c still bothers me, though. maybe it's TP36 plus something else than pin 34.
<wolfspraul>
let's see how the fix2b goes on other boards now
<wolfspraul>
I thought we had mostly exhausted what we can learn on 32/3c/77 etc. so far I haven't been proven wrong.
<DocScrutinizer>
I figured what component structure on the die might have been damaged initialy and with wild handwaving came up with a burn out the bug procedure that fused away another component thus "fixing" the short
<wolfspraul>
of course if there would be something it would be very significant, so it's good to have a little paranoia
<DocScrutinizer>
amazingly it actually worked on one of the three chips
<DocScrutinizer>
those were pretty simple chips though, where you actually could get even structure photos
<DocScrutinizer>
in the very early days of integrated circuits
<wpwrak>
wolfspraul: what i'm looking for is a non-statistical test for a defect there. so far, TP36-pin34 marks 0x77 as a clear outlier. but 0x3c proves that it doesn't show all the outliers. but if we could have a reasonably small set of pins to probe, that would help to catch damaged boards before they fail in the field.
<DocScrutinizer>
on sth like SN74xx series components
<wpwrak>
wolfspraul: also, the other NOR problems we've observed could be linked to this
<wolfspraul>
I don't think any of the boards made it very far
<wolfspraul>
I will look at that again, but not so worried
<wolfspraul>
we can also make the test a little stronger and catch them that way :-)
<wolfspraul>
it seems to fail quite early
<DocScrutinizer>
are you using JTAG boundary scan?
<wpwrak>
DocScrutinizer: the day wolfgang comes across some money, he'll hire you as the magic ESD healer for the fab ;-)
<DocScrutinizer>
lol, no thanks
<wolfspraul>
what drove me crazy before fix2b was boards failing on render cycle 2, 5, 9 etc. that meant even if we would have increase the render cycles to 20, 50, etc. we would have ignored something major (and driven up the fail rate sky high in the process)
<wolfspraul>
this is all far far better now, all under control
<wpwrak>
wolfspraul: yes, failures that late would be very bad
<wolfspraul>
I think tomorrow Adam should really move 80% or more of his time to fix2b and getting 30 full units ready for sales
<DocScrutinizer>
are you using JTAG boundary scan?
<wolfspraul>
after that day the time pressure goes away too
<wpwrak>
DocScrutinizer: (boundary scan) i don't think there's any test of that sort
<DocScrutinizer>
running "normal" programs and deciding go/no-go isn'T a proper QA
<wpwrak>
(atben/atusb production test has sort of a boundary scan ;-)
<DocScrutinizer>
proper QA consists of putting DUT into pathological operation modes to see how it reacts. The results of these operation modes are not like works/doesn't-work but more like "this error pattern is expected, so the board is ok" vs "duh it refuses to fail, so it might be defect"
<wpwrak>
(not that it actually caught anything - all three defects where outside the scope of these tests. but the tests are there, in all their cryptic beauty :)
<wpwrak>
wolfspraul: let's get the 0x3c signals and measure resistance on a few more pins on it then. that shouldn't take long. then adam can finish the fix2b and testing, and then we'll also see if more boards to decide to join the cluster.
<DocScrutinizer>
e.g you reduce VDD to a level where you expect a certain error to occur. If you got flaky output loads the error pattern will be different
<wpwrak>
DocScrutinizer: hmm, that seems tricky. lots and lots of undocumented behaviour.
<DocScrutinizer>
indeed it's undocumented, but you always do your own documentation on a known-good device anyway
<DocScrutinizer>
lots of QA are mere comparative tests between DUT and reference device
<wolfspraul>
agreed, some 3c pins and that's it
<wpwrak>
DocScrutinizer: what i do is pull up/down all pins, then set one to the opposite level. see if all the rest still reads as set. etc.
<wolfspraul>
DocScrutinizer: yes :-) comparative testing, lots and fast
<DocScrutinizer>
yup, when you also precisely log the VDD current etc then that's a good way
<wolfspraul>
and then even a dumb guy like me can make a thumb up/down decision :-)
<wpwrak>
DocScrutinizer: i skipped current this time (atben/atusb). too hard to pull it off (would have had to make a test rig, etc.)
<DocScrutinizer>
good way to follow and do more alike that
<DocScrutinizer>
yes, without a test rig you can not do decent QA
<wolfspraul>
calling it a day, n8
<DocScrutinizer>
that's the whole purpose of all those test pads after all
<wpwrak>
DocScrutinizer: (current test) of course, with my lab it would have been easy. maybe a simple passive board, but that would have been all. alas, tuxbrain wouldn't have had the instruments to make use of this ...
<DocScrutinizer>
night wolfspraul
<wpwrak>
wolfspraul: untroubled dreams ! :)
<wpwrak>
hmm, i think i'll see if i can find some food. government sneaked in some public holiday today. the things that happen in election years under a government that sees blatant populism as its strongest card :-(
<wpwrak>
well, if all else fails, i can still live on crackers and cheese for a week or so ...
<wpwrak>
DocScrutinizer: thanks for your help ! so, no urgent xray session for adam :)
<wpwrak>
DocScrutinizer: have you been a bad boy ? ;-)
<DocScrutinizer>
nah
<urandom__>
why did he hack nokia? for fun?
<wpwrak>
urandom__: maybe to show solidarity with elop ? :)
<urandom__>
wpwrak well he doesn't write anything about solidarity in his message
<larsc>
probably he was just bored
<urandom__>
if he is bored he should do something productive and not hack other peoples websites just to show off how cool he is
<DocScrutinizer>
I appreciate he kicked develper.nokia.com webmins' butts, as that website had several issues with authentication etc before he came
<DocScrutinizer>
e.g it dropped a session-end cookie that made subsequent logins always fail
<DocScrutinizer>
doesn't show up if you use firefox with standard configuration for cookie management :-P
<DocScrutinizer>
for those users prefering a different browser and/or more sophisticated cookie management, it failed boldly
<DocScrutinizer>
I'd prefer he had found a way to snitch Nokia's root signing key used in OMAP3630 "TPM"
<DocScrutinizer>
but I guess iven if you find that file, it'd be secured with a passphrase that needs special treatment of Elop to get hold of it
<wpwrak>
DocScrutinizer: competent security ? don't count on it ;-)
<roh>
hm.. john wasnt seen here lately eh?
<roh>
.s
<roh>
meh
<DocScrutinizer>
hah
<DocScrutinizer>
just thought of you, while closing a 'mail.om.org: connection timed out' requester
<wpwrak>
roh: isn't it nice when people always think of you in their moments of distress ? you must feel like some superhero, better than superman ;-)
<DocScrutinizer>
well, I'm almost used to it. This warning requester poping up every 5 min, for the best part of last 2 weeks
<DocScrutinizer>
and each time I have to close it by clicking "OK", just to find it popping up 3min later again. Alas it's a modal requester which sucks big time
<DocScrutinizer>
ptobably I should stop polling of that account completely
<DocScrutinizer>
:-/
<roh>
meh
<roh>
tried re-setting the box again 2 times today but it doesnt look good. maybe we really need a human on the other end now to get it up again
<roh>
just entered a  verbose 'support-ticket'
<roh>
the will check the machine for visible damage now and if there is none run some tests
<DocScrutinizer>
roh: is it colocated?
<DocScrutinizer>
roh: ...or Hetzner hw?
<wpwrak>
rented, i think
<DocScrutinizer>
if it's rented then they shall damn replace it for a working iron there
<DocScrutinizer>
When I rent a car, I won't accept extended repair times in the garage. If we rented a server, that was meant to be a working one
<DocScrutinizer>
I thought they even guarantee a certain uptime percentage