<bb-m-labs>
build #787 of artiq-win64-test is complete: Warnings [warnings python_unittest] Build details are at http://buildbot.m-labs.hk/builders/artiq-win64-test/builds/787 blamelist: Robert J?rdens <rj@quartiq.de>, Thomas Harty <thomas.harty@physics.ox.ac.uk>
<sb0>
and whitequark is going to look into a few other things such as running at a lower freq
<rjo>
running the ddr3 at lower freq?
<sb0>
yes
<sb0>
there is still some margin within the spec, and also those chips generally seem to be OK with running at a lower freq
<whitequark>
rjo: more specifically, keeping placement in the fpga the same and changing frequency
<sb0>
not a great solution but may provide a data point and/or an interim solution
<whitequark>
to see if maybe it's a signal integrity issue inside the FPGA or something like that
<rjo>
hmmm. wouldn't we suspect that that (a) doesn't help identifying the root cause and (b) doesn't resolve the issue?
<whitequark>
well, i'm just guessing, i don't have a lot of experience with these things
<rjo>
sb0: what about going through datasheets, reviewing the code?
<sb0>
I also don't have much experience with this particular bug; on every other board where we have this kind of system it doesn't do that
<sb0>
also this is definitely on the ultrascale, not sdram side, there aren't many things that can go wrong when just reading the page buffer
<sb0>
the DRAM is basically a SRAM in this mode
<sb0>
well, unless the SDRAM IO pins are misconfigured or something like that... but Greg did electrical measurements with reads, correct?
<rjo>
sb0: let's set a benchmark first. on the designs (not sayma) that work, the transition regions are how wide? 5 taps peak-peak?
<sb0>
the sayma taps aren't the same as the others
<rjo>
sb0: there is the kcu1005.
<rjo>
*kcu105
<rjo>
ah. not
<rjo>
ddr4
<sb0>
it's unusable, Joe didn't want to pay for ddr4 (which in hinsight was a terrible decision) and we though (incorrectly) that ddr3 would be easier
<rjo>
yes. but since we are now using TIME mode, we have a good idea how wide they are.
<sb0>
also, the eyes are no longer noisy and the sdram no longer failing when sawg is disabled, correct?
<rjo>
sb0: could you form yourself a view of what greg tested? i know exactly as much as you do.
<rjo>
sb0: what's with the preamble/postamble thing? you mentioned it and iirc _florent_ tested it but didn't see a difference. are you happy to exclude that?
<sb0>
it's not affecting reads
<rjo>
ok
<rjo>
on the top of my list (limited by not knowing much about sdram) to test is (1) with tom's board whether the issue exists after the VTC/reset sequence changes i added (2) setting a benchmark on what "no problem" looks like on the eye scan.
<sb0>
rjo, when reading the page buffer there is really not much to know about sdram
<sb0>
you issue a read command, then after CAS latency you get 8 bits per pin of read data, that is all
<sb0>
and that read data is stored in a static buffer, since we're not issuing any other command to the SDRAM during the eye scan
<rjo>
those are "musts" from my perspective. on the other items i have much less experience regarding likelyhood and cost-benefit. (3) generate the xilinx mig and check io settings (4) add the xilinx mig on the 64 bit ddram with sawg on the 32 bit sdram.
<rjo>
if you say it's worthwhile to check lower frequencies and if whitequark can do that quickly then fine. but i'd like to know what the conclusions would be if it either works then or if it doesn't work. just from an electrical perspective, moving the sampling points away from the edges will obviously improve things but is it likely to give insight into the origin and is it likely to be enough to get the
<rjo>
improvement we expect and need?
<sb0>
this ultrascale also has 21 megabits of RAM, we can also look into that as an interim solution
<sb0>
and likely the same bug will manifest itself on the TTL IOSERDES
<rjo>
sb0: where would it be easiest to test? TTL IOSERDES? serwb? DDR3?
<sb0>
(and we cannot get more RAM with bitstream tricks, the next die ku060 is different)
<rjo>
sb0: what do *you* want to do? implement the BRAM interim? test TTL IOSERDES integrity?
<sb0>
yeah I can look into that
<sb0>
_florent_, can we easily run serwb without a SERDES? 1 bit per cycle
rohitksingh has joined #m-labs
<rjo>
sb0, _florent_: is there anything for me to try? if you don't have anything, then i'll try to determine a metric for how good our eyes are on kc705/kasli/sayma and look at the xilinx mig io constraints.
rohitksingh has quit [Quit: Leaving.]
rohitksingh has joined #m-labs
<sb0>
rjo, also get some hard data on how good it is with/without sawg
<_florent_>
sb0: i also wanted to see if we could get the dram working at lower speed. I'll maybe do a test later today.
mumptai has joined #m-labs
<_florent_>
rjo: we could also improve the algo to find the middle of the eye. By doing an eyescan, storing all results in memory and run an algo on these results instead of taking a decision at each step of the scan. It could be more reliable. (on the last results hartytp posted, we clearly saw that the algo we would have chosen manually)
<rjo>
_florent_: do you think that finding the middle of the eye is good enough?
<_florent_>
rjo: that's one of the test i want to do with hartytp's board: set fixed read delays values in the bootloader (chosen manually by looking at the scan) and see if things are working fine
<_florent_>
rjo: i can't answer the question for now but i think on every memtest that was reported to be failing, the algo was not choosing the best read delay values
<hartytp>
_florent_ that probably would work
<hartytp>
but, I still fear that the dead spots in the middle of the eye are a sign of something sinister going on
<_florent_>
hartytp: yes, we would still need to understand that, but would probably provides an interim solution
<hartytp>
true
<hartytp>
one test would be to run a very long eye scan with good statistics
<hartytp>
if that has large working regions then clearly a good enough algo can make it work
<rjo>
i don't think we need multiple interim solutions. if we have decided that we need one, we should pick the simplest. isn't that BRAM?
<rjo>
assuming a huge BRAM actually works. sb0: is that known to work?
<rjo>
currently, we are talking about at least three potential interim solutions: lower freq, better algo, bram.
<sb0>
not known to work, I'm not sure how much memory we actually need, we may need to optimize the runtime for size a bit, and also a huge bram may cause timing issues
<rjo>
sb0: what's the best interim solution then?
rohitksingh has quit [Ping timeout: 240 seconds]
<sb0>
there is no best but bram may be the least bad
<sb0>
or xilinx mig phy, if it works
<whitequark>
sb0: I don't think the runtime is going to fit into 2.6 MB
<whitequark>
also if you use all the BRAM on a chip that sounds like a timing closure problem waiting to happen
<whitequark>
especially given how congested amc-standalone already is
<whitequark>
hm, maybe it will fit
<whitequark>
the runtime itself is ~600k
<whitequark>
but the socket buffers will need to be shrunk, the heap will be very small, limiting the size of DMA traces, and then there's the worst part
<sb0>
yeah DMA and analyzer will be disabled
<whitequark>
with RAM this small, if you allocate too much in kernels, they smash down into the runtime
<whitequark>
so that will need to be addressed properly
<sb0>
as I said there is no good solution
<hartytp>
how much work is it to use the xilinx mig?
<hartytp>
worry is that if this is some obscure timing bug then none of these solutions will help, since something else will start crashing randomly
<rjo>
hartytp: yeah. i wouldn't put that high on the list either. i'd think adding the xilinx mig to the heavy wheight artiq design and having it run its test pattern would be a good ddx.
<rjo>
i.e. taking greg's design, fusing it with artiq (either at the verilog level or earlier) and maybe run artiq as a parasitic load.
<hartytp>
yes
<rjo>
_florent_: i checked the io constraints w.r.t xilinx and they look good to me.
rohitksingh has joined #m-labs
<_florent_>
rjo: ok thanks
<rjo>
sb0: the RTM FPGA on sayma-1 reports 105 C.
rohitksingh has quit [Quit: Leaving.]
<rjo>
i increased the eye scan repetitions to 1024 to increase sensitivity. if i am not mistaken that's 8192 tests for each delay tap. and given the minimum "IDELAY chain resolution" of 2.5ps that's >110 taps of continuous "1" and >270ps eye width minimum (the idelay taps seem to range from 2.5ps to 15ps). that's a good benchmark. i don't expect more.
hartytp has quit [Quit: Page closed]
<_florent_>
rjo: do you have a better eyescan with that?
<rjo>
_florent_: that's just for me to evaluate signal quality.
<_florent_>
rjo: ok
<rjo>
_florent_: on boards and in situations where the signal is like that (with 110k reps resulting in 110 valid taps) i would not bother looking for a problem in hardware or gateware with that tool. if there is still a problem i'd expect it to be related to the difference of operating conditions between eye scan and actual operation (either through V, T or through gateware/firmware issues with memory operations).
<rjo>
_florent_: i'd be interested in what tom's board spits out right now.
<whitequark>
rjo: fwiw I was the last to touch sayma-1
<rjo>
_florent_: just to check my understanding of write levelling and the frequencies we are using: if there are ~325 taps from the beginning of one "zone of 1s" to the next during write levveling, that means the average tap delay is 1ns/325=3ps?
<whitequark>
I put allaki on channel 1
<rjo>
whitequark: ack.
<whitequark>
not sure why it's so hot
<whitequark>
RTM is at 85deg, AMC is at 105deg
<rjo>
whitequark: and you are using the board locks as well for that?
<whitequark>
er, actually, not right now, I should
<rjo>
whitequark: right. amc is 105.
<rjo>
whitequark: i have the lock. are you working on it right now? do you want to?
<whitequark>
I can't
<whitequark>
had to return from the lab before the subway stopped working
<whitequark>
so go ahead
<rjo>
ok. then i'll keep it.
<rjo>
you guys should cool it before 10k go up in smoke.
<whitequark>
is the FPGA itself that expensive?
<rjo>
i think the fpga is around 1k-2k but the rework and lost time would hurt much more.
<whitequark>
ouch, 2k.
<whitequark>
that's a lot of money for a small piece of silicon.
<_florent_>
rjo: yes between 2 zones of 1s you have 1 sdram clock cycle
<rjo>
yes.
<_florent_>
rjo: so 2ns
<_florent_>
rjo: also the DQS initial delay that is indicated is 500ps
<rjo>
_florent_: ok. so 2ns/325=6.2ps average tap delay, matches the 500ps/82 initial delay.
<_florent_>
yes, last time i looked at that it matched
<rjo>
_florent_: then <1e-5 error over 110 taps = 0.680ns is a good window. that's the benchmark then.
<rjo>
_florent_: ah. you had calculated that before. sorry for missing it.
<rjo>
_florent_: can i assume that there will only be one zone of correct reads during read leveling (b/c of the PRBS)? or can there be two?
<_florent_>
rjo: yes we should have only one zone
<rjo>
_florent_: then from a mathematical perspective the following algorithm will be good: find the first and last tap that give many correct reads. take the mean.
<rjo>
_florent_: before it was (wrongly) biasing to shorter taps (taking the first valid and the first failing one that's at least a bunch of taps after that).
<_florent_>
rjo: yes we could try that
<rjo>
_florent_: it's a bit more than just trial and error ;) this makes the algorithm robust and unbiased (if the predicates hold). i can't think of more requirements that would be needed.
<rjo>
whitequark: the gateware you flashed onto sayma-1, does it have sawg?
<whitequark>
no
<whitequark>
I didn't want to figure out how to get sawg to generate something useful
<whitequark>
triangle seemed fine
<rjo>
whitequark: ack.
<whitequark>
rjo: is it normal that even without sawg it builds for something like two hours?
<whitequark>
on the lab.m-labs.hk machine
<rjo>
whitequark: if someone else is building something at the same time then yes.
<whitequark>
oh
<rjo>
whitequark: on murray without sawg isn't that bad IIRC
<whitequark>
btw do you happen to know the sweet spot for vivado?
<whitequark>
does it want lots of RAM, or lots of cores, or a few really fast cores?
<rjo>
whitequark: and you had a flterm spinning (i took the liberty to kill it).
<whitequark>
because if i can get fast ultrascale compiles by buying a pair of xeons and pouring LN2 on them i'm going to do that
<rjo>
whitequark: i think it's a single fast core and cache. on murray it's usually using a single core for the most time (p&r).
<whitequark>
so if that core runs at, say, 5 GHz instead of 2.5...
<rjo>
whitequark: IME tweaking and improving the code is a much better approach than brute forcing it.
<whitequark>
but even compiling an empty project is excruciatingly slow
<whitequark>
code doesn't get better than that!
<rjo>
yes. but even if you were to get it to 5 GHz it would still only be half the time.
<whitequark>
that means i can do twice as much work if i'm bottlenecked by vivado
<whitequark>
that sounds worthwhile to me.
<whitequark>
how many days did sb0 spend rebuilding the ethernet bitstream?
<rjo>
whitequark: exactly. that's partly irrelevant because in the end looking at the datasheet and the code was at least as important.
<rjo>
whitequark: i guess we agree that we shouldn't let ourselves be bottlenecked by vivado. and there are different ways to get there.
<whitequark>
bruteforcing it is not mutually exclusive with optimizing the code, too
<rjo>
if both need work and given fixed time then it usually is.
<whitequark>
but getting fast hardware is mostly not time but money
<rjo>
setup and maintenance cost. and you were talking about ouring ln2 on them.
<GitHub-m-labs>
artiq/master 4b3f408 Robert Jordens: sdram: simplify read level scan
<GitHub-m-labs>
artiq/master 845784c Robert Jordens: kusddrphy: use first and last tap that yield many valid reads
<GitHub-m-labs>
artiq/master ed2e0c8 Robert Jordens: sayma/sdram/scan: test each tap 1024 times
<rjo>
_florent_: could you try it on tom's board with that ^^
<rjo>
_florent_: https://hastebin.com/labikejiki.pas is what i am getting with sawg, on sayma-1. if it is anything as good as that on tom's board we have run out of boards to reproduce the problem on.
<GitHub-m-labs>
artiq/master 65379b1 Robert Jordens: conda: bump migen, misoc...
<rjo>
jbqubit: could you also try that? yours is afaik the only other board that had/has the issue. and make absolutely certain that you are using the current migen/misoc.
<GitHub-m-labs>
[artiq] jordens commented on issue #908: Could everyone who currently has access to a board with this issue try the current artiq/misoc/migen with sawg (and otherwise presumed and/or known worst case conditions) and report the console messages? @jbqubit @enjoy-digital @marmeladapk https://github.com/m-labs/artiq/issues/908#issuecomment-374329773
balrog has quit [Ping timeout: 240 seconds]
<rjo>
on sayma-3 the taps are 8ps while on sayma-1 they are 6.2ps. the read window is about 700ps on both. sayma-1 with RTM, sayma-3 without, same gateware. both deliver robust and good read leveling.
<trxw>
Hello I am having trouble making phaser environment flash kc705:
<trxw>
$ artiq_flash -t kc705 -m phaser Design name: b'top;UserID=0XFFFFFFFF;COMPRESS=TRUE;Version=2017.4\x00' Partname b'7k325tffg900\x00' Date b'2018/01/09\x00' Time b'17:32:42\x00' found binary data length: 5950452 Open On-Chip Debugger 0.10.0-00013-gbb7beda (2018-02-13-15:56) Licensed under GNU GPL v2 For bug reports, read http://openocd.org/doc/doxygen/bugs.html none separate Info : auto-selecting first available session transport "jtag".
<trxw>
Any ideas?
kragniz552 has joined #m-labs
kragniz552 has quit [Client Quit]
<trxw>
Also this is the output of ftdi:
<trxw>
dmesg | grep ftdi [40505.053168] usbcore: registered new interface driver ftdi_sio [40505.053274] ftdi_sio 1-8:1.0: FTDI USB Serial Device converter detected [40505.053458] ftdi_sio 1-8:1.1: FTDI USB Serial Device converter detected [48580.650551] ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0 [48580.650563] ftdi_sio 1-8:1.0: device disconnected [48580.650648] ftdi_sio ttyUSB1: FTDI USB Serial Device conv
<rjo>
whitequark: what's the difference between 'for &offset in [n, n + DQS_SIGNAL_COUNT].iter()' and 'for offset in n..n + DQS_SIGNAL_COUNT'?
<trxw>
The LEDs on the board do flash however while I get the error message
<rjo>
trxw: does artiq_flash hang after that line?
<rjo>
trxw: you haven't shown any error message.
<trxw>
The artiq_flash ends with the msg
<rjo>
trxw: but please don't paste more than ~4 lines of terminal output into irc.
<trxw>
Also this:
<trxw>
$ artiq_flash -t kc705 -m phaser -f flash_storage.img proxy storage start Open On-Chip Debugger 0.10.0-00013-gbb7beda (2018-02-13-15:56) Licensed under GNU GPL v2 For bug reports, read http://openocd.org/doc/doxygen/bugs.html none separate Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'. adapter speed: 25000 kHz Info : ftdi: if you experience problems at higher adapter clocks
<rjo>
trxw: use one of the many pastebin sites.
<trxw>
ok, sure
<rjo>
_florent_: on kc705, the read leveling does wrap around.