03:00 <sb0> rjo, do you think it would be acceptable, EMI-wise, to send one of the Artix-7 transceiver clocks over the RTM connector?

03:01 <sb0> an alternative is to use this picxo thing to build a si5324 equivalent inside the fpga, but this is more complex

03:03 <sb0> (since we need two loop timings, and we have only one 5324 on rtm)

10:13 <rjo> sb0: yes. i think that's fine.

10:20 <rjo> sb0: EMI-wise, afaict, digital signals are only problematic if they non-scrambled and are in close proximity/same die/package as the sensitive ones (clocks or actual analog signals). i.e. clocks on the rtm connector are ok. they give us fixed frequency spurs but that's it. no data dependent noise.

10:32 <sb0> I like the elegance of the PICXO DPLL, but (1) not interfacing it to Xilinx hardblocks (2) Xilinx seems to be particularly protective of their reference design (3) Joe will complain about paying for it

10:34 <sb0> pragmatically, we can operate a synchronous IOSERDES on the LVDS lines instead

10:39 <rjo> picxo?

10:40 <rjo> that xapp?

10:41 <rjo> let me read that.

11:11 <rjo> sb0: why do we need two loop timings? for the two links? can't they be the same?

11:12 <sb0> RTIO is asynchronous to the wishbone bus

11:12 <sb0> and we can't tunnel Wishbone in DRTIO because we need wishbone to initialize the clocks required for DRTIO

11:13 <rjo> sb0: ah ye.

11:13 <rjo> ah yes.

11:13 <sb0> for only one loop timing, we can reconfigure the link at a new data rate after the clock chips are initialized

11:13 <sb0> but this sounds fragile and cause difficult debugging situations

11:15 <sb0> we also lose control completely if the clock chips are misconfigured

11:15 <rjo> can't we just bootstrap that? i.e. have the AUX FPGA configure the 5324 using the sequencer and then wait for link/drtio establishment and then transfer?

11:16 <rjo> i guess that's what you mean.

11:16 <rjo> don't we have to do that same mechanism anyway for the AMC FPGA?

11:17 <sb0> you always need to do that to implement loop timing.

11:17 <sb0> and the two clocks you are switching need to be at almost the same frequency

11:17 <sb0> this is a big assumption for wishbone/rtio clocks.

11:18 <rjo> sure.

11:18 <sb0> also, I don't like the sequencer that much for configuring the 5324 - not enough visibility/debuggability/control

11:19 <sb0> better use a softcore cpu

11:20 <sb0> and the automatic clock switching built into the 5324 doesn't work well as I mentioned, better have the CPU switch it manually when appropriate

11:20 <rjo> i agree. did you change the drtio master/satellite so to configure in runtime?

11:20 <sb0> yes

11:20 <sb0> this is how I found that long-standing drtio bug

11:21 <rjo> then, since wishbone-over-GT* is jittery and large latency anyway, why don't we AsyncFIFO it to the RTIO clock domain?

11:22 <sb0> the reason we need loop timing for wishbone is because the IOSERDES on the AMC side does not have a CDR

11:23 <sb0> this is simply a matter of clocking the RTM -> AMC transceiver synchronously to the AMC -> RTM and with a fixed phase offset

11:25 <rjo> the PICXO sounds good and very adequate for DRTIO in general.

11:25 <sb0> if there was a CDR on the AMC side, we could do even simpler than what you propose - clock the two ways asynchronously and add CDCs

11:26 <sb0> PICXO may not be so adequate for DRTIO because it seems you cannot extract the corrected clock and send it into the fabric to implement TTLs etc.

11:26 <sb0> but I may be wrong

11:26 <sb0> the only clock you have is the jittery CDR output

11:27 <sb0> AFAICT PICXO only does *transceiver* loop timing

11:27 <rjo> you get TXOUTCLK.

11:28 <sb0> right...

11:28 <rjo> which is not the serial clock. right.

11:28 <rjo> so you'd have to slip any SERDES high speed clock to align.

11:29 <sb0> it is the serial clock when you bypass the elastic buffers but I think you cannot do that when using PICXO (as usual, Xilinx know how to fuck up their silicon)

11:29 <sb0> there *may* be a trick to extract the corrected clock, but one has to dig into Xilinx docs and pray that there isn't some obscure silicon bug lurking somewhere

11:31 <larsc> the PICXO is using the same resources like the CDR, since you can get the CDR clock, you should be able to get that clock as well

11:31 <sb0> well, there's a trick, route a transceiver's TX to another transceiver clock input on the PCB, and transmit K28.7 :-)

11:32 <sb0> larsc, aren't the CDR resources already occupied by the receiving section of the same transceiver?

11:33 <larsc> looks like there is one for TX and one for RX

11:36 <larsc> https://www.xilinx.com/support/documentation/user_guides/ug476_7Series_Transceivers.pdf#page=150

11:38 <larsc> TXOUTCLKPMA

11:38 <larsc> that's the corrected clock

11:38 <rjo> oh. and even with only the slow TXOUTCLK it wouldn't be a problem because you can use a regular PLL to get a well-phased faster clock.

11:39 <sb0> the PICXO appnote says "Transmit buffer bypass is not supported"

11:39 <rjo> so in short I'd acctually like to see PICXO.

11:39 <sb0> larsc, ah yes! good find

11:41 <sb0> if we have TXOUTCLKPMA we can perhaps hack that idiotic elastic buffer into fixed latency by having TXUSERCLK=TXOUTCLKPMA+some empirically determined phase offset. assuming there are no reset issues.

11:41 <larsc> I think you can bypass the buffer

11:41 <sb0> not when using PICXO, according to xapp589

11:42 <sb0> without yes, and the latency is fixed, I've been doing that already

11:43 <larsc> if you run your fabric with the same clock as the SERDES you should be able to bypass it

11:43 <sb0> yes sure

11:43 <sb0> see http://www.xilinx.com/support/documentation/application_notes/xapp589-VCXO.pdf p. 27

11:44 <sb0> Mandatory Conditions and Limitations - Transmit buffer bypass is not supported.

11:44 <sb0> now this doesn't say why, it could be simply that the soft design doesn't support it

11:49 <larsc> it could be that the phase offset of the TXOUTCLKPMA is too large, who knows

11:49 <sb0> what do you mean?

11:53 <larsc> I think normally when running in buffer bypass mode the PMA clock is aligned to the USRCLK using the PI, but since in this case the PMA clock is the user clock that wouldn't work

11:54 <larsc> so I guess bypass does not work

11:54 <larsc> I take back what I said before

11:54 <sb0> ooh i see+

11:59 <sb0> but when they use that elastic buffer they manage to clock one end synchronously to the PMA still (without PI), so why didn't they just put a register there and give the other end to the fabric

11:59 <sb0> sigh

12:01 <larsc> a register wouldn't work, would it? I mean your clocks still have a phase offset so setup and hold wouldn't be met

12:01 <larsc> you have to route the TXOUTCLKPMA to a BUFG first and then back to the USRCLK input

12:02 <sb0> a register gives a wide timing margin (and a small setup+hold window), then the phase requirement becomes rather relaxed

12:03 <sb0> that USRCLK input is useless, it should be a regular TXOUT -> BUFG -> fabric FF -> transceiver input FF

12:03 <sb0> I don't see why USRCLK is needed at all and why it would be hard at all to meet timing on the path above

12:09 <sb0> anyway, all FPGA transceivers are pieces of shit and we don't have much of a choice here

12:11 <sb0> I think the elastic buffer latency jumps should happen every 10 line UI? maybe having TXUSRCLK offset from TXOUTCLKPMA is a working hack to fix latency ...

12:18 <sb0> actually I think some White Rabbit folks inconsciously do that, and it works for them