sb0 changed the topic of #m-labs to: ARTIQ, Migen, MiSoC, Mixxeo & other M-Labs projects :: fka #milkymist :: Logs http://irclog.whitequark.org/m-labs
<jbqubit> I still see core device crashing almost every time I write to frequency1 or frequency2
balrog has quit [Ping timeout: 260 seconds]
q3k has quit [Ping timeout: 240 seconds]
<GitHub-m-labs> [artiq] sbourdeauducq reopened issue #794: Clocking, DAC support and JESD synchronization on one Sayma card https://github.com/m-labs/artiq/issues/794
<jbqubit> sb000: Leaving UMD Sayma on and attached to scope. Feel free to login.
<GitHub-m-labs> [artiq] whitequark commented on issue #1065: No, this is definitely memory corruption. Look at this line:... https://github.com/m-labs/artiq/issues/1065#issuecomment-396096430
<GitHub-m-labs> [artiq] whitequark commented on issue #1065: No, this is definitely memory corruption. Look at this line:... https://github.com/m-labs/artiq/issues/1065#issuecomment-396096430
sb0 has joined #m-labs
<sb0> jbqubit, did you rework the hmc7043 reset?
sb0 has quit [Quit: Leaving]
q3k has joined #m-labs
balrog has joined #m-labs
kaolpr has quit [Ping timeout: 264 seconds]
kaolpr has joined #m-labs
<GitHub-m-labs> [artiq] sbourdeauducq pushed 2 new commits to release-3: https://github.com/m-labs/artiq/compare/36204a97d8ff...e8092f6f1142
<GitHub-m-labs> artiq/release-3 e8092f6 Sebastien Bourdeauducq: conda: use h5py 2.8...
<GitHub-m-labs> artiq/release-3 408734b whitequark: artiq_rpctool: use inspect.formatargspec instead of a NIH formatter....
sb0 has joined #m-labs
<sb0> _florent_, have you given more thought to the "simplest thing that could possibly work
<sb0> + maximum robustness" serwb approach?
<bb-m-labs> build #1636 of artiq-board is complete: Success [build successful] Build details are at http://buildbot.m-labs.hk/builders/artiq-board/builds/1636
<bb-m-labs> build #2439 of artiq is complete: Failure [failed python_unittest_2] Build details are at http://buildbot.m-labs.hk/builders/artiq/builds/2439 blamelist: whitequark <whitequark@whitequark.org>, Sebastien Bourdeauducq <sb@m-labs.hk>
X-Scale has quit [Quit: HydraIRC -> http://www.hydrairc.com <-]
sb0 has quit [Ping timeout: 265 seconds]
<GitHub-m-labs> [artiq] sbourdeauducq commented on issue #1067: Can you try by using the startup kernel? Likely there is intermittent memory/gateware corruption (did you rework the HMC7043 reset?) and running the whole TCP/IP stack is more likely to crash than running simpler code. Also disconnect Ethernet during the test so that the CPU doesn't have to process network broadcasts. https://github.com/m-labs/artiq/issues/106
<GitHub-m-labs> [artiq] sbourdeauducq commented on issue #1067: It could be that it's longer kernels loaded through Ethernet that cause crashes, not frequency1 itself. https://github.com/m-labs/artiq/issues/1067#issuecomment-396114196
<GitHub-m-labs> [artiq] sbourdeauducq commented on issue #1067: Can you try by using the startup kernel? Likely there is intermittent memory/gateware corruption (did you rework the HMC7043 reset?) and running the whole TCP/IP stack is more likely to crash than when running simpler code. Also disconnect Ethernet during the test so that the CPU doesn't have to process network broadcasts. https://github.com/m-labs/artiq/issue
sb0 has joined #m-labs
sb0 has quit [Quit: Leaving]
<_florent_> sb0: i'll fix the reset/init of low speed serwb
sb000 has joined #m-labs
<sb000> but low speed serwb is still quite complex for what it does. and aren't there still unexplained bugs with it?
<_florent_> sb000: except the reset/initialization issue, i'm not aware of others bugs
sb000 has quit [Ping timeout: 260 seconds]
<GitHub-m-labs> [artiq] hartytp commented on issue #794: @jbqubit thanks for checking that more carefully. I don't think I can reproduce this on my board...... https://github.com/m-labs/artiq/issues/794#issuecomment-396136685
<GitHub-m-labs> [artiq] hartytp commented on issue #794: @enjoy-digital since we don't control the phase of `jref` do we need a multireg here?https://github.com/m-labs/jesd204b/blob/25fd79dbf9d5b9536482f11a4d716747479191ec/jesd204b/link.py#L279 https://github.com/m-labs/artiq/issues/794#issuecomment-396138669
<GitHub-m-labs> [artiq] hartytp commented on issue #794: @enjoy-digital since we don't control the phase of `jref` do we need a multireg here?https://github.com/m-labs/jesd204b/blob/25fd79dbf9d5b9536482f11a4d716747479191ec/jesd204b/link.py#L279... https://github.com/m-labs/artiq/issues/794#issuecomment-396138669
<GitHub-m-labs> [artiq] hartytp commented on issue #1065: @whitequark is there anything useful in the second crash I posted? https://github.com/m-labs/artiq/issues/1065#issuecomment-396139085
<GitHub-m-labs> [artiq] hartytp commented on issue #1065: @whitequark @sbourdeauducq How do we go about debugging this?... https://github.com/m-labs/artiq/issues/1065#issuecomment-396139192
FabM has joined #m-labs
<GitHub-m-labs> [artiq] sbourdeauducq commented on issue #1065: If it's memory corruption, gdb cannot help. https://github.com/m-labs/artiq/issues/1065#issuecomment-396157855
hartytp has joined #m-labs
<hartytp> _florent_: fyi, I have one other crazy idea about some of the Sayma issues I've seen
<hartytp> it occurs to me that a lot of the debugging I've done has been with the board upside down to probe the clock chips
<hartytp> that reduces airflow to the top
<hartytp> yesterday when I couldn't make serwb crash, the board was the right way up
<hartytp> openocd tells me that the FPGA temps are quite low, but still...
<hartytp> will try gathering statistics with the board in both orientations and see if there is a real effect
<_florent_> hartytp: that can be interesting yes, if we figure out that the issues are related to temperature, that's already a first step.
<hartytp> I finally got round to refactoring the 830 code and implementing some fixes
<hartytp> (setting icp correctly for our loop filter design, running the PFD at a higher frequency, etc)
<hartytp> testing now, then will submit a PR unless you have any objections?
<_florent_> hartytp: you now probably know the hmc830's internals better than any one else here, so no objections.
<hartytp> hopefully that will make it easier to play around with different clock freqs/interpolation etc
Shikadi has quit [Quit: Leaving]
kaolpr has quit [Ping timeout: 256 seconds]
<GitHub-m-labs> [artiq] whitequark commented on issue #1065: > @whitequark is there anything useful in the second crash I posted?... https://github.com/m-labs/artiq/issues/1065#issuecomment-396228462
X-Scale has joined #m-labs
<GitHub-m-labs> [artiq] jordens commented on issue #1065: Remnants from the memory test at boot? Maybe clear all sdram or set it to some known value after testing it. https://github.com/m-labs/artiq/issues/1065#issuecomment-396238738
<GitHub-m-labs> [artiq] jordens commented on issue #1065: But either way. Probably just noise. 0x440249cc is also pretty high. https://github.com/m-labs/artiq/issues/1065#issuecomment-396243089
<GitHub-m-labs> [artiq] hartytp commented on issue #1065: So....what do we do about this? https://github.com/m-labs/artiq/issues/1065#issuecomment-396244748
<hartytp> okay, I'm officially sick of the HMC830
<hartytp> changing the r divider to put the PFD at 100MHz (which should be okay according to the data sheet)
<hartytp> it is clearly locked according to a spectrum analyzer
<hartytp> but, lock detect register reads 0
<hartytp> aah, no, Greg's settings put the lock detect window too high
<GitHub-m-labs> [artiq] gkasprow commented on issue #1065: we can run more advanced memory test that scans all memory.... https://github.com/m-labs/artiq/issues/1065#issuecomment-396271398
<GitHub-m-labs> [artiq] hartytp commented on issue #1065: > I've noticed that on one of our boards the memory test does not pass anymore... https://github.com/m-labs/artiq/issues/1065#issuecomment-396275424
<GitHub-m-labs> [artiq] marmeladapk commented on issue #1065: @hartytp We'll check without RTM tommorow. We used the same build on several boards and only two showed such issues so it's more likely to be some kind of hardware failure. https://github.com/m-labs/artiq/issues/1065#issuecomment-396281132
<GitHub-m-labs> [artiq] hartytp commented on issue #1065: :+1: https://github.com/m-labs/artiq/issues/1065#issuecomment-396285022
<_florent_> hartytp: have you been able to reproduced serwb issues today?
<hartytp> nope
<hartytp> well, that's not quite true
<hartytp> I haven't seen the board getting stuck in an infinite init loop
<hartytp> but I have seen small numbers of errors reported during the init tests (say 8 or 6 or that kind of level)
<hartytp> a few random cases of the board crashing during init
<hartytp> that's about it IIRC
<hartytp> that's both board orientations
<hartytp> am planning to gather statistics over night
<hartytp> sec
<_florent_> ok
<GitHub-m-labs> [artiq] hartytp opened pull request #1068: Sayma HMC830: update interface and register writes. (master...hmc830_tidy) https://github.com/m-labs/artiq/pull/1068
<hartytp> _florent_ or sb0: could you have a look at that ^
<hartytp> give feedback and test on your board
<hartytp> seems to work reliably for me!
hartytp has quit [Quit: Page closed]
<GitHub-m-labs> [artiq] gkasprow commented on issue #1065: @marmeladapk we should load some old version of the gateware. Both boards used to work. It's hard to believe they got broken by themself. https://github.com/m-labs/artiq/issues/1065#issuecomment-396313164
<GitHub-m-labs> [artiq] jordens commented on pull request #1068 2621782: That part of the comment shouldn't be removed. https://github.com/m-labs/artiq/pull/1068#discussion_r194484235
<rjo> bb-m-labs: force build --props=package=artiq-board,artiq_target=kasli,artiq_variant=opticlock artiq-board
<bb-m-labs> build forced [ETA 41m46s]
<bb-m-labs> I'll give a shout when the build finishes
<bb-m-labs> build #1637 of artiq-board is complete: Failure [failed conda_build] Build details are at http://buildbot.m-labs.hk/builders/artiq-board/builds/1637
<GitHub-m-labs> [artiq] hartytp commented on pull request #1068 2621782: Extra reviews are always welcome. However, in this case I don't think it's necessary. I've checked this all through against the data sheet carefully and tested it on my hardware. I'm inclined to say that if it works reliably on other HW as well then further review isn't worth the time it would take (but if you have spare time then go for it!). htt
<GitHub-m-labs> [artiq] hartytp commented on pull request #1068 2621782: thanks! will fix.... https://github.com/m-labs/artiq/pull/1068#discussion_r194511930
<GitHub-m-labs> [artiq] hartytp commented on pull request #1068 2621782: will fix. https://github.com/m-labs/artiq/pull/1068#discussion_r194511985
<GitHub-m-labs> [artiq] hartytp commented on pull request #1068 2621782: Okay, I'm happy to revert that change.... https://github.com/m-labs/artiq/pull/1068#discussion_r194513789
<GitHub-m-labs> [artiq] hartytp commented on pull request #1068 2621782: Okay, I'm happy to revert that change.... https://github.com/m-labs/artiq/pull/1068#discussion_r194513789
<GitHub-m-labs> [artiq] hartytp commented on issue #1068: @marmeladapk @jbqubit @sbourdeauducq it would be great if one or more of you could check that this works on your boards before we merge this. Thanks! https://github.com/m-labs/artiq/pull/1068#issuecomment-396360368
<GitHub-m-labs> [artiq] hartytp commented on pull request #1068 ddb4752: To check, which bit of the comment do you want back?... https://github.com/m-labs/artiq/pull/1068#discussion_r194522641
<GitHub-m-labs> [artiq] jordens commented on pull request #1068 ddb4752: Sorry. I meant just the hmc7043 comment should not get lost. https://github.com/m-labs/artiq/pull/1068#discussion_r194528176
jbqubit has quit [Ping timeout: 260 seconds]
<GitHub-m-labs> [artiq] hartytp commented on pull request #1068 ddb4752: Okay, I'll reintroduce that if you'd prefer.... https://github.com/m-labs/artiq/pull/1068#discussion_r194529661
<GitHub-m-labs> [artiq] hartytp opened pull request #1069: Sayma: disable unused HMC7043 outputs (master...hmc7043_outputs) https://github.com/m-labs/artiq/pull/1069
hartytp has joined #m-labs
<hartytp> _florent_: finished a final review of the HMC7043/HMC830 code
<hartytp> currently running https://github.com/hartytp/artiq
<hartytp> master (10b87ac) which combines my two PRs
<hartytp> without SAWG
<_florent_> hartytp: interesting, i'll look at taht
<hartytp> but, of course, that bricks Sayma
<hartytp> no uart
<hartytp> sigh
<_florent_> hartytp: but these are firmware changes only no?
<hartytp> yes
<hartytp> we've seen this before
<hartytp> the UART death seems to be a build-build variation where the code differences between builds have no connection whatsoever with anything UART related
<hartytp> I'm wondering if it's a vivado bug, but can't test that atm because Sayma doesn't meet timing on 2017.x
<hartytp> well, I can fix that by changing ethernet clock constraints, but I haven't done that yet
<hartytp> we're definitely rooting out Sayma bugs, but it's bloody slow
<hartytp> need to confirm this, but I'm pretty sure that the build can be fixed by rebuilding only the fw
<hartytp> with a slightly different version of ARTIQ
<hartytp> i.e. no recompile gateware
<hartytp> anyway, will look at that in the morning
<GitHub-m-labs> [artiq] hartytp commented on issue #1064: Reproduced with https://github.com/hartytp/artiq/commits/master 10b87ac. Files here [when upload finishes].... https://github.com/m-labs/artiq/issues/1064#issuecomment-396396485
<hartytp> actually, I don't want to wake up to this...
<hartytp> will investigate a little
<hartytp> hmm rebuilding from master didn't fix it this time, so maybe something has happened to my hw. will have to check in the morning
hartytp has quit [Quit: Page closed]
<GitHub-m-labs> [artiq] hartytp commented on issue #1064: Resetting artiq to current master (0b86225) and rebuilding gateware and firmware fixed this. Rebuilding just the firmware (with `--no-comiple-gateware`) did not seem to (although I will double check those observations tomorrow.... https://github.com/m-labs/artiq/issues/1064#issuecomment-396405861
<GitHub181> [smoltcp] squidarth commented on issue #83: Alright, so today I spent some time digging into how Linux handles cache eviction:... https://github.com/m-labs/smoltcp/issues/83#issuecomment-396406078
<GitHub185> [smoltcp] squidarth commented on issue #83: Alright, so today I spent some time digging into how Linux handles cache eviction:... https://github.com/m-labs/smoltcp/issues/83#issuecomment-396406078
<GitHub21> [smoltcp] squidarth commented on issue #83: Alright, so today I spent some time digging into how Linux handles cache eviction:... https://github.com/m-labs/smoltcp/issues/83#issuecomment-396406078
<GitHub161> [smoltcp] squidarth commented on issue #83: Alright, so today I spent some time digging into how Linux handles cache eviction:... https://github.com/m-labs/smoltcp/issues/83#issuecomment-396406078
<GitHub18> [smoltcp] whitequark commented on issue #83: > also think that an approach that uses another thread to garbage collect in the background would add a lot of complexity... https://github.com/m-labs/smoltcp/issues/83#issuecomment-396407228
[X-Scale] has joined #m-labs
X-Scale has quit [Ping timeout: 240 seconds]
[X-Scale] is now known as X-Scale
Gurty has quit [Excess Flood]
Gurty has joined #m-labs
Gurty has joined #m-labs
Gurty has quit [Changing host]