<sb0>
so I tried to build the conda package locally with conda, not just from source, and OF COURSE this is what happens:
<sb0>
An unexpected error has occurred. Conda has prepared the above report.
<sb0>
If submitted, this report will be used by core maintainers to improve
<sb0>
future releases of conda.
<sb0>
so, conclusion: I cannot build on the buildbot because it takes the old misoc files for some crazy reason, and I cannot build locally because then the generated package crashes conda when installing
<whitequark>
that sounds like my experience with conda
<whitequark>
"it takes the old misoc files for some crazy reason" I think I've hit this at *least* three times
<whitequark>
always for some different incoherent reason
<whitequark>
sometimes upgrading would fix it, sometimes it would break it further
rohitksingh_work has quit [Ping timeout: 244 seconds]
<sb0>
rjo: you might have missed it because of this new avalanche of conda bugs, but on the 'new' branch where conda happens to work, test_urukul.UrukulTest.test_att is failing
<GitHub-m-labs>
misoc/master 714ea68 Sebastien Bourdeauducq: attempt to work around conda problems
<sb0>
how about this plan: llvmlite and artiq are still built with "conda build", the firmware ad gateware is built with nix and then some hack generates a conda package, and all other conda packages (migen, misoc, etc.) are deprecated
<GitHub-m-labs>
artiq/master f5befba Sebastien Bourdeauducq: conda: bump misoc (attempt to WA conda problem)
<sb0>
bb-m-labs: stop build artiq conda
<bb-m-labs>
build 2706 interrupted
<bb-m-labs>
build #2706 of artiq is complete: Exception [exception interrupted conda_remove] Build details are at http://buildbot.m-labs.hk/builders/artiq/builds/2706 blamelist: Sebastien Bourdeauducq <sb@m-labs.hk>
<sb0>
bb-m-labs: force build artiq
<bb-m-labs>
build forced [ETA 41m09s]
<bb-m-labs>
I'll give a shout when the build finishes
<mithro>
sb0 / whitequark: How does migen sim and Records interact? I'm getting "NotImplementedError: <Record data at 0x7faa365c0e48>" -- but I'm probably doing something stupid?
<whitequark>
are you forgetting .raw_bits() somewhere?
<whitequark>
you can't directly assign records to each other
<rjo>
sb0: i saw the test failures a while ago. but that clearly looked like it preferring the bitstreams from the new branch over master. sounded like user error to me.
<rjo>
bb-m-labs: force build --props=package=artiq-board,artiq_target=kasli,artiq_variant=opticlock artiq-board
<bb-m-labs>
build forced [ETA 17m51s]
<bb-m-labs>
I'll give a shout when the build finishes
<rjo>
bb-m-labs: force build --props=package=artiq-board,artiq_target=kasli,artiq_variant=ptb artiq-board
<bb-m-labs>
The build has been queued, I'll give a shout when it starts
<rjo>
sb0: please don't delete 4.0.dev packages that have large numbers of downloads as long as ther e is no 4.0 release. that breaks things and makes reproducing any of the kasli setups very hard.
<whitequark>
why can't we host our own packages anyway? isn't it as simple as putting tarballs in a directory and running conda index on it?
<whitequark>
1 TB hard drive is cheap and fits a lot of packages
<rjo>
someone needs to maintain it.
<whitequark>
someone has to maintain the anaconda stuff, too
<whitequark>
what with all the "delete all the old packages, except the ones that are actually used"
<sb0>
rjo: I deleted them because the "new" branch makes conda mad and causes it to install random versions...
<rjo>
sb0: i think they are just sorted higher than master.
<sb0>
rjo: turns out, the att failure also happens on master, but maybe not when the tests are run outside the buildbot
<sb0>
rjo: it possibly is triggered by the higher version. but the end result is, conda installs firmware from new and software from master, even though the firmware recipes specify an exact software version
<whitequark>
^ yep, this is the actual bug here
<whitequark>
the conda constraint solver is some NIH thing that is completely brokey
<whitequark>
they should just use aspcud
<sb0>
rjo: this caused a *different* ad9910 test to fail (in fact most tests would fail when that happens), the att test issue is something else
<sb0>
okay, the same package that crashes conda when installed from a local file is installable after a round-trip to anaconda.org
<rjo>
the choice of just incrementing build number for different versions is probably bad. iirc the conda documentation implies that.
rohitksingh_wor1 has quit [Ping timeout: 246 seconds]
rohitksingh_work has joined #m-labs
rohitksingh_wor1 has joined #m-labs
rohitksingh_work has quit [Ping timeout: 268 seconds]
rohitksingh_wor1 has quit [Read error: Connection reset by peer]
<sb0>
oh this is dumb, test_att actually fails because it attempts to run it on the kc705
<GitHub-m-labs>
artiq/master 22a223b Sebastien Bourdeauducq: examples/master: clean up remnants of early urukul tests
<sb0>
rjo: do you have anything in particular for the 4.0 release announcement?
Gurty has joined #m-labs
<sb0>
rjo: we can switch the infrastructure to nix, which never has any such problems IME, and the make more frequent releases with the most standard version strings so conda won't ever choke on them
<sb0>
llvmlite versioning still uses build numbers and will need changing
m4ssi has quit [Remote host closed the connection]
m4ssi has joined #m-labs
<rjo>
could you explain the details of the plan and how the different use cases are handled?
<rjo>
for the release announcement i'd also mention the new features and big changes (not just the breaking changes). maybe distill the reports and the git log.
<rjo>
and i would test build all bitstreams before tagging
<sb0>
rjo: so, all CI moves to nix; we only ever touch conda when spinning a release, and the conda interaction will be limited
<sb0>
i.e. we don't build bitstreams with conda anymore, this becomes nix (or source) only
<sb0>
for spinning a release it is built semi-manually with a script, not in buildbot which is more difficult to debug
<sb0>
the bitstreams and firmware are built outside of conda, then some other script puts them into a conda package either by producing the conda archive format directly, or generating a conda recipe on the fly and running conda-build
<sb0>
before releasing we run unit tests on windows (manually) - there aren't many windows failures, so we can take that out of CI
<sb0>
developers can use nix or source
<rjo>
sb0: the conda installation/packaging (which is the only one for windows and the dominant for linux) won't be tested.
<sb0>
the tests we are doing right now aren't good anyway since conda is non-deterministic
<sb0>
but we can do some manual testing when releasing
<rjo>
running tests manually and building manually leaves nothing in terms of continuous integration.
<sb0>
nothing? it leaves everything other than the packaging
<sb0>
also, nix should become the dominant package management solution on linux.
<sb0>
CI with conda is just hell, but with nix it integrates nicely
<sb0>
it just speaks volumes against conda that you think packaging is the main source of problems
<rjo>
how are users supposed to use other packages in conda? i see a several problems with what you are proposing.
<sb0>
they either use the artiq conda releases, or install those packages with nix instead
<sb0>
stuff like matplotlib, pandas, jupyter, etc. is already packaged for nix and it works
<rjo>
hundreds of other packages people are using are not.
<sb0>
like what?
<rjo>
like the opticlock packages.
<sb0>
anything else?
<sb0>
also, that's on linux, right?
<rjo>
pretty sure that every user has a bunch of packages that they want.
<sb0>
just port them to nix
<sb0>
(opticlock packages)
<rjo>
and maintain both packages? nope.
<sb0>
nix also has a much bigger community than conda
<sb0>
and a more competent one, it would seem
<sb0>
maintaining nix packages is actually pretty trouble free. unlike other solutions and conda in particular, it is perfectly deterministic
<rjo>
improving conda sounds easier than maintaining two build infrastructures, giving up CI on windows testing and windows builds, and forcing users into systems that they don't use.
<sb0>
definitely not. the conda code is messy, buggy, and poorly maintained. issues and PRs rot for years. new bugs pop up all the time.
<rjo>
i'm not nearly convinced and opposed to this currently.
<sb0>
whereas I have not found a single bug in nix while giving it levels of abuse that would make conda completely explode
<sb0>
what percentage of artiq users were already using conda?
<rjo>
from what i can see this would have a bad effect on practical use of artiq and would drive away users. i'd rather look at properly using conda and making changes there. you need to look not just at you developing artiq but also at the users installing, upgrading, testing and using it. also on windows, also on conda, also with other packages
<sb0>
generating dumb packages (e.g. with very standard version strings, and with simple dependencies) goes in the direction of "properly using conda"
<sb0>
keep the complicated stuff like bitstream building in a more competently designed infrastructure
<sb0>
and again, I'm not proposing removing conda, but having conda only for user releases (without firmware or bitstream build capabilities), which we can do more frequently
<sb0>
Daniel mentioned seaborn and cvxopt as "special" python packages people are using. I just tested that they work ouf-of-the-box with nix.
<sb0>
it's already packaged....
<sb0>
oh and btw, when there is a nix problem, you can go to the IRC channel and get an answer in minutes. when there is a conda problem, it's like years, if ever...
<sb0>
i suppose many people have a "bah, whatever" attitude towards conda, whereas nix is actually supposed to work
<whitequark>
nix is actually not very good
<whitequark>
conda is just far worse
<whitequark>
e.g. the nix language is very ad-hoc and probably should be redesigned from scratch
<sb0>
yeah, the language is not the friendliest part. but at least it works correctly.
<whitequark>
the difference is that nix is written primarily by people who are tired of this shit from "traditional" package managers, and conda is written by people who are happy if they don't need to build numpy on windows manually
rohitksingh has quit [Ping timeout: 250 seconds]
<cr1901_modern>
nix sounds like it might make sense for artiq, but I'm not a fan of it's hyperstatic approach to deps
<cr1901_modern>
for normal use*
cr1901_modern1 has joined #m-labs
tmbinc2 has joined #m-labs
cyrozap has quit [Ping timeout: 268 seconds]
tmbinc1 has quit [Ping timeout: 268 seconds]
cr1901_modern has quit [Ping timeout: 268 seconds]