sb0 changed the topic of #m-labs to: ARTIQ, Migen, MiSoC, Mixxeo & other M-Labs projects :: fka #milkymist :: Logs http://irclog.whitequark.org/m-labs
_rht has joined #m-labs
rohitksingh_work has joined #m-labs
cr1901_modern has quit [Ping timeout: 240 seconds]
cr1901_modern has joined #m-labs
rohitksingh_work has quit [Quit: Leaving.]
<rjo>
whitequark: a guy from CERN mentioned the swift type inferencer yesterday and wondered whether we (you) had considered it. he said that is was two passes and scaled nicely.
<whitequark>
rjo: it's extremely bad
<whitequark>
you've complained that arrays of 100_000 elements (or what did you have in that issue) take a long time to typecheck
<whitequark>
swift's type inference is *exponential* on real-world, not obviously wrong code, and an array of 50 elements can take a minute
<whitequark>
swift crashes on let a: Double = -(1 + 2) + -(3 + 4) + 5
<whitequark>
it crashes on that expression for precisely the same (root) reason that ARTIQ Python's error messages are sometimes unintuitive around numeric operations, integer literals as well as arithmetics is generic in Swift
<rjo>
whitequark: aha. interesting! i'll let him know if i hear of him again.
<whitequark>
however instead of handling this in an ad-hoc but compatible with good old HM way they decided to implement it via typeclasses
<whitequark>
if I'm not mistaken this basically turns type inference into a SAT solving problem
<whitequark>
no, scratch that, it's an SMT problem in presence of subclassing
<rjo>
whitequark: by the way, they were really pleased, impressed, and interested with the work on and around the compiler. especially and specifically pythonparser and documentation.
<whitequark>
excellent!
<whitequark>
means the not-insignificant time I put into that early on has paid off.
<rjo>
one idea that i had: there is cppyy which dynamically binds C++ libraries for use as python extension modules (using LLVM Cling+reflection). wouldn't that be nice to replace llvmlite?
<whitequark>
that sounds horrifying
<rjo>
whitequark: i think it already paid of with us and grumpy using it.
<whitequark>
you're using a bad idea (introspecting C++ code to generate bindings at runtime) with a bad idea (cling) to do a bad idea (using LLVM's C++ API)
<rjo>
whitequark: well that other llvm wrapper you mentioned also generates stuff dynamically.
<whitequark>
these are bad because: 1) generating bindings at runtime via introspection is a deployment nightmare, you have to get a complicated set of libraries to align just right
<rjo>
whitequark: but if the C++ API is bad and the C API is bad, there is no good API to LLVM left.
<whitequark>
it *could* work at compile time (so package build time, with conda), it would be tricky to get but it could be done, ruby-llvm bindings are done that way
<whitequark>
2) cling. cling is a really weird set of hacks on top of C++ that doesn't actually implement C++ but some sort of language whose featureset intersects with it. it's also really large, and we are already deploying many large packages
<rjo>
whitequark: the pyROOT guys seem really happy with it. you have to weigh having to maintain/write a wrapper yourself against maintaining the dynamic binder.
<whitequark>
well a more concrete argument against cling is that we'll have to tie it to our LLVM fork
<whitequark>
so it's even more work around LLVM upgrades, packaging, etc
<whitequark>
(cling is the least bad idea of all these three)
<rjo>
doesn't that also apply to the other (all other) llvm bindings?
<whitequark>
so that's the third aspect of it
<sb0>
whitequark, did you hear back about the license?
<sb0>
for the llvm bindings
<whitequark>
LLVM's C API is its public interface. it is not fully stable (that would actually be undesirable) but it is stable enough to build on top of it painlessly
<whitequark>
LLVM's C++ API routinely has massive refactoring landed onto it just for the sake of makign something a little bit cleaner
<whitequark>
our use case is ideal for consumption of the C API
<rjo>
whitequark: i remember the opposite statement: "don't use llvm-c as it is not stable, use IR or llvm-c++"
<whitequark>
from whom?
<rjo>
the llvmlite creators IIRC.
<whitequark>
I could certainly have never said any such thing because it is demonstrably and obviously false
<whitequark>
well then I doubt their competence
<whitequark>
and/or reading comprehension. this is the first thing you read in any guide for integrating LLVM into your compiler, including the ones on llvm.org.
<whitequark>
the IR is not stable either btw
<whitequark>
much less stable than the C API at that. it's been broken in every minor release since 3.7 in a significant way
<sb0>
my experience is also that llvm-c is quite straightforward to use whereas llvm-c++ is noticeably messier, and any docs you may get are out of date
<rjo>
anyway. i am not pushing something here (cppyy), just want to make shure you are aware of it and get a chance to form an opinion. i also get uncomfortable hearing the words "C++ interpreter".
<whitequark>
the *bitcode* is *backwards compatible*, but the bitcode format is so atrociously hard to generate (and undocumented) that you only really can do that if the C API is unusable for you (e.g. .NET cannot call C code quickly, so ki9a is generating bitcode instead)
<whitequark>
yes. I am aware of the possibility. I don't have to use cling anyway, there are many C++ binding generators for Python
<whitequark>
it just doesn't make any sense
<whitequark>
sb0: correct.
<whitequark>
the C++ API doesn't really have docs for the most part
<whitequark>
rjo: cling is not a C++ interpreter despite its name
<rjo>
whitequark: are there other dynamic C++ binding gens than pyROOT, cppyy?
<whitequark>
it's a JIT compiler with a slightly different grammar and instrumentation that lets it catch certain errors eagerly
<whitequark>
rjo: why dynamic specifically? that's a disadvantage in my book, for our use case. more complexity and far slower
<rjo>
the binding generator may be more complex. sure. but that seems to be saved as you need zero (redundant) code per binding new library you want to use.
<whitequark>
wait, are we talking about the same thing?
<whitequark>
by "static" vs "dynamic" binding generator I understand "generating a .py/.cpp/whatever files" vs "generating data structures in the guts of the python process at every load"
<rjo>
whitequark: i think we are talking about the same thing. "static": cffi code, llvmpy, etc. "dynamic" introspect some c++ library with llvm/cling (magically, don't really know how they do it) and dynamically make python entry-points for stuff.
<rjo>
... at runtime.
<whitequark>
they need headers
<whitequark>
well that or DWARF debug info but it's literally hundreds of megabytes in case of LLVM
<whitequark>
there's no other way to interop with C++
<rjo>
ok. i had the feeling that it would not work with "just the .so".
<whitequark>
yes, and the header approach is a nightmare
<whitequark>
so for example it's not enough to just ship LLVM's headers, you also have to have a valid STL, libc, etc
<whitequark>
that means that on Windows you'll need an MSVC or MinGW setup or something like that. or you cannot do the #include's and you blow up on preprocessor defines you don't know
<whitequark>
rjo: btw we are not talking about the exact same thing
<whitequark>
there is a third approach
<whitequark>
which I call "static" here
<whitequark>
it introspects some C++ library in the same way (usually libclang, not cling, it shouldn't make any difference anyway) and then generates C++ and Python stub code
<rjo>
well. they will probably talk about it again and promote it. now you/we can respond adequately ;)
<whitequark>
the same thing it would have generated at runtime, but serialized to files, so that you can pack it up, compile it, and ship it
<rjo>
whitequark: is there a tool that does that?
<whitequark>
well there is ffi_gen for ruby, that's how the ruby llvm bindings work
<whitequark>
note that they still wrap the C API, it just takes less effort than manually
<rjo>
if that works, i'd also prefer that one over the "other static" and "dynamic" approaches above.
<whitequark>
rjo: and yes, there is a tool that does this, curiously enough
<rjo>
the "other static" (cffi, ctypes, cython, swig, python.boost) are all terrible IMHO already because of the code redundancy.
<whitequark>
it's Alessandro's LLVM bindings (the one that we want to integrate)
<rjo>
ok. nice.
<whitequark>
it's a 600-line file that parses LLVM-C's headers with regexps (!) and emits Python code using cffi
<rjo>
iirc the chances are very good that we can do this very soon.
<whitequark>
kinda gross but does the job well enough
<whitequark>
I would personally go with libclang but it's probably not worth rewriting
<rjo>
scratch that "iirc".
<whitequark>
sb0: no reply yet
<rjo>
is libclang then "another wrapper for LLVM-C(++)"?
<sb0>
whitequark, oh yes, and does it work on windows?
<whitequark>
sb0: worst case I can rewrite the exact same thing from scratch in 2-3 days top
<sb0>
including windows support?
<whitequark>
what windows support?
<whitequark>
why would something need to be done for windows?
<sb0>
famous last words :)
<sb0>
at the minimum, it has to be packaged
<whitequark>
it's a noarch package
<sb0>
how does it open the llvm library?
<sb0>
how is the llvm library packaged?
<whitequark>
.so/.dll
<sb0>
so dlopen() and LoadLibrary() called from python?
<whitequark>
cffi has a wrapper for that
<sb0>
how does it find the path to the library?
<whitequark>
including, iirc, the logic for handling file extension
<whitequark>
on windows the library is in the python library search path
<whitequark>
on linux it might actually be trickier
<sb0>
yes. so there are packaging issues.
<whitequark>
it's definitely easier to package than llvmlite, for one
<whitequark>
and we coped with that. I don't see a problem
<sb0>
ok
<sb0>
how much time would that take?
<whitequark>
packaging? or complete migration?
<sb0>
including the usual amount of conda breakage
<sb0>
full migration, including packaging and linux+windows
<whitequark>
to package it we need to flip one flag in the LLVM build process and figure out how to handle conda/non-conda search path
<whitequark>
it might actually be easiest to just copy the entire libLLVM.so into the python package dir
<whitequark>
full migration is more complicated because the API differs quite a bit and we'll need to adjust the lowering code for that, and then sort out all ensuing crashes
<whitequark>
the one downside of using the LLVM-C API (vs generating IR) is that the punishment for violating LLVM's API contract is an assertion failure and not some sort of nice message
<whitequark>
I do not expect this to be a problem for our users, and I can demonstrate that: when was the last time someone reported an ARTIQ bug where the compiler tried to put invalid IR into LLVM?
<whitequark>
(I think we had one or two long ago)
<whitequark>
anyway, I think a week should be sufficient
<whitequark>
the lowering code update is mostly mechanistic
<whitequark>
sb0: oh and one more thing, I'd like to upgrade LLVM to 4.0 before that as well
_rht has quit [Quit: Connection closed for inactivity]
<whitequark>
the attribute API that we use fairly heavily was changed in 4.0
<rjo>
whitequark: ack. we need license clarification for those bindings. could you ping him again?
<whitequark>
just did
<rjo>
sb0: are you waiting for that license clarification before sending the quote to ken/jungsang? we might want to un-bundle that.
<rjo>
whitequark: thx.
<sb0>
rjo, yes, waiting for that
<sb0>
whitequark, how long would the 4.0 upgrade take?
<GitHub100>
[artiq] whitequark commented on issue #723: Or more precisely it is an LLVM binding generator, not LLVM bindings. This also greatly reduces (ideally to zero) the amount of work binding-side that needs to be done when upgrading LLVM in the future. No more hacking around llvmlite limitations \o/ https://github.com/m-labs/artiq/issues/723#issuecomment-300752602
<GitHub4>
[artiq] jordens commented on issue #535: @whitequark could you add a short assessment of the state/quality/existence of support for or1k FP in the toolchain (artiq compiler, llvm, llvmlite/new-pyllvm, runtime) and the biggest roadblocks to this issue? https://github.com/m-labs/artiq/issues/535#issuecomment-300774053
<GitHub157>
[artiq] jordens commented on issue #535: @whitequark could you add a short assessment of the state/quality/existence of support for or1k FP in the toolchain (artiq compiler, llvm, llvmlite/new-pyllvm, runtime) and the biggest roadblocks to this issue? https://github.com/m-labs/artiq/issues/535#issuecomment-300774053
<GitHub66>
[artiq] whitequark commented on issue #655: @jordens The upstreamed patch is the equivalent of our patch but for function calls rather than loads and it is immediately useless to us in its current form. (It might be useful for other things, not sure yet.) However the argument that allowed it to get in can be used when revisiting !unconditionally_dereferenceable and we should do that. https://github.com/m-labs/artiq/issues/655#issuecomment-30079