cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
dddddd has quit [Read error: Connection reset by peer]
jvesely has quit [Quit: jvesely]
<mattip>
tumbleweed: I wonder if setuptools would accept a PR to add a pypy limited abi tag, and we could sneak in both a pypy2 and a pypy3 one
<kenaan>
cfbolz expose-jsonmap 8504b741f563 /pypy/module/: expose a way to create a dict structure from applevel, and instantiate it. can be used in csv module and ot...
<mattip>
then we could add it to EXTENSION_SUFFIXES on pypy2, pypy3, and get cffi to import setuptools (to monkeypatch distutils build_ext)
<mattip>
arigato: is there a reason cffi/ffiplatform does not import setuptools at the top of the file?
<mattip>
and use limited_api to build API c-extension modules via FFI().compile() ?
<arigato>
tumbleweed: would it help if cffi compiled its .so in such a way that it's identical everywhere (all versions of pypy and cpython)?
<cfbolz>
arigato: hey
<arigato>
mattip: the reason for not importing setuptools is that setuptools is not available everywhere
<arigato>
cfbolz: hi
<arigato>
mattip: as far as I know, extensions on cpython 3.x are compiled in limited_api mode
<arigato>
mattip: ...except on windows because mess
kenaan has quit [Ping timeout: 276 seconds]
<mattip>
not in simple API mode ffi.cdef(...); ffi.set_source(...); ffi.compile(...)
<cfbolz>
arigato: I am going to be intermittently online in the next couple of days, but if you feel like looking at the record-known-result branch at any point that would be awesome
<arigato>
mattip: OK maybe only when you're using a setup.py
<arigato>
cfbolz: OK
<mattip>
also, until we change our c prefix from PyPy to Py, any c-extension .so built on cpython cannot be imported to pypy and visa-versa
<arigato>
of course, I meant only the cffi ones
<arigato>
to compile cffi extensions only once and import them everywhere, it probably needs some hacks so that it doesn't try statically to use any PyXxx function
<arigato>
probably not #including Python.h at all
<arigato>
but it's doable
<arigato>
I'm just saying the possibility exists, I don't know enough to know if it's a good idea (e.g. cpython 2.7 won't try to import a file called ...-abi3.so)
<arigato>
maybe a slightly saner idea would be to generate a .py file instead of a .so, which then does something to import a real standard .so (which has nothing to do with cpython); but again, how you'd build that is messy
<arigato>
so in my previous sentence, "to import a real standard .so" => I meant load a standard .so, not import it in any pythonic sense
<mattip>
the _cffi_backend could handle some of the messiness, maybe?
<arigato>
that's the purpose of the code in your paste: when you import (in the python sense) the .so, then all it does is find _cffi_backend and ask it to handle the situation
<arigato>
so that's not the mess I was refererring to, I meant how are you supposed to build a standard .so with distutils/setuptools
<arigato>
anyway
<mattip>
numpy.distutils.command has build_clib
<arigato>
if the .abi3 tag is meant to be used for compiled modules that work on *any* implementation, then pypy lost, and we can't sanely use .abi3 for C extension modules
<arigato>
so although maybe we can abuse it for cffi extensions, for normal C extensions we can't
<mattip>
I think no-one considered pypy in setuptools, and we can legitimately submit a PR to have a pypy-specific abipp or (abi-pp2, abi-pp3) tags
<arigato>
that sounds like a better plan to me
<mattip>
even without cffi, we should do that
<mattip>
is it worth playing with a limited API project to see if the .so can be shared across pypy2/pypy3, then we can ask for a single "abipp" tag
<mattip>
or does someone already know why it won't work (string/bytes int/long API changes?)
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
<arigato>
for non-cffi modules, I think it's a lost cause, because indeed of all the string/bytes int/long changes
<arigato>
to do that, we'd need to introduce brand new parts of the API, like PyText_Xxx() and PyInteger_Xxx()
<arigato>
e.g. PyText_FromString() would return a string on pypy2 and a unicode on pypy3
<arigato>
it's common I think for C extension modules that are meant to be portable between cpython2 and cpython3, to #define PyText_FromString to either PyString_FromString or PyUnicode_FromString
<arigato>
so (sorry for not saying it clearly) I mean that it can't really work, because C extension modules already do this kind of #defines, which means a single compiled .so can't work on both py2 and py3
<arigato>
but I think it's fine if we give up on "abipp" and add only "abi-pp3", which would then be used just like it is on cpython3. also for the HPy (pyhandles) project, which would also emit "abi-pp3" on pypy3 I guess
<arigato>
of course it would be cool if there was also a tag that means "works everywhere", which we could then use for cffi
<arigato>
(have cffi produce completely untagged .so's? these are recognized both by cpython2 and cpython3)
<mattip>
ok, so the TODO list has: 1) add abi-pp3 and abi-pp to py3.6, 2) add abi-pp to default, 3) convince setuptools to add abi-pp3 to setuptools.command.build_ext. 4) get cffi to use abi3 and abi-pp
xcm has quit [Remote host closed the connection]
<mattip>
I think cpython2/cpython3 is beyond reach
<arigato>
I'm not sure I see the point of "abi-pp"
<mattip>
so that on pypy cffi can produce .so that can be used on both
xcm has joined #pypy
<mattip>
pypy2/pypy3
lritter has joined #pypy
<mattip>
getting back to FFI().compile, which platforms don't have/use setuptools?
<arigato>
ah, so imp.get_suffixes() on, say, pypy3 would include '.abi-pp3-71-x86_64-linux-gnu.so' and also '.abi-pp-1-x86_64-linux-gnu.so'
<arigato>
i.e. abi-pp3 with the C extension version, and .abi-pp with another version that never changes?
<mattip>
right, but the 71 is troubling, it means we need to carefully manage limited_api version numbering
<arigato>
setuptools: I just wanted a way that works without setuptools, because it's not in the base cpython. I'd be fine nowadays if we required setuptools in order to use ffi.compile(), as long as we don't require it to use a compiled module
<mattip>
ok, I will try to do a PR
antocuni_ has joined #pypy
<arigato>
mattip: if we want to seriously implement limited_api like it was meant to be used, it's more work and it freezes us into the current solution (e.g. what we put in the PyObject structure)
<arigato>
so for now I'm not exactly sure what limited_api would give to pypy
<mattip>
maybe once hpy is in, we can rethink it
<arigato>
yes
<arigato>
so I'm not sure I see the point of "abi-pp3" actually
<mattip>
ok. At least we thought it through.
<arigato>
I *think* we need to check what occurs if we compile on pypy3 a C extension module which says "limited_api"
<arigato>
because if the result is ".abi3.so" then there's a problem
<mattip>
no, it won't produce that today
<arigato>
OK
<mattip>
setuptools ends up asking importlib.machinery for EXTENSION_SUFFIXES, which is set in importlib._bootstrap_external to _imp.extension_suffixes, which on pypy3 does not contain abi3
<arigato>
ah OK
<arigato>
so in summary, we're only adding the versionless '.abi-pp.so' tag inside pypy2 and pypy3 and having cffi produce that
<mattip>
sounds good, but will require a cffi-specific build_ext class. Not too hard, and might also avoid requiring setuptools
<arigato>
OK
<mattip>
we can document it in cpython_differences
<arigato>
cfbolz: record-known-result looks reasonable to me, good job
kenaan has joined #pypy
<kenaan>
arigo record-known-result 7e2f7fad5601 /rpython/rlib/: Test and fix annotation for record_exact_value()
<cfbolz>
arigato: cool, thanks. I was trying to do something relatively pragmatic
<arigato>
it's pragmatic code and not too much of it, but it opens the door to optimizations we didn't think about previously
<arigato>
rbigint.fromint(x).toint(), just to give one example
<cfbolz>
arigato: yes
<cfbolz>
arigato: all sorts of these things
<cfbolz>
But I'd like to introduce them on a by-need basis
<cfbolz>
(eg is it useful that a+b-b==a for rbigints?)
CrazyPython has joined #pypy
mattip has quit [Ping timeout: 276 seconds]
<arigato>
can it reasonably do strengh reduction, too? e.g. at app-level "mylong + myint" is special-cased in longobject.py, but if it weren't, then could we in theory implement it all in rbigint.py?
<arigato>
i.e. reduce f(g(x)) to h(x)
CrazyPython has quit [Ping timeout: 245 seconds]
<cfbolz>
arigato: hm, good question
mattip has joined #pypy
<cfbolz>
arigato: I don't quite see how
Rhy0lite has joined #pypy
<arigato>
def g(x): if we_are_jitted(): record_known_result(h(x), f, y)
antocuni has joined #pypy
<arigato>
we_are_jitted is guarding against running h(x) for nothing
<cfbolz>
arigato: yes
<cfbolz>
But unsranslated it is run, to make it testable that the known result is actually correct
<arigato>
ah, then "if not we_are_translated() or we_are_jitted()" would be best here?
<cfbolz>
I am confused, let me look at something
<arigato>
I'm taking the example of rutf8.py in this branch
<arigato>
you replace "@elidable def g(x)" with:
<arigato>
def g(x):
<arigato>
y = _g(x)
<arigato>
record_stuff
<arigato>
return y
<cfbolz>
arigato: I thought that I was already making sure that the call is only done only when jitting
<arigato>
well, if you write in rpython record_known_result(h(x), f, y)
<arigato>
then h(x) is going to be executed
<cfbolz>
Right
<cfbolz>
So far only f(y)is is not executed
<arigato>
yes
<arigato>
if h(x) is a messy function, even if the C code doesn't actually use the result, it's probably not going to disappear
<cfbolz>
I think we have a backend opt that does it but I am not totally sure
<arigato>
ah, but even if we do, I guess it needs to be taught about record_known_result, because that goes all the way to C (where it is #defined as an empty macro)
<cfbolz>
arigato: true
<cfbolz>
So we'd need a backend opt that removes it, together with the we are jitted symbolic
<arigato>
I'm confused, isn't it enough to write "if we_are_jitted(): record_known_result(f(y), ..)"?
<kenaan>
JulianB default 6c7852e9615a /README.rst: Update some links to HTTPS.
<kenaan>
JulianB default f8f6fe914790 /LICENSE: Merge some license gap fixes.
<cfbolz>
arigato: yes, but then the invariant isn't tested
<arigato>
hence my next suggestion to write "if not we_are_translated() or we_are_jitted(): ..."
<cfbolz>
arigato: btw, I don't see how the example could work for rbigint.add(x, rbigint.fromint(y)) -> rigibint.add_int(x, ')
<cfbolz>
You can't add a record to fromint(y), since you don't know the x
<arigato>
ah, right
<cfbolz>
It's a pity, isn't it
<arigato>
but it's true for any x, so all you need is "for i in range(-sys.maxint-1, sys.maxint): record..."
<cfbolz>
Hahaha
<arigato>
:-)
<cfbolz>
Nono, x is an rbigint
<arigato>
damn
<cfbolz>
:-)
<cfbolz>
arigato: it's cool that we can express idempotence, though. Eg abs(abs(x)) is fine
<arigato>
right
<cfbolz>
arigato: I'm a bit worried about the testability specifically for rbigint, though. Because only the value is going to stay the same, not the pointer
<arigato>
not sure I follow
<cfbolz>
If you run it untranslated with record in rbigint, then you get an assertion error
<cfbolz>
During testing
<cfbolz>
Because the record machine uses == to compare two rbigints, not rbigint.eq
<arigato>
ah, yes
<cfbolz>
So we may have to add a comparison function for testing, later
<cfbolz>
(when we actually want to add the first rbigint hint)
<arigato>
makes sense
<cfbolz>
Essential property: str.lower is idempotent!
<arigato>
unicode.lower, I'm not sure
<arigato>
yes, seems so
<arigato>
.upper too
<arigato>
with exactly one possible exception though!
jvesely has quit [Quit: jvesely]
njs has joined #pypy
<njs>
quick cffi question. so I'm trying to do: obj = ffi.new("MYTYPE *"); print(ffi.cast("uintptr_t", obj)). But this gives: TypeError: '_cffi_backend.CData' object cannot be interpreted as an integer
<njs>
I've tried like a dozen variations and I cant' find the magic incantation to just get the stupid point value out of this object. Is there a trick I'm missing?
<njs>
(this is just for some debug prints)
<arigato>
exactly what you entered seems to work fine for me
<tumbleweed>
arigato: so, yeah, from a quikc reading of the PEPs. Nobody ever thought about alternative python implementations implementing a stable ABI
<tumbleweed>
Either we need to define an interface for that, or to claim abi3 you need to support cpython's abi3
<tumbleweed>
and if the goal is wheel compatibility for cffi across releases, you'll need pip, wheelhouse, etc. to support this too