cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
lritter has quit [Ping timeout: 258 seconds]
lritter has joined #pypy
jcea has joined #pypy
Rhy0lite has quit [Quit: This computer has gone to sleep]
lritter has quit [Ping timeout: 264 seconds]
energizer has quit [Ping timeout: 256 seconds]
jcea has quit [Quit: jcea]
speeder39_ has quit [Quit: Connection closed for inactivity]
pmp-p has quit [Ping timeout: 258 seconds]
pmp-p has joined #pypy
energizer has joined #pypy
oberstet has joined #pypy
forgottenone has joined #pypy
rjarry has quit [Ping timeout: 246 seconds]
rjarry has joined #pypy
<bbot2> Started: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/8248 [mattip: force build, win-unicode]
<bbot2> Started: http://buildbot.pypy.org/builders/own-win-x86-32/builds/2377 [mattip: force build, win-unicode]
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/7084 [mattip: force build, win-unicode]
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/5388 [mattip: force build, win-unicode]
<bbot2> Failure: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/8248 [mattip: force build, win-unicode]
<rjarry> arigo: I tried using that callback idea to fix my performance issue but that is not possible
<rjarry> the problem is that the function that parses the dictionary to a C struct or parse a C struct to a dictionary is recursive
<rjarry> so I simply cannot forward an opaque handle to a single dict
<rjarry> isn't there any way to use the python C api?
<rjarry> maybe in a separate compiled extension
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/7084 [mattip: force build, win-unicode]
<arigo> unsure why you could use the Python C API but not cffi
<arigo> also, I don't know the details so I don't understand what you're saying---e.g. why can't you use several handles to store several dicts
<rjarry> hi arigo :)
<arigo> you need to pass the handles into C code, and pass them back to the Python callbacks
<arigo> hi :-)
<rjarry> I line profiled it, and I spend a lot of time accessing to C struct fields
<rjarry> so I figure it would be faster if I did everything in pure C
<rjarry> btw, I managed to gain about 30% perf with this patch: https://dpaste.com/E629E9DB4
<rjarry> it basically removes a lot of python object instantiation
<rjarry> but that is still too slow compared to what it should be if I implemented this in C
<arigo> sorry, can't really help you here concretely
<rjarry> I do not expect help on this specific optimization, don't worry
<rjarry> but I wonder if there is any way I could write a separate (non-cffi) compiled extension
<rjarry> that uses the python C api
<rjarry> and somehow allow that separate extension to access to CData objects
<rjarry> I realize this would be hacking around cffi low level stuff, but I do not see any other solution that would bring significant improvement
<rjarry> using callbacks with ffi handles will still mean doing a lot of back & forth between C and python
<rjarry> and my profiling shows that is what is costing here
<Hodgestar> cfbolz: A debugging technique that you thought Armin might not know of seems definitely worth a blog post!
<bbot2> Failure: http://buildbot.pypy.org/builders/own-win-x86-32/builds/2377 [mattip: force build, win-unicode]
<arigo> rjarry: I *guess* you could redeclare the struct CDataObject in your own extension, and use that to access directly the "char *c_data" field of any CData object from C
<arigo> that's a hack that is not portable, e.g. to PyPy
<rjarry> ah, that's what I wondered :(
<arigo> but for CPython it should be stable
<rjarry> to PyPy the ABI is different but can I determine that when compiling the extension?
<arigo> yes
<rjarry> and access another offset in the structure
<arigo> no :-/
<arigo> on pypy there is nothing stable like that
<rjarry> that's unfortunate
<arigo> and anyway, on pypy it's going to be faster to not run CPython C API code at all
<rjarry> hmm
<arigo> so you could write an optional CPython C API extension for CPython only
<rjarry> I could keep the pure python version for pypy yes
<rjarry> I need to experiment a bit to see if it is worth the hassle
<rjarry> and is that realistic to modify the char *c_data field of any CData object from C ?
<arigo> no, it's supposed to be immutable
<rjarry> ok, I give up then :)
<arigo> really, if you're aiming for top performance on CPython, sometimes the best results are achieved by writing a plain CPython C extension---don't try to mix in cffi
<arigo> unfortunately, that's slower on PyPy
<arigo> with cffi you're supposed to get reasonable performance on CPython and top performance on PyPy
<arigo> (which is supposed to be what people using CPython or PyPy are looking for)
<rjarry> I understand
<rjarry> I had thought trying out Cython, but that does not look like an easy solution either
oberstet has quit [*.net *.split]
[Arfrever] has quit [*.net *.split]
dansan has quit [*.net *.split]
ebarrett has quit [*.net *.split]
xcm has quit [*.net *.split]
[Arfrever] has joined #pypy
jacob22 has quit [Read error: Connection reset by peer]
dansan has joined #pypy
ebarrett has joined #pypy
oberstet has joined #pypy
jacob22 has joined #pypy
xcm has joined #pypy
<arigo> right
<arigo> for performance on both cpython and pypy, there's the upcoming "hpy" project, but it's not ready yet
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/5388 [mattip: force build, win-unicode]
<rjarry> arigo: seems interesting indeed
<rjarry> are you involved in this?
t4nk304 has joined #pypy
t4nk304 has quit [Client Quit]
Rhy0lite has joined #pypy
<arigo> roughly, yes
<arigo> you can join the #hpy channel if you're interested
<rjarry> why not :)
dmalcolm has quit [Remote host closed the connection]
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/7085 [arigo: testing, list-with-longs]
dmalcolm has joined #pypy
lritter has joined #pypy
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/7085 [arigo: testing, list-with-longs]
jcea has joined #pypy
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/7086 [arigo: testing, list-with-longs]
<bbot2> Started: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/8249 [arigo: testing, list-with-longs]
Dejan has joined #pypy
Dejan has joined #pypy
Dejan has quit [Changing host]
BPL has joined #pypy
BPL has quit [Remote host closed the connection]
<bbot2> Failure: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/8249 [arigo: testing, list-with-longs]
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/7086 [arigo: testing, list-with-longs]
BPL has joined #pypy
mattip_ has joined #pypy
mattip has quit [Ping timeout: 256 seconds]
mattip_ is now known as mattip
Rhy0lite has quit [Ping timeout: 240 seconds]
Dejan has quit [Quit: Leaving]
Rhy0lite has joined #pypy
larstiq_ has quit [Ping timeout: 272 seconds]
BPL has quit [Quit: Leaving]
forgottenone has quit [Quit: Konversation terminated!]
lritter has quit [Quit: Leaving]