cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "the modern world where network packets and compiler optimizations are effectively hostile"
rokujyouhitoma has quit [Ping timeout: 240 seconds]
tbodt has quit [Read error: Connection reset by peer]
yuyichao has joined #pypy
tbodt has joined #pypy
the_drow has joined #pypy
<the_drow>
I just saw your announcement on GiL-less PyPy. I have donated some money to the STM project initially so I'd really like to see it used for that project if possible.
<the_drow>
fijal, ^^ :)
<fijal>
the_drow: as in you would prefer stm or gilless?
<the_drow>
The GIL-less approach is what everyone in the Python community wants I think so yeh me too :)
<fijal>
Cool thanks!
<the_drow>
With lock elision it could be as fast as non-threaded PyPy as well
<fijal>
If you read yesterday chat you'll see why things are hard :)
<the_drow>
fijal, I never thought the problem would be the CPU cache
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
asmeurer__ has quit [Quit: asmeurer__]
kenaan has quit [Read error: Connection reset by peer]
exarkun has quit [Ping timeout: 240 seconds]
exarkun has joined #pypy
marr has joined #pypy
kolko has quit [Read error: Connection reset by peer]
<exarkun>
I fixed some stuff to do with the imp module tests in default yesterday. Then I was going to merge default into py3.5 before fixing the new problems there but `hg merge` defeated me.
<exarkun>
Anyone planning to merge default into py3.5 soon and save me the trouble? :)
yuyichao has quit [Ping timeout: 246 seconds]
yuyichao has joined #pypy
adamholmberg has joined #pypy
<mjacob>
exarkun: it can get quite complex :)
<mjacob>
exarkun: where do you get the merge conflicts, imp module or somewhere else?
Remi_M has quit [Quit: See you!]
<exarkun>
in the imp test module that I modified and also in cpyext which I guess someone else did some work on
<ronan>
mattip, exarkun: merged, but I didn't fix all the failures in imp and cpyext
<exarkun>
ronan: thanks :)
rokujyouhitoma has joined #pypy
tbodt has quit [Read error: Connection reset by peer]
rokujyouhitoma has quit [Ping timeout: 240 seconds]
the_drow has joined #pypy
<LarstiQ>
exarkun: lazy decoding could work
yuyichao_ has joined #pypy
tbodt has joined #pypy
yuyichao has quit [Ping timeout: 255 seconds]
<arigato>
the Windows console returns an error (12: not
<arigato>
depending on heap usage).
<arigato>
binary and the length is greater than 66,000 bytes (or less,
<arigato>
enough space error) on writing into stdout if stdout mode is
<arigato>
you don't say.
tbodt has quit [Ping timeout: 240 seconds]
<cfbolz>
:-(
<kenaan>
arigo default eaa386e4970b /rpython/rlib/rposix.py: Issue #2636 Add the same workaround as CPython for os.write() on Windows on a tty
gclawes has joined #pypy
<kenaan>
exarkun py3.5 6ee00ca3924e /pypy/module/imp/test/test_app.py: The `U` mode is deprecated on Python 3; use r instead.
<mattip>
ronan: after the merge cpyext tests seem to basically work AFAICT
raynold has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 248 seconds]
the_drow has quit [Ping timeout: 240 seconds]
tbodt has joined #pypy
ronan has quit [Ping timeout: 246 seconds]
antocuni has quit [Ping timeout: 240 seconds]
rokujyouhitoma has joined #pypy
gclawes has quit [Ping timeout: 246 seconds]
rokujyouhitoma has quit [Ping timeout: 248 seconds]
asmeurer__ has joined #pypy
asmeurer__ has quit [Quit: asmeurer__]
marky1991 has quit [Ping timeout: 240 seconds]
marky1991 has joined #pypy
marky1991 has quit [Ping timeout: 276 seconds]
marky1991 has joined #pypy
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tbodt has joined #pypy
yuyichao_ has quit [Ping timeout: 240 seconds]
yuyichao has joined #pypy
rokujyouhitoma has joined #pypy
dunpeal has joined #pypy
<dunpeal>
Hi! I'm considering writing a performance-sensitive project to target PyPy. Wondering what my options are if I need to rewrite some key performance-critical parts (e.g. hot loops) for extra performance.
lritter has quit [Quit: Leaving]
<dunpeal>
In CPython, I'd extend the VM, probably using Cython.
<Alex_Gaynor>
dunpeal: the standard approach is to write some C code and call it using cffi
rokujyouhitoma has quit [Ping timeout: 240 seconds]
<dunpeal>
Alex_Gaynor: anything else? I'd rather not write C if I can help it. Also, it's essentially a "calling out" model, no? Is there something more like "inlining"?
<dunpeal>
(I'd prefer C++ to C)
<Alex_Gaynor>
I mean, you can write it in any language that exposes a C ABI; Rust, C++, sure.
<dunpeal>
OK, so basically, cffi is my only options with PyPy.
<exarkun>
"extend the VM, probably using Cython" is an interesting way to put it.
<mjacob>
should we close the py3.3 branch? the last commit is from october 2016
<dunpeal>
exarkun: if I write a C-API extension, am I not "extending the VM"?
<dunpeal>
also, any update on numpypy status?
<exarkun>
dunpeal: All Cython does is generate some C and then expose it as a CPython extension module, which your Python code can call via an extremely specialized ffi system.
<cfbolz>
dunpeal: nowadays the C-extension support is good enough to support regular numpy in pypy
exarkun has quit [Read error: Connection reset by peer]
marky1991 has quit [Ping timeout: 240 seconds]
<dunpeal>
cfbolz: wasn't that always the case, if you're willing to accept a big performance hit? I take it the difference is this performance hit is smaller now (how bad is it)?
<dunpeal>
thanks. I'm planning to use scikit-learn, which I've been told might expose some bugs in cpyext because of the non-trivial way libraries in that stack call into C
rokujyouhitoma has quit [Ping timeout: 240 seconds]
<exarkun>
wow cool how'd I do that. (application-level) AssertionError: BytecodeCorruption: << unknown opcode, ofs=6, code=32, name=<module>
<dunpeal>
LarstiQ: not sure I understand your answer. Maybe I'm missing some key detail, but if the problem is that numpy utilizes C-API instead of cffi, can't the numpy project itself decide to port itself to cffi?
<cfbolz>
yes, but why?
<cfbolz>
it would be a huge amount of work
<dunpeal>
cfbolz: some folks over at #python thought it may happen, since apparently some people consider C-API archaic or generally Bad
<cfbolz>
yes, I am one of those people. still, a rewrite is a major undertaking
<LarstiQ>
dunpeal: so starting a new project with cffi over C-API is indeed a good idea
<LarstiQ>
dunpeal: and rewriting something small enough might too
<tos9>
dunpeal: (FWIW I meant something not exactly that. I mean, I did mean that, but I meant more broad things than that.)
<LarstiQ>
dunpeal: but numpy is huge
<tos9>
E.g., Cython eventually caving and growing a CFFI backend
<tos9>
I just said that it will help put pressure on the scientific community
<dunpeal>
tos9: yeah, no worries
<dunpeal>
the other argument is that if pypy is the future, there may be pressure on numpy to port
antocuni has joined #pypy
<cfbolz>
not if cpyext performance is good enough
<cfbolz>
antocuni: around?
<antocuni>
cfbolz: just for few minutes
<cfbolz>
antocuni: I read your cpyext mail properly now
<cfbolz>
cool work :-)
<antocuni>
thanks, although I didn't do much after that
marky1991 has joined #pypy
<cfbolz>
antocuni: I think the slowness in the "new object every call" case is that the arguments need C structures allocated and attached
<cfbolz>
I think that doesn't appear in the traces maybe
zware has quit [Quit: No Ping reply in 180 seconds.]
zware has joined #pypy
<antocuni>
"C structures allocated and attached": I assume you mean the cpyext proxies which corresponds to the W_Root objs?
<mjacob>
exarkun: assert rewriting in app-level tests is a big hack that breaks sometimes
<exarkun>
yay.
<exarkun>
so simplify the assert?
<antocuni>
cfbolz: however, note that also CPython needs to do one allocation at each call, I don't know why we are that much slower
<cfbolz>
antocuni: we need a dict lookup for every argument too
<antocuni>
I also tried to compile CPython **without** pymalloc, but the speed is the same
<cfbolz>
for the mapping w_obj -> c proxy struct
<antocuni>
right
<antocuni>
although the vmprof data clearly showed that the bottleneck was the malloc, IIRC
<cfbolz>
hm
<mjacob>
exarkun: not sure what broke exactly, but i wouldn't be surprised if it works when putting the assert on one line or something strange like that ;)
<cfbolz>
exarkun: you can try to run with --assert=plain
<exarkun>
rewriting it is easy enough ... and I don't want to make everyone in the future run pytest with special arguments, do I?
<cfbolz>
exarkun: it's just a problem when it fails ;-)
<mjacob>
exarkun: you put the py3.5 bookmark on 92129:c38befdc824e... please remove it
<exarkun>
what
<exarkun>
I don't know how to use hg sorry
<exarkun>
All I did was commit and push
jamesaxl has quit [Quit: WeeChat 1.8]
<mjacob>
well, then fix your broken hg-git bridge ;)
<exarkun>
exactly as I did for several previous revisions :(
<exarkun>
I threw that in the trash already
<exarkun>
I'm just using hg.
<mjacob>
ok, i'll try to remove it ;)
<exarkun>
I don't even know how to see bookmarks except sometimes hg seems to randomly drop them in unexpected places
<exarkun>
oh 92129 is from days and days ago
<kenaan>
mjacob py3.5 995cc5bdbad5 /pypy/objspace/std/: Rename bytes.translate's 'deletechars' argument to 'delete' to match CPython.
<kenaan>
exarkun py3.5 6eb780259953 /lib-python/3/test/test_sysconfig.py: If the machine can be [3-6], allow the suffix to be [3-6]. It's hard to tell but the intent (https://www.python.o...
<njs>
even if numpy wanted to port to cffi, the cffi api is missing tons of stuff that numpy would need. This is by design (cffi is strictly a C foreign function interface, not a way to interact with the interpreter at the C level), but I don't see how it would work given numpy's weird needs.
ronan has joined #pypy
kolko has quit [Ping timeout: 255 seconds]
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 248 seconds]