<njs>
(latter slides are aimed at a meeting of folks from big companies using python, so there's some stuff about sustainability at the beginning you can skip past)
<simpson>
njs: At least two other alternatives present themselves, after considering your slide deck. First, numpy could move to the stdlib. Second, numpy could stop using C/API; IOW C/API could stop existing altogether.
<simpson>
I agree 100% that the situation is currently ridiculous.
<simpson>
And I like that we're continuing to get people to sit up and say so.
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
<njs>
simpson: it's very widely agreed that the C API is a problem. the core devs definitely feel the pain, because they're the ones who have to keep it stable :-). complaining about it was like a constant theme at all the language summits since I started going. The problem is that it's really really hard to find viable alternatives.
<njs>
unfortunately "move numpy to the stdlib" and "write numpy in pure python" are not really viable for several reasons...
xcm has quit [Remote host closed the connection]
<simpson>
Are the reasons Conda, Continuum, and cash~? We may have to choose the community over them.
<simpson>
...OTOH I shouldn't say "we" when I've already left.
<njs>
simpson: heh, actually Travis is like 112% behind the idea of throwing out numpy and rewriting from scratch and putting it in the stdlib
<simpson>
And, as you've pointed out, the JITs already have each done part of that work.
<njs>
simpson: but like... throwing out solid 25 year old codebases with tens of millions of users is just not a... realistic thing
<simpson>
njs: I used to work on Xorg a while ago. Throwing out decades-old code was cause for celebration. The Deletion Fairy was beloved for their blessings.
<njs>
simpson: yeah but this isn't that, this is like Xorg->Wayland
<simpson>
I agree that, for Python in particular, doing anything that breaks folks who type `from numpy import *` (bless their scope-polluting hearts!) is not a workable solution.
<njs>
after literally decades of discussion, and then a full decade of actual work, Wayland is... almost sort of replacing X, finally?
<simpson>
Oh! Nope, don't worry about that, but X11 isn't going anywhere. An Xorg can be hosted inside Wayland, for the support of legacy clients, who will continue to exist for a while.
<njs>
yeah exactly
<njs>
I'm not worried about it, I'm saying this is a good example of what this kind of transition actually looks like in practice :-)
<simpson>
Wayland was deliberately not API-compatible nor designed to be a full replacement, though, whereas things like PyPy's micronumpy are certainly intended to match the numpy API.
<simpson>
Anyway, sorry for the tangent. Just saying that code isn't inherently worth anything when it comes to these sorts of decisions, other than that it's easy to just do nothing because code is code.
<njs>
micronumpy hasn't exactly been a fabulous success story though
<njs>
anyway yeah, that's the context
<simpson>
Well, how could it have been? PyPy usage is at what, still 2% or something like that across the Python community?
<simpson>
Ugh, sorry. I'm grumpy.
<njs>
I'm not saying micronumpy *should* have failed, just that if we want a different outcome we might need to try something different :-)
<simpson>
Whatever gets buy-in from numpy's owners, I guess. Still waiting for my modest proposal of "write Python" to be adopted by the Python community.
adamholmberg has quit [Remote host closed the connection]
<simpson>
(This slow-motion multigenerational train wreck is what flashes through my mind every time somebody asks me why Monte has no FFI nor C API.)
<energizer>
simpson: your proposal is to completely port numpy to pure python and run it on pypy?
<simpson>
energizer: Somewhat. In fact, skip the numpy. I've personally experienced massive (60x) speedups by switching away from numpy to pure Python and running on PyPy, and so I became a believer.
<energizer>
simpson: in scientific-like code?
<simpson>
More deeply, what I want to emphasize is that *every* JIT implementation has gotten caught up on the C/API in one way or another. It's not good. It's not healthy. And I like PyPy.
<simpson>
energizer: Yeah, this was on a real-world numeric workload, generating level data on-the-fly for a Minecraft game server. With PyPy, it was fast enough to host thousands of (simulated) clients on a single laptop.
<mattip>
simpson: as someone who wrote alot of numpypy and now works on numpy, I would love to work on njs's proposal
<mattip>
there has to be a migration path
dddddd has quit [Remote host closed the connection]
oberstet has quit [Remote host closed the connection]
lritter has joined #pypy
jacob22__ has quit [Ping timeout: 252 seconds]
xcm has quit [Ping timeout: 240 seconds]
xcm has joined #pypy
Taggnostr2 has quit [Quit: Switching to single player mode.]
Taggnostr has joined #pypy
jacob22__ has joined #pypy
antocuni has joined #pypy
lritter has quit [Ping timeout: 245 seconds]
themsay has quit [Ping timeout: 268 seconds]
Taggnostr has quit [Quit: Switching to single player mode.]
Taggnostr has joined #pypy
antocuni has quit [Ping timeout: 246 seconds]
<vstinner>
mattip: morning. i should read the backlog, i talked 1h about the C API with a colleague, and then with arigato about PyHandle :)
<mattip>
vstinner: well, if you can't read the backlog my appreciation for your work still stands :)
<vstinner>
mattip: serge told me that the problem of optimization is that if the C API becomes a blackbox, the optimizer is blocked. it cannot introspect C calls
<vstinner>
mattip: i should rephrase my website. there are different goals
<vstinner>
mattip: i do care about portable ABI where everything is blackbox, but i don't want to loose support for numpy nor prevent Cython to use CPython internals to emit efficient machine code
<vstinner>
mattip: in short, portable ABI should be the default, but users should remain able to access directly internals if they are care about performance
<vstinner>
i don't want to force the full Python community to move towards portable ABI, it doesn't work
<vstinner>
people use internals for best performances, maybe for bad reasons, but i'm not interested to try to convince everybody :)
<mattip>
hmm. My take from "optimizer is blocked" is that if users care about performance, they should not be encouraged to use internals
<mattip>
we should have a story that enables performance without accessing internals
<vstinner>
mattip: tell me that story :)
<mattip>
at least without *manually* accessing internals
<vstinner>
right now, it's hard to build a C extension into portable ABI. i want to make it easy, and let people choose
<vstinner>
for me the reasonables options are: (1) to fork the current API and slowly remove/rework functions which cause issues (2) write a new C API from scratch (ex: arigato's PyHandle idea)
<mattip>
most people I know prefer to use tools: cython, numba, SIP, pybind11
<vstinner>
mattip: i understood that cython can produce different code depending on options, that's cool :)
<vstinner>
mattip: PyQt is one of the only user of the stable ABI no?
<mattip>
I assume you are already, but I think it would be nice to meet up with all the tool makers and discuss
<vstinner>
mattip: cython should use something to discuss with CPython, no?
<mattip>
there is also mypyc, pypy of course, swig ...
<vstinner>
mattip: i migrated the capi-sig mailing list to Mailman3, I started to talk on that list which is now waiting for you!
<vstinner>
mattip: and i'm here to talk to you, PyPy devs
<vstinner>
mattip: i read once that tagged pointer is not interesting in PyPy. i guess that PyPy has difference tradeoff because it already has an efficient JIT compiler
<vstinner>
mattip: but in CPython, it's a different story. the price of boxing/unboxing is quite high
<vstinner>
mattip: i didn't abandon the idea of small enhancements of the C API to more towards being able to experiment tagged pointers in the master branch of CPython
<vstinner>
mattip: but i have been asked to write a PEP for that, and well, CPython has some governance issues :) we chose the PEP 8016 governance, but we will not have steering committee before next february
<vstinner>
mattip: since last july, no PEP can be approved :-(
<vstinner>
mattip: i made concrete changes in CPython: i separated the API in 3 parts: portable ABI, cpython API and internals API: Include, Include/cpython/, Include/internals/
<vstinner>
i'm making small steps, but it's moving forward :)
<cfbolz>
I am a tagging-skeptic, personally ;-)
<cfbolz>
even for interpreters
<cfbolz>
the big downside of tagging is that you make a bit everything more expensive, including most specifically incref and decref
<vstinner>
cfbolz: i don't really care if it's a success or not
<vstinner>
cfbolz: it's a nice example to explain the issues of the current API :)
<cfbolz>
right :-)
<vstinner>
cfbolz: i would like to allow people to experiment cool things
<vstinner>
cfbolz: right now, nothing cool can be tried in CPython, we are limited to micro optimizations, because of the C API
lritter has joined #pypy
jcea has joined #pypy
dddddd has joined #pypy
themsay has joined #pypy
themsay has quit [Read error: Connection reset by peer]
<fijal>
vstinner: have you heard njs ideas about cython-something
<vstinner>
fijal: no, i didn't
marky1991 has joined #pypy
<fijal>
so the idea (called cython-ir) is, in some form that we have a cython-like language or IR that can be understood
<fijal>
so it's *just* high level enough that you can express things like numpy
<vstinner>
fijal: hum. once he told me that he wants to optimize "a + b + c" to avoid temporary array, and he needs a different language for that. but i forgot the details :)
<fijal>
but then can compile to either C+C API but also understand it in terms of JIT
<vstinner>
fijal: is it just an idea? or does it exist?
<fijal>
just an idea
<vstinner>
many people have many ideas
<vstinner>
and nobody does anything :)
<fijal>
that is very true
<fijal>
that is less true
<vstinner>
"i have an idea: let's make python faster!" "i like your idea" "+1000"
<vstinner>
well well well :)
<fijal>
right so this is a step beyond
<fijal>
this is "this is how we can make python faster"
<vstinner>
fijal: last time i asked njs about his idea, he told me that he is too busy to implement it
<fijal>
fair
<vstinner>
fijal: what does use the IR?
<fijal>
well, what do you think about an idea as an idea?
<vstinner>
fijal: which tool is supposed to use IR to emit faster code?
<fijal>
pypy, numba
speeder39_ has joined #pypy
<fijal>
next-VM
<vstinner>
fijal: i have no opinion because i don't understand what is the input language. is it python? python-like, like cython?
<fijal>
like cython
<fijal>
something that numeric people can be happy with, I guess
<vstinner>
fijal: the IR idea sounds interesting
<vstinner>
fijal: but i think that it's unrelated to C API issues
<fijal>
why is it unrelated?
<fijal>
it's related in a sense that maybe we can make C API more private that way
<antocuni>
vstinner: it's complementary. For example, I can imagine having a cython-like tool which emits this IR; the IR contains C-API calls, and it's both architecture and language-version independent
<antocuni>
then you have a tool which takes this IR, convert to C and generate a "real" cpython extension
nunatak has quit [Quit: Leaving]
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
Zaab1t has quit [Quit: bye bye friends]
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 246 seconds]
adamholmberg has joined #pypy
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
speeder39_ has quit [Quit: Connection closed for inactivity]
Zaab1t has joined #pypy
michaelgreene_ has quit [Read error: Connection reset by peer]
jacob22__ has quit [Ping timeout: 246 seconds]
antocuni has quit [Ping timeout: 250 seconds]
xcm has quit [Read error: Connection reset by peer]
xcm has joined #pypy
marky1991 has quit [Ping timeout: 250 seconds]
jacob22__ has joined #pypy
themsay has quit [Ping timeout: 250 seconds]
Zaab1t has quit [Quit: bye bye friends]
marky1991 has joined #pypy
Rhy0lite has quit [Quit: Leaving]
marky1991 has quit [Remote host closed the connection]
themsay has joined #pypy
themsay has quit [Ping timeout: 244 seconds]
themsay has joined #pypy
themsay has quit [Remote host closed the connection]