cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "the modern world where network packets and compiler optimizations are effectively hostile"
rokujyouhitoma has quit [Ping timeout: 240 seconds]
slackyy has quit [Read error: Connection reset by peer]
raynold has quit [Quit: Connection closed for inactivity]
marky1991 has joined #pypy
rokujyouhitoma has joined #pypy
<fijal>
njs: ping
<njs>
fijal: ?
<fijal>
njs: how often people use CPython C API in cython?
<fijal>
like
<njs>
uh, not sure, probably depends on the cython
<fijal>
I wonder if a system that supports cython & numpy would be considered viable for python numeric people
<njs>
I've certainly done it...
<njs>
but certainly there's a lot of useful things you can write without it
<njs>
you'd be targeting a subset of cython regardless, b/c really cython the language exposes the fact that it's going through a C compiler as part of its semantics
<fijal>
right
<fijal>
I think that part is ok
<fijal>
somehow
<njs>
but this is kinda-sorta the idea we were talking about at the compilers summit exactly a year ago, and then I dropped the ball when I got sick :-/
<fijal>
well maybe its worth restarting
<fijal>
I think it's a bit sad (leaving pypy aside) that it's so hard to come up with a faster python
<fijal>
and part of the problem is C API
<fijal>
web world largely moved to cffi and numeric world largely moved to cython
rokujyouhitoma has quit [Ping timeout: 248 seconds]
<fijal>
njs: thoughts?
<njs>
I definitely think it's worth restarting
<njs>
everyone there seemed very interested, including several teams from intel etc
<njs>
of course "interested" != "actually going to follow through with $$" as we both know well but.
<fijal>
right
<fijal>
PSF would throw *something*
<njs>
but like continuum is spending full-time salaries on maintaining numba's reimplementation of numpy, the status quo is plainly ridiculous
<fijal>
would continuum support such efforts?
<njs>
potentially
<njs>
numba has the same pain you do with dealing with numpy/pandas/etc.
<fijal>
right
<fijal>
we haven't really started yet tbh
<fijal>
but we're thinking about cython
<fijal>
seems gilless is easier to find money for
<njs>
and of course they'd love to be able to sell anything that makes python numerics faster. if pypy could do that then hey.
<fijal>
right
<njs>
the general idea we sort of settled on at the workshop was, come up with some sort of intermediate language with cython-ish semantics (i.e. opcodes for low-level bit-twiddling and C ffi, sort of wasm-inspired maybe, + python-level stuff like attr lookup and reference tracking, the stuff that you don't want to use the C API for)
<njs>
come up with a compiler from something like a subset of cython to this language
<fijal>
right
<njs>
and then you (a) compile to C + the Python C API using cython, (b) also compile to the IL, and (c) distribute them together with some metadata that says like "hey, pypy/numba/whoever, if you're about to jump across the C API boundary to call <this function pointer>, there's an identical implementation of the same functionality over at <this offset in the IL>"
<fijal>
so the key question is "how much does it give you"
<njs>
with the idea being that this way you can incrementally convert something like numpy into this language, while keeping full functionality and speed on CPython, but more and more of the hotspots become visible the JIT in Numba/PyPy/whoever.
<fijal>
riight
<fijal>
it's touch tricky to secure funding for us, but maybe not impossible
<bremner>
what's the main issue with CPython in pypy, that it blocks jit?
jcea has joined #pypy
nimaje1 has joined #pypy
nimaje1 is now known as nimaje
nimaje is now known as Guest39948
<fijal>
bremner: sorry?
<njs>
fijal: the next step I've had on my todo list since last year is to write up a clean summary of all this and post it to the python-compilers list to try and start organizing something
<bremner>
fijal: err. I mean the Python C API, sorry.
<fijal>
bremner: it leaks all kinds of internal details
<bremner>
is it fixable with time / money?
<fijal>
it's improvable
<bremner>
or a conceptual problem
<njs>
fijal: there were a whole bunch of different potentially-interested parties there, and ideally we could get multiple groups actually interested in putting work in
<LarstiQ>
bremner: details that are different between CPython and pypy/force pypy to emulate things
<fijal>
but it's a lot of time and a lot of effort
<fijal>
bremner: there is also the fact that JIT can't see it
<bremner>
right.
<njs>
bremner: as far as speed goes, yeah, if you have something written to the CPython C API like numpy, it will never go fast on pypy, because the JIT can't do anything with black box C code, and if all you're doing is issuing a bunch of calls to CPython C API methods then... CPython is really fast at that actually
<njs>
they're not super coherent as is. what we need is some thought leadership :-)
<fijal>
ok
<fijal>
njs: it's hard for me to see how we can find funding, but I'm keen to do the work
<njs>
yeah, that's a big question mark for me
<fijal>
if we can find funding, I can take over some of the aspects of that, but a lot has to be done before
<njs>
it's ridiculously ambitious, but... also ridiculously awesome, maybe?
<fijal>
maybe a $5k or $10k exploratory grant to numfocus/PSF?
_whitelogger has joined #pypy
<fijal>
njs: ok, who would we ask?
<fijal>
your plan seems plausible tbh
fenrrir has joined #pypy
<fijal>
the exact details of il are maybe not even that repevant
<njs>
fijal: oh yeah, I'm sure the exact details will change when we get into it
<njs>
fijal: but a page or two that makes the central idea clear and makes it obvious that it's the only thing that will work, would do a lot I think to focus discussion and start consensus building?
<njs>
it's a lot easier to get funding once your plan is inevitably where everyone thinks the ecosystem is going, I feel like :-)
Tiberium has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 260 seconds]
ronan_ has joined #pypy
ronan_ is now known as ronan
ELFrederich has joined #pypy
<fijal>
njs: heh
<fijal>
njs: ok, do you want me to write that page or two?
<fijal>
or maybe collaborate?
<kenaan>
arigo py3.5 3d878f5a314f /pypy/module/sys/system.py: Fix for Linux distributions on which os.confstr('CS_GNU_xxx') doesn't work
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 246 seconds]
rokujyouhitoma has quit [Ping timeout: 240 seconds]
fenrrir_ has joined #pypy
fenrrir has quit [Ping timeout: 255 seconds]
cstratak has joined #pypy
<fijal>
plan_rich: you around?
<plan_rich>
fijal, yes
<fijal>
the jit trace viewer is completely wrong
<fijal>
in vmprof
<fijal>
shows the entry numbers for the entry not for the loop
<fijal>
does not collapse the preamble
<fijal>
etc
<fijal>
plan_rich: feel free to tell me "then fix it"
<fijal>
:)
ronan__ has joined #pypy
ronan__ is now known as ronan
<plan_rich>
can you fix it?
<plan_rich>
yes the preamble is not collapsed, but I thought that the loop numbers should be correct
<plan_rich>
do you have an example?
<Remi_M>
fijal: pong
<fijal>
plan_rich: where is it showing? just preamble on the left
<fijal>
Remi_M: do you have prospective students? we were thinking about an interpreter for ir in the jit
<realitix>
Hi fijal, how are you ?
<Remi_M>
fijal: not right now, but I could put the project on our website
Fleche has joined #pypy
<fijal>
realitix: hi
<plan_rich>
the loop numbers are always at the label
<fijal>
Remi_M: cool
<fijal>
plan_rich: yes but the sorting by preamble on the left makes zero sense
<realitix>
for the demo cube simulation we talked about, is it to test pypy or cffi? I ask because the "official" bullet physics wrapper is not with cffi and would have poor performance with pypy. In that case, I have to wrap it with cffi
<Remi_M>
fijal: if you can give me a description and make it sound scientific :P
<fijal>
Remi_M: ok
<fijal>
realitix: pypy
<realitix>
ok, more work for the demo then but it can be cool !
<fijal>
realitix: but we can have our own basic physics in python. or something
<realitix>
I know bullet physics pretty well so it's simpler to use (for me)
<realitix>
I will ask the author it he can use cffi
<plan_rich>
a plain loop doing arith and running for 2000 iterations
<plan_rich>
ok, I see it should say; entered ~900 times no?
<plan_rich>
and we should collapse the preamble? as in not display it by default?
<plan_rich>
(and show the entries when the user clicks on it)
<plan_rich>
I can fix that if that is a good plan
<fijal>
yes I think so
<fijal>
for all of the above
cstratak has quit [Ping timeout: 240 seconds]
<fijal>
plan_rich: asm should not show guard code
<fijal>
label on "opt" (as opposed to rewritten) does not show numbers
<plan_rich>
the one at the very bottom?
<fijal>
yes
<fijal>
and why is the driver name "" and function ""?
<plan_rich>
probably an old pypy, I should retry with a new version
<plan_rich>
opt and rewrite does not show the numbers because increment_debug_counter is inserted only for the last stage. which means we would need to carry the information from asm to opt+rewrite
<fijal>
ok
<fijal>
but increment_debug_counter is still referencing something in the loop, can be addressed no?
<plan_rich>
each label emits a inc_debug_counter
<fijal>
well it's trivial to map it to the corresponding label from the previous tab, common
<plan_rich>
ah, I'm wrong. seems like increment_debug counter is already in rewritten, so it is missing in OPT
rokujyouhitoma has joined #pypy
<plan_rich>
just to confirm: I'm fixing this now
rokujyouhitoma has quit [Ping timeout: 260 seconds]
<mattip>
bendlas: so maybe "python pytest.py pypy/module/test_lib_pypy"
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
kipras`away is now known as kipras
<mattip>
bendlas: if you really want to prove it is working, you should copy what the buildbot does for "own" (untranslated) tests, it takes an hour or two
<mattip>
what is os.name on your platform? that file has a "skip" if it is not "posix"
<bendlas>
we're running `pypy-c ./pypy/test_all.py --pypy=./pypy-c -k 'not ( test_ssl or test_urllib2net or test_urllibnet or test_urllib2_localnet or test_socket or test_shutil or test_zipfile64 or test_epoll or test_tkinter )' lib-python`
<bendlas>
it's `posix`
<mattip>
I'm backing up to where you ran "python pytest.py pypy/module/test_lib_pypy"
<mattip>
and looking for test_resource.py
<mattip>
or even "python pytest.py pypy/module/test_lib_pypy/test_resource.py"
<bendlas>
mattip: here is for `python ./pypy/test_all.py --pypy=./pypy-c lib-python 2>&1 | tee /tmp/python.check` https://hastebin.com/hotaholepu.log
<mattip>
got it, that only runs translated, i.e. ./pypy-c pytest.py -A pypy/module/test_lib_pypy/test_resource.py
vkirilichev has quit [Remote host closed the connection]
fenrrir has quit []
<mattip>
ok, so it seems your platform does not support the resource module properly
<bendlas>
hm, it's a pretty standard linux kernel with a non-standard userland
<mattip>
bendlas: you aren't runnig as root by chance?
<bendlas>
not me, and I think our build processes are owned by unprivileged users as well
<bendlas>
here is an ipython session. I think it shows, that I can set the soft_limit, but it doesn't seem to have an effect: https://hastebin.com/cineliheqe.py
<mattip>
does it work in cpython?
<mattip>
the limit is set (line 18 shows you can no longer write to a file) but that os.urandom is not respecting the setting
<mattip>
ok, so at least that does not seem to be unique to PyPy
<bendlas>
btw, the tkinter failures are probably phony and due to me running a tiling window manager
<mattip>
cool. So we are left with being unable to lock /dev/urandom with resource.setrlimit
<mattip>
?
<bendlas>
pretty much
panni_ has quit [Ping timeout: 240 seconds]
<bendlas>
`python pytest.py lib-python/2.7/test/test_os.py 2>&1 | tee /tmp/test_os.log` fails in a weird way. Am I doing it wrong? https://hastebin.com/zudewezoca.log
Fleche has quit [Ping timeout: 240 seconds]
<mattip>
no, our stdlib is slightly different than cpython's, that is a known difference
<mattip>
if you really want ot run cpython's tests, you have to find them in /usr/lib somewhere
<mattip>
on Ubuntu the tests are part of some non-installed package, I have them as part of a git checkout of cpython
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<mattip>
bendlas: back to urandom, it seems we do not have an untranslated test for setting RLIMIT_NOFILE on /dev/urandom
<mattip>
it should IMO be in rpython/rlib/test/test_rurandom.py