cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
oberstet has quit [Remote host closed the connection]
rich_ has joined #pypy
rich__ has quit [Ping timeout: 240 seconds]
YannickJadoul has joined #pypy
dddddd has joined #pypy
nimaje has quit [Remote host closed the connection]
nimaje has joined #pypy
<mattip>
interesting problem with virtualenv setting sys.path on pypy2.7
<mattip>
if you change the lib_pypy/__extensions__ then import cStringIO
<mattip>
it imports the lib_pypy pure python one instead of the builtin module
<nimaje>
is there any specific reason why the usession dir when building from the 7.3.0 source release tarball contains rc4? it seems to be usession-release-pypy2.7-v7.3.0rc4-0
<mattip>
where did you get the release tarball?
<mattip>
the sys.path thing is due to cStringIO not being in MODULES_THAT_ALWAYS_SHADOW
<arigato>
in sys.path, it's only a marker, nothing more
<arigato>
we can replace it with a different marker, which would make that code in virtualenv happy
<arigato>
that looks random though
<mattip>
virtualenv fixed the problem by not modifying it, but that mechanism looks brittle
<arigato>
yes, if we replace it with a constant string like "/pypy/extensions" then that virtualenv code works, but it's brittle indeed
<arigato>
there may be code somewhere that detects that "/pypy/extensions" or "../lib_pypy/__extensions__" is not a real path and removes it
<arigato>
I've got no clue how to fix that
<mattip>
maybe we should just rip it out and always use our pseudo c-extensions
<arigato>
unsure how it helps
<mattip>
in module/imp/importing.py we have code that explicity avoids importing builtins if they are not in MODULES_THAT_ALWAYS_SHADOW
<mattip>
(sorry about the two negatives)
<arigato>
yes, such modules are imported later, precisely at the time where we see lib_pypy/__extensions__ in sys.path
<arigato>
see comment above the definition of MODULES_THAT_ALWAYS_SHADOW
<mattip>
so what if we short curcuit, and always return delayed_builtin ?
<arigato>
then "import math" will fail to find "math.py" in the current directory
<arigato>
("wat?" "yes that's what cpython does")
<arigato>
the alternative is to add a file in lib_pypy for each module *not* in MODULES_THAT_ALWAYS_SHADOW
<arigato>
though what this file should be is unclear, maybe an empty file with a custom extension?
<arigato>
or a .py file with just a marker inside, or something
<mattip>
IMO ignoring math.py when it is imported as "import math" and forcing "from somewhere import math" would be nicer anymway
<mattip>
even if it breaks compatibility
<arigato>
well, there is a reason for this hack
<arigato>
I don't remember what the reason was but I think we kept seeing code breaking because of that
<mattip>
maybe we could first check that our marker is still in sys.path, and if not error out
<arigato>
or simply, we kill the marker from sys.path, but instead we check the filesystem: if the sys.path entry contains a file with a special name then we look for extension modules
<arigato>
and of course we make an empty file of that name inside lib_pypy/
<mattip>
that would still allow something like virtualenv to mess things up if they didn't copy the file
<arigato>
are they copying a random subset of lib_pypy/* somewhere else? that won't work anyway
<mattip>
good point
<arigato>
we can call the marker something.py, too, if that helps virtualenv copying it
<mattip>
lib_pypy/__extensions__.py, with a README inside it?
<arigato>
yes, or lib_pypy/_pypy_extensions.py
marky1991 has quit [Ping timeout: 260 seconds]
<mattip>
I wonder which code required the hack, and if it is still relevant
rubdos has joined #pypy
<mattip>
the commit to add the hack was a72429e0e0ed
<mattip>
I can't create a math.py or cStringIO.py and convince cpython2.7 to use them
rubdos has quit [Quit: WeeChat 2.4]
rubdos has joined #pypy
rubdos has quit [Ping timeout: 248 seconds]
rubdos has joined #pypy
<arigato>
uh? if I make math.py and type "python2 -c 'import math'" it imports this local file
<arigato>
though maybe you're using a version of python2 that was compiled differently---it's a mess
<arigato>
if there is a math.so then in python2.7, "python -c 'import math'" will import a local math.py instead
<arigato>
but the math module may also be built-in in some compilations of python
<mattip>
ubuntu 18.04's python2 has builtin math and cStringIO
<arigato>
of course
<arigato>
the point was more about other rarer modules
<arigato>
typically a module obscure enough that you don't even know it exists as an .so, and you have a .py of the same name and it works fine
<arigato>
I think I've seen it occur at least with "parser.py"
<arigato>
as well as "grp.py" maybe
<mattip>
ok
<mattip>
looking at set(cpython2 sys.builtin_module_names) ^ set(pypy2 sys.builtin_module_names) is a mess
<arigato>
cpython2 sys.builtin_module_names depends on compilation options, too
<mattip>
right, so even more mess
<mattip>
but it seems like once your CPython has those builtins (like mine has grp built in), you cannot override it with a py file
<arigato>
yup
<mattip>
PyPy goes above and beyond
<arigato>
which makes the python language ill-defined because it depends on compilation options
<mattip>
is it well defined on python3? I don't see the same mechanism there
<arigato>
it's possible that ronan killed it when porting to the new importlib
<mattip>
I wonder how to even search for issues around this in CPython or PyPy
<mattip>
maybe we are being too strict, and could simplify things by always using buitins if they exist, like CPython
<arigato>
if you prefer, it's also possible to rename all pypy/module/*/moduledef.py (with applevel_name="_pypy_math" for example) and add a file lib_pypy/math.py containing "from _pypy_math import *; from _pypy_math import __doc__"
<arigato>
mattip: I'm -1 because of the example of "parser.py"
<arigato>
well at this point I'm only -0 I guess. I'm sure it will crash for someone, like it did when I fixed the problem, but I'm only -0 on reverting all the mess and documenting in cpython-differences
<arigato>
so I'm +1 on adding "lib_pypy/parser.py" and applevel_name="_pypy_parser"
<arigato>
and doing that for all modules that are at least not very common
<mattip>
will that slow down start-up time noticably?
<arigato>
I doubt it
<arigato>
I'm OK if we leave 'math' out of it too, i.e. consider math a real built-in like it is on ubuntu
<arigato>
and cStringIO
<mattip>
ok, that sounds like a better approach: take Ubuntu as a guide, add py files for the c-extensions
<mattip>
and document it in cpython differences
<arigato>
and for py3.x I have no clue
<mattip>
maybe do the same?
<mattip>
my so modules are in /usr/lib/python2.7/lib-dynload
<arigato>
if cpython 3.x has got the same behavior, then indeed we can also replicate it by doing the same trick
<mattip>
and 3.6 so modules are in /usr/lib/python3.6/lib-dynload
<arigato>
if we do, we could write a test too, which checks somehow that all module names enabled by default have the string "pypy" in them or are listed in cpython's sys.builtin_module_names
<arigato>
with special cases and exceptions
<arigato>
(like for math on python 2.7 if we're running py.test with a non-ubuntu cpython 2.7)
<mattip>
we could hard code the list of pseudo c-extensions: MODULES_THAT_NEVER_SHADOW in addition to MODULES_THAT_ALWAYS_SHADOW
<mattip>
their union would be sys.builtin_module_names
lritter has quit [Remote host closed the connection]
<arigato>
(ok, we can just move the list MODULES_THAT_ALWAYS_SHADOW in the test I was talking about, and check that all enabled modules either start with "_pypy" or "__pypy__" or are in that list)
<arigato>
(the point being, I want to avoid the case where a new module "foobar" I've never heard about is added to CPython 3.n+1, and we make it a built-in in pypy because that's the default behaviour, without failing any test)