cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
<antocuni>
finally I am going somewhere in my quest to speedup cpyext tests on the py3.6 branch
<antocuni>
so, it seems that the major cause of the slowness is because we are importing half of the stdlib just to initialize the space
<mattip>
YannickJadoul: sounds about right. It should be just wget 7.3.0rc3; tar -xf; pypy -mensurepip;
<antocuni>
however, there is not a single source which triggers all these imports. I manage to greatly reduce the initialization (from ~55s to ~24s) only if I apply ALL these changes simultaneously:
<antocuni>
1. comment out "from pickle import Pickler" in cpyext/moduledef.py
<antocuni>
2. comment out "preload_builtins" in test_cpyext.py:LeakCheckigTest (because it trigger "import types", I think)
<antocuni>
3. use _dummy_importlib intead of _frozen_importlib
<mattip>
is there a way to to (1) differently or avoid it in "not runappdirect", and do it only for pickle tests?
<antocuni>
I think that the proper way would be to do the change inside pickle.py itself
<mattip>
with an "if config.with_cpyext" or so
<antocuni>
and for (2), probably we can find a way to get types.FunctionType etc without having to do the full "import types", which on py3k triggers the import of e.g functools and collections.abc :(
<antocuni>
mattip: where do you want to put "if config.with_cpyext"?
<mattip>
pypy/module/_pickle_support
<YannickJadoul>
mattip: Thanks! I'm trying to test it out on the cibuildwheel PR that will soon add PyPy, though. So I probably need to fork the manylinux repo
<antocuni>
mattip: yes, or maybe directly inside pickle.py, probably with a "try: import cpyext" guard
<mattip>
YannickJadoul: what about Dockerfile that starts "FROM quay.io/pypa/manylinux2010_x86_64:latest"
<mattip>
antocuni: somehow mixing cpyext and lib-python leaves a bad taste, but I did it in datetime.py as well :(
Rhy0lite has joined #pypy
<antocuni>
I agree, but on the other hand also the current code in cpyext.Module.startup looks like a hack, since it changes the global state of another module :(
ronan has joined #pypy
adamholmberg has joined #pypy
<mattip>
I think the code in cpyext.Module.startup looks like the code in pypy/module/_pickle_support/maker.py, but I am not sure
<antocuni>
mattip: I don't think so. It seems that maker.py is about unpickling, while Pickler.dispatch is about pickling
<YannickJadoul>
mattip: Yes, I'll probably do that, and add 7.3.0rc3 to it. Just curious if there was a shortcut :-)
<mattip>
antocuni: ahh, right
<antocuni>
ok, this is even weirder. So, if I apply only (1) and (2) in the py3.6 branch, running a single test still takes ~55s
<antocuni>
if I apply (1) and (2) in the dummy-importlib2 branch BUT I disable _dummy_importlib, it takes ~24s
<kenaan>
cfbolz py3.7 f9a14309d6d1 /: start implementing PEP-0567 Python code is taken from there, pure Python apart from two functions in __pypy__ to re...
<kenaan>
cfbolz py3.7 4f0d7c92cfdd /: pure python HAMT immutable dict implementation from the immutables package taken from rev 11863b29e3fbcd7d25335befc...
<kenaan>
cfbolz py3.7 4f3a9b9e4db1 /lib_pypy/_contextvars.py: use the _immutable_map.Map implementation in the _contextvars module
<kenaan>
mattip release-pypy2.7-v7.x 724f1a7d62e8 /: merge default into release
<kenaan>
mattip default 0e6c068722c0 /.hgtags: Added tag release-pypy3.6-v7.3.0rc4 for changeset 1608da62bfc7
<kenaan>
mattip default 16369979c2f9 /.hgtags: Added tag release-pypy2.7-v7.3.0rc4 for changeset 724f1a7d62e8
<antocuni>
there is no need to go through imp, because it ends up calling create_extension_module anyway
<antocuni>
so, at the end of the day, when I avoid importing "imp", "types" and "pickle", running a cpyext test takes ~23s, only 3s more than on default
<antocuni>
more generally speaking, I wonder whether we should try harder to avoid importing modules from lib-python during tests
<antocuni>
like, disabling it by default unless you enable an option
<antocuni>
although the problem is that certain things like encodings needs to be imported anyway :(
<ronan>
yes, inadvertently importing from the stdlib is the cause of a lot of avoidable slowdowns
<antocuni>
how essential is to import "encodings"?
<antocuni>
can we think of writing an inter-level fake replacement for tests which don't need weird encodings?
<ronan>
I'm not sure
<ronan>
some encodings have interp-level implementations
<ronan>
but IIRC, the encodings module is needed on windows
<antocuni>
well, speeding up tests on linux would be already very good :)
<ronan>
yes
<ronan>
I think CPython have also added hacks recently to avoid importing encodings
<Alex_Gaynor>
(SpaceCompiler is such a cool name)
<antocuni>
ronan: why? For performance as well?
<ronan>
yes
<antocuni>
Alex_Gaynor: somehow I think of it as a compiler whose source language consists only of whitespaces :)
<antocuni>
how good is mercurial at handling cherry-picking across branches? If I develop some code against py3.6, then I cherry-pick to default, will we have problems at the next merge?
<ronan>
usually, it's fine, but I typically do a merge from default to py3.6 immediately after pushing both commits just to be sure
<ronan>
btw, CPython have added an "optimisation" to bypass importlib in common cases in 3.6
<antocuni>
:facepalm:
<antocuni>
so basically, first they moved a lot of logic to applevel, then now they are undoing it one piece at a time?