<fling>
need to rebuild with cpython afterwards :<
<fling>
emerge[30029]: segfault at 0 ip 00007f42ca464ecd sp 00007ffc6e731f40 error 6 in libpypy-c.so[7f42c8900000+2a6c000] Code: b4 00 eb ab 90 66 48 8d 3d 38 41 12 01 66 66 48 e8 c8 20 93 ff 48 8b 3d f1 40 12 01 48 89 ee 48 8b 40 38 48 89 05 23 0f 7e 02 <4c> 89 23 e8 db 43 93 ff 4c 8b 23 48 89 1d 99 ba 30 01 48 8b 1d 1a
ricekrispie2 has left #pypy ["Leaving"]
<mattip>
fling: is there a stacktrace (please use a pastebin)?
<fling>
mattip: no, idk how to get one.
<mattip>
back up a step, what do you call "update it with itself" - what version did you have and to which did you update?
<mattip>
this is gentoo?
<fling>
7.0.0 -> 7.1.0 ; yes, gentoo.
<mattip>
on 64 bit x86?
<fling>
64
<arigato>
fling: unless you can extract from the gentoo system which exact command was run, with what version of pypy, ending in the segfault, then I'd say your best bet is to report it to gentoo first
<fling>
arigato: /usr/lib/python-exec/pypy/emerge with pypy 7.1.0
<arigato>
yes, but that's what I mean: we don't know what this command does, because we don't have a gentoo system at hand
<fling>
emerge for ebuild is like apt for dpgk
<arigato>
I know, but you have to realize that for us it's like saying "I run pypy on a program that you don't know and I get a segfault with no more information"
<arigato>
what we need is either for someone to extract from "emerge" a small reproducer, i.e. a small Python program that when run segfaults; or else to have access to a linux machine with everything installed just right so that "emerge pypy" there segfaults
<fling>
So what program to run? :P
<fling>
Which one you know? I could install anything I think
<arigato>
unsure what you're asking, but I'm saying, we need a machine set up so that typing "emerge pypy" segfaults, and then we need ssh access to that machine (with gdb etc. but I guess that's common on gentoo)
darkman66 has joined #pypy
_whitelogger has joined #pypy
<fling>
arigato: mattip: looks like deps break and need to be rebuild after each pypy upgrade unlike with cpython.
<fling>
I'm trying to fix this with `eix -IUc python_targets_pypy --only-names | parallel -j32 --lb /usr/lib/python-exec/python3.6/emerge -1Oq`
<arigato>
as I said, you should ask on the gentoo channel
<arigato>
I have no clue what this does
<fling>
I'm just keeping you informed ;P
<mattip>
arigato: any thoughts about windows dll resolution? While the microsoft rules are clearly outlined
<mattip>
I don't see what changed between the python2 releases
<mattip>
hmm, when did I convert to using the externals subrepo, and recompiled all the dependencies?
<arigato>
I also failed to find the difference
<arigato>
I guess we'll need to bisect, downloading many versions of the pypy-c-win32 builds and hoping there is no big gap just where it matters
<mattip>
externals was part of the msvc14 branch merged 496d05d4758e in feb 2018
<arigato>
v6.0.0 is later, April 2018
<arigato>
and still works
<mattip>
good, I guess
<arigato>
makes little sense, because "The directory from which the application loaded." is the first listed search directory, and as far as I can tell that should be the directory containing pypy.exe in all cases
<mattip>
maybe by "application" they mean the pyd
<arigato>
I think not
<arigato>
I have no clue
lritter has joined #pypy
<mattip>
in 80ba133be397 I added LoadLibraryEx, which has different search rules
<mattip>
then started using it in 8894e5f1d5c0
<arigato>
ah, and the dates make it look like a likely cause
<arigato>
progress
<arigato>
ah and we pass LOAD_WITH_ALTERED_SEARCH_PATH
<arigato>
the flag
<mattip>
yup
<mattip>
so we are using "SafeDllSearchMode"
<mattip>
(which is what cpython does)
<arigato>
right
<arigato>
indeed, this flag no longer looks at "The directory from which the application loaded"
<arigato>
good
<arigato>
as in, good, it makes sense
<arigato>
we need to fix it somehow
<mattip>
package.py can put the dlls in lib_pypy
<mattip>
which is more compliant with what Microsoft (at least according to the issue I linked) recommends
<arigato>
yes, but it's still a backward-incompatible break
<arigato>
I would try to use not LoadLibraryEx but SetDllDirectory
<arigato>
because this gives a search order with both "The directory from which the application loaded",
<arigato>
and then a user-specified directory
<mattip>
that breaks os.environ['PATH'] for all subsequent users, so will break numpy, scipy, tensorflow, ...
<mattip>
and will be the default in cpython 3.8
<arigato>
ah, but SetDllDirectory also causes "." to no longer be searched
<mattip>
so by then all those c-extension projects will need to adapt
<arigato>
I'm unsure what you mean with os.environ['PATH']
<fling>
arigato: mattip: rebuilding things depending on pypy helped. No errors anymore.
<mattip>
once you use SetDllDirectory, it freezes the os.environ['PATH']. Further modifications are not searched
<arigato>
where do you see that? it seems to me that they are still searched normally
<mattip>
or something like that. Anaconda tried using it and broke numpy, so we were getting many complaints until they reverted it
<arigato>
OK
<arigato>
well it seems to be a cffi-only issue, so we should look at what occurs with cffi on CPython too
<arigato>
ah no, it's not only cffi but also C extension modules on pypy
<arigato>
I think that CPython works the same way for both, too
<arigato>
I'm trying not to include any PATH issues in this, because I *think* it's a problem that is independent
* mattip
off, bbl
<arigato>
CPython 2.7 uses LoadLibraryEx(LOAD_WITH_ALTERED_SEARCH_PATH), just like pypy does now
<mattip>
for cffi, ctypes, and c-extension loading
<arigato>
yes. so let me check what occurs with CPython and cffi on Windows
<mattip>
sorry, gotta go
<arigato>
sure
<arigato>
so no: CPython finds a dll located in the same directory as python.exe, even when running from somewhere else and importing a cffi module that is also somewhere else
<arigato>
CPython 2.7.13 at least does
<arigato>
ah no, my python.exe is also in PATH
<arigato>
OK even if it's not, it behaves like that
<arigato>
I HAVE NO CLUE
<arigato>
there is some extra Windows magic around LoadLibraryEx in CPython. I think it may be about adding the path of python27.dll to the DLL search path
<arigato>
OK so with CPython it seems to always look in the python.exe and python27.dll directory, at least I can't make it stop even if I don't understand why
<arigato>
at this point I'd also be fine if we give up and put "sqlite3.dll" inside "lib_pypy", and the same with "tk*.dll"
<arigato>
as far as I can tell, python 2.7 *looks* like it shouldn't look in its own directory, but it does at least for me, so maybe it's not really intended?
<kenaan>
rlamy jit-cleanup d35167b4bf4e /rpython/jit/metainterp/: kill cast_instance_to_base_ref, cast_to_ref and simplify some code
<kenaan>
rlamy jit-cleanup 4ee751fdcc36 /rpython/jit/metainterp/: More cleanup
darkman66 has quit [Remote host closed the connection]
forgottenone has quit [Quit: Konversation terminated!]
darkman66 has joined #pypy
darkman66 has quit [Ping timeout: 268 seconds]
antocuni has quit [Ping timeout: 246 seconds]
Zaab1t has joined #pypy
Zaab1t has quit [Client Quit]
oberstet has quit [Quit: Leaving]
lritter has quit [Ping timeout: 245 seconds]
marky1991 has joined #pypy
fling has quit [Ping timeout: 255 seconds]
fling has joined #pypy
darkman66 has joined #pypy
darkman66 has quit [Remote host closed the connection]
darkman66 has joined #pypy
glyph has quit [Quit: End of line.]
glyph has joined #pypy
<arigato>
mattip: tcl85.dll and tk85.dll are not found if they are put in lib_pypy. They need to go where the .pyd is, which is in lib_pypy/_tkinter
<mattip>
:(
<mattip>
thanks
<arigato>
also, no, cffi uses LoadLibraryEx too when loading pyd's
<arigato>
the code is shared with cpyext
<arigato>
cffi uses LoadLibrary only for dlopen(), which is different
<mattip>
in cpython?
<arigato>
I'm talking about pypy here, but let me check cpython
<arigato>
yes, on cpython too
<mattip>
hmm. Then I don't understand "CPython it seems to always look in the python.exe and python27.dll directory"
<arigato>
yes, I don't understand that either
<mattip>
unless it changes the flag
<arigato>
I don't think so
<mattip>
sorry, just making things up now. No idea
<arigato>
the flag is hard-coded inside the .c source of CPython calling LoadLibraryEx
<arigato>
but there is no way to be sure of anything here because it's a large mess, and we might miss something else set up by some other code
<mattip>
the PR for python3.8 to disallow searching PATH has been merged, and there are still no debug tools to determine the current search path
<mattip>
arigato: any thoughts about gc.get_objects ?
fling has quit [Ping timeout: 246 seconds]
fling has joined #pypy
marky1991 has quit [Ping timeout: 250 seconds]
<arigato>
no, we need to decide something
<njs>
mattip: arigato: a useful trick for working around DLL search issues: if you do LoadLibrary("/absolute/path/to/some.dll"), then that forces that exact file to be used for all *future* calls to LoadLibrary("some.dll")
<njs>
(within the same process)
<mattip>
njs: +1
<mattip>
sometimes you don't know what the name is, like MKL - which loads a dll per-architecture
<mattip>
so it used to be convenient to add the MKL bin dir to the PATH / LD_LIBRARY_PATH (dependent on OS)
<arigato>
I don't see how that helps, but thanks
darkman66 has quit [Remote host closed the connection]
<mattip>
arigato: if you know your pyd will need a dll, you can preload it into the exe
darkman66 has joined #pypy
<mattip>
but only if you know the exact name
<arigato>
I suppose, but that's unrelated to the fact that we're trying to fix a general problem
<njs>
I don't really understand the problem, but it sounded like you were struggling to get your own binary to load the dlls that you were shipping alongside it
<arigato>
no, this was a symptom of a difference when cffi's .pyd files are linked with more .dll's
<arigato>
a difference between two versions of PyPy
<njs>
ah yeah, for loading .pyd's your only hope is to do exactly whatever cpython does
<arigato>
the real goal is to understand this difference, not hack around to make the current case work
<mattip>
that is the problem - what exactly is cpython doing? It succeeds to load a dll in the python.exe directory,
<mattip>
even though it, by our understanding, should not
darkman66 has quit [Ping timeout: 245 seconds]
<mattip>
see "SafeDllSearchMode"
<njs>
can you copy/paste cpython's library-loading code into cpyext :-)
<arigato>
the code is already exactly the same
<njs>
odd
<njs>
this is about where I would usually start hassling steve dower
<arigato>
except cpython does more stuff we don't understand, somewhere else
<arigato>
or at least we guess so
darkman66 has joined #pypy
marky1991 has joined #pypy
infernix has quit [Ping timeout: 245 seconds]
infernix has joined #pypy
<mattip>
amazon reached out to me as a result of our ARM32 buildbot request for help, and have given me access to a arm64 machine
<mattip>
to try to build arm32 on. I set up a docker but the platform detection fails to correctly identify arm32
Ai9zO5AP has quit [Quit: WeeChat 2.4]
<cfbolz>
mattip: nice!
<mattip>
python -c "import platform; print platform.machine()" prints aarch64 so our rpython/translator/platform gets confused
<mattip>
I wonder how debian does it
<mattip>
tumbleweed: ^^^
<mattip>
I don't see any patches for arm32 platform detection
<njs>
(the kernel has a bunch of flags you can set, that tell it to lie to particular processes about what platform the kernel is. the setarch command lets you spawn a process with some of those flags set.)