cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "nothing compares to the timeshifter, my personal polar expedition in software" - pedronis
<nedbat>
Alex_Gaynor: thanks for the implicit support in the commit message :)
<Alex_Gaynor>
nedbat: previous iterations of that commit message exhibited more frustration
<nedbat>
Alex_Gaynor: i remain baffled that people argued against disabling the optimizer
<Alex_Gaynor>
nedbat: Our situation is particularly exhausting because it's flaky, because pypy's optimizer does _not_ affect coverage, but our coverage merger apparently sometimes loses's pypy's coverage
<nedbat>
coverage merger?
<Alex_Gaynor>
codecov.io, it receives all of our test job coverage reports nad merges them, but it has bugs
<nedbat>
i see
<Alex_Gaynor>
Anyways, yes, very frustrating and I can't believe folks have argued against it for 9 years
<kenaan>
antocuni cpyext-avoid-roundtrip ccf12107e805 /pypy/module/cpyext/: WIP: copy the tupleobject freelist logic from CPython, and whack around until we can use it from ...
<kenaan>
antocuni cpyext-avoid-roundtrip a79aad3bfd78 /pypy/module/cpyext/test/test_api.py: add a proper repr for this class, else its repr() is StdObjSpace, which is veeeery confusing. Hoo...
<kenaan>
antocuni cpyext-avoid-roundtrip aab2478d4f85 /pypy/module/cpyext/test/test_tupleobject.py: call PyTuple_New from here
adamholmberg has quit [Remote host closed the connection]
arigato has joined #pypy
arigato has quit [Ping timeout: 255 seconds]
arigato has joined #pypy
TheAdversary has joined #pypy
yuyichao has quit [Ping timeout: 255 seconds]
<kenaan>
arigo unicode-utf8 cb0586abb276 /pypy/interpreter/: Implement and test these
arigato has quit [Quit: Leaving]
yuyichao has joined #pypy
cstratak has quit [Quit: Leaving]
<nedbat>
When i run my test suite under pypy3, I get random ResourceWarnings about unclosed files. They appear in different places on different runs. Once it was on this line: https://github.com/nedbat/coveragepy/blob/master/coverage/python.py#L28 . Is there a way to make these warnings reliable? Should I just disable them?
lritter has joined #pypy
cstratak has joined #pypy
raynold has joined #pypy
kolko has quit [Ping timeout: 260 seconds]
arigato has joined #pypy
<kenaan>
arigo unicode-utf8 f7ce6bd07d6a /pypy/objspace/std/listobject.py: Re-enable this
oberstet has quit [Ping timeout: 264 seconds]
mvantellingen has quit [Read error: Connection reset by peer]
mvantellingen has joined #pypy
arigato has quit [Ping timeout: 258 seconds]
tnorth has joined #pypy
arigato has joined #pypy
arigato has quit [Client Quit]
<gsnedders>
nedbat: I presume not, given they happen when GC is run and the GC is AFAIK non-determinsitic?
<kenaan>
arigo unicode-utf8 28266792f75a /pypy/module/_codecs/interp_codecs.py: Re-enable this
<kenaan>
arigo unicode-utf8 4ab3528a9b0c /rpython/rlib/runicode.py: Revert runicode to be exactly like in default
<kenaan>
arigo unicode-utf8 6c1d3144913e /pypy/module/_codecs/interp_codecs.py: Another fix by reverting this piece of code to default
<nedbat>
gsnedders: how did you determine that they happen when the gc is run? Seems like the message would be about an object being reclaimed while open, but these can't be being reclaimed yet, right?
<gsnedders>
nedbat: my memory is the warnings happen when the object's destructor is called and the underlying resource isn't closed
<gsnedders>
nedbat: and you can definitely reclaim objects where the underlying resource is still open; consider def f(): x = open("foo"). after calling f, the file object can be reclaimed as its unreachable despite the resource still being open, and you'd get a wraning then
<gsnedders>
nedbat: (note CPython also uses the object's destructor for this, but because of refcounting it's mostly pretty predictable when the warning is triggered)
<nedbat>
gsnedders: I don't know how to know if it's the same fd: ResourceWarning: unclosed file <_io.FileIO fd=11 mode='rb' closefd=True>
<nedbat>
gsnedders: it's not 0, 1, or 2, so it's something that the program opened i guess.
<gsnedders>
LarstiQ: that's all consistent with it being triggered by the GC, at least
<LarstiQ>
gsnedders: hmm, could be.
<nedbat>
gsnedders, LarstiQ: what can I do to help track this down?
<nedbat>
btw, this is PyPy 3.5.3 (pypy 5.9.0.beta.0).
<gsnedders>
nedbat: do you get a similar warning on CPython BTW?
<nedbat>
gsnedders: no
<LarstiQ>
and I don't suppose pypy2 is helpful here? (Does it even have ResourceWarnings?)
<gsnedders>
(Not for unclosed files)
<nedbat>
today, it looks like every pypy3 run produces a warning, but in different places each time.
<exarkun>
Running with strace would probably tell you which file the warning relates to
<exarkun>
(unless running with strace changes the behavior so as to eliminate the warnings)
<nedbat>
gsnedders: seems like it's always fd 10 or 11
<LarstiQ>
pypy/module/_io/interp_fileio.py has an 'unclosed file', the other two possibly relevant ones are in lib-python/3/asyncore.py and lib-python/3/_pyio.py
<LarstiQ>
actually, pypy2.7 has -X track-resources it seems
<nedbat>
LarstiQ: looks like pypy2 doesn't have ResourceWarning, just like CPython 2 doesn't.
<LarstiQ>
nedbat: could you try with -X track-resources?
* LarstiQ
thinks it's W_FileIO._dealloc_warn_w
<nedbat>
LarstiQ: is there a way to do that with an envvar?
jacob22 has joined #pypy
<LarstiQ>
nedbat: does not look like it. It calls `sys.pypy_set_track_resources(True)` so you could try that by hand. But if you're asking for an env var, not sure that is easier
* nedbat
tries it
jacob22_ has quit [Ping timeout: 248 seconds]
* LarstiQ
confirms that generates resource warnings in pypy2 console
<nedbat>
LarstiQ: setting that seems to generate no warnings for me in pypy2
<LarstiQ>
nedbat: run a gc.collect()
<nedbat>
LarstiQ: where/when?
<LarstiQ>
nedbat: don't know if there is a particular good spot, but at least once to force the reaping of unreferenced open files
<nedbat>
LarstiQ: i added a .collect() after each test, still no warnings on pypy2
<LarstiQ>
nedbat: ok, starting to sound like a pypy3 specific issue
<nedbat>
LarstiQ: fun: pypy3 doesn't have that sys. entry
<LarstiQ>
nedbat: so at this point I feel comfortable labelling it a pypy3 bug, and my next step would be to try and get it reproducable enough for a bug report
<nedbat>
LarstiQ: i tried using the warnings module to turn the warning into an error. No warning! :(
<nedbat>
LarstiQ: is "run the coverage.py test suite" reproducible enough? Or do we need to make it smaller?
<LarstiQ>
nedbat: no need to make smaller afaik, just 'git clone ..; <command to run test suite>'
<nedbat>
LarstiQ: do you want me to write a bug? On bitbucket?
adamholmberg has quit [Remote host closed the connection]
<LarstiQ>
nedbat: or I can do that if you give me the raw ingredients
<pjenvey>
are you guys aware of -X track-resources
marky1991 has quit [Remote host closed the connection]
<LarstiQ>
pjenvey: yeah
<pjenvey>
(i haven't really read this conversation, just throwing it out there)
<nedbat>
pjenvey: we played with sys.track_resources_whatever(True)
marky1991 has joined #pypy
<LarstiQ>
nedbat: did we (you) try -X track-resources only on pypy2 or pypy3 too?
<LarstiQ>
in case it does more than what cpython defines
<nedbat>
LarstiQ: i wasn't sure how to use -X through tox/etc, so i used the sys function, which isn't on pypy3
<LarstiQ>
ah right
* LarstiQ
looks at the pypy3 code
<LarstiQ>
hmm, no code hits
<syntaxman>
As it turns out the OOM killer killed it. Is this even avoidable? (4GB of RAM)
<LarstiQ>
syntaxman: maybe with enough swap, though you're trading that for time
pilne has joined #pypy
marr has quit [Ping timeout: 248 seconds]
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
cstratak has joined #pypy
cstratak has quit [Remote host closed the connection]
tbodt has joined #pypy
tbodt has quit [Client Quit]
tbodt has joined #pypy
antocuni has joined #pypy
mattip has joined #pypy
jamesaxl has quit [Read error: Connection reset by peer]
jamesaxl has joined #pypy
forgottenone has joined #pypy
antocuni_ has joined #pypy
antocuni has quit [Ping timeout: 240 seconds]
mattip has quit [Ping timeout: 255 seconds]
marky1991 has quit [Ping timeout: 240 seconds]
marky1991 has joined #pypy
tnorth has quit [Ping timeout: 246 seconds]
<syntaxman>
I'm confused... terminal output stopped (assertion error) right after the c compile began. Pdb+ was killed. But top shows that the c compile is still going on.
adamholmberg has joined #pypy
adamholmberg has quit [Remote host closed the connection]