cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
<jredondo>
dash: ImportError: No module named 'psycopg2._psycopg'
<jredondo>
Same for plyvel._plyvel
<dash>
jredondo: yes, for psycopg you can use psycopg2cffi
<jredondo>
dash: Yes, thanks. My main problem is now plyvel.
<jredondo>
dash: Actually with psycopg2cffi it imports as expected so far. But what could I do with plyvel?
<dash>
not sure there's a plyvel version using cffi, you might look into leveldb-py or such
jamesaxl has quit [Read error: Connection reset by peer]
jamesaxl has joined #pypy
amaury__ has joined #pypy
LooCfur has joined #pypy
<LooCfur>
so I was annoyed that my python 3 code was so slow, did some googling and stumbled on this. What takes minutes in python 3 is taking under 2 seconds in pypy3
<LooCfur>
what in the world is wrong with vanilla python that it's so retardedly slow?
<LooCfur>
if you guys can python so much faster, why isn't python even trying? I don't get it
<LooCfur>
make
<simpson>
Yep.
<LooCfur>
simpson, do you know what's going on with that? I honestly can't make sense of it
<hawkowl>
okay so I still don't understand what is holding on to this frame
TheAdversary has quit [Ping timeout: 256 seconds]
inad922 has joined #pypy
<kenaan>
Raemi guard-compatible 84015df039c9 /rpython/jit/metainterp/optimizeopt/unroll.py: fix missing part of a previous commit 931f5eaed82f Apparently we missed another place where ResumeAtPosit...
oberstet has joined #pypy
<cfbolz>
Remi_M: wow
<cfbolz>
I'm impressed
<cfbolz>
I can take care of the test failures, if you want
<cfbolz>
Remi_M, arigato: can I ask how you managed to find it?
<Remi_M>
cfbolz: looking at trace trees and seeing that the bridges connected to the wrong places
<Remi_M>
so now, on richards, guard_compatible is 20% slower than some default version of pypy
<Remi_M>
we're going for a hike soon, so if you want to fix tests while I'm away, feel free :)
<cfbolz>
Ok
<Remi_M>
also arigato had suspicions about the disabled value-profiling for guard_compatible. That (and possibly suboptimal assembly ) may account for the 20% slowdown
<cfbolz>
Right
<cfbolz>
Because he suspects there are clashes?
<hawkowl>
I have a nasty feeling some frames are holding things and not letting it get GC'd
<cfbolz>
It's possible to re-enable the value profiling, but needs extra cleverness
<cfbolz>
Remi_M: anyway, that was already a huge help, I tried to find this bug for days and days already
<Remi_M>
I think Armin's concern was that when a counter for a value reaches the compilation-threshold, then maybe other counters should be reset so that we are not compiling too many bridges too fast. but it's not clear to me right now
<hawkowl>
hence holding on to arguments in threads forever
<cfbolz>
hawkowl: right, that would indeed be my suspicion, that some code in the thread module is just not careful at all
<hawkowl>
hence both not GCing them ever and also failing twisted's test suite
<hawkowl>
and it doesn't look like I can reach in to delete the frame?
* Remi_M
-> lunch
<hawkowl>
if I can find where it's storing it, I can hack around it
<cfbolz>
hawkowl: where do you see it creating the frame?
dddddd has joined #pypy
<hawkowl>
but I think this is actually a symptom of a memory leak
<cfbolz>
(warning, I don't know this code at all)
<hawkowl>
cfbolz: I don't see it creating one, but it never releasing one
<hawkowl>
as in
<hawkowl>
I don't know where in this code it picks up the frame
<cfbolz>
so how do you know it's this file?
<hawkowl>
but I know it's holding it
<hawkowl>
so referants to the object not freed includes a frame, which still has it in the context for some reason
<hawkowl>
referants to that frame are thread._local
<hawkowl>
so either I am barking up the craziest tree
<hawkowl>
or I don't understand how gc works
<hawkowl>
..but weirdly
<hawkowl>
it says it is referred to by... a dict? and also a tuple
<cfbolz>
hawkowl: can you manage to make a small file that runs quickly that shows the problem? if yes, you could try to run it on an untranslated pypy
<hawkowl>
so maybe the thread local is a red herring
<hawkowl>
let me see if i can reduce it to not use twisted
<cfbolz>
that would be super helpful
<cfbolz>
fwiw, my gut feeling still says that you are right and thread.local is at fault
<cfbolz>
hawkowl: hm, all this occurs after some thread finished, right?
illume has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<hawkowl>
well it's supposed to yeah
<hawkowl>
but even if i stop and join the threads, doesn't help
<hawkowl>
oh what
<hawkowl>
this is a race condition
<cfbolz>
?
<hawkowl>
this might be an assumption by the test
<hawkowl>
yeah it is
<hawkowl>
it's a GC bug
<hawkowl>
it relies on the GC running after it's freed
<hawkowl>
it's supposed to have a test saying "oh this works here" by dumping the result of the weakrefs
<hawkowl>
but of course we haven't got rid of it
<hawkowl>
...and so it creates a new reference.
<hawkowl>
gc.collect() in the thread callback does it
Remi_M has quit [Quit: See you!]
Remi_M1 has joined #pypy
<hawkowl>
this is a dumb test
Remi_M1 has quit [Client Quit]
Remi_M has joined #pypy
Hasimir has joined #pypy
<hawkowl>
a dumb test with gc assumptions
<hawkowl>
okay so there isn't a leak there, cfbolz
<hawkowl>
at least here
Remi_M has quit [Client Quit]
<cfbolz>
hawkowl: ok
<hawkowl>
cfbolz: thanks
<cfbolz>
hawkowl: it was just rubber duck services ;-)
<hawkowl>
cfbolz: I mean, yes
<hawkowl>
but you also made me inspect _thread.local more
<cfbolz>
👍
jamesaxl has quit [Read error: Connection reset by peer]
Hasimir has quit [Ping timeout: 248 seconds]
lritter has joined #pypy
jamesaxl has joined #pypy
<hawkowl>
cfbolz: btw this is the only real test failure on pypy3
<hawkowl>
so next twisted release may support pypy3 as a supported platform
<cfbolz>
awesome!
<hawkowl>
...not pypy2.7 yet...
<cfbolz>
hawkowl: thanks for doing this!
<hawkowl>
pypy3 doesn't have the gnarly trash that pypy2.7 has to run
<hawkowl>
in twisted, that is
<hawkowl>
cfbolz: thanks for pypy3!
<hawkowl>
i am real excited
<hawkowl>
*leaves CI to run for six thousand years*
raynold has quit [Quit: Connection closed for inactivity]
illume has joined #pypy
codehotter has quit [Ping timeout: 264 seconds]
illume has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<kenaan>
cfbolz guard-compatible 7825277c4910 /pypy/objspace/std/: re-enable support for immutability tracking of attributes of user-defined instances. This works by piggyb...
<kenaan>
cfbolz guard-compatible d8319beb1fe4 /rpython/jit/metainterp/test/test_compatible.py: change the test slightly to show the real problem: calls aren't getting removed in bridges
codehotter has joined #pypy
jcea has joined #pypy
Jonno_FTW has quit [Remote host closed the connection]
pf_moore has joined #pypy
TheAdversary has joined #pypy
Hasimir has joined #pypy
Rhy0lite has joined #pypy
inad922 has quit [Ping timeout: 245 seconds]
jredondo has joined #pypy
inhahe has joined #pypy
illume has joined #pypy
inhahe has quit [Ping timeout: 268 seconds]
orion has joined #pypy
orion is now known as Guest30818
marky1991 has joined #pypy
marky1991 has quit [Ping timeout: 240 seconds]
amaury has joined #pypy
mcyprian has joined #pypy
marky1991 has joined #pypy
illume has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]