cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
<kenaan>
antocuni gc-hooks ba293bc808ee /pypy/module/gc/: start to add app-level gc hooks; the test pass, but we still need to polish stuff a bit
jamesaxl has quit [Read error: Connection reset by peer]
jamesaxl has joined #pypy
energizer has quit [Ping timeout: 256 seconds]
marr has quit [Ping timeout: 256 seconds]
AndrewBC has quit [Ping timeout: 255 seconds]
<kenaan>
antocuni gc-hooks def5c83256cc /rpython/memory/gc/: move the if gc_*_enabled checks inside the fire_* methods
<kenaan>
antocuni gc-hooks e854ddff70bd /rpython/memory/: turn gc_*_enabled from attributes into overrideable methods. This is needed because the normal rpython code CAN...
<kenaan>
antocuni gc-hooks 91b33ec9b345 /pypy/module/gc/: enable/disable the low-level hook according to the value of w_on_gc_minor
antocuni has quit [Ping timeout: 240 seconds]
marr has joined #pypy
<mjacob>
arigato: i'm not 100% sure, there are some other, possibly related problems
<looCfur__>
I try so hard to love python but it just has to be so difficult. Then there is pypy, that almost fixes the problems with python except things like numpy aren't compatible with pypy3... so you're stuck being limited no matter what you try to do
<cfbolz>
looCfur__: numpy works with pypy3, afaik
<looCfur__>
really? the documentation said it didn't so I didn't even try
<looCfur__>
well, it said it was buggy TBH
dddddd has joined #pypy
<cfbolz>
looCfur__: I would try it and see
marr has quit [Ping timeout: 276 seconds]
drolando has quit [Remote host closed the connection]
drolando has joined #pypy
antocuni has joined #pypy
<kenaan>
antocuni gc-hooks 5bf5fddf26f6 /pypy/goal/targetpypystandalone.py: blindly try to use the new gchooks in the pypy target
<cfbolz>
lazka: heh. That's a lot less laggy in CPython, I assume?
<kenaan>
antocuni gc-hooks ce5745894545 /pypy/module/gc/: pass a 'stats' object to the hook instead of passing all the values individually
<kenaan>
antocuni gc-hooks 6d1d852ff79f /pypy/module/gc/hook.py: try to fix annotation
lazka has quit [Ping timeout: 240 seconds]
<fijal>
I'm incredibly impressed cpyext works at all tbh
<kenaan>
cfbolz default a145a9a4581b /pypy/module/pypyjit/: like this, I hope
<cfbolz>
fijal: always the pessimism :-P
jamesaxl has quit [Read error: Connection reset by peer]
<fijal>
yeah sure
<fijal>
aren't you?
<fijal>
you would have said it's impossible 10 years ago :-P
jamesaxl has joined #pypy
<cfbolz>
fijal: ten years ago even the jit was only barely possible ;-)
jcea has quit [Quit: jcea]
mattip has joined #pypy
<arigato>
some more serious snow
energizer has joined #pypy
<cfbolz>
:-)
raynold has quit [Quit: Connection closed for inactivity]
<mattip>
arigato: I think what i am seeing goes like - there is c-api code that allocates a buffer,
<mattip>
then does multiple "buf = x; y = x; x = buf" calls (all still in c-api
<mattip>
) to allow memory overlap btwn x,y
<mattip>
the buffer is modified via pure python code to have an additional attribute (mask)
<mattip>
during allocation
<mattip>
the w_obj representation of buffer is collected at some gc call during the loop of multiple calls,
<mattip>
so when the w_obj is reconstructed from the buffer's pyobj, the mask attribute is missing
<mattip>
</spam>
* mattip
trying to write a test using the cpyext/test/arrary.c module
<arigato>
unclear why the w_obj would be collected by the GC if the pyobjects are still alive
<mattip>
it there specific code to check tp_refcnt > "0" (for our equivalent of 0) and prevent collection?
<mattip>
s/it/is
<mattip>
s/tp_refcnt/ob_refcnt/
<mattip>
from what I can see in rpython/memory/gc/incminimark.py, _rrc_minor_free(), there is a specific case to make ob_pypy_link NULL
<mattip>
if the refcnt > "0"
asmeurer_ has joined #pypy
<arigato>
rrc_minor_trace() first takes all young pypy objects with an associated pyobject
<arigato>
and if the refcnt is not exactly 0 (plus the REFCNT_FROM_*_PYPY flag)
<arigato>
then it marks it
<arigato>
which means rrc_minor_free() will then see it as surviving
<mattip>
.. and if it is not young (assuming that can happen)?
<arigato>
mattip: maybe try to comment out "def _cpyext_attach_pyobj", i.e. don't overwrite "cls._cpyext_attach_pyobj"
<arigato>
there might be something wrong here
<arigato>
it's an optimization which allows the pypy object to be freed even though the pyobject side is still there
<mattip>
ahh, no only the young list goes to _rrc_minor_free in rrc_minor_collection_free
<arigato>
it may be the case that, precisely, Python-defined subclasses of C-defined classes get to use _cpyext_attach_pyobj() too, even though they should not
<mattip>
ok
<arigato>
ah no, wait
<arigato>
you need to keep the function, but inside it, call rawrefcount.create_link_pypy() instead of rawrefcount.create_link_pyobj()
<arigato>
or else, comment out the whole function add_direct_pyobj_storage() and its 4 callers just afterwards
* mattip
reading the code and trying to udrestand the code paths
<arigato>
...yes, that looks like it might give the behaviour you're describing
<mattip>
w_obj._cpyext_attach_pyobj is called in track_reference, which is called to tie a new pyobj to a w_obj
<arigato>
also, I suspect the three lines in cpyext/pyobject.py:131-133 (on default) are not needed, because track_reference() would do it already
<arigato>
mattip: yes. the fact that create_link_pyobj() is *ever* called is an optimization, which might be applied too broadly here
<mattip>
cool, that gives me the hook I was looking for to progress
<mattip>
still not clear to me how the object ends up in the young list
ceridwen has quit [Ping timeout: 240 seconds]
<arigato>
likely because it's just freshly allocated
<arigato>
it goes away because cpyext thinks it's not needed to keep the pypy version
<arigato>
which is true for purely CPython-C-type objects with no data in the pypy version
<arigato>
should correctly use create_link_pyobj() only on the exact classes, not on subclasses
* mattip
trying
<mattip>
thanks
ceridwen has joined #pypy
<arigato>
...ok, that doesn't really work as we'd like
<arigato>
but continue trying it out
<arigato>
if it fixes the problem it should tell us we've understood why
<arigato>
the problem looks related to some pypy issue we have that complains that if 'x' is of type 'some_type_from_cpython_api', then x has got a __dict__
<arigato>
that means that we can't use create_link_pyobj() right now because otherwise whatever is stored in this __dict__ disappears mysteriously
<arigato>
(which we do so far, so whatever you store in the unexpected '__dict__' *will* mysteriously disappear sometimes)
<arigato>
I think it's exactly like what you're seeing, where an attribute of the app-level subclass mysteriously disappears
<antocuni>
arigato: FWIW, our plan for triggering GC hooks doesn't work
<antocuni>
the problem is that AsyncAction.fire() can allocate (because it appends to self.fired_actions), so I cannot call it directly from the hook
<arigato>
ah
<kenaan>
antocuni gc-hooks f261ec7406c6 /pypy/module/gc/hook.py: this seems to fix annotation; however, it still cannot translated because AsyncAction.fire() can allocate, and ...
<arigato>
well, time to hack a little bit
<antocuni>
one possible way is to use a fixed-size list I suppose?
<arigato>
or a bitfield of integer bits
<antocuni>
arigato: each field indicating whether "action N" needs to be triggered?
<arigato>
ah no, just use a linked list instead of the python list
<antocuni>
ah, good idea; so each action would contain a reference to its next
<arigato>
I think it should be safe to manipulate that linked list in no-gc code, because it's anyway all prebuilt stuff
<arigato>
yes
<antocuni>
ok, I'll try
<antocuni>
a bit annoying, but too bad :)
<arigato>
probably means we have the same problem in cpyext's dealloc_trigger()
<arigato>
yes
inhahe_ has joined #pypy
<antocuni>
ah, because if we trigger the collection in the middle of PyObjDeallocAction.perform, we are in an inconsistent state?
<arigato>
no, in the middle of .fire()
<arigato>
because it also calls append()
<arigato>
so I guess in practice it can never occur that just after a minor collection, the small amount of memory needed for .append() causes troubles
<antocuni>
ah, and fire() is called by the rawrefcount machinery, I suppose
<arigato>
yes, immediately after a minor collection too, like you do
<antocuni>
well, in my case it DOES create troubles because when you do an incremental step, you first do a minor collection
inhahe__ has quit [Ping timeout: 256 seconds]
<arigato>
so indeed we should also add @no_collect here...
<antocuni>
so if the hook allocates, the collect-step expects the nursery to be empty but it's not
<arigato>
yes
<arigato>
ah no
<antocuni>
(I know this because an assert failed when running tests untranslated :))
<arigato>
there's only the fire() here, right?
inhahe has quit [Ping timeout: 256 seconds]
<antocuni>
where?
<arigato>
at the end of minor collections
<arigato>
in your branch
<antocuni>
yes, a couple of setfields and a fire
inhahe has joined #pypy
<arigato>
...ah, and an untranslated test fails because of only the extra list allocated
<arigato>
good, tests :-)
<antocuni>
no, unfortunately this precise test does not fail untranslated (mostly because I don't think I can trigger proper gc hooks inside applevel tests, so I fake them)
<antocuni>
but I had another test failing for that, so I added @rgc.no_collect