cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
<sysfault>
is there anything someone who's looking to work on their python skills can do to contribute? something small i can do until my knowledge is strengthened
dddddd has quit [Remote host closed the connection]
AndrewBC has quit [Read error: Connection reset by peer]
AndrewBC has joined #pypy
AndrewBC has quit [Read error: Connection reset by peer]
forgottenone has quit [Quit: Konversation terminated!]
jamesaxl has joined #pypy
inhahe_ has joined #pypy
inhahe__ has quit [Ping timeout: 248 seconds]
<kenaan>
cfbolz default fe40f4eff666 /pypy/module/pypyjit/: test and fix: in the on_abort hook, the descrs of guards are still None as it runs before optimization
<cfbolz>
mattip (logs): David's ideas fixed the readline failures :-)
<cfbolz>
sysfault: something to do is always "run your favourite python project on pypy and see whether it crashes and how its performance changes"
<kenaan>
cfbolz pyparser-improvements dd9a56219993 /lib-python/2.7/test/test_genexps.py: adapt to new error message
illume has joined #pypy
wallet42 has joined #pypy
oberstet has joined #pypy
<antocuni>
do we have any special rpython decorator to assert that a certain function CANNOT allocate anything?
<antocuni>
uhm, maybe @rgc.no_collect
energizer has quit [Ping timeout: 240 seconds]
<kenaan>
antocuni gc-hooks ee3944e10c6c /: start a branch in which to implement app-level hooks to signal major (and possibly minor?) collections
<kenaan>
antocuni gc-hooks 7b2f669acfa4 /rpython/memory/gc/: add a hook interface to GC, which is implemented by a prebuilt object. Implement the hook for incminimark gc-minor
<kenaan>
antocuni gc-hooks 5fbf8b32c70c /rpython/memory/gc/: add a hook for gc-collect-done
<kenaan>
antocuni gc-hooks f16482adff0b /rpython/memory/gc/: add a hook for gc-collect-step
<kenaan>
antocuni gc-hooks 9bb5a3a3e292 /rpython/: WIP: hack around until we can pass a prebuilt instance of gchooks to the GC transformer, which will then forwar...
<kenaan>
antocuni gc-hooks e6a1a204ce7a /rpython/memory/gc/: make it possible to disable the gc hooks
<kenaan>
antocuni gc-hooks 5cb980c10b8c /rpython/memory/: add a translated test for gc hooks
antocuni has quit [Ping timeout: 260 seconds]
illume has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<mattip>
putting a watch on __pyx_v_buf->ob_pypy_link, it gdb stops in the GC with __pyx_v_buf->ob_pypy_link ==0
<mattip>
so it is collecting the w_obj even though the __pyx_v_buf->ob_refcount is REFCNT_FROM_PYPY + 5
illume has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
iko has quit [Remote host closed the connection]
marky1991 has quit [Read error: Connection reset by peer]
marky1991 has joined #pypy
TheAdversary has quit [Disconnected by services]
Hasimir has quit [Ping timeout: 248 seconds]
Hasimir has joined #pypy
Cheery_ is now known as Cheery
TheAdversary has joined #pypy
marky1991 has quit [Read error: Connection reset by peer]
antocuni has joined #pypy
marky1991 has joined #pypy
Rhy0lite has joined #pypy
<mattip>
so it seems this object is being erroneously collected
<mattip>
It is allocated via a c-api c function (np.empty_like(x) ) where x is an app-level class (MaskedArray)
<mattip>
the construction does not go through MaskedArray.__new__
<mattip>
but through a numpy mechanism
<mattip>
but is it ever correct for a w_obj with a pyobj and refcount > 0 (well, > REFCNT_FROM_PYPY) to be collected?
<antocuni>
mattip: I think it's not correct
<antocuni>
no idea why it is collected, but I suppose it should not
planrich has quit [Ping timeout: 256 seconds]
marky1991 has quit [Ping timeout: 240 seconds]
iko has joined #pypy
marky1991 has joined #pypy
TheAdversary has quit [Disconnected by services]
Hasimir has quit [Ping timeout: 248 seconds]
TheAdversary has joined #pypy
Hasimir has joined #pypy
dddddd has joined #pypy
wallet42 has quit [Quit: Connection closed for inactivity]
<kenaan>
antocuni gc-hooks 3d2b6d04026f /rpython/memory/gc/: call the hooks through helpers which are marked as @rgc.no_collect, to ensure that we cannot allocate anything ...
<kenaan>
antocuni gc-hooks 696e7ef11214 /rpython/: make it possible to define gchooks in the target, and use the new interface to print some stats inside targetgc...
tayfun26 has quit [Remote host closed the connection]
jredondo__ has quit [Ping timeout: 248 seconds]
antocuni has joined #pypy
energizer has joined #pypy
<arigato>
mattip (logs): maybe it's Cython playing dark tricks and the object's ob_refcnt used to be really smaller at the point where the GC ran, then the GC decided to kill it, but then Cython increased the ob_refcnt again
<arigato>
although, that explanation doesn't work if you have a gdb watchpoint that triggers *inside* the GC directly
<arigato>
(assuming, maybe wrongly, that Cython does no crazy things like changing ob_refcnt from another thread that doesn't hold the GIL)
inhahe_ has quit [Ping timeout: 256 seconds]
<arigato>
mjacob: ok, so the problem is really not the weakref?
<antocuni>
arigato: as you might have seen, I'm working on a branch for implementing GC hooks at app-level
<antocuni>
(so far I have done only the rpython-level part)
<antocuni>
my idea is to write the rpython hooks so that they record some stats and set a flag when triggered
jamesaxl has joined #pypy
asmeurer__ has joined #pypy
<antocuni>
then have a PeriodicAsyncAction which checks the flag and trigger the real app-level hook (if it's set)
<antocuni>
does it sound reasonable?
<arigato>
yes
<arigato>
ah, not PeriodicAsyncAction
<antocuni>
how often do you expect a minor collection to be triggered between one PeriodicAsyncAction and the next?
<arigato>
it should instead be done e.g. like the cpyext's callback
<arigato>
from the rpython lib you invoke some callback, but that callback cannot do much of anything; instead it just calls my_action.fire()
<arigato>
then my_action is triggered very soon
<antocuni>
where is the code for cpyext callbacks? And what is "my_action"?
<arigato>
some non-periodic async action
tbodt has joined #pypy
<antocuni>
ah, so a subclass of executioncontext.AsyncAction
<arigato>
yes, it's PyObjDeallocAction in cpyext
<arigato>
in state.py
<antocuni>
yes, found it
asmeurer__ has quit [Ping timeout: 240 seconds]
<arigato>
and it's self.dealloc_trigger that gets passed to the rlib.rawrefcount for callback
<antocuni>
one thing which I don't understand about actions is whether they are supposed to be singleton, or I need to instantiate a new one each time
<arigato>
that's singletons
<antocuni>
how do you pass params then?
<arigato>
well "singletons" but created with the space as argument
<arigato>
so translation-time singletons
<antocuni>
yes, but I see that perform takes only executioncontext and frame. Like, if I want do to "fire on_gc_collect(memory_used=1024)"
<arigato>
you can probably give them attributes, and then you'd call the .fire() method only if a flag is set, for example, and check more parameters in perform()
<antocuni>
ah ok
<antocuni>
not very clean, but I suppose it might work
<arigato>
you can do whatever you want in perform(), and a few cheap tests earlier to avoid calling fire()
<antocuni>
I suppose the worst that can happen is that another gc hook overrides the parameters in case it is called before the action had a chance to be fired
<arigato>
right
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<antocuni>
and this brings back again my previous question: how often should I expect minor collections to be done between two performs()?
<arigato>
you can't really allocate stuff before calling fire(), but you can somehow combine the values with the previously recorded values, if you want to go fancy
<arigato>
unknown? usually zero I guess
<antocuni>
yes, I was thinking of something like that; at least, having a counter which tells me how many events happened
<antocuni>
ok good; then maybe combining the values it's not even needed; if by chance it happens, too bad and we don't care
<arigato>
following a fire(), a perform() is usually called quickly. I would be a bit more careful about calling fire() for every minor collections, as this can slow things down
<antocuni>
yes sure
<antocuni>
my plan is to add support for it, but only calling fire() if the hook is actually enabled
<antocuni>
and only if it doesn't slow down benchmarks
<arigato>
ok
<arigato>
I guess some people would like the hook even if it slows down a bit
<antocuni>
my real need is recording gc-collect and gc-collect-step; but while I was at it, adding support for gc-minor looked easy enough :)
<arigato>
(particularly if it can be enabled for 1 minute every hour, or something)
<antocuni>
yes; in my real-world scenario, we have some code which periodically shows a slow down, and we don't know why. If we can record gc activity, we can at least figure out if it's gc-related or not
<antocuni>
also, if someone wants to be advanced, I suppose you could use such a hook to record the traceback and then see which places allocates the most
<arigato>
ah, advanced indeed
<antocuni>
actually, I didn't think about it before, but now it seems like a very good idea to me :)
<arigato>
starts to feel too much like Mono in Unity, where they used an old Boehm-based Mono until recently and allocating anything was to be avoided :-)
<antocuni>
if you manage to write python code which does not allocate, I buy you as many beers as you want
<arigato>
while True: i += 1
dddddd has quit [Remote host closed the connection]
<kenaan>
antocuni gc-hooks 2492c826dee0 /rpython/memory/test/test_transformed_gc.py: reset the counters at the beginning of this test, else we count also the colletions occurred in the previous tests
<kenaan>
antocuni gc-hooks ecb86e9ba2d6 /rpython/: simplify MyGcHooks: instead of using a global GC_HOOK_STATS, we store the stats as an attribute and we ensure t...