antocuni changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "PyPy: the Gradual Reduction of Magic (tm)"
songww has quit [Remote host closed the connection]
songww has joined #pypy
marr has joined #pypy
antocuni has joined #pypy
fryguybob has quit [Remote host closed the connection]
lesshaste has quit [Remote host closed the connection]
<fijal>
ronan: eh, do you have any clue about the errorhandlers in codecs on py3k?
<fijal>
arigato: so I think I can find a way without touching __eq__
<fijal>
I'm half worried existing mechanisms are not enough, but let's start there
<kenaan_>
antocuni default 1cac28ee833b /pypy/module/_continuation/test/test_stacklet.py: cherry-pick a small part of the continulet-no-frame-loop branch and make stack() available to all tests; fix tes...
zmt00 has quit [Ping timeout: 248 seconds]
<arigato>
antocuni: continulet-no-frame-loop: you should review and minimize a bit the diff, but other than that:
<arigato>
bottomframe.f_backref is jit.vref_None
<arigato>
might be a little bit dangerous
<antocuni>
why?
<arigato>
because you don't know if there is really always a frame below the bottom frame when running
<antocuni>
how can it happen?
<arigato>
it takes a contrieved example, but it could happen
<antocuni>
I think that in that case you have _greenlet_start or something like that
<antocuni>
but yes, I see the general problem
<antocuni>
what do you suggest? To keep an explicit "is_running" flag?
zmt00 has joined #pypy
<arigato>
hum, there is already something like that
<antocuni>
if there is, I couldn't/cannot find it
* arigato
still looking
<arigato>
a bit confused, because I would expect post_switch() to always run with 'origin' being stopped and 'self' being started (which might occasionally be the same object)
<arigato>
"py.test -x" gives me Invalid call to virtual_ref_finish, anyway...
<antocuni>
did you put DEBUG=True, by chance?
<arigato>
no
<arigato>
no diff
<antocuni>
ah yes
<antocuni>
it's test_permute
<antocuni>
I didn't fix it yet
<antocuni>
but, pytest -k not permute should pass
<arigato>
it doesn't, test_zpickle fails
<antocuni>
true :(
<antocuni>
yes, I tested only test_stacklet
<antocuni>
but well, my point is that before fixing all the remaining things, I wanted to check whether the approach was reasonable
<arigato>
ok
<antocuni>
sorry for the confusion
<arigato>
I would look if you need the two different cases in post_switch(), if f_backend is vref_None or not
<arigato>
because this looks like logic that might work only in simple cases
<antocuni>
I tried hard to find a way to write it without the if, but I couldn't find it
<antocuni>
it would be easy to do if you swap two continulets
<antocuni>
in that case, you swap their bottomframe.f_back: one becomes the existing stack, the other becomes vref_None
<antocuni>
but this is not the case, because we have only ONE continulet, and we "swap" it with the "current" stack
<antocuni>
arigato: also, about your remark "bottomframe.f_back could be None even if it's running": I claim that if it is the case, it means that on default you can construct a case in which sys._getframe(0).f_back.f_back... results in a cycle
<arigato>
hum... why?
<arigato>
there is no "is vref_None" kind of comparison on default
<antocuni>
yes, but I think that "is vref_None" happens in the same cases in which on default you have a frame cycle
<antocuni>
in other words: if you remove the if, you end up with a thing which always does "origin.bottomframe.f_backref = current"
asmeurer__ has quit [Quit: asmeurer__]
<antocuni>
so, in some cases instead of setting vref_None, it builds a cycle
<antocuni>
if THIS ends up running, you get a cycle which is visible through sys._getframe
<arigato>
(someone should really kill _continuation and write greenlet directly in RPython...)
<arigato>
every time we need to change 3 lines it takes half a day
<antocuni>
eh
<antocuni>
arigato: note also the collections of diagrams which I had to do to understand what was going on :)
<antocuni>
why did we choose to write greenlets this way? Just because it was theoretically nicer, or we had some concrete ideas about other ways to use continulets?
<arigato>
just because it's theoretically nicer
* antocuni
wonders who to blame :-P
<arigato>
so, I *think* that both self and origin must be non-running continulets in post_switch(), but I may be wrong
<arigato>
so both have (in default) a loop
<antocuni>
so, I should break both loops?
<arigato>
there are probably still two cases, but that depends on 'origin is self' or not
<antocuni>
also, this origin vs self thing is very confusing to me (actually, its the most confusing part). It seems it's needed only to implement switch(to=...), isn't it?
<arigato>
yes
<antocuni>
and this feature is never used by anybody
<arigato>
uh
<arigato>
no
<arigato>
greenlet.py uses "to=target"
<antocuni>
aaah
<antocuni>
I couldn't find it because it's not calling "switch" directly, of course
<arigato>
obviously
<arigato>
so, in the three lines that are now in "if ORIGINAL:"
<arigato>
there is the currently running stack frame in sthread.ec.topframeref, and two continulets 'self' and 'origin' that should both be non-running and a loop of frames
<arigato>
or I'm very confused
<antocuni>
ok, but then it means that on my branch, they will be both non-running and with "f_backref is vref_None"
<arigato>
I might be very confused
<antocuni>
I wonder whether we should just add an "is_running" attribute a puts asserts here and there
<antocuni>
s/a puts/and put
<arigato>
yes, looks like a plan
<antocuni>
on the branch or on default?
<arigato>
ah wait
<arigato>
ah, maybe the new attribute W_Continulet.topframeref can play this role nicely
<arigato>
it should be vref_None when running
<arigato>
I am still very confused
<antocuni>
ah indeed
<arigato>
I think we can end up with more complicated situations in default
<arigato>
e.g. two non-running continulets, but they form a single bigger loop of frames
<antocuni>
how? If they are non-running, their bottomframe.f_back should be None
<arigato>
I mean, in default
<antocuni>
ah, so you mean that it's a situation which works in default but would be broken in the branch?
<arigato>
yes (maybe)
<antocuni>
I'm having a really hard time to visualize this stuff
<arigato>
yes, it's all very simple but somehow impossible to visualize
<antocuni>
arigato: if two non-running continulets form a cycle, it means that a.bottomframe.f_back ==> b.topframeref
<arigato>
yes, and vice-versa
* antocuni
wonders how to build such a case
<arigato>
then the problem is that "running" or "paused" cannot be easily known
<arigato>
because a continulet might be part of a large cycle,
<arigato>
and somewhere else we tweak the bottomframe.f_back to explode this cycle into the running frame stack, or back
<antocuni>
I **THINK** that if this case happens on the branch, we just break the last link of the cycle
<antocuni>
i.e., the "top" continulet will be link the the ones below it, but then at some point you hit f_back is None
<arigato>
right
<antocuni>
I really need to go because they are waiting for me; if you are there after lunch, we can continue later
<arigato>
it seems there is no good solution at all
<arigato>
I wonder what would break if we simply never changed bottomframe.f_backref, which would stay None all the time (and used a new attribute W_Continulet.backref)
<arigato>
all tracebacks would stop after one greenlet
<arigato>
W_Continulet.bottomframe might be killed, too
<arigato>
it definitely simplifies things, although not the "horribly simple" code in default's post_switch()
jcea has joined #pypy
<kenaan_>
arigo cffi/cffi d5661822dee4 /testing/cffi1/test_recompiler.py: Issue #343 [patch by david naylor] Fix test_recompiler for libc++
<kenaan_>
arigo default 34aff140932c /pypy/module/test_lib_pypy/cffi_tests/cffi1/test_recompiler.py: CFFI Issue #343 [patch by david naylor] Fix test_recompiler for libc++
oberstet has joined #pypy
songww has quit [Quit: songww]
nimaje has quit [Quit: WeeChat 1.9]
nimaje has joined #pypy
kolko has joined #pypy
kolko has quit [Ping timeout: 250 seconds]
kolko has joined #pypy
raynold has quit [Quit: Connection closed for inactivity]
Rhy0lite has joined #pypy
adamholmberg has joined #pypy
<antocuni>
arigato: back
<antocuni>
well, but then if we have an exception inside a greenlet we will have a different traceback than CPython
<antocuni>
which might not be too bad, for all I know
<antocuni>
arigato: ah no
<antocuni>
on CPython, greenlets seems NOT to set their f_back
<arigato>
then yes, it seems we should not set f_back at all
<antocuni>
so basically: we don't even save bottomframe and we just use topframeref?
<arigato>
yes, but not called "topframeref"
<arigato>
but "backframe" or something (also, I'm not sure it should be a vref at all)
<arigato>
(no, it should be a vref)
<arigato>
(otherwise creating a greenlet forces the current frame in a bad way)
<antocuni>
why backframe?
<arigato>
something to replace ".bottomframe.f_backref"
<antocuni>
well, but it's something which later will be used to set ec.topframeref, isn't it?
<arigato>
it's the topframe of some other greenlet, usually, but as an attribute of 'self', it is below self's bottommost frame
<arigato>
yes, but
<arigato>
"topframeref" seems to mean "here's the topmost of my frames"
<antocuni>
yes, exactly: isn't this the case?
<arigato>
no
<arigato>
it is ".bottomframe.f_backref", which may often point back in a loop to the topmost of my own frames, but not always
<arigato>
e.g. not if there are several continulets in a loop
<antocuni>
I am again very confused
<arigato>
it's .bottomframe.f_backref in its regular role as "the next frame below .bottomframe"
<antocuni>
yes, but we just concluded that we don't want ANY frame below bottomframe
<antocuni>
ah, maybe I see
<arigato>
no, we concluded that we don't want to *show* that in the app-level f_back
<antocuni>
you mean, the python frame which happens to be below bottomframe in the *C* stack
<arigato>
but we don't want any change in the internal structure
<arigato>
I don't think we can say anything about the C stack here
<arigato>
I just mean that we should do a very minimal change from default, which is to store ".bottomframe.f_backref" into ".backframeref" instead
<arigato>
and thus make the f_back at app-level return None
<antocuni>
yes, but I claim that "backframeref" is a bad name, at least in my understanding of things
<antocuni>
when I tried to understand how it works, at some point I had the eureka moment, in which I said: "I know, we abuse bottomframe.f_back to store a reference to the top"
<arigato>
it's a name chosen to match the internal implementation (given that the attribute is not visible at app-level anyway)
<arigato>
it's not an abuse
<arigato>
it's because of this nice (and theoretical and useless) model
<antocuni>
then it means that my mental model is wrong
<antocuni>
one thing that I don't understand is:
<antocuni>
if we have a BOTTOM frame, how is it possible that there is a frame below it?
<arigato>
so, as I said, the basic model is that all frames in the current thread (including bottomframe and others) have f_back, and the property is that: there is one top-level frame that is not the f_back of anyone else, and that is running; and there is a single f_back=None, which is the bottommost
<arigato>
if you take this as a definition, you see it includes the case of a regular stack, but also a regular stack + any number of loops
<arigato>
of any size
<antocuni>
ok
<arigato>
to answer your question more precisely, ".bottomframe" is the bottommost frame that belongs to the current continulet
<antocuni>
but then "bottomframe" is a bad name, because it's not bottom at all
<antocuni>
ah ok
<arigato>
right, so I suggest ".backframeref" or something, that points to the next frame immediately below the ones that belong to the current continulet
<antocuni>
ok, I think that now I get one source of my confusion: is that the top->bottom relationship is not totally ordered at all
<arigato>
no, it's loops
<antocuni>
right
<arigato>
in theory you can have a loop of several continulets, each with .backframeref pointing to the topmost frame in the next one
<arigato>
and from there the user can ask to resume any of the continulets
<arigato>
...which is admittedly very hard to understand
<antocuni>
and then they all run in order, but from a different starting point depending on which precise continulet you resume
<arigato>
yes
<antocuni>
ok, it makes some kind of sense
<arigato>
"some kind of sense", indeed
<antocuni>
aah, now I also understand why it's called "permute"
<arigato>
yes, it exchanges two ".backframeref"
<arigato>
in theory switch() can be implemented with permute(): you create a continulet and run it, and inside, it permute() its own backframeref, and returns
<antocuni>
oh, I see
infinite has quit [Ping timeout: 268 seconds]
<antocuni>
ok, I think I got it now
<antocuni>
thanks to some other diagrams which I just drawn
<antocuni>
so basically, in a given thread: we have a single "top->to->bottom" acyclic sequence of frames, which is the currently running stack
<antocuni>
and N loops of frames, each of which composed by one or more continulets
infinite has joined #pypy
<arigato>
yes
<antocuni>
cool, now I understand 99% of it and I only miss the remaining 99%
<antocuni>
thanks :)
<antocuni>
anyway, back to the fix: ok, we use .backframeref instead of .bottomframe.f_backref; but then what happens to the current bottomframe? Do we still need it?
<arigato>
mostly not
<arigato>
we only need the two lines inside the pure-Python code that call continulet.switch()
<arigato>
but we can move them to RPython code inside new_stacklet()
<antocuni>
so, inside new_stacklet() we call self.switch() instead of self.bottomframe.execute_frame()?
<arigato>
not instead of
<arigato>
we call switch(), and then we need to really run w_callable from W_Continulet.__init__
<arigato>
so we store w_callable and __args__ on the W_Continulet I guess
<arigato>
and use space.call_args()
<antocuni>
ah, I see
<antocuni>
basically, I call switch() immediately so that I give back the execution to the caller of continulet.__new__
<antocuni>
and then whey they switch() back to me, I call the actual function
<arigato>
yes
<antocuni>
ok
Garen has quit [Read error: Connection reset by peer]
Garen has joined #pypy
<kenaan_>
mattip refactor-PyFloat_FromString 57019e77c377 /pypy/: refactor possible recursion in PyFloat_FromString
* mattip
working out new numpy failures due to cpython quirks
<antocuni>
arigato: re "the 2nd ec.topframeref = None": do you mean the one immediately before global_state.origin = self?
<arigato>
yes, the one after "try: except Exception:"
<antocuni>
ok
<arigato>
if you also set it before, then afterwards it should be None too
<arigato>
but better safe than sorry, maybe in some exceptional cases it won't be
<antocuni>
I'm not sure to follow
<antocuni>
currently, frame.execute_frame sets f_backref = ec.topframeref, then calls the switch
<arigato>
I mean that if you set ec.topframeref = vref_None before, then afterwards, it should be the same value: all calls to frames change and then restore it when they return
<antocuni>
ah ok
<arigato>
but better safe than sorry, you can keep the line afterward unmodifed too
<arigato>
fijal: if you mean "which version introduced support for ffi.set_source()" then it was 1.0
<kenaan_>
antocuni continulet-no-frame-loop-2 5dfc7af8c0ff /: a branch where to try to fix issue 2683 in a different (and simpler) way than continulet-no-f...
<kenaan_>
antocuni continulet-no-frame-loop-2 37701890010b /pypy/module/_continuation/test/test_stacklet.py: cherry pick two failing tests from the branch continulet-no-frame-loop
<kenaan_>
antocuni continulet-no-frame-loop-2 55154dad821a /pypy/module/_continuation/interp_continuation.py: WIP: (antocuni, arigato): refactor things so that we no longer need a bottomframe, and that t...
<fijal>
arigato: but ffi.verify still works right?
<antocuni>
arigato: ^^^ r55154dad821a
<arigato>
fijal: yes
<fijal>
ok
<fijal>
is it possible to package wheel in a way where verify() builds the thing?
<fijal>
if so, is there documentation how to do this?
<arigato>
no, that's the point of using set_source() instead
<arigato>
with verify(), you can hack around, but there is no clean solution
<kenaan_>
mattip default 107848cb5acc /lib-python/2.7/test/test_urllib2net.py: graft oneliner from stdlib from 2.7.14 to fix tests
oberstet has quit [Ping timeout: 240 seconds]
asmeurer_ has joined #pypy
<mattip>
anyone on macosX, feel like taking a look at a nightly 2.7 build and pypy.module.posix.test.test_posix2 failures?
<nanonyme>
mattip, isn't it just macOS these days?
<mattip>
you already are demonstrating a knowlege much beyond mine
<nanonyme>
Seems like they're trying to put non-fans off scent by changing name all the time
<nanonyme>
Doesn't mean I actually run Apple hardware
<nanonyme>
(nor operating systems)
asmeurer_ has quit [Quit: asmeurer_]
<nanonyme>
Sorry :(
marky1991 has quit [Read error: Connection reset by peer]
<antocuni>
arigato: I fear that bottomframe is needed to implement pickling
<kenaan_>
antocuni continulet-no-frame-loop-2 e8f933d33b7e /pypy/module/_continuation/test/test_stacklet.py: fix and simplify test_f_back_*: now that we hide the frames below bottomframe, a part of the ...
<kenaan_>
antocuni continulet-no-frame-loop-2 8c14e037eea6 /pypy/module/_continuation/: fix permute, and rewrite the corresponding test since we can no longer check what is the 'bac...
drolando has quit [Read error: Connection reset by peer]
<njs>
"so it means that on CPython, with greenlets, we get exceptions that print normally but that contain some frames with f_back == None, right?" <-- note that this isn't so weird -- you can also see this with simple generators.