cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "the modern world where network packets and compiler optimizations are effectively hostile"
<kenaan>
arigo default c4f7ee9d1f93 /lib-python/2.7/ctypes/test/test_byteswap.py: test_struct_struct now passes
<arigato>
njs: hi
<njs>
arigato: hello!
<njs>
good... morning, I guess
<arigato>
:-)
<njs>
(it's just after midnight here, so it works for both of us :-))
<arigato>
"is it trivial for the JIT to optimize option 1 into the same thing as it would do for option 2?"
<arigato>
yes, the JIT should optimize that
<arigato>
but there are a few "buts"
<arigato>
notably, get_queued_items() must not contain any loop
<arigato>
if you want to be sure, then option 2 is safer
<njs>
hmm. it could internally be written as 'if not have_queued_items(): return [] else: ...'
<arigato>
right, but if the RPython code in the "else:..." part contains a loop, then the whole RPython function will not be entered by the JIT at all
<njs>
else: make_the_real_list() I suppose
<njs>
the context is that I'm considering a proposal for adding some functions to sys in python 3.7, and wondering if I should propose 2 functions or 1. So we can certainly imagine the rpython implementation is as careful as we like :-)
<arigato>
then yes
exarkun has quit [Ping timeout: 252 seconds]
tilgovi has joined #pypy
<njs>
(specifically: right now if you meant to write 'await foo(1)' and accidentally instead write 'await foo()' you get a nice TypeError, but if you accidentally write 'foo(1)' then you get a warning message printed to the console at some arbitrary later date. it turns out fixing this is really hard w/o breaking asyncio, but I'd like to at least be able to query the interpreter for unawaited coroutines so I can give a proper error that's at least close to the prope
<njs>
r place.)
exarkun has joined #pypy
<arigato>
heh. this is going to be fun in pypy, for reasons unrelated to that one
<njs>
(but this means that I have to ask the interpreter for the unawaited coroutines on every context switch -- thousands of times a second -- and in a correctly written program there are never any.)
<njs>
oh?
<arigato>
not even sure how you'd implement that on CPython. with weakrefs?
<njs>
I was thinking we'd keep a thread-local double-linked intrusive list in the coroutine objects
<njs>
they insert themselves when constructed, remove themselves again when iterated or gc'ed
<arigato>
remove themselves when GC'ed? that prevent "foo(1)" from being detected, no?
<arigato>
anyway, these GC operations are typically more costly in PyPy than in CPython, so no, I wouldn't worry about a detail in how the JIT optimizes a call
<njs>
you also need a hook that gets called if they're gc'ed before being detected the other way
adamholmberg has joined #pypy
<arigato>
great, I can see how libraries are going to rely on that hook *only*
<arigato>
as it is sufficient with refcounting to detect 'foo(1)'
<arigato>
and then we'll have libraries that are naturally incompatible with PyPy
<arigato>
if I can make a suggestion that really would help PyPy, it is that the hook is not visible at all from the C or Python API
<njs>
arigato: (a) it isn't sufficient actually, because you also want to catch accidentally writing 'blah = foo(1)' when you mean 'blah = await foo(1)'
<njs>
arigato: and (b) I don't really expect anyone to use these hooks except for me so.... :-)
<njs>
uh. how would the hook be used then?
<arigato>
it would record the fact in an internal list
<arigato>
which is queried with the same has_xxx() as you suggested above
adamholmberg has quit [Ping timeout: 252 seconds]
<arigato>
it doesn't really change much, admittedly
<arigato>
but you have an interface that has a clear meaning, at least
<arigato>
without talking about "the time at which __del__ is called"
<arigato>
the meaning is: when has_xxx() is called, it returns a list of all coroutines never awaited, which may be now freed or not
<njs>
arigato: yeah, that would work too -- I hadn't fully decided which was better :-). The downside with that approach is that you have to flip the interpreter into the "hold onto all coroutines" state or not and if you accidentally leave it in that state then they just leak. But it could certainly work, it's just a question of which API is nicer.
<njs>
it also has the minor advantage that you can skip maintaining the list entirely when it's turned off, but that's pretty cheap
<arigato>
ok, then if you're asking the question about which API is nicer for PyPy, the answer is clearly this one
<arigato>
because calling arbitrary Python code when objects are GC-ed is costly and error-prone---even on CPython where the call could occur in some random unrelated thread *if* you have cycles
<arigato>
you usually don't have cycles, which makes such issues *usually* invisible
<njs>
unawaited coroutines already call arbitrary python code when GCed, they enter the warning machinery
<arigato>
fine
<njs>
unawaited coroutines are never supposed to be GCed, it's a bug when it happens. so the cost at that point doesn't matter very much :-). But maybe there are other issues e.g. with maintaining the references through the compacting collector?
<arigato>
sorry, "GC'ed" means "freed from memory" anywhere else than on CPython :-)
<arigato>
but yes, never mind, I'm mostly ranting at this point. we have grown enough support to emulate CPython pretty efficiently too
<njs>
no, that's not the issue -- they literally are not supposed to be freed until after they have transitioned into awaited coroutines
<arigato>
well, with a normal GC you can't transition from "needs a random callback when freed" to "doesn't need one after all". but I'm ranting, because pypy's own GCs actually support that nowadays, for generators already
<arigato>
for when generators contain "try:finally:" around "yield", but their execution is now finished
<njs>
yeah
<cfbolz>
arigato: all that is still not really timely in pypy
<cfbolz>
(but maybe the fix for the recent GC bug made it less terrible?)
tilgovi has quit [Ping timeout: 246 seconds]
<njs>
arigato: so does that mean that it doesn't matter after all whether the API is callback-on-gc + way-to-find-live-objects vs. a modal thing where the list of unawaited coroutines holds strong refs?
<arigato>
cfbolz: heh, maybe. I didn't think about that issue in gc-del-limit-growth
<cfbolz>
arigato: I think that branch might have solved a number of "why are my files never closed" issues, no?
_main_ has joined #pypy
<arigato>
njs: it does, because right now generators have a __del__ only if they contain a "yield" inside "try:finally:"
<njs>
arigato: no, your coroutines do issue a warning if they get GCed without being iterated
<arigato>
ok
<njs>
arigato: I have an example of it in my docs and everything :-)
_main_ has quit [Read error: Connection reset by peer]
_main_ has joined #pypy
<arigato>
I would still consider Jython and IronPython too, and in general trying to avoid __del__ as much as possible is a Good Thing (tm)
<arigato>
cfbolz: I'm not completely sure that the branch really has an impact on large programs
_main_ has quit [Read error: Connection reset by peer]
<cfbolz>
arigato: right, that's possible
<njs>
hard to know what to make of jython here, they're still in the "maybe we should think about python 3 some day" stage :-/
__main__ has quit [Ping timeout: 240 seconds]
<njs>
but fair point, one can imagine that someday we might be able to get rid of that warning or something
<cfbolz>
njs: that might change soon
__main__ has joined #pypy
<arigato>
I'm mostly trying to keep the job of making a new Python interpreter reasonable, but well, I know that it's a lost cause already
rokujyouhitoma has joined #pypy
<arigato>
...sorry if that sounded mean, that wasn't meant to. and thank you for checking with #pypy when designing new things
<njs>
we should probably nag you for thoughts on PEP 550 at some point, though there are still a bunch of more basic semantic issues flipping back and forth
rokujyouhitoma has quit [Ping timeout: 248 seconds]
marr has joined #pypy
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 240 seconds]
adamholmberg has joined #pypy
adamholmberg has quit [Remote host closed the connection]
Coldstrom has joined #pypy
<Coldstrom>
Hi
adamholmberg has joined #pypy
<Coldstrom>
i have an issue with the running python code
<the_drow>
Do we serialize traces and pass them between subprocesses? Is that possible? Or is it part of the "traces are not cached" problem
<LarstiQ>
Coldstrom: you're not providing enough information for anyone to judge what's going on
exarkun has quit [Ping timeout: 252 seconds]
exarkun has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 248 seconds]
antocuni has joined #pypy
<cfbolz>
the_drow: the latter
<the_drow>
Thanks
adamholmberg has joined #pypy
<njs>
arigato: oh duh, just remembered -- coroutines have to have a __del__ method regardless of the unawaited warning nonsense, because they also do throw-GeneratorExit-to-clean-up thing that generators do :-/
<njs>
well, I guess you could imagine dropping that at some point because no-one is supposed to rely on it, but blah.
adamholmberg has quit [Ping timeout: 248 seconds]
<fijal>
njs: I guess the usual first question would be "does it work well with cyclic GC in cpython"?
<fijal>
er, misplaced quotes
<njs>
fijal: I don't think any of the variants have any trouble with cycles or anything on cpython
<fijal>
other than stuff not happening immediately?
raynold has quit [Quit: Connection closed for inactivity]
<Coldstrom>
guys i need to solve this,m working for a company where my task is to run python code in a sandbox env
<Coldstrom>
and i have tried docker based and all the other sandbox mechanism but all were not upto the mark
<Coldstrom>
m relaying on this sandbox n dead line is very close
<Coldstrom>
please help
rokujyouhitoma has quit [Ping timeout: 252 seconds]
<the_drow>
Hi guys. I'm on PyPy 5.1.1 on a client's machine and I'm trying to use boto. I'm getting an "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed" error when trying to fetch a s3 bucket by name.
<the_drow>
I don't think upgrading is a problem
<the_drow>
but it will take some time to do so
<the_drow>
so right now I need to workaround this somehow
rokujyouhitoma has quit [Ping timeout: 248 seconds]
jacob22_ has quit [Ping timeout: 252 seconds]
nimaje has quit [Remote host closed the connection]
nimaje has joined #pypy
exarkun has quit [Ping timeout: 246 seconds]
exarkun has joined #pypy
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 248 seconds]
jacob22_ has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 248 seconds]
marky1991 has joined #pypy
raynold has joined #pypy
tormoz has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
exarkun has quit [Ping timeout: 248 seconds]
exarkun has joined #pypy
<kenaan>
mattip default e45fdeb7813a /pypy/module/cpyext/: test, fix - any str subtype should never have tp_as_a_number.* functions set
<kenaan>
mattip default fc115074c233 /pypy/module/cpyext/test/test_bytesobject.py: failing test ' ' + np.string_('abc') should not call np.string_.__radd__
<mattip>
that test fails since we do lookup at runtime, and find in the base classes one with a __radd__ implementation, even though str has no __radd__
<mattip>
likewise for __rsub__, __rdiv__, __sub__, ... which are all found in the np.generic parent class
<mattip>
any ideas what would be the "correct" fix?
<simpson>
Hm, what's the right way to divmod() in RPython?
<mattip>
but a cextension PyStringMy_Type that has bases = (str, class-with-__radd__) will not expose the __radd__ on cpython
rokujyouhitoma has joined #pypy
<kenaan>
mattip default 23392d66a346 /pypy/module/cpyext/test/test_bytesobject.py: Backed out: fc115074c233, not clear if this is a PyPy bug or a CPython bug
nimaje1 has joined #pypy
nimaje1 is now known as nimaje
nimaje has quit [Killed (moon.freenode.net (Nickname regained by services))]
rokujyouhitoma has quit [Ping timeout: 246 seconds]
leto_ni has quit [Ping timeout: 276 seconds]
<mattip>
found a way to avoid the problem - doing ' ' + str(np.string_('abc')), the conversion to str removes the ambiguity
rokujyouhitoma has joined #pypy
<mattip>
but I would prefer to somehow be cpython compatible
rokujyouhitoma has quit [Ping timeout: 240 seconds]
<kenaan>
arigo default 2032a21a8320 /rpython/translator/sandbox/rsandbox.py: "Fix" the most glaring issue about sandboxing
<arigato>
Coldstrom (logs): sandbox issue fixed, now sandboxed pypy runs again (as long as you don't try to start the interactive mode)
exarkun has quit [Ping timeout: 240 seconds]
exarkun has joined #pypy
tbodt has joined #pypy
rokujyouhitoma has joined #pypy
leto_ni has joined #pypy
mattip has quit [Ping timeout: 255 seconds]
rokujyouhitoma has quit [Ping timeout: 248 seconds]