cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
<kenaan>
antocuni hpy 05f41db32719 /pypy/module/hpy_universal/: resolve an XXX and use the proper way to include src/getargs.c as a separate compiled module
oberstet_ has quit [Remote host closed the connection]
xcm has quit [Read error: Connection reset by peer]
xcm has joined #pypy
<kenaan>
mattip cppyy-dev 76fdc207c387 /pypy/doc/whatsnew-head.rst: document branch to be closed
<kenaan>
mattip default 490e127a5973 /pypy/doc/whatsnew-head.rst: remerge cppyy-dev into default which updates the backend
<arigato>
petronny: I think we fixed that in trunk
<arigato>
or in the py3.6 branch, if you want python 3.6
fijal has quit [Ping timeout: 264 seconds]
fijal has joined #pypy
idnar has quit [Ping timeout: 264 seconds]
idnar has joined #pypy
jvesely has quit [Quit: jvesely]
<petronny>
mattip_: QEMU cross-compile
<petronny>
arigato: ok then, I will wait for the next stable release.
dddddd has quit [Remote host closed the connection]
<antocuni>
mattip_: I don't know much about sockets. I tried to follow a bit CPython's and PyPy logic, and I think that where they differs is in rsocket.py:recvinto
<antocuni>
it seems that pypy calls "self.wait_for_data(for_writing=False)", I don't see anything like that in CPython
<antocuni>
could it maybe explain the difference?
tsaka__ has quit [Quit: Konversation terminated!]
tsaka__ has joined #pypy
<antocuni>
mattip_: also, if you look at socketmodule.c:sock_recv_guts in CPython2.7, you see that it has a very different logic than CPython3. At a first glance, it looks similar to what we do in rsocket
<antocuni>
so this is probably the original source of incompatibility
<created>
Assuming I'm not doing anything wrong (which I might be)
<created>
It seems like pypy is not doing garbage collection, up and until the process runs out of memory and fails due to failed malloc in 3rd-party code
<created>
Adding manual "gc.collect()" calls avoids the "leak" and crash
<created>
The objects needing to be garbage collected have a __del__ (which frees up some unmanaged resources - if you pardon the expression)
<created>
Is that expected?
<created>
I mean, I'm just going to force delete these objects, which is a sane thing to do anyway
<created>
But I'm curious if this is expected behaviour
<created>
(I guess pypy has no way of knowing how much unmanaged memory those objects "hold" and so can't make any sensible choices here
<antocuni>
what you describe is a reasonable explanation: suppose you have a small Python object with a __del__, which keeps alive a large malloc()ed memory region
<created>
antocuni - yeah, but I didn't really get the relevant part
<created>
Why is forcing a full GC when getting close the OS limit bad? Not doing so will cause the process to crash, no?
<antocuni>
pypy has no way to know that this object is keeping alive a lot of memory, so you need to allocate MANY of them before triggering the gc
<created>
Though I do agree that "close to the OS limit" is hard to define
<created>
Yeah, that's what I figured
<antocuni>
the correct way to handle the problem is to explicitly release resources when they are no longer needed. Usually this is done with a "with" statement
<antocuni>
or with a .close() call
<created>
And though I don't understand the webpage's explanation, I do agree with its conclusion
<antocuni>
note that even on CPython, relying on __del__ is dangerous: there are cases in which __del__ are not called at all!
<antocuni>
what is that you don't understand in the webpage explanation?
<created>
The part where it says: "we’re getting dangerously close to the OS’s limit can be very bad in some cases."
<created>
While I agree that a frequent GC collection due to pent up junk is bad, isn't crashing due to reaching the OS limit worse?
<antocuni>
yes
<antocuni>
the point of the page is that you should find ANOTHER way to dealing with it
<created>
Ah
<antocuni>
also, that paragraph is speaking only about the concrete example of file descriptors
<antocuni>
but the concept can be generalized to all kind of resources
<created>
But for native python objects, there'll be no out-of-memory before a full gc run, right?
<antocuni>
right
<antocuni>
if pypy's GC can manage all the memory needed by a process, it will always run the collection "at the right moment"
<antocuni>
the problem arises when there are resources which are unknown to the GC; like e.g. file descriptors or externally-managed memory like in your case
<created>
Thanks, I get it now
<antocuni>
you're welcome
mattip_ has joined #pypy
jvesely has quit [Quit: jvesely]
<kingsley>
Does the "numba" compiler use the industry standard LLVM to speed up python math on POWER9?
<cfbolz>
kingsley: numba uses llvm, I suppose it's the same on power
<cfbolz>
but that's really the wrong channel for this question :-)
<kingsley>
cfbolz: Thanks for indulging my curiosity.
<kingsley>
I'd be happy to learn of a numba IRC channel, and ideally, POWER9 benchmarks.