cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "the modern world where network packets and compiler optimizations are effectively hostile"
vkirilichev has quit [Remote host closed the connection]
vkirilichev has joined #pypy
<njs>
fijal: I would say that gone would be one possible fix, yes :-)
<njs>
fijal: not one we can start with though
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
antocuni has joined #pypy
lritter_ has quit [Quit: Leaving]
salonen has joined #pypy
arigato has joined #pypy
<salonen>
I have a problem with pypy when using multiprocessing SimpleQueue, pypy is not releasing the underlying file descriptors and my system is eventually running out of fd's. I have a very simple testbench with which to recreate the problem at least on macOS and Linux. Could someone take a look and comment?
<salonen>
antocuni I read that earlier, my problem is that I don't know how to explicitly close the underlying file descriptors (which are used for pipes in SimpleQueue)
<antocuni>
I don't know the details of course, but you might have to patch SimpleQueue
<salonen>
I see. I'm new to pypy and the gc issues, so I'm a bit baffled - shouldn't standard library modules close files properly?
<antocuni>
uh sorry, I also did not realize that SimpleQueue is in the stdlib :)
<antocuni>
indeed, SimpleQueue.__init__ creates a new Pipe but never closes it :(
<salonen>
The same problem exists with multiprocessing.Queue as well
ronan has joined #pypy
<antocuni>
salonen: do you create tons of queues?
<arigato>
the Pipe is a pair of objects, each of which with a .close() method
<arigato>
so we might need to add a close() method to Queue and SimpleQueue
<salonen>
My program creates quite a few, and then I ran into this problem when testing my program with various options
<antocuni>
arigato: yes but then it means that pypy's queues will have a close() and CPython's won't
<arigato>
antocuni: too bad, "if hasattr(queue, "close"): queue.close()"...
<njs>
CPython Queue has a close()
<njs>
at least in 3.7-dev
<arigato>
right, found it too in 2.7
<njs>
no __enter__ or __exit__ though, sigh
<njs>
type.__new__ should warn if it notices that a class has close() but no __enter__ and __exit__
<antocuni>
the close() doesn't not close self._writer though (not sure if it's needed)
<arigato>
antocuni: it's done by the background thread
<antocuni>
ah, saw it
<antocuni>
salonen: well, this probably means that you should just call .close() on your queues
<arigato>
SimpleQueue don't have close(), still
<njs>
salonen: anyway the workaround is to call sq._reader.close(); sq._writer.close() when you're done with the queue
<njs>
but there's definitely a cpython bug here yeah
<salonen>
Ok, closing _reader and _writer explicitly solves the problem, thanks
<arigato>
:-)
<salonen>
could one of you create the bug report for cpython, I think you'd be able to explain in better
<njs>
(technically this is dangerous because those are internal attributes and in a future python release they might move, but realistically I doubt multiprocessing will ever change)
rokujyouhitoma has quit [Ping timeout: 255 seconds]
ronan has joined #pypy
marky1991 has quit [Ping timeout: 255 seconds]
Ryanar has joined #pypy
adamholmberg has joined #pypy
<salonen>
another observation from my testbench regarding file descriptors, w/ pypy3.5 multiprocessing.Process.sentinel is not closed and when I run bunch of cases in my testbench that alone is enough to drain fd's on my system
haypo has left #pypy [#pypy]
ronan has quit [Quit: Ex-Chat]
ronan__ has joined #pypy
ronan__ has quit [Read error: Connection reset by peer]
salonen has quit [Ping timeout: 240 seconds]
yuyichao_ has joined #pypy
yuyichao has quit [Ping timeout: 276 seconds]
ronan has joined #pypy
Ryanar has quit [Ping timeout: 246 seconds]
jamescampbell has joined #pypy
magniff has joined #pypy
<magniff>
sup chat) it is a pleasure join this channel
jamescampbell has quit [Ping timeout: 268 seconds]
rokujyouhitoma has joined #pypy
jamescampbell has joined #pypy
rokujyouhitoma has quit [Ping timeout: 255 seconds]
asmeurer_ has joined #pypy
<kenaan>
smihnea pypy_bytearray b43a6e2c0ea1 /pypy/objspace/std/bytearrayobject.py: bytearray performance fix(ported from PyPy3)
<kenaan>
rlamy default 94b71132a96e /pypy/objspace/std/bytearrayobject.py: Merged in smihnea/pypy_bytearray/pypy_bytearray (pull request #559) bytearray performance fix(ported from PyPy3)
jamescampbell has quit [Quit: Leaving...]
cstratak has quit [Quit: Leaving]
cstratak has joined #pypy
marky1991 has joined #pypy
realitix has quit [Ping timeout: 268 seconds]
yuyichao_ has quit [Ping timeout: 255 seconds]
mattip has joined #pypy
asmeurer_ has quit [Ping timeout: 276 seconds]
yuyichao_ has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 255 seconds]
magniff has left #pypy [#pypy]
magniff has joined #pypy
asmeurer__ has joined #pypy
realitix has joined #pypy
magniff_ has joined #pypy
magniff has left #pypy [#pypy]
asmeurer__ has quit [Ping timeout: 248 seconds]
realitix has quit [Ping timeout: 260 seconds]
magniff_ has quit [Ping timeout: 260 seconds]
asmeurer_ has joined #pypy
asmeurer_ has quit [Ping timeout: 260 seconds]
rokujyouhitoma has joined #pypy
asmeurer_ has joined #pypy
rokujyouhitoma has quit [Ping timeout: 260 seconds]
cstratak has quit [Quit: Leaving]
asmeurer_ has quit [Quit: asmeurer_]
<kenaan>
rlamy py3.5 85b1a22bbd65 /pypy/module/cpyext/: Add missing function PyObject_LengthHint
raynold has joined #pypy
vkirilichev has quit [Remote host closed the connection]
<ronan>
bytearray.extend() is supposed to call operator.length_hint, but that's not very relevant to us (jitting the iteration over a random sequence is way more important)
<ronan>
the test would pass if I called operator.length_hint and ignored the result, but that's a bit too silly, isn't it?
<fijal>
njs: an honest question, could you get that effect with ast rewrite?
<fijal>
the KeyboardInterrupr one
<pjenvey>
ronny: can't you still get the jit speedup *and* do the length hint? more or less the appexec change is equivalent to adding a jitdriver in that loop
<fijal>
pjenvey: ronan
<fijal>
ronan: I would say skip the test
oberstet has quit [Ping timeout: 240 seconds]
<pjenvey>
you can actually use the result of the length hint to resize_list or whatever its called on the underlying bytearray list
<pjenvey>
resize_list_hint i think
asmeurer_ has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 268 seconds]
<njs>
mattip: yeah, but I don't know what connection he sees between this and AST rewriting. Or even what exactly he has in mind when he says AST rewriting :-)
<njs>
though I'm pretty sure there's nothing wrong with using the stack frame here, it *can* be done without any speed penalty, though it may not quite be possible currently
<mattip>
ok, it's out of my league anyway, esp at this hour
* mattip
plugging away at removing refcount semantics from numpy
<mattip>
gnite
mattip has left #pypy ["bye"]
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 248 seconds]