cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
t3chn0punk has quit [Ping timeout: 268 seconds]
antocuni has quit [Ping timeout: 244 seconds]
t3chn0punk has joined #pypy
redj has joined #pypy
t3chn0punk has quit [Ping timeout: 260 seconds]
asmeurer__ has quit [Quit: asmeurer__]
raynold has quit [Quit: Connection closed for inactivity]
<fijal>
_aegis_: right, so your goal would be to parallelize parts of it?
dan- has quit [Ping timeout: 260 seconds]
dan- has joined #pypy
dan- has quit [Changing host]
dan- has joined #pypy
t3chn0punk has quit [Ping timeout: 240 seconds]
oberstet has joined #pypy
t3chn0punk has joined #pypy
t3chn0punk has quit [Ping timeout: 256 seconds]
t3chn0punk has joined #pypy
altendky has quit [Quit: Connection closed for inactivity]
<Remi_M>
_aegis_: it's unclear if STM would have helped with latency problems. But if you simply need more frequent switching between threads, you can play around with sys.setcheckinterval(). Setting it to a lower value should switch more frequently, but may add some overhead (due to more frequent switching).
<Remi_M>
(or I guess sys.setswitchinterval() for pypy3)
<LarstiQ>
well, quite generally speaking different languages have different designs to better fit some tasks, having 1 language for the complete spectrum of what humas program for seems a bit much
voldmar has quit [Client Quit]
voldmar has joined #pypy
<toaderas>
LarstiQ: Most languages fail to get syntax/semantics right for optimum readability, Python does that as good as possible
voldmar has quit [Client Quit]
t3chn0punk has joined #pypy
antocuni has joined #pypy
t3chn0punk has quit [Ping timeout: 276 seconds]
Rhy0lite has joined #pypy
<LarstiQ>
toaderas: so slightly less ambitious: widening the performance range where python is usable
<LarstiQ>
toaderas: have you tried pypy?
<toaderas>
LarstiQ: I didn't say that pypy is not good enough - I use it
t3chn0punk has joined #pypy
t3chn0punk has quit [Ping timeout: 260 seconds]
<cfbolz>
of course it's not good enough ;-)
t3chn0punk has joined #pypy
marky1991_2 has joined #pypy
hotpot33 has joined #pypy
marky1991_2 has quit [Read error: Connection reset by peer]
marky1991_2 has joined #pypy
t3chn0punk has quit [Ping timeout: 244 seconds]
voldmar has joined #pypy
voldmar has quit [Client Quit]
t3chn0punk has joined #pypy
t3chn0punk has quit [Ping timeout: 240 seconds]
stevenja_ has joined #pypy
t3chn0punk has joined #pypy
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
marky1991_2 is now known as marky1991
marky1991 has quit [Changing host]
marky1991 has joined #pypy
marky1991 has quit [Read error: Connection reset by peer]
t3chn0punk has quit [Ping timeout: 240 seconds]
marky1991 has joined #pypy
t3chn0punk has joined #pypy
stevenja_ has quit [Ping timeout: 240 seconds]
stevenja_ has joined #pypy
hotpot33 has quit [Ping timeout: 265 seconds]
t3chn0punk has quit [Ping timeout: 268 seconds]
t3chn0punk has joined #pypy
<_aegis_>
Remi_M: the low-latency code in particular is a call from CFFI that's waiting for the GIL
<_aegis_>
any idea if sys.setcheckinterval will allow a CFFI callback to trigger more often?
<_aegis_>
in a profiler the C code calling the callback just spent a ton of time sitting on a lock
<_aegis_>
I mitigated the first-party case where it can happen by putting a sleep in the background job that was holding the GIL for too long
<_aegis_>
but users can cause it themselves by writing a compute intensive script
hotpot33 has joined #pypy
<_aegis_>
the idea with STM is that the CFFI callback wouldn't need to take a lock to get a thread slot (as the thread count will be going from 1-3), thus removing the latency added by the GIL during parallel workloads
marky1991 has quit [Read error: Connection reset by peer]
marky1991 has joined #pypy
marky1991 has quit [Read error: Connection reset by peer]
mattip_ has quit [Read error: Connection reset by peer]
mattip has joined #pypy
marky1991 has joined #pypy
inhahe_ has joined #pypy
inhahe__ has quit [Ping timeout: 244 seconds]
<ronan>
idnar: on pypy, "undefined" usually means that it's an RPython function
marky1991 has quit [Read error: Connection reset by peer]
<simpson>
LarstiQ: Is this a rename of ZipPy perhaps?
DIRTY has quit [Max SendQ exceeded]
DIRTY has joined #pypy
<cfbolz>
simpson: no, it's a new project
<cfbolz>
Got started about a year ago
<cfbolz>
Oracle has three or so people working on it
<Alex_Gaynor>
Is this different from the py3 graal VM that some folks at one of the California universities were doing 5 years ago?
<Alex_Gaynor>
UC Irvine or Davis I think
<cfbolz>
Yes, that's zippy
<Alex_Gaynor>
Ah
<Alex_Gaynor>
thanks
<cfbolz>
It's different in that graalpython are trying harder to support frame objects and other internal things
<idnar>
ronan: hmm, the calls are inside lxml
marky1991 has quit [Read error: Connection reset by peer]
marky1991 has joined #pypy
marky1991 has quit [Read error: Connection reset by peer]
tayfun26 has joined #pypy
marky1991 has joined #pypy
stevenja_ has quit [Remote host closed the connection]
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
asmeurer has joined #pypy
tayfun26 has quit [Quit: tayfun26]
<arigato>
_aegis_: I think that we should look at your problem more directly with the regular pypy
<arigato>
in theory, cffi callbacks should only sleep in the lock if there is another thread running python code, and that other thread should release the gil regularly (or even, with pypy, other pypy threads should actively wake up a different pypy thread)
<arigato>
pypy received some optimizations for the total throughput, but that was assuming most of the time is spent running python code, and threads do possibly many short calls that release the gil
stevenja_ has joined #pypy
<arigato>
your case might be the opposite, where most time is spent in C but there are many short calls to python
<arigato>
it's a case that we could look into in more details
stevenja_ has quit [Ping timeout: 260 seconds]
stevenja_ has joined #pypy
stevenja_ has quit [Remote host closed the connection]
stevenja_ has joined #pypy
stevenja_ has quit [Ping timeout: 240 seconds]
<_aegis_>
ok so I have a desktop application that has a bunch of OS bindings: https://talonvoice.com/
<_aegis_>
there are a lot of reasons for pypy to be doing something
<_aegis_>
for example, I have a "cron" thread that schedules events to run later (like to upload a batch of app metrics to my server)
<_aegis_>
I'm likely doing on the order of hundreds of calls from C to pypy per second for various reasons
<arigato>
that's not too much
<_aegis_>
ok, so the issue where it was stalling on the GIL, was the metrics upload
<_aegis_>
metrics are being loaded from disk and gzip compressed
<_aegis_>
this was happening during a vsync callback from C to pypy to draw to the screen
<_aegis_>
*gzip decompressed
<_aegis_>
there's an initial metrics checkin that happens 1 minute after startup, so if I draw a circle that moves around, I see huge frame drops during the metrics upload
<arigato>
ah, so C calls pypy which calls gzip.decompress, which should release the GIL again, but doesn't release it instantly
<arigato>
right?
<_aegis_>
no, cron is a pypy thread that is just running
<_aegis_>
it does a queue.get(timeout=time_to_next_event), then calls the next event function
<_aegis_>
so there's a queue.get(timeout=1 minute) on startup, in a threading.Thread, which calls metrics.collect(), which decompresses metrics files from disk (line at a time!)
<_aegis_>
if gzip is C, I bet the line-at-a-time part causes a ton of FFI activity too
<_aegis_>
so there's a vsync thread running in C, that calls a ffi.callback roughly in the middle of each render frame
<_aegis_>
so we'll drop frames if pypy blocks for more than 8ms or so
<_aegis_>
but it dropped a lot of frames, basically for the whole duration of the gzip loop
<arigato>
right, yes that's possible
<_aegis_>
I worked around it by putting a 5ms sleep for each line in the gzip loop
<arigato>
if one pypy thread does many many very short calls to C, the GIL is not actually released, because otherwise the performance would drop by a huge factor if we have two pypy threads doing that
<arigato>
it's possible that there's no limit to this effect, and if one thread does it all the time, then it can completely hog the gil
<_aegis_>
my roughly "worst case" for ffi callbacks per second is... 3 monitors at 60hz, 3 eye trackers at 90hz w/ 2-3 event streams, and an audio thread (recording 1 mic at ~86 callbacks per second for 512-sample chunks at 44100hz)
<arigato>
(but we'd need to check)
<_aegis_>
3*60+(3*90*3)+86 = ~1076
<_aegis_>
all of which I want to finish pretty quickly
<_aegis_>
hence my thought that a second dedicated thread would be really nice, or building something out to transparently run user code in a whole separate pypy instance with some rpc + code migration shenanigans
<tos9>
in case there are any photographers in the building, https://github.com/Julian/libraw-cffi was pretty easy to just throw together and already using to pull some metadata
<_aegis_>
dunno, I guess I could go the other way and make people move batch jobs out of the main pypy process
<_aegis_>
(the reason I can't just hand-optimize the metrics job and call it a day is the whole point of this is to run a bunch of unvetted user code too)
<_aegis_>
oh it's not line at a time, it's file at a time
<_aegis_>
so I was just doing gzip.open().split('\n') on 100 files, ~30kb compressed, ~100kb uncompressed
<arigato>
playing with parameters in my test script, I've found a case where the gil switches between two threads only every ~20ms
marky1991 has quit [Read error: Connection reset by peer]
marky1991 has joined #pypy
marky1991 has quit [Read error: Connection reset by peer]
marky1991 has joined #pypy
oberstet has joined #pypy
dddddd has joined #pypy
hotpot33 has quit [Read error: Connection reset by peer]
asmeurer has joined #pypy
<kenaan_>
wlav cppyy-packaging a3af4486e271 /pypy/module/_cppyy/interp_cppyy.py: some more asserts for the annotator (gives a small performance improvement on bound function calls)
<kenaan_>
wlav cppyy-packaging 1214d76af168 /pypy/module/_cppyy/: support low-level views of arrays that allow user-side updates to shape
<kenaan_>
wlav cppyy-packaging dbc2035e00e4 /pypy/module/_cppyy/: more pythonizatin of std::vector and more STL tests
raynold has joined #pypy
Rhy0lite has quit [Quit: Leaving]
asmeurer has quit [Quit: asmeurer]
asmeurer_ has joined #pypy
asmeurer_ has quit [Quit: asmeurer_]
eregon has quit [Remote host closed the connection]