cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "the modern world where network packets and compiler optimizations are effectively hostile"
<yuvipanda>
it's a tornado application! I spent some time profiling and fixing low hanging fruit, and it feels like now I can try out pypi :)
<yuvipanda>
err pypy
<yuvipanda>
Profiling with pyflame showed we were spending significant time in a loop that was doing auth cookie validation, plus it's a general tornado app - and I saw some great benchmark numbers :)
<yuvipanda>
I'll keep channel posted on how it goes! I'm using the 'sparky portable binaries', and running the 3.5 compatible version
rokujyouhitoma has quit [Ping timeout: 255 seconds]
<yuvipanda>
preliminary results are exciting :) we know we've some race conditions we haven't chased down yet, but pypy seems fast enough that these races are being hit less often :)
_whitelogger has joined #pypy
<njs>
yuvipanda: hey yuvi :-)
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
<yuvipanda>
hey njs!
<yuvipanda>
njs: If this is the njs who I think it is I sent you an email a while ago ;)
<yuvipanda>
njs: didn't realize you were involved in pypy!
<njs>
yuvipanda: uh... yeah, I was just thinking that
<njs>
yuvipanda: well, I kibbitz and complain about things
<yuvipanda>
:D
<yuvipanda>
njs: so there's probably going to be an EdX data8 course with about 10k or more students, and so I've to scale JupyterHub up an order of magnitude or more :D
<yuvipanda>
njs: so I've been in full on profiling mode to optimize the single hub case, and also doing work on sharding it out
<njs>
yuvipanda: woo / uhoh
<yuvipanda>
njs: yeah, and it starts in end of september soo....
<njs>
yuvipanda: but you use kubernetes so it automatically scales right
<yuvipanda>
njs: :P
forgottenone has joined #pypy
<yuvipanda>
njs: yeah, we haven't had problems there yet. but the hub process loop is too busy - the singleuser servers can't connect to it to authenticate
<yuvipanda>
since it's too busy spawning new stuff
<yuvipanda>
njs: profiling also showed that our authentication related code is a CPU hotspot, which is never good in a tornado app
<yuvipanda>
njs: do you think you can leave a comment there? :D
<njs>
yuvipanda: commented
<yuvipanda>
ty, njs! I'm trying to use passlib to send a PR just now
<njs>
yuvipanda: note that pypy is much happier with extension modules implemented using cffi than anything else
<njs>
(dragging this slightly back towards the channel topic :-))
<yuvipanda>
njs: right, so I could use the aragon cffi implementation
<yuvipanda>
njs: I was surprised that pycurl just worked on pypy!
<yuvipanda>
(even though I'm not using it in this particular implementation)
<yuvipanda>
njs: I'm very new to pypy, so am still a bit unclear on your comment about moving it to a thread - will that help at all?
<yuvipanda>
or rather, I am unsure right now on how the GIL works in PyPy
<yuvipanda>
just the same, different in some ways, etc.. should read up
<yuvipanda>
(not counting STM)
<njs>
yuvipanda: recent pypy has gotten *much* better at handling arbitrary CPython extension modules (thanks to mattip and others); even numpy mostly Just Works now (though scipy etc. don't quite). But classic C extension modules have a lot of overhead the cffi eliminatees.
<njs>
yuvipanda: basically pypy's GIL is just like cpython's
<yuvipanda>
right. so if the implementation I have drops the GIL then threading is gonna gimme a boost, otherwise it's not going to
<njs>
(the exact implementation is a little different, but you don't care about that. mostly it just means it avoids some weird scheduling issues that cpython can sometimes get into.)
<yuvipanda>
right
<njs>
I believe cffi always drops the GIL while calling into C
<yuvipanda>
am not sure if it's related to pypy or the library.
<yuvipanda>
njs: ooooh, cool!
<yuvipanda>
njs: do you know why the py3.5 compatible implementation is considered beta? I couldn't find specifics on the site
<njs>
yuvipanda: ...that kubernetes-client issue is very odd. Maybe something going wrong in X.509 parsing? the pypy 3.5 ssl module in particular is a lot of complicated new code that hasn't been tested much
<njs>
(I guess this also answers your last question :-))
<yuvipanda>
njs: haha :D yes, that's possible. it only shows up under heavy load tho - most of the requests succeed
<njs>
it's definitely not a kubernetes-client issue. It looks like that corrupted string is being returned by the pypy ssl module's getpeercert() code
<yuvipanda>
njs: right.
<yuvipanda>
ooh, pypy is on bitbucket!
<njs>
yuvipanda: you might want to try using urllib3.contrib.pyopenssl.inject_into_urllib3() to switch urllib3 to using pyopenssl instead of the default ssl and see if that helps
<yuvipanda>
ugh, the login system won't let me log in :|
<yuvipanda>
to bitbucket, that is. I think it is confused about my email based login and my google auth based login and won't let me in on either
<yuvipanda>
njs: ah, ok! I'll try that.
<njs>
hmm, you're using threads too... maybe there's some accidental global variable problem happening in the X.509 parser
<yuvipanda>
yeah, that's possible. I've multiple threads that are using ssl all the time
<yuvipanda>
njs: oh yeah, the second biggest CPU hotspot was ssl (with cpython). will pypy automatically help with that or should I explicitly switch it to pyopenssl?
<njs>
yuvipanda: probably there is nothing to be done about ssl, it's all happening in openssl no matter what
<njs>
basically it's beta because they just wrote like 3 releases worth of python features in a few months and you get to be the first person running them in production :-)
<yuvipanda>
njs: I discovered that we used to use passlib but switched it to the custom implementation 3y ago becuase it was a big dependency. I guess that no longer is true because of wheels
<yuvipanda>
njs: yeah, that's understandable and even acceptable :)
<yuvipanda>
njs: haha, right :D
<yuvipanda>
njs: looks like replacing the custom implementation with passlib is going to be a bit of work since we've to be backwards compatible...
<yuvipanda>
njs: I'm guessing just switching it to run on a thread is already going to be useful since I assume hashlib will drop GIL... not sure if pypy's hashlib is in python or not
<njs>
yuvipanda: I think their hashlib is using the same cffi openssl wrapper as their ssl module
* yuvipanda
nods
rokujyouhitoma has joined #pypy
ronan has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
<yuvipanda>
could not load libunwind at runtime. error: libunwind.so: cannot open shared object file: No such file or directory
<yuvipanda>
this is on latest pypy release ([PyPy 5.8.0-beta0 with GCC 6.3.0])
<yuvipanda>
and with pip install vmprof inside a virtualenv created with pypy
<yuvipanda>
ah, seems to be a red herring - I am getting the web URL
<yuvipanda>
the webservice is awesome
gclawes has quit [Read error: Connection reset by peer]
<LarstiQ>
yuvipanda: that's with the portable binaries right? What platform?
gclawes has joined #pypy
<yuvipanda>
LarstiQ: linux (amd64)
<yuvipanda>
Ubuntu 17.04
<yuvipanda>
LarstiQ: and yep, with the portable binaries
<LarstiQ>
hmm, not sure what the exact problem is then, maybe something for squeaky to look into
* yuvipanda
nods
<LarstiQ>
yuvipanda: iirc libunwind is for profiling into C frames, if you don't need that, no problem
* yuvipanda
nods
<yuvipanda>
yeah I don't care in this instance
arigato has joined #pypy
<kenaan>
arigo py3.5 a92ebe04ee55 /lib_pypy/_cffi_ssl/_stdssl/certificate.py: Issue #2615 Don't use a static buffer. Corruption in a multithreaded environment!
<LarstiQ>
good morning arigato :)
<arigato>
hi
<yuvipanda>
arigato: thanks for fixing the bug! I am probably going to be running load tests all week, so if you've a new binary I can try :)
<yuvipanda>
I'm not set up to build pypy unfortunately, and my bitbucket login is in some weird state
<LarstiQ>
yuvipanda: you can kick off a build on the buildbot, or wait for a nightly
<yuvipanda>
ok! I'll wait for the nightly!
<yuvipanda>
thank you for the quick fix!
raynold has joined #pypy
<arigato>
yuvipanda: you don't actually need a new binary
<arigato>
just updating the file "certificate.py" would be enough
<yuvipanda>
fijal: yeah! I'm emitting those as JSON event streams from my stress tester
<yuvipanda>
a bunch of folks are working on making this more realistic - using a poisson process to load it rather than my current '100 users start every minute, go!'
<fijal>
so better/worse than cpy?
<yuvipanda>
fijal: better!
<yuvipanda>
very definitely
<yuvipanda>
I don't know how much better yet exactly.
<fijal>
so orm is the biggest issue
<fijal>
we kinda struggle with orms
<yuvipanda>
I see
<yuvipanda>
fijal: for what reason?
<yuvipanda>
this is just talking to an in-memory sqlite db btw
<yuvipanda>
I haven't switched it out to pg yet
<fijal>
because the way they're written makes it hard to optimize
<yuvipanda>
(am running another 1000 users through the same process to see how that's like)
<yuvipanda>
fijal: ah, right. Just the usual magic making JIT life harder story?
<fijal>
you should really generate code instead of going through dynamic dispatch
<fijal>
Yeah, also unpredictable etc
<yuvipanda>
right
<fijal>
you want assembler copy per call site
<yuvipanda>
I can't change sqlalchemy tho unfortunately
<fijal>
ya i understand
<fijal>
jupyterhub is likely not flush with cash, but we can make you the online version of mprof
<fijal>
vmprof
<fijal>
that is continuous gathering
<yuvipanda>
in production?
<yuvipanda>
fijal: hmm I'm getting a 'connection aborted by peer' if I try to upload a > 100M file btw
<yuvipanda>
doesn't even wait, happens immediately
<antocuni>
yuvipanda: that's a false alarm
<antocuni>
if you ignore it, it just works :(
<yuvipanda>
antocuni: yeah, seems to!
<yuvipanda>
arigato: btw, I can't repro the SSL exception now with your patched file
<arigato>
good
antocuni has quit [Ping timeout: 248 seconds]
ronan has quit [Ping timeout: 255 seconds]
rokujyouhitoma has joined #pypy
<yuvipanda>
alright, thanks all! I've a >500MB profile file I'll figure out what to do with tomorrow :) this has given us concrete steps to move forward :)
rokujyouhitoma has quit [Ping timeout: 255 seconds]
rokujyouhitoma has quit [Ping timeout: 248 seconds]
<plan_rich>
vmprof global variable signal_handler_value has one bit (LSB) that indicates one should ignore signals and all other bits are counters how many threads entered the signal handler
<plan_rich>
it can frequently occur on pypy that e.g. pypy_codemap_invalidate_set or some other functions called from the jit enable the signaling again after some one else has called vmprof_ignore_signals(1)
<plan_rich>
can I turn the signal_handler_value into a counter that counts how often vmprof_ignore_signals is called?
asmeurer__ has joined #pypy
* plan_rich
turns that into a counter for now
<plan_rich>
well, no. I need another solution for that...
kipras`away is now known as kipras
asmeurer__ has quit [Quit: asmeurer__]
<mjacob>
ronny: you implemented most of build bot test summary, right?