cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
<kenaan>
mattip default 8d3df17fc5ee /pypy/: remove the _ssl module; we now use the cffi-based one
<mattip>
let's see what breaks
jcea has quit [Remote host closed the connection]
jcea has joined #pypy
<Dejan>
i smell success
<Dejan>
:)
<cfbolz>
mattip: Yay, deleting code is good!
edd[m] has quit [Write error: Connection reset by peer]
bendlas has quit [Write error: Broken pipe]
agates[m] has quit [Write error: Connection reset by peer]
extraymond[m] has quit [Remote host closed the connection]
Ashleee has quit [Ping timeout: 240 seconds]
Ashleee has joined #pypy
dddddd has joined #pypy
jcea has quit [Remote host closed the connection]
antocuni has joined #pypy
jvesely has joined #pypy
jvesely has quit [Quit: jvesely]
jcea has joined #pypy
BPL has joined #pypy
BPL has quit [Remote host closed the connection]
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
BPL has joined #pypy
BPL has quit [Remote host closed the connection]
antocuni_ has joined #pypy
antocuni has quit [Ping timeout: 252 seconds]
Nik8 has joined #pypy
<Nik8>
Hi, Im trying to benchmark some code on pypy using pytest-benchmark (that using cprofile), and im getting different results in different runs, lets say this exmaple: https://pastebin.com/gq5T9vUs
<mattip>
I don't know details of pytest-benchmark, but using cprofile is going to interfere with the JIT.
<Nik8>
Im benchmarking the code in the same environment if i wanst clear, just the "same" run twice (or more)
<Nik8>
with some warmup runs.
<Nik8>
what will be recommended instead of cprofile? and maybe do you know any reference of code tath does benchmarking of pypy code?
<mattip>
it depends what you are trying to do
<mattip>
microbenchmarking is nice for comparing two versions of pypy, not for comparing across implementations
<mattip>
if you want to compare cpython to pypy, you need to do it with your actual code
<mattip>
for profiling, we recommend statistical profilers like vmprof
Nik8 has quit [Ping timeout: 260 seconds]
firespeaker has quit [Quit: Leaving.]
BPL has joined #pypy
BPL has quit [Remote host closed the connection]
<mattip>
pytest asserts no longer print the values on default as well as py3.6, is that on purpose?
<mattip>
ronan: ^^^
<cfbolz>
mattip: yes, I noticed too, sounds wrong
<cfbolz>
nik8 is gone again, hm
altendky has joined #pypy
YannickJadoul has quit [Quit: Leaving]
Nik8 has joined #pypy
<Nik8>
Im trying to benchmark an action which take 300-400 ms, that considered as micro-benchmark? (im using 9-10 warmup iterations)
<cfbolz>
Nik8: what does your *actual* workload look like
<cfbolz>
Nik8: also, what do you see for this concrete example that you consider "getting different results"?
<Nik8>
im profiling and benchmarking creating big python object, which is created from json and includes some more logic processing, i dont have a good special way to describe it, just normal bussiness logic
<Nik8>
different results = +- 15% of the time it took
<Nik8>
Today, the described action is taking "too long" and i want to make it faster, im searching for a way to measure if my changes improve anything. if there is any other automated recommended way ill be glad to hear
<Nik8>
And if you know any project which use pypy and does those things (benchmarking) ill be glad to read about it also
<cfbolz>
+-15% in the above example is unfortunately pretty common
<cfbolz>
on my machine, do() is 30ms, lots of noise sources on the ms scale
<cfbolz>
you always need to run several processes, and average the results
<cfbolz>
and look at the variation, to see whether your improvement is statistically significant
<Nik8>
the benchmark plugin of pytest run several iterations and does that
<Nik8>
(not processes)
<cfbolz>
Nik8: in the same process? or starting a new process
<cfbolz>
the latter is really needed
<Nik8>
why?
BPL has joined #pypy
<cfbolz>
because a lot of the variation gets reset when you start a new process. you could get unlucky with address space randomization, for example
<cfbolz>
(or with the order in which pypy compiles stuff)
<cfbolz>
(fwiw, I see the same relative variation on the pasted example in CPython, about 5%)
<cfbolz>
but I was just running it as a script, not with pytest-benchmark
<Nik8>
Vmprof, as i saw is a profiler which show me the plot of the code time consuming, What did you meant to use it for benchamarking?
<cfbolz>
Nik8: vmprof is great for showing you which parts of your code are slow
<cfbolz>
But it will give you slightly slower results than running outside of vmprof
<Nik8>
How the measuring of speed.pypy are done?
<cfbolz>
Just a loop, with time.time calls around
<Nik8>
Ill look for the code
<Nik8>
WIth several processes and so on?
<mattip>
bitbucket.org/pypy/benchmarks
<cfbolz>
Nik8: unfortunately not
<cfbolz>
That's why the day to day results are so noisy
<Nik8>
That using cprofile -_-
<cfbolz>
Nik8: if you manage to extract a piece of your actual workload into a benchmark, that would be quite interesting for us, fwiw
<cfbolz>
Nik8: where do you see cprofile usage?
<Nik8>
My bad, just an option to use, configurable
xcm has quit [Read error: Connection reset by peer]