antocuni changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "PyPy: the Gradual Reduction of Magic (tm)"
<kenaan>
antocuni keep-debug-symbols 005844269621 /pypy/tool/release/: add a smartstrip tool, which can optionally keep the debug symbols in a separate file, instead of jus...
<kenaan>
antocuni keep-debug-symbols 7108fe737121 /pypy/tool/release/package.py: remove the nostrip option; introduce a --no-keep-debug option; use smartstrip(keep_debug=True) by def...
tbodt has quit [Read error: Connection reset by peer]
adamholmberg has quit [Read error: No route to host]
adamholmberg has joined #pypy
antocuni has joined #pypy
Remi_M has joined #pypy
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
kenaan has joined #pypy
<kenaan>
antocuni keep-debug-symbols f29f0f12ffa8 /: close branch to be merged
<kenaan>
antocuni default 77fff565d382 /pypy/tool/release/: merge the branch keep-debug-symbols: - symbols are stripped from the executable and placed in a file libpypy-c....
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
<arigato>
fijal: pong
<fijal>
arigato: I need a few things. One is guidance of where to take utf8 branch next
<fijal>
the other thing is the total memory GC allocates
<fijal>
and the third is review of the intro materials, I'm about to finish the third part
<fijal>
the third can wait, pick at most one of the previous two :)
<arigato>
I answered on #pypy-sync about the total memory, right?
<fijal>
9:58 AM <arigato> yes, everywhere in incminimark.py and minimarkpage.py (it's not many places but don't forget one)
<fijal>
9:58 AM <arigato> e.g. minimarkpage.allocate_new_arena
<arigato>
yes
<fijal>
so, what is "everywhere"?
<fijal>
everywhere that calls malloc?
<arigato>
llarena.stuff, I think
<fijal>
llarena.arena_malloc
<fijal>
so the other bookkeeping data should be small right?
AndrewBC has joined #pypy
adamholmberg has quit [Ping timeout: 248 seconds]
<arigato>
from allocate_new_arena? yes
<fijal>
ok
<arigato>
ah, or you mean, all the lists and deques of addresses
<cfbolz>
if you have a lot of code in python, that part will be faster
<lesshaste>
but aren't those measuring the speed of calling (empty) C functions?
<cfbolz>
yes, but there is still a loop on the python side
<cfbolz>
that loop is slow in cpython
<lesshaste>
oh I didn't realise that was where the speedup was
<lesshaste>
it isn't clear from the blog post at all
<lesshaste>
if anyone would like to use that code I pasted as a benchmark I can add some simple data generation too
<antocuni>
cfbolz: actually from what I can see by running the branch on simple benchmarks, the pure-python speedup is huge enough that you get an overall win even if your benchmark is cpyext-intensive
<lesshaste>
please just let me know
<fijal>
lesshaste: if you want to submit stuff like that, PLEASE provide a self-running example
<fijal>
lesshaste: this is a function which I have no clue whats' the input
<antocuni>
for example, the numpy version of code which I showed in my last blog post: on CPython is ~6.5 seconds; on PyPy is ~33 seconds; on cpyext-avoid-roundtrip is ~5.5 seconds
<fijal>
lesshaste: but it should be as easy as downloading the nightly and trying, too
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
<antocuni>
there is no recent nighly of the branch, AFAIK
danieljabailey has quit [Ping timeout: 258 seconds]
<arigato>
fijal: strategies: what cfbolz said is a good (and maybe only) first step
adamholmberg has quit [Ping timeout: 240 seconds]
<lesshaste>
I hope it's helpful
<antocuni>
note that this particular example might not be fast on the branch. I **think** that special methods like __getitem__ goes trough a code path which is not currently optimized
<lesshaste>
antocuni, do you mean my example?
<antocuni>
yes
<fijal>
arigato: so no lists?
<lesshaste>
antocuni, ah ok. I hope it will be of some use in the future. My timing claims were a bit off. for n = 20 it's about 7 secinds
<lesshaste>
but it increases exponentially with n
<lesshaste>
so it's very slow by n = 25
<lesshaste>
fijal, it computes the permanent of a matrix
<arigato>
fijal: for lists, it may make sense anyway to have a strategy, *just* to remove the type check when we get elements
<lesshaste>
antocuni, I have a pure python version of the function too if that's of interest
<cfbolz>
arigato: if we want that, we might want to enable the general strategy of 'all elements have the same rpython type'
<antocuni>
lesshaste: yes please
<cfbolz>
Which exists somewhere (a branch?)
<arigato>
cfbolz: yes, but it might be the case that it's good to enable it for unicodes only, and less good for everything (for all I know)
<cfbolz>
No, enabling this changed nothing measurable, in either direction :-(
<antocuni>
can't we just have a list of unwrapped unicodes and forget about indexes?
<lesshaste>
antocuni, it's sped up very nicely by pypy
<arigato>
cfbolz: it may help u''.join() for example
<arigato>
cfbolz: (not in the jit)
<cfbolz>
Indeed
<antocuni>
and maybe, if we somehow detect that we recompute the index again and again, we switch the strategy
<cfbolz>
antocuni: it's not just indices, it's also the length
<arigato>
cfbolz: but no real clue
<antocuni>
ah
<antocuni>
then we keep the lengths in a parallel rpython lists, or something
danieljabailey has joined #pypy
<antocuni>
but indeed, I'm not sure it's worth
<arigato>
the length can also be recomputed, too
<lesshaste>
antocuni, you will see that the implementation is not identical to the numpy version which may not be ideal..sorry. The numpy version was written to try to use numpy functions as much as possible
<arigato>
but I think that the regular W_UnicodeObject has the length non-lazily
adamholmberg has joined #pypy
<antocuni>
lesshaste: as usual :)
<lesshaste>
antocuni, :)
<lesshaste>
if I can explain any aspect of the code please just ask
<lesshaste>
it is interesting to me how slow the numpy version is even using cpython. It seems that cpython has some large overhead to access array elements too
<arigato>
I think you should detect and ignore AssertionError and NotImplementedError
tbodt has quit [Client Quit]
drolando has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<arigato>
(these two are not officially catchable in RPython)
<cfbolz>
when do we raise NotImplementedError? in abstract base methods?
<fijal>
yeah
drolando has joined #pypy
<cfbolz>
rindolf: my very short looking around suggests that what is slow is this computation: ((sq*sq) % powers[n]) != sq
<rindolf>
cfbolz: probably
<arigato>
maybe rewriting it as pow(sq, 2, powers[n]) ? don't know if it's going to be faster
<cfbolz>
the modulo is the problem
<arigato>
yes, but pow() exists with three arguments (doing a modulo) for a reason
<fijal>
uh
<fijal>
how do I get type of class from block.inputargs again?
<arigato>
cfbolz: don't know if this reason is only valid for greater powers than 2
<arigato>
fijal: likely, you need to track it down a bit
<Alex_Gaynor>
arigato: it is; it interleaves the modulos to "reduce" the value every once in a while. with only 1 mul there's nothing to interleave the modulos with
<arigato>
thanks
<xorAxAx>
yes, pow() will use square and multiply
tbodt has joined #pypy
<fijal>
arigato: and is there a way to get it from struct* assertion error to AssertionError?
<fijal>
rclass.some_cast?
* arigato
digs in rtyper/
<fijal>
arigato: ok, if you don't remember I can dig too :)
<arigato>
get_standard_ll_exc_instance_by_class()
<arigato>
in rtyper/exceptiondata.py
<arigato>
looks promizing
TheAdversary has joined #pypy
asmeurer_ has quit [Quit: asmeurer_]
<rindolf>
cfbolz: arigato : the gmpy-based port runs much faster in cpython 2.7 - about 1m17s and i am unable to build gmpy or gmpy2 for pypy using pip
<arigato>
rindolf: thanks. indeed
<arigato>
gmpy and gmpy2 are both supported in pypy via cpyext, and they both access the internals of python's long objects
<arigato>
it could be regarded as a bug of cpyext, but well
<arigato>
we'd need to emulate the whole CPython way to represent long integers, which is slightly different in PyPy
<arigato>
if you only need a few functions, and really want to try PyPy, then try this:
<arigato>
I think it contains enough asserts, but I'm concerned about whether all existing tests pass (and pypy translates)
<fijal>
right
<fijal>
so thats's another question - I can put it on branch and run buildbots
<fijal>
cfbolz: uh, there is not a single test for can_raise that's different than what I wrote
<cfbolz>
Even worse :-P
<fijal>
yeah I know, but what can I do?
<cfbolz>
fijal: sorry, what I mean is really just test a function that contains an assert statement, as opposed to one that contains an assert many levels away
<arigato>
run all the tests anyway
<arigato>
in that directory
<arigato>
as a first step
<fijal>
cfbolz: ah :)
<cfbolz>
And definitely a test about NotImplementedError, with the standard pattern we use (an abstract method in the base, and concrete implementations)
<fijal>
ok so the raise statements are not always 4 operations of course
<fijal>
eh
<arigato>
I think you should ask the question about "preblock.exits[0].args[0]", resolving the same_as() as long as it's a Variable
<arigato>
ah I guess it's "preblock.exits[0].args[1]"
<fijal>
so now there is a question what do we do with a block that has direct_call(something_always_raises) -> except_block
<arigato>
that's fine if RaiseAnalyzer thinks this precise function can't raise, because it calls something else that always raises
<arigato>
so it will still say Yes
<arigato>
(but test!)
<fijal>
yes that's what I did
<fijal>
(write a test)
<arigato>
I wonder though
<arigato>
if we drop the guard_exception in the JIT
<arigato>
but an AssertionError is really raised
<arigato>
I think that nonsense can follow
<fijal>
you need a debug build right?
<arigato>
no, failing asserts are always there
<arigato>
they cause fatal rpython errors
<fijal>
right
<arigato>
but only at the point where they are "caught" I think
<fijal>
so maybe we should propagate them like MemoryError?
<arigato>
maybe
jcea has quit [Remote host closed the connection]
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
drolando has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
tbodt has joined #pypy
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<mattip>
fijal: do you mean run_subprocess ? AFAICT it forks once before lots of memory is alloctaed, then forks the smaller-footprint process again at each Popen-like call
<mattip>
but still forks each time
<fijal>
ah right, because we have to call exec
<fijal>
is there a way to cheaply call compiler?
<mattip>
not with python AFAICT, I looked around a bit.