cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
BPL has quit [Quit: Leaving]
marky1991 has quit [Ping timeout: 276 seconds]
_whitelogger has joined #pypy
Rabid_Python has quit [Quit: KVIrc 4.9.3 Aria http://www.kvirc.net/]
dddddd has quit [Remote host closed the connection]
cjwelborn has quit [Quit: sys.exit(0)]
<mattip> cfbolz: there is an own failure, some interaction with _readline and utf8
<cfbolz> mattip: thanks, will fix in a bit
<mattip> cool, thanks for the speedups and fixes
<cfbolz> mattip: after this change is merged to py3.6 we are beating CPython3 on AliReza's issue
<cfbolz> Not massively, but by 20% or so
<mattip> there is not alot of overhead in "read from stdin, write to stdout" in cpython, so even pulling even is an achievement
<cfbolz> mattip: we are jitting the loop, so I think our io and unicode infrastructure still has some overhead
<cfbolz> Also, we are hitting the ASCII case in this example, things are probably different for 'real' unicode
antocuni has joined #pypy
ajlawrence has joined #pypy
<kenaan> cfbolz default 6e25c50447f0 /pypy/module/_io/: fix corner case about readline with limit that goes beyond the end of the string
BPL has joined #pypy
antocuni has quit [Ping timeout: 245 seconds]
<mattip> yay for hypothesis and corner cases
<cfbolz> mattip: yes, what would we do without it
dddddd has joined #pypy
ajlawrence has quit [Remote host closed the connection]
<kenaan> cfbolz py3.6 6e832892a7f7 /: merge default
Masklinn has joined #pypy
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
<Dejan> cfbolz, this paper looks like they are still far from compressed pointers in V8
antocuni has joined #pypy
<cfbolz> Dejan: I read it as they are getting there ;-)
astronavt has quit [Quit: ...]
astronavt has joined #pypy
<Dejan> Exactly 6h ago: "<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-linux-aarch64/builds/60 [py3.6]"
<Dejan> and it is still waiting for lock
Masklinn has quit []
<Dejan> Do you guys need more aarch64 builders?
<Dejan> I have an AArch64 VPS that is mostly idle
<Dejan> on scaleway.com
<cfbolz> Dejan: talk to mattip, we might!
Dejan has quit [Quit: Leaving]
Dejan has joined #pypy
xcm has quit [Read error: Connection reset by peer]
xcm has joined #pypy
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
marky1991 has quit [Ping timeout: 265 seconds]
<astronavt> do packages that don't work with pypy generally fail to pip install? or would i have to try it and find out the hard way
<astronavt> what _doesn't_ work with pypy anyway at this point? just certain C extensions?
<simpson> There are some documented quirks of CPython as well: http://pypy.readthedocs.io/en/latest/cpython_differences.html
<simpson> Generally, code will install fine, if its dependencies are compatible, but only fail at runtime.
<astronavt> ah thats too bad. i guess youd need the developer to be a good citizen and check for non-cpython in setup.py
<astronavt> or maybe theres a way to specify in pyproject.toml
<simpson> The most common failure mode you might not already know about is file-descriptor exhaustion, which occurs when people forget to close what they've open()'d.
<Dejan> astronavt, that is why there are thousands of tests to run
<astronavt> simpson that would mean the library developer isnt using 'with open'? i never want to install those packages anyway :P
<astronavt> the weakref stuff looks more problematic
<astronavt> and __del__ not being called reliably
<simpson> astronavt: Such code is depressingly common IRL. And folks don't always come to that conclusion; I recall a session with some folks from Intel a few years ago, where they weren't sure whether their code was buggy, or PyPy was buggy.
<simpson> Well, as you say, there's style, and then there's smell. .__del__() is already a risky method.
<astronavt> good point
<astronavt> "For example, a generator left pending in the middle is — again — garbage-collected later in PyPy than in CPython." this is where i imagine stuff like memory leaks could arise even if you arent messing around with weak references and such
<simpson> In usual use, no, it's not a problem. You'd have to leak the generator in order to leak its frame. What's really being communicated here is a meta-pattern: PyPy doesn't have reference-counting, so unlike with CPython, sometimes objects aren't destroyed immediately upon becoming inaccessible.
<astronavt> i see. so it will still be gc'ed if it goes out of scope, just not _immediately_
<Dejan> I am not an expert, but i think CPython does not use RC either
<Dejan> It uses generational GC
<simpson> Right. It's like any other GC'd language with finalizers; the finalizers run when the objects are reaped, but that's disconnected from when the objects are dereferenced.
<astronavt> i thought reference counting was like the heart and soul of cpython semantics
<simpson> Dejan: CPython has a GC and also RC. The GC chiefly is used to break cyclic references.
<astronavt> thats the only reason i know what reference counting even is
<astronavt> "C-API Differences" <- so other than stuff that uses CPython internals, a "correct" C extension should work, albeit maybe with extra overhead
<Dejan> how that hybrid even works?
<astronavt> looks like basic cython code actually works w/ cpyext, cool! http://docs.cython.org/en/latest/src/userguide/pypy.html
<Dejan> with if both GC decides to free some memory referenced by RC?
<Dejan> or vice-versa
<simpson> Dejan: All objects are registered with the GC and also have an RC field. RC happens on every access. GC runs when memory is low.
<simpson> GC can read the RC field.
<Dejan> aha!
<Dejan> thanks!
<simpson> astronavt: "It's about time." It's really too bad that the Cython team insists on integration on their terms; it could be a lot smoother and simpler if they stopped insisting on so much C.
<astronavt> what do you mean "insisting on so much C"
<simpson> Cython's promise is that we can write code that is basically Python, and it goes fast. PyPy's promise is that we can write Python, and it goes fast.
<simpson> There's another part of Cython focused on CFFI, but really folks should be using cffi for that.
<Dejan> I do lots of D coding for my personal stuff mostly, at work I use Python predominantly... I came here as a Java programmer...
<Dejan> to me, PyPy is the reason why I started coding Python
<Dejan> CPython is just too slow for some use-cases, and I do not want to have to re-write pieces of code in Cython, or C/D
<Dejan> PyPy makes me believe Python has future :)
ronan has joined #pypy
Dejan has quit [Quit: Leaving]
ronan has quit [Ping timeout: 240 seconds]
antocuni has quit [Ping timeout: 276 seconds]
<arigato> cfbolz: compressed pointers: yay?
<arigato> this document throws some possible plans around
<cfbolz> arigato: yes, I agree that we don't know it will actually work in the end
<arigato> it limits the total memory to 4GB though
<arigato> the old branch in which I played with that compressed pointers to still allow for 32GB of memory
marky1991 has joined #pypy
<cfbolz> arigato: right
<cfbolz> arigato: did our branch work with the jit, do you remember?
<arigato> no, it didn't
<arigato> (i.e. yes I remember)
<cfbolz> Right
<arigato> also it did the thing that can't work on OS X, which is convincing the kernel to give us the initial 4 GB of addressable space
<arigato> note that the stm branches we did later implement something similar to the more general indirection needed
<arigato> basically they implement everything we'd need except they still used 64 bits to represent the modified pointers
<arigato> ("modified" as in made relative to a single global address (or thread-local in the case of stm))
<cfbolz> arigato: right, good to know if we ever want to try again
<cfbolz> arigato: and the jit was supporting stm, right?
<arigato> yes
<arigato> so we'd know exactly the places to fix, at least
<cfbolz> Cool
<arigato> I'm still a bit skeptical, but maybe a plain max-4-GB-for-everything-allocated-by-the-GC would work to avoid most speed overheads (and maybe be faster in the end)
<cfbolz> arigato: why is that so much faster than 32gb?
<arigato> it would of course consume less RAM, but because the limit is 4 GB, it's not very useful for machines with a lot more than 4 GB
<arigato> the 32 GB version has got overhead because it needs to shift by three bits, and then also (as the V8 page discusses) it's a simplification if someone can look at 8 bytes of memory and get its value as a GC pointer whether it is "compressed" or not
<arigato> but maybe that's not a great argument, it might only say we were lazy somewhere
<cfbolz> arigato: the overhead was a small percentage only in any case, iirc
<arigato> OK
<arigato> then maybe we should (one day) try again with the 32 GB limit, but this time including all GC arrays? I think GC arrays were allowed to be anywhere
<arigato> in theory because it makes the memory limit much higher, but in practice because our GC uses plain malloc for large arrays
<cfbolz> arigato: right
<arigato> (the stm branch has got that fixed, because large arrays need to be allocated together with the rest)
<arigato> (and the GC header was still 64 bits when we could also force that inside 32 bits, etc.)
<cfbolz> arigato: are any of the changes there generally useful and we should merge them at some point?
<arigato> I'm not sure
<arigato> remember that it had its own GC written in C
<cfbolz> Right
Dejan has joined #pypy
dansan has quit [Remote host closed the connection]
dansan has joined #pypy
marky1991 has quit [Ping timeout: 240 seconds]
<Dejan> the failed test is weird
marky1991 has joined #pypy
<Dejan> where can i find the source: pypy/interpreter/test/apptest_coroutine.py
<Dejan> (I am looking at http://buildbot.pypy.org/builders/pypy-c-jit-linux-aarch64/builds/60/steps/shell_12/logs/pytestLog and I found the test in this file failed)
<Dejan> thing is... the test should not fail
<Dejan> the result is the same what i have on CPython
<Dejan> unless with PyPy we expect different behaviour?
<arigato> so we do gc.collect() in this test to make sure the coroutines are collected and the warnings emitted, but we also find some coroutines left around from previous tests
<cfbolz> Dejan: fwiw, it fails on x86 too
<arigato> yes, my quick fix was enough (see b8f000ca2554)
<kenaan> arigo py3.6 b8f000ca2554 /pypy/interpreter/test/apptest_coroutine.py: Emit warnings from unrelated older tests before catching the warnings from this precise test
<arigato> so there were other coroutine objects left lying around, which were waiting for the next GC to emit a warning
energizer_ is now known as energizer
<arigato> that's why the test failed by getting 3 warnings where only 1 was expected
<cfbolz> arigato: I've been wondering, this is the struct for the utf8 index storage:
<cfbolz> UTF8_INDEX_STORAGE = lltype.GcArray(lltype.Struct('utf8_loc_elem',
<cfbolz> ('ofs', lltype.FixedSizeArray(lltype.Char, 16)),
<cfbolz> ('baseindex', lltype.Signed),
<cfbolz> ))
<cfbolz> arigato: due to alignment, every array entry will be 2*wordsize big, right?
<cfbolz> ah, sorry. no that works out fine
<cfbolz> it's three words per array entry on 64 bit
marky1991 has quit [Ping timeout: 265 seconds]
<arigato> yes
<cfbolz> arigato: I feel there are still some optimizations missing somehow. eg if we have a string of length 1 and we index it, we make such an array of size 1
<cfbolz> (probably to not make a bridge that checks the size?)
<arigato> maybe create_utf8_index_storage could use a cache for very small string sizes?
<cfbolz> arigato: to share the structures?
<arigato> yes
<cfbolz> right
<arigato> maybe just for utf8len==1
<cfbolz> for size 1 there are four different ones, for size 2 16
<cfbolz> right
<arigato> yes
marky1991 has joined #pypy
<arigato> or for utf8len <= 3
<cfbolz> ah no, it's less
<cfbolz> the ascii ones are never created
<arigato> yes, and there is only the position of indices 1, 5, 9,...
<cfbolz> true
<cfbolz> arigato: I think the other optimization that would be useful is to not create a structure at all for things like s[1:] and s[:-1]. we could only do that optimization in the jit, with jit.isconstant
<arigato> ouch, yes
<cfbolz> ok, will look into these
<arigato> s[:-1] could be done always, by just copying the _index_storage of the bigger string and assuming we're not really loosing memory
<arigato> s[1:] is quite a bit harder
<arigato> ah no, I see what you mean
<cfbolz> arigato: the goal for both would be to never even create the index
<arigato> yes
<cfbolz> by just calling next_codepoint_pos
<arigato> yes
<cfbolz> ok, on it :-)
<arigato> :-)
<arigato> right now, even doing s[0] builds the index I think
<cfbolz> of course
<cfbolz> arigato: but again, I would only do it under an isconstant guard, right?
<cfbolz> otherwise bridges
<arigato> yes
marky1991 has quit [Ping timeout: 265 seconds]
<Dejan> Time to go home, have a nice weekend
Dejan has quit [Quit: Leaving]
speeder39_ has joined #pypy
marky1991 has joined #pypy
edd[m] has quit [Write error: Connection reset by peer]
bendlas has quit [Remote host closed the connection]
bendlas has joined #pypy
edd[m] has joined #pypy
marky1991 has quit [Ping timeout: 265 seconds]
speeder39_ has quit [Quit: Connection closed for inactivity]
CrazyPython has joined #pypy
CrazyPython has quit [Client Quit]
speeder39_ has joined #pypy
CrazyPython has joined #pypy
CrazyPython has quit [Read error: Connection reset by peer]