zmt00 has quit [Read error: Connection reset by peer]
zmt00 has joined #pypy
arigo is now known as arigato
<arigato>
trying to get out the cffi 1.12 release
<arigato>
there is a rare race condition, though, that only shows up sometimes on Windows only
<arigato>
with CPython
<cfbolz>
arigato: hey!
<cfbolz>
got home ok?
<arigato>
hi! yes, thanks
<cfbolz>
cool :-)
<arigato>
still veeeeeery tired
<cfbolz>
same, actually
<cfbolz>
arigato: I re-remembered what the problem with the more compact integer sets is: there is no cheap way to compute their length
<arigato>
ah
<arigato>
can you keep the length as a separate field?
<cfbolz>
arigato: yes, but then you need to update it a bit everywhere (using lots of popcounts, I suppose?)
<arigato>
or not really, because it makes operations like unions and intersections much slower?
<arigato>
right
<cfbolz>
not sure it's bad, popcount is probably very cheap nowadays
<arigato>
it would probably be fine performance-wise but it needs extremely careful tests
<cfbolz>
yep
<arigato>
if it can get out of sync in one corner case, boom
<cfbolz>
yes
<cfbolz>
anyway, it's more work that I am willing to do at this point ;-)
<arigato>
OK :-)
<cfbolz>
arigato: I also looked more into stringsearch: normal searches become 5% slower, whereas constant search strings speed up by 2x. worth it or not?
<arigato>
meh, probably
<cfbolz>
hah
<cfbolz>
I think the answer is no, and I should rather at some point try to speed up string search more generally
<arigato>
OK
<arigato>
unicode-utf8: note that it will often make 4 bridges for the 4 possible lengths of characters
<cfbolz>
"ah" :-(
<cfbolz>
arigato: is that a problem for the merge?
<arigato>
if we hide that code inside @jit.dont_look_inside or @jit.elidable, then it ends up building more temporary string objects that were removed with optimizeopt/vstring
<cfbolz>
right
<antocuni>
let's build a method jit!
* antocuni
runs away
<arigato>
precisely :-)
<arigato>
so I don't know, but I think it's fine to merge anyway
<arigato>
the vstring problem show up when we do things like ord(unicode[index]): the unicode[index] part builds a string of length 1 to 4, which is then consumed by ord()
<arigato>
right now, the unicode[index] branches for all lengths, but then the ord() on the correct branch is super-fast
<cfbolz>
right
<arigato>
I would guess it's the same with "unicode[index] == 'X'"
<cfbolz>
arigato: I fear we really need experience in the wild to know whether it's a problem
<arigato>
yes
_whitelogger has joined #pypy
<mattip>
arigato: is there a concise sample of the windows cffi cpython code that causes a race condition?
<arigato>
mattip: it's in testing/embedding/test_thread.py. I'm working on it
<arigato>
the C program is in thread3-test.c
<arigato>
it's a C program that embeds two different bits of Python code from two different threads
<arigato>
...it seems to be a false alarm somehow: the final printf("done\n") goes nowhere, but adding a fprintf(stderr, ...) afterwards shows up on the screen
<mattip>
can you release with "known issues" and /solve it/let someone else solve it/ later?
<arigato>
yes, I could do that
<mattip>
it might be a windows problem with the console output handling
<arigato>
yes, it seems to be the case
<arigato>
it's very obscure but I really think there is nothing wrong with my code here
<arigato>
the printf("done\n") gets executed by the main thread after it is done waiting for the other threads, but it just doesn't get output at all
<mattip>
+1 for calling it a windows bug
<arigato>
cool. release!
<kenaan>
arigo cffi/cffi 262a22d485b7 /testing/embedding/test_thread.py: Add comments about this test failing occasionally on Windows
antocuni has quit [Ping timeout: 268 seconds]
<arigato>
@Alex_Gaynor: are you still ready to do the binary releases of cffi?
<arigato>
(last release was almost one year ago!)
<xorAxAx>
arigato, why not flush explicitly?
<kenaan>
arigo cffi/cffi[release-1.12] 4c6a0cf68db6 /: make release branch
<xorAxAx>
(by fflush())
<arigato>
xorAxAx: why would a missing fflush() not matter in 99% of the cases in a situation where all other threads are finished already? but I can try
<xorAxAx>
because the main thread flushes and then joins the other threads?
<xorAxAx>
(a guess)
<xorAxAx>
(in other words, the join is implicit and the flush is explicit in the libc end-function)
<arigato>
ok, indeed the main thread doesn't join the other threads (it only waits until they activate a semaphore, which they do just before finishing)
<Ninpo>
mattip: I got side tracked the last few days with something that needed to happen NOW NOW at work. I'm happy to tweet something, a blog post may take a while though, got a _lot_ on
<Ninpo>
Also public writing about work stuff has to be cleared because my employers are dicks, despite being happy to stand on the back of open source.. :|
<cfbolz>
Ninpo: tweet would already be awesome
<cfbolz>
Ninpo: you tweet I retweet?
<Ninpo>
Sure, though I'm not technically running a build on the latest branch :D
<kenaan>
arigo cffi/cffi 7eba738b73fb /testing/embedding/: Found out a way that appears to fix the problems on Windows
<Ninpo>
what hashtags do you guys use for pypy stuff?
<Ninpo>
Running the fantastic PyPy project's unicode-utf8 branch on a complicated text analysis project cut our runtime from 14+ hours to less than 10. Amazing work! #python #pypy #utf8 #unicode
<Ninpo>
That alright?
<cfbolz>
Ninpo: wonderful :-)
<cfbolz>
Ninpo: ah, do you have some cpython numbers to compare too, by chance?
<fijal>
Ninpo: "now now" is a saying in south africa that means "not immediately" ;-)
<arigato>
xorAxAx: thanks for the idea, it seems to have fixed the problem
<Ninpo>
Running the fantastic PyPy project's unicode-utf8 branch on a complicated text analysis project cut our runtime from 14+ hours to less than 10 (CPython 4 times slower still) Amazing work! #python #pypy #utf8 #unicode
<xorAxAx>
mjacob, the branch ann-systemexit seems to fix my initial problem
<Ninpo>
btw, when building pypy, is the creation of a "python" symlink something I can ask the package script to do? Or is it something I need to do somehow post build
<Cld>
mattip, fijal: well in fact I am the one that do the wrapper to a C++ library. The C++ function give me a std::string and for the moment i do an allocation of a new char * array and do a strcpy...
<fijal>
Cld: right, so the "more official" way to do it is to pass an allocated string
<fijal>
but it really does not matter
<arigato>
yes, common C APIs receive a "char *" buffer for the output string and the length of that buffer, and there is a mess if the string is too large
<arigato>
if you just call malloc() and expect the receiver to call free(), then yes, you simply need to declare free() in the cffi.cdef
<Cld>
ok
<Cld>
in all case a can't give a the pointer to an already allocated 'bytes' to reduce the number of copy ?
<Ninpo>
cfbolz: Does that branch include your jit improvements?
antocuni has joined #pypy
<Ninpo>
Oh yeah, how _does_ pypy do with recursion? I know it's an issue in cpython...
antocuni has left #pypy [#pypy]
<Ninpo>
As in generally, don't
antocuni__ has joined #pypy
<Ninpo>
I enjoy playing with FP you see
<arigato>
Cld: no, you can't do that, but there might be a way to remove the strcpy you do in the C wrapper function
<arigato>
it's not very easy because it involves keeping the C++ std::string object alive
fling has quit [Ping timeout: 250 seconds]
<arigato>
if you manage to do that, then just return "my_std_string_that_wont_die_just_now.c_str()" from the C wrapper
<fijal>
Ninpo: as bad as cpython
<fijal>
as in, recursion consumes C stack which is not infinite
<fijal>
it also does not play well with the JIT
<arigato>
Cld: there are also solutions involving callbacks: you call a C/C++ function that calls back to Python code, with the c_str() as argument, and you call ffi.string() from the callback. there is overhead in doing that though
<cfbolz>
Ninpo: it is, yes
<cfbolz>
fijal: it's still better than cpython
<fijal>
yes, speed wise
<Cld>
arigato: and the C wrapper function return a PyObject * ?
<arigato>
Cld: no, there is no "PyObject *" with cffi
<arigato>
the C wrapper function returns nothing
<arigato>
the callback invokes ffi.string() and stores the result somewhere in Python code
<Cld>
ok
<arigato>
either in some global, or if you want to avoid that, look at ffi.new_handle()
<Ninpo>
fijal: right ok :)
<arigato>
(it is the clean solution---but the callbacks and handles make some overhead)
<Cld>
arigato: yep, but the strcpy and the call to the C free are not free neither...
<Alex_Gaynor>
arigato: wheels uploaded; I'll go remove py33 from the wheel builder for the future
<arigato>
thanks!
<fijal>
Alex_Gaynor: and people want to drop 2.7 support....
<fijal>
bit crazy
<Alex_Gaynor>
fijal: people have weird ideas 🤷
<tos9>
Alex_Gaynor: isn't 3.4 also already out of security support from upstream?
<fijal>
when? end 2020?
<tos9>
or will be at end of month I guess
<Alex_Gaynor>
tos9: idk, in general I think python has few enough security issues that very few people meaningful use upstream support as a proxy for whether they should drop it
oberstet has quit [Quit: Leaving]
<tos9>
Hee hee, I think I had exactly that conversation with njs a month ago, while trying to make this decision for my own software
<fijal>
tos9: mornin
<tos9>
fijal: howdy
* cfbolz
waves
<mattip>
interesting, cffi and numpy have about the same number of pypi installs per month: 11 - 15 million. Lots of CI work?
<mattip>
maybe travis should cache popular downloads
fling has joined #pypy
<fijal>
mattip: pyopenssl relies on cffi
fling is now known as Guest65761
<fijal>
that's 1/3 or so of downloads
<fijal>
maybe more
<Alex_Gaynor>
fijal: it's also every certbot (let's encrypt client), paramiko, AWS encryption tools, etc.
mosajjal has quit [Quit: Ping timeout (120 seconds)]
<fijal>
ya
<fijal>
tos9: I'm chatting with FIT NYC later today
<fijal>
I wonder what I can ask them
marky1991 has joined #pypy
illume has joined #pypy
<kenaan>
stevie_92 cpyext-gc-cycle a316d3f47e9f /: Added interface for rawrefcount finalizers to the gc Added support for rawrefcount finalizers to dot-te...
<kenaan>
rlamy default 6b234ecc047d /pypy/doc/index.rst: Update link to IRC logs
marky1991 has quit [Read error: Connection reset by peer]
<tos9>
fijal: cool -- about what? are you quitting software and joining the fashion industry?
<tos9>
(if so, good idea, should have told me that was an option)
<fijal>
tos9: would not you given an opportunity?
<fijal>
VR Sketch - they're using it
<fijal>
tos9: I think rhettinger used to be a fashion photographer before becoming software engineer
<Ninpo>
btw, when building pypy, is the creation of a "python" symlink something I can ask the package script to do? Or is it something I need to do somehow post build?
<arigato>
Ninpo: as far as I know it's not something that can be done using our scripts
<Ninpo>
ok ta :)
<LarstiQ>
Ninpo: if you use a virtual env, that will do it
Rhy0lite has joined #pypy
<Ninpo>
LarstiQ: yeah, pyenv doesn't (and lost me a day on Monday by not doing so)
<Ninpo>
I plan to raise an issue/PR
beystef has quit [Quit: beystef]
rajgopal has joined #pypy
Garen has quit [Read error: Connection reset by peer]
Garen has joined #pypy
rajgopal has left #pypy [#pypy]
Garen has quit [Read error: Connection reset by peer]
xcm has quit [Remote host closed the connection]
Garen has joined #pypy
xcm has joined #pypy
iomari has quit [Ping timeout: 272 seconds]
iomari has joined #pypy
lazka has quit [Remote host closed the connection]
Garen has quit [Read error: Connection reset by peer]
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
realitix has quit [Ping timeout: 246 seconds]
xcm has quit [Remote host closed the connection]
epelesis has joined #pypy
xcm has joined #pypy
<fling>
emerge -DNkpu world is ~40 seconds faster than with python3.6
<mattip>
fling: out of how many total?
<cfbolz>
fling: nice!
<fling>
3:15 with python
<Ninpo>
Huh I forgot you can use pypy with portage
xcm has quit [Ping timeout: 268 seconds]
Zaab1t has quit [Quit: bye bye friends]
<fling>
Ninpo: I enabled pypy3 as the default and pypy as a fallback
xcm has joined #pypy
<Ninpo>
fling: gonna play with that in a sec :D
<Ninpo>
fling: just pypy ebuilds or?
<fling>
Ninpo: what?
<Ninpo>
as in how did you install pypy for portage to use it
<Ninpo>
Did you use ebuilds or build it yourself or squeaky's build etc
<fling>
done! :D Now emerging using pypy and pypy3 is the default
<Ninpo>
sexy
<fling>
It is always simplier to write an ebuild than to maintain all the deps and to build things running make and installing them to /usr/local somewhere
<Ninpo>
Yeah. I miss gentoo, but I was terrible at keeping up with updates, then I'd run into dep hell when I finally got around to it
<fling>
Ninpo: no need to update everything if you don't want.
<Ninpo>
Very tempted to put it back on my home desktop after using Arch/Antergos for a while. *glares at systemd*
<fling>
Ninpo: just update what glsa-check suggests and you will be fine.
<fling>
Ninpo: you could install gentoo fast with `lxc launch images:gentoo gentoo`
<Ninpo>
:o
<Ninpo>
fling: Don't suppose you've got ebuilds for 7 handy have you?
<fling>
Ninpo: 7 handy?
<Ninpo>
As in, I don't suppose you have easy access to share an ebuild for pypy(3) v7?
<Ninpo>
I'm working on the assumption you made an ebuild for version 7
<Ninpo>
Apologies if not
<fling>
Ninpo: upload it to bugzilla if you have one.
<fling>
Ninpo: there is an ebuild for HEAD version in portage anyway.
<fling>
Ninpo: or are you asking for an ebuild for your own usage?
<Ninpo>
It's ok, if you didn't make one I'll carry on
<fling>
Ninpo: you could just copy the existing ebuild to your local repo. It could be enough to just rename the ebuild to change the version, no need to edit probably if pypy3 7 is not really that different from 6 or HEAD
<fling>
Ninpo: have not you tried building 7 version using the existing ebuild?
<Ninpo>
Yeah, gentoo will build it with cpython though which takes forever, was after bin packages
<Ninpo>
I was about to. I only got reminded that pypy can be used with portage about 20 minutes ago!
<fling>
do I need pypy3 7 too?
<Ninpo>
_need_? No idea :D It's the latest pypy release.
<kenaan>
cfbolz default 8ad869bf0714 /rpython/jit/backend/zarch/regalloc.py: try to fix s390x
<cfbolz>
mattip: after we merge we should write a blog post about the unicode-utf8 branch too. I volunteer doing/helping with that
<fling>
cfbolz: what is so special about this branch?
<cfbolz>
fling: in a sense nothing, we do this with most major branches
<cfbolz>
but otoh it's a pretty fundamental change that a lot of people have opinions about
<tumbleweed>
cfbolz: what is that s390x commit trying to fix?
<Ninpo>
some small/med/large datasets from my overall run
<Ninpo>
small is too small to see a lot of difference but boy does it rack up as you get up there in the minutes
Rhy0lite has quit [Quit: Leaving]
<fijal>
fling: on our blog post in the future :)
<Ninpo>
Anyone interested in looking at a cprofile output of my app that I'm getting those numbers with? I've never looked at it before and the learning curve seems steep, any hints as where to consider spending some time with improvements would be very much appreciated. Not expecting a full code review, but if anything jumps out as "Have a look in that function" or what not would be appreciated?
<Ninpo>
No worries if not, what I _think_ I'm understanding is there's not a lot I can do further
<fijal>
Ninpo: generally speaking I would need a program I can ryn
<fijal>
run
<fijal>
the cProfile alone is not helpful
<Ninpo>
I feared that might be the case. Stupid work :/
<Ninpo>
I've found a tool that brings it up in kcachegrind, trying to find good docs to grok it
<Ninpo>
I think I may bite the bullet and make a non sqlalchemy executor branch
dddddd has joined #pypy
<Ninpo>
woah, snakeviz is nice
<fijal>
well, you can generate data right?
<fijal>
that's not your work data
<fijal>
or isolate something
<Ninpo>
I can't give you anything runnable without it not being what I'm running. Work are very clear on that. I've asked for an open source policy to be implemented (we've recently been acquired) :/
<Ninpo>
I can share snippets but that doesn't help you run anything :/
<Ninpo>
It's no worries, I understand I'm in a difficult situation, just wasn't sure if this was useful to anyone that understood the output
<mattip>
cfbolz: it would be good to somehow update or recreate the image at the bottom of speed.pypy.org for a blog post and in general
<mattip>
I guess we could do a run of the pypy benchmarks vs cpython2.7.2 just to get the latest bar
<cfbolz>
mattip: yes, that would be useful
<cfbolz>
mattip: but I think we should also do some microbenchmarks (I volunteer for that)
epelesis has quit [Remote host closed the connection]
<Ninpo>
vs python3 would be super useful too considering after next year etc
<Ninpo>
the fact py2 is supported by pypy will be a prime driver after then, performance still the key one for py3 stuff imo
<Ninpo>
Hell, it's why I'm here :D
<Ninpo>
mattip: any ETA on the unicode stuff merging in with the 3.6 stuff? I do miss my async comprehensions
<Ninpo>
again happy to build/test etc
<cfbolz>
"soon" (probably at most a week or two)
<cfbolz>
Ninpo: ^^
<Ninpo>
ace :)
<Ninpo>
Oh, don't think I said this in here before so let me say it now: THANK YOU for back porting fstrings :D
<Ninpo>
between that and njs' async generator library, async comprehensions are the only thing I lack
<Ninpo>
no tracemalloc in pypy3.5?
<Ninpo>
or is that a build time option
<Ninpo>
ImportError: No module named '_tracemalloc'
<arigato>
Ninpo: no, tracemalloc is measuring mallocs, but that doesn't really makes sense in pypy because objects are not allocated with malloc
<cfbolz>
I kind of want a stochastic version of it, fwiw
<Ninpo>
oh
<Ninpo>
How to memory profile with pypy then?
<Ninpo>
holy crap balls, running a malloc profile with cpython on this code is slooow (that or cpython is ridiculously slower at this bit than pypy on top of that)
<mattip>
Ninpo: working on it now
<Ninpo>
mattip: which bit?
<Ninpo>
I've asked a bunch of inane questions tonight D:
<mattip>
unicode stuff merging in with the 3.6 stuf
<Ninpo>
ahhh
<mattip>
but it will be broken for a few iterations
<Ninpo>
rog, gimme a poke when you need something running in anger
realitix has joined #pypy
realitix_ has joined #pypy
realitix has quit [Ping timeout: 250 seconds]
realitix_ is now known as realitix
<cfbolz>
mattip: is the merge to 3.6 very annoying?
<mattip>
yes
<mattip>
hg merge is doing weird things again, like at the sprint a year ago
<cfbolz>
argh
<mattip>
I think I will push the default and py3.5 for now, and leave py3.6 for tomorrow
<cfbolz>
super cool :-)
<kenaan>
mattip unicode-utf8 2418468b1574 /pypy/doc/whatsnew-head.rst: document branch to be merged
<kenaan>
mattip unicode-utf8 98fe98e6e00d /: close branch
<kenaan>
mattip default ba081fb468f4 /: merge unicode-utf8 into default
<cfbolz>
wooooooooooooooo
<kenaan>
mattip unicode-utf8-py3 e6b04cfc1802 /: close branch
<kenaan>
mattip py3.5 0bbb64dc7f98 /: merge unicode-utf8-py3 into py3.5
<cfbolz>
mattip: super cool, congratulations :-)
<kenaan>
mattip py3.5 f468bb5ec141 /: merge default into branch
<cfbolz>
energizer: those are language design questions. For PyPy the easy answer is: because CPython does it that way
<ronan>
mattip: hurrah!
<energizer>
cfbolz: if i wanted to experiment with changing these semantics on a personal fork, are those features relatively straightforward to fiddle with, or more complex?
<cfbolz>
energizer: probably not too hard
kipras has quit [Read error: Connection reset by peer]