arigato changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | mac OS and Fedora are not Windows
iomari has quit [Quit: http://quassel-irc.org - Chat comfortably. Anywhere.]
rubdos has quit [Ping timeout: 250 seconds]
Kipras_ has quit [Read error: Connection reset by peer]
jcea has quit [Quit: jcea]
dddddd has quit [Remote host closed the connection]
xcm has quit [Read error: Connection reset by peer]
xcm has joined #pypy
themsay has joined #pypy
speeder39_ has joined #pypy
beystef has joined #pypy
beystef has quit [Quit: beystef]
mattip has joined #pypy
<mattip> cfbolz: benchmarks of regalloc branch are here https://gist.github.com/mattip/df7bf7c5e4ffedb2ebfaf973f4d6efc6
<mattip> (the logging one seems garbage - 0.1 ns?)
mattip_ has quit [Ping timeout: 240 seconds]
mattip_ has joined #pypy
themsay has quit [Ping timeout: 244 seconds]
mattip has quit [Remote host closed the connection]
speeder39_ has quit [Quit: Connection closed for inactivity]
bivab has joined #pypy
mattip has joined #pypy
rubdos has joined #pypy
<ronan> mattip: that benchmark is a no-op, it gets jitted down to an empty loop
<mattip> ronan: thanks. looking at the source of deflateCopy, it seems v1.2.8 does not validate input, so we should skip that test
<mattip> if ZLIB_VERSION == '1.2.8'
<ronan> how comes it doesn't segfault on CPython, though?
<fijal> mattip: the spectral-something is also not a very reliable benchmark
<mattip> ronan: which test of deflateCopy?
<mattip> we have two failures - one in rpython/rlib/test/test_rzlib.py -k test_unsuccessful_compress_copy
<mattip> that sets zalloc to NULL to test failure
<mattip> one in pypy/module/zlib/test/test_zlib.py::AppTestZlib::()::test_compress_copy
<kenaan> cfbolz default a362dbfc3e89 /pypy/doc/whatsnew-head.rst: tests are good
<ronan> mattip: I'm thinking of test_cannot_copy_compressor_with_stream_in_inconsistent_state
tinix has quit [Quit: Leaving.]
lritter has joined #pypy
mattip has quit [Ping timeout: 246 seconds]
xcm is now known as Guest6169
Guest6169 has quit [Killed (hitchcock.freenode.net (Nickname regained by services))]
xcm has joined #pypy
<tos9> ronan / mattip: hey -- I assume you guys are back to trying to merge the zlib stuff to py35... anything I can help with?
mattip has joined #pypy
<tos9> (considering I am up at 6AM :)
<fijal> tos9: heh
<tos9> fijal: oy
<fijal> tos9: what form do you want vmprof proposal in?
<tos9> fijal: will send you what I started, probably something like this
<tos9> fijal: (3 page word doc, scope of what to be delivered etc)
<fijal> ah cool
<fijal> please do
<tos9> also maybe I decided I might care about cpython eventually :(
<tos9> but yeah "eventually"
<tos9> fijal: obviously out of date with the discussion, but yeah maybe it is a start?
<fijal> 👍
dddddd has joined #pypy
<ronan> mattip, tos9: I think the failing tests are wrong, because they check a condition that should never occur and are incompatible with zlib 1.2.8
<tos9> ronan: the "inconsistent state" ones were meant to handle what zlib.h says happens when somehow stuff isn't fully initialized
<tos9> ronan: which still seems to be there at least in 1.2.11: "Z_STREAM_ERROR if the source stream state was inconsistent (such as zalloc being Z_NULL)."
<tos9> (I couldn't see how to get zlib itself to give me that invalid state, so I manufactured it myself)
<ronan> AFAICT, that comment was wrong in 1.2.8
<tos9> ronan: you're reading the zlib source, or? This seems to be the function it calls right? https://github.com/wenzhilv/reactOS/blob/6d912629f7d050213ad0b933a8ed0395da3ed653/sdk/lib/3rdparty/zlib/inflate.c#L105-L117
<ronan> inflateStateCheck doesn't exist in 1.2.8
<mattip> it seems the documentation does not reflect the code, also in the failing rpython test
<tos9> ronan: looks like they just did the check inline: https://github.com/fmrico/zlib-1.2.8/blob/master/inflate.c#L1448-L1450
<ronan> which doesn't check zalloc
<ronan> that explains why it's only deflateCopy() that segfaults, and not inflateCopy()
<tos9> ah, indeed -- in 1.2.11 the two look more similar
<kenaan> arigo unicode-utf8 fb5c9143c946 /pypy/objspace/std/unicodeobject.py: Add a conditional_call_elidable here
xcm has quit [Read error: Connection reset by peer]
xcm has joined #pypy
mattip has quit [Ping timeout: 250 seconds]
gsnedders has quit [Quit: leaving]
gsnedders has joined #pypy
gsnedders has quit [Client Quit]
<Ninpo> Anywhere I can grab the 7.0 cut/cheat and pass it to pyenv to run some tests? I'm mid dev on a project currently using 3.5 6.0 as it's orders of magnitude faster than cpython. If it'd help to have a 24h runtime on some heavy data I'd happily help
<Ninpo> I've got lines that'd test the async comprehensions etc as well I can easily put back
<Ninpo> also async generators
gsnedders has joined #pypy
kipras has joined #pypy
squeaky_pl has joined #pypy
_whitelogger has joined #pypy
<squeaky_pl> are there any problems known problems on the current tag release, I'm seeing some zlib chat above?
<Ninpo> ronan: ah it's the 3.6 I'm interested in for myself, but I'll happily test both if it's useful
<Ninpo> Going to see if I can get pyenv to build the nightlies thanks for the links ronan
<ronan> squeaky_pl: no, the zlib stuff is about a new feature that isn't in the release
<squeaky_pl> Ninpo, why would you need pyenv? can't you just untar?
<kenaan> fijal arm64 dba5b910fb11 /rpython/jit/backend/aarch64/: (arigo, fijal, rodolph, bivab) fight until the test nearly passes
<Ninpo> squeaky_pl: tbh I was just thinking of the quickest way I could drop it into my virtualenv setup and switch easily
<Ninpo> if it's prebuilt I can just drop it into versions
<squeaky_pl> you can still untar, run new virtualenv on top of new pypy and install your program there
<squeaky_pl> I updated all the dependencies and successfully translated 3.5 7.0 for portable builds on top of GCC 8.2, the resulting binary seems to work fine. I guess I am ready for the release.
<Ninpo> squeaky_pl: I copied the pyenv installer helper for pypy3.5 nightlies and modified it :)
<squeaky_pl> mattip_, are you gonna officially release on Monday?
<ronan> squeaky_pl: Antonio is the release manager this time
<Ninpo> :facepalm: pyenv's nightly installer is broken anyway.../me sighs, does it the "hard" way
<squeaky_pl> ronan, thanks for the info
<fijal> squeaky_pl: matti is in the plane right now I believe
<squeaky_pl> I have enough RAM on my laptop to run 3 translations in parallel, how times changed.
beystef has joined #pypy
<kenaan> fijal extradoc de18def1f720 /blog/draft/2019-02-sprint.rst: add sprint report
<Ninpo> Trying to run latest nightly and getting error while loading shared libraries: libtinfo.so.5
<squeaky_pl> Ninpo, you need something like portable builds.
<Ninpo> is that not what I get from http://buildbot.pypy.org/nightly/release-pypy3.5-7.x/ ?
<squeaky_pl> Ninpo, those are built on Ubuntu, if your os version doesnt match exactly you might have problems.
<Ninpo> oh I see.
<squeaky_pl> Ninpo, are you trying to use 3.6 7.0?
<Ninpo> yes
<Ninpo> on Arch
<squeaky_pl> Ninpo, I will have a portable build in more or less 1 hour, I will ping you.
<Ninpo> squeaky_pl: oh wow thank you so much!
<squeaky_pl> It will work or any x86_64 Linux that is 10 years or newer.
<Ninpo> 'tis why I was fiddling with pyenv and hoping to get it to work, but it falls over on 3.5 nightlies as well which have a install option provided
<Ninpo> Appreciate the effort squeaky_pl and happy to help test
<squeaky_pl> Ninpo, libtinfo comes from ncurses, you must have ncurses 6 installed on your distro or your distro doesnt provide libtinfo because the symbols are inside libncurses
<Ninpo> ah probably
<Ninpo> rolling release and all that
<Ninpo> I take it the pypy3.5v6 is a "portable build"
<Ninpo> that works fine with pyenv install
<squeaky_pl> PyEnv can install my builds automatically, they package portable pypy
<squeaky_pl> https://github.com/squeaky-pl/portable-pypy the repo is here
<kenaan> cfbolz extradoc 6cdc3695cb72 /blog/draft/2019-02-sprint.rst: some typos
<kenaan> fijal arm64 392250be4182 /rpython/jit/backend/aarch64/: (bivab, rodolph, arigo) Make the first test of test_runner pass on ARM64. Yay!
<tos9> cfbolz: changing hyphens to en dashes -- fight the good fight!
mattip has joined #pypy
<kenaan> mattip unicode-utf8-py3 902af9e81bcc /: refactor unicode_to_decimal to use only utf8
<kenaan> mattip unicode-utf8 6cea85fa01f6 /pypy/objspace/std/: deal with s.encode() differently (where s can be either bytes or unicode)
<kenaan> mattip unicode-utf8 624ea5b754de /pypy/module/pypyjit/test_pypy_c/test_string.py: update improved traces in tests
<squeaky_pl> I just found that idle is broken on 3.6 7.0
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/6009 [mattip: force build, unicode-utf8]
<bbot2> Started: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/7283 [mattip: force build, unicode-utf8]
<bbot2> Started: http://buildbot.pypy.org/builders/rpython-linux-x86-64/builds/146 [mattip: force build, unicode-utf8]
<squeaky_pl> running pypy -m idlelib.idle on 3.6 7.0 results in an explosion, but probably not a release blocker
<squeaky_pl> it looks like you are missing idlelib.pyshell module
<squeaky_pl> I will open a bug
<cfbolz> squeaky_pl: 3.6 still needs more work. It's more of a preview/alpha
<cfbolz> But yes, issue sounds good
<mattip> maybe an entry in the wiki, if there is a failing buildbot test https://bitbucket.org/pypy/pypy/wiki/Status%20of%20PyPy3%20support%20for%203.6
mattip has quit [Ping timeout: 245 seconds]
<squeaky_pl> Ninpo, if you wanna try 3.6 7.0 portable it's here: https://bitbucket.org/squeaky/portable-pypy/downloads/pypy3.6-7.0.0-alpha-20190209-linux_x86_64-portable.tar.bz2 Just untar and bin/pypy should work for you.
<Ninpo> squeaky_pl: thank you so much :D
<Ninpo> Is there a changelog for 3.5 v7.0 anywhere?
<Ninpo> cor, 7 is going to be worth it for me whether it's 3.5 or 3.6. Very very nice work, thanks so much pypy folks
<Ninpo> squeaky_pl: your repo was the missing piece of the puzzle, piece of cake creating pyenv install files now (and both versions installed)
<Ninpo> Thank you very much. Off to test!
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
themsay has joined #pypy
beystef has left #pypy [#pypy]
<Ninpo> Overall runtime seems relatively the same which is nice. I've noticed processing of a db result set into a json structure is a bit slower, I'll test that locally on the DB in case it's network messing with it and provide a test case later if confirmed. Back later, thanks again squeaky_pl
<cfbolz> Ninpo: thanks, if you find a slowdown please report it
<Ninpo> cfbolz: I will, will take a little bit to scrub the data (lots of PII) but absolutely
<Ninpo> This project would have been a non starter without pypy
squeaky_pl has quit [Ping timeout: 244 seconds]
<fijal> Ninpo: can you share more with us about the project?
<fijal> If you already did I'm sorry, on the phone
<Ninpo> I haven't, at least not today :) I'm processing a bunch of text type fields in a mysql database looking for incorrectly encoded bytes (utf8 inside mysql latin1 fields) and saving the results to a json file per DB. Over 2TB of data
<Ninpo> Using trio and sqlalchemy/sqlalchemy_aio for the heavy lifting
<Ninpo> 4 workers for db, 8 table workers per db worker
<Ninpo> cpython managed 95k tables in about 10 hours last time I ran it (about 25% through), pypy35-6 does a complete scan in about 14 hours
<Ninpo> There's a refactor run on the main logic coming up Monday now that a full successful pass has been done
<Ninpo> Long running, all python, perfect for pypy
<fijal> The slowest part is probably SQL alchemy
<fijal> We never figured out how to speed up current orms
<Ninpo> I'm not ORMing with it. I use the query builder in one spot, the rest is conn.execute
<Ninpo> makes use of threads with sqlalchemy_aio
<Ninpo> Part of the refactor will be to test trio_mysql which is a trio-ised pymysql
<fijal> Yeah still
<fijal> Try without it on a small data
<Ninpo> 'tis on the list :)
<fijal> I'm happy to look into simple examples how to speed it up
<Ninpo> One of the todos next week is come up with some varying size test data so the tests can be written, I'll make those available
<Ninpo> bulk of the processing time is feeding from an async generator off the result set and binascii().decode operations
<Ninpo> It was my intent to open source it, was on my personal github but my employer got upset :|
<bbot2> Failure: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/7283 [mattip: force build, unicode-utf8]
<Ninpo> testing against individual schemas atm, I'll leave a run going overnight to see how the over time and memory performance is, I've got good 3.5v6 baselines for that
squeaky_pl has joined #pypy
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/6009 [mattip: force build, unicode-utf8]
squeaky_pl has quit [Remote host closed the connection]
mattip has joined #pypy
mattip has quit [Client Quit]
<kenaan> mattip unicode-utf8 b96e9dd44e46 /: merge default into branch
<kenaan> mattip unicode-utf8 322380023a4a /: remove most imports of runicode, except mbcs (win32). TBD: cpyext, micronumpy
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/6010 [mattip: force build, unicode-utf8]
<bbot2> Started: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/7284 [mattip: force build, unicode-utf8]
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/4408 [mattip: force build, unicode-utf8]
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-32/builds/5047 [mattip: force build, unicode-utf8]
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-jit-macosx-x86-64/builds/4219 [mattip: force build, unicode-utf8]
<bbot2> Failure: http://buildbot.pypy.org/builders/rpython-linux-x86-64/builds/146 [mattip: force build, unicode-utf8]
ajlawrence has joined #pypy
ajlawrence has quit [Ping timeout: 256 seconds]
<bbot2> Failure: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/7284 [mattip: force build, unicode-utf8]
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-32/builds/5047 [mattip: force build, unicode-utf8]
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/6010 [mattip: force build, unicode-utf8]
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-macosx-x86-64/builds/4219 [mattip: force build, unicode-utf8]
beystef has joined #pypy
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/4408 [mattip: force build, unicode-utf8]
beystef has quit [Quit: beystef]
lritter has quit [Quit: Leaving]
Ai9zO5AP has quit [Ping timeout: 272 seconds]
ajlawrence has joined #pypy
kanaka has quit [Read error: No route to host]
agronholm has quit [Read error: Connection reset by peer]
agronholm has joined #pypy