antocuni changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "PyPy: the Gradual Reduction of Magic (tm)"
danieljabailey has quit [Quit: ZNC 1.6.5+deb2build2 - http://znc.in]
* arigato
looks at _json.encode_basestring_ascii and _json.encode_basestring
<arigato>
2.7's encoder.py also starts with "from __pypy__.builders import StringBuilder, UnicodeBuilder", not 3.5's
* arigato
looks at the diff between standard 2.7 json/ directory and our own
<arigato>
pf pf pf
<gcbirzan>
The pypy one is based on simplejson.
<arigato>
why does the stdlib-2.7.13 branch contain the pypy changes in json/??
<cfbolz>
arigato: maybe there was a complex manual merge necessary or so?
* arigato
diffs directly CPython
<arigato>
gcbirzan: unsure what you mean. the lib-python's json/ is still almost the same as 2.7's json/
<arigato>
ok, encoder.py in particular was modified quite a lot
<gcbirzan>
Hm. It seems simplejson _is_ the cpython one. Maybe I confused it with another version...
<cfbolz>
arigato: note that json_bench is purely a benchmark of json.dumps (not json.loads). so the changes to encoder.py are the relevant ones
<arigato>
yes
<arigato>
I also notice that I am responsible for all the "recent" changes in encoder.py
<arigato>
although that dates back to 2013 and 2014
<cfbolz>
heh :-)
<arigato>
ah, found someone else to blame :-) fijal in 2011 did the StringBuilder/UnicodeBuilder
<gcbirzan>
loads() is almost the same speed as the cpython one, btw. And almost the same as the pypy2 one.
<arigato>
(of course I'd also like to blame whoever forgot all these changes in the py3.x branch)
<arigato>
gcbirzan: thanks for the note!
<gcbirzan>
Ugh. Getting the benchmark thing to run under python3 is painful. 15 minutes into it, one fails. :P
<gcbirzan>
I guess there should be a way to annotate the tests that should work under python3 as well. Currently, the whole thing dies when a test fails.
<arigato>
gcbirzan: as noted repeatedly yesterday, we're unlikely to accept changes that don't come with a bit of thinking about where we want to go, and it's likely to be "start from cpython's performance module"
forgottenone has quit [Quit: Konversation terminated!]
<kenaan>
arigo py3.5 4d7c200aa842 /pypy/module/_pypyjson/: Re-enable raw_encode_basestring_ascii(). Not used so far
dddddd has joined #pypy
altendky has joined #pypy
oberstet has joined #pypy
Rhy0lite has joined #pypy
jcea has joined #pypy
<kenaan>
arigo py3.5 18c3825bad60 /lib-python/3/json/encoder.py: This is encoder.py from pypy 2.7, with the changes in CPython 2.7--3.5 manually applied.
<kenaan>
arigo py3.5 4e1c19053bd9 /: A bit closer to pypy2.7
<cfbolz>
arigato: given that you put "guard-compatible" on the sprint topic list, I plan to do a bit of work on the branch before the sprint - at least bring it up to date with the current default
<cfbolz>
(some of the hacks on the branch became unnecessary in the meantime)
<nedbat>
cfbolz: we talked about tweeting for a Mac helper person. Maybe it would be better for you to tweet it, and me to retweet?
yuyichao_ has quit [Ping timeout: 255 seconds]
deathsparton has joined #pypy
deathsparton_ has joined #pypy
deathsparton has quit [Ping timeout: 256 seconds]
deathsparton_ has quit [Ping timeout: 264 seconds]
<cfbolz>
nedbat: I could, but it sounded like you have understood the requirements better. something like "PyPy is looking for help with building Mac binaries" + what?
<nedbat>
cfbolz: hmmm
<nedbat>
cfbolz: "PyPy is looking for someone to help with Mac builds. Interest and energy are more important than expertise!"
<nedbat>
cfbolz: ?
<cfbolz>
ok
deathsparton_ has joined #pypy
<cfbolz>
nedbat: sent
<nedbat>
retweeted (twice)
ebarrett has quit [Ping timeout: 240 seconds]
<Rotonen>
what are you struggling with in regards to the mac builds?
<Rotonen>
just build on 10.13 and declare you only support 10.13 for now and keep adding targets carefully one by one towards the past over time?
<nedbat>
Rotonen: you were very energetic the last time I was here talking about this. Could you help with the Mac builds?
<cfbolz>
Rotonen: yes, we want somebody to do that work (since at the moment it seems nobody in the core dev group has the cycles to do that)
<Rotonen>
i've provided a script to get around those issues, just use that as a scaffold?
<Rotonen>
i'll tackle win64 first, but not having too much time
<cfbolz>
hence the tweet ;-)
<Rotonen>
we can suppose most of your mac users use a standard homebrew installation of pypy and install via homebrew so that's less fruitful than bringing forth a 64bit windows build, IMO
<nedbat>
Rotonen: i'm sure no one would mind if you help with the win64 builds. :)
deathsparton_ has quit [Remote host closed the connection]
<Rotonen>
i may be poked for advise in regards to the mac builds, that cake has many layers and i'd start from the simplest possible ones (and preferably with runtime checks and early aborts when not running on an intended target system to reduce the amount of end user confusion)
adamholmberg has joined #pypy
<mattip>
+1 for recommending Homebrew and leaving the macosx builds the way they are
<mattip>
but the way to move the macosx build problem forward is to engage with the build slave owner
<nedbat>
mattip: "recommending homebrew" means what exactly? Using brew to install pypy?
<Rotonen>
or make the build script robust in regards to runtime (as in just wrap everything with xcrun and apple recommended practices thereof)
<nedbat>
Rotonen: you've used the word "just" a few times now :)
<mattip>
Rotonen: use brew to install a binary PyPy rather than the downloads we put on our site
<mattip>
sorry, nedbat
yuyichao_ has joined #pypy
<tos9>
the current binaries are either broken or they're not though aren't they?
<tos9>
if they're broken it seems a bit odd to leave them up
<Rotonen>
nedbat: it's like 3 things or so one needs to get right: 1) set the deploy target in the xcode way 2) use xcrun cc as your compiler 3) make sure you do not use any syscalls which are not available on older systems when you target older than you compile on (xcode cc has a warning flag for this and you can -Werror)
<nedbat>
Rotonen: you can see why i keep suggesting you would be perfect for this gig :)
bachmann has joined #pypy
<Rotonen>
tos9: broken is not black and white in this case, 'comes with assumptions baked in' is closer
<arigato>
the problem is really to find someone with a mac that wants to care. none of the core developers with that category
<nedbat>
Rotonen: you mean like, "I assume json will work properly" ?
<Rotonen>
nedbat: i have no need for a distributable pypy on macos myself, whereas i do for 64bit on windows
<nedbat>
mattip: i might be a little afraid to switch installation mechanisms atm....
<nedbat>
bachmann1234: minimally, yes. Both started with me :)
<tos9>
nedbat: what if you just send me all the code you want to run and I execute it for you and send back the results
<nedbat>
tos9: i don't know what you mean: the bug has the repro case
<nedbat>
bachmann1234: btw, o/
<arigato>
jafd: hi! people here are discussing the Mac issues, as you noticed. we're looking for someone with a Mac who would be willing to fix the Mac-specific installation issues and build the releases
<jafd>
arigato: how many different macOS versions do you need to support?
deathsparton_ has joined #pypy
<tos9>
nedbat: that was a joke for an alternate way you'd get your work done.
<nedbat>
tos9: that was my other possible interpretation :)
<tos9>
nedbat: (I'm waiting for the Sierra laptop to not be dead. Clearly with too much time on my hands.)
<mattip>
we have at least two open macosx issues that need attention
<nedbat>
mattip: i brew installed it, and the json thing looks good for me on 5.10.1
<jafd>
arigato: if time's not pressing, I'd assemble a kind of build pipeline over the next weekend (not having a rent source yet)
<nedbat>
jafd: the other issue is a process one: the downloads page has two mac builds, for different osx versions. But one is 5.10.0, the other is 5.10.1
<jafd>
nedbat: that sounds evil. Nothing in that version number implies platform version...
<arigato>
it's the first time we gave two Mac builds, following issues that showed up apparently only now
<mattip>
nedbat: it has many downloads for macosx, there are two for 5.10.0 and one for 5.10.1, with a reccomendation to use neither of them
<arigato>
I'd personally prefer if you could make a single working build
<mattip>
but to prefer Homebrew
<jafd>
mattip: on #2731, it would _really_ help if pypy didn't roll its own ways of finding libraries, and I clearly remember submitting a PR that got merged and dealt with OpenSSL
<nedbat>
mattip: personally, i don't care how many there are, or what the recommendations are. I just want clear instructions so I can support PyPy with coverage.py
<arigato>
I think we should distribute a fully-non-homebrew version with as much libraries as we can included, so that it just works
<bachmann1234>
I think suggesting the brew version is fine in some cases. But may be problematic when someone needs to work with multiple pypy versions for testing/tox
<jafd>
arigato: is using ncurses, libz, libssl etc. that are supplied with the OS too much of PITA?
deathsparton_ has quit [Ping timeout: 256 seconds]
<arigato>
jafd: ssl in particular is very much of a pain to use
<arigato>
I think because the official Mac-distributed one is not supposed to be used by 3rd-party apps, so it changes widely from one OS release to the next
pilne has quit [Quit: Quitting!]
<nedbat>
bachmann1234: fwiw, i only test against "latest" pypy and pypy3 for coverage.py
<bachmann1234>
ah, ok
<tos9>
nedbat: OK, can reproduce on Sierra
<tos9>
with that binary.
<nedbat>
ugh, brew installs pypy3 5.10.1, but pypy 5.10.0_1 ?
<mattip>
jafd: do you mean the py3.5-mac-embedding branch? That only dealt with the _ssl cffi import library
<mattip>
not with the issues mentioned in issue 2731
jcea has quit [Quit: jcea]
ebarrett has joined #pypy
<jafd>
https://bugs.python.org/issue17128 strongly suggests embedding OpenSSL if you want to go standalone. It has its smells (once in a while they find a vulnerability there), but every other alternative feels worse
<arigato>
jafd: yes, I think we should do that. that's also why it would be cool if there's someone like you that would react to openssl vulnerabilities and updates the embedded version quickly
<Rotonen>
i recommend going for libressl when you embed ssl
<jafd>
I don't know if I'm going to have all that's needed, but IMO every little bit helps.
<Rotonen>
you also want to embed readline, lzma and a few others
<Rotonen>
(readline is debatable, but can get in the way across a wide range of versions)
<jafd>
Rotonen: aren't there modules on PyPI which imply Python was linked against OpenSSL?
<jafd>
Last I checked, LibreSSL wasn't a drop-in replacement
<Rotonen>
jafd: they've done their stuff wrong, then
<Rotonen>
for the past year i've been using it, it's been a drop-in for me
<jafd>
Well, I don't want to be that guy, but I'm a strong supporter of "we don't break userspace" mode of work
<Rotonen>
macos 10.13 ships with it as well so it's a drop-in for apple as well, but they do not provide the ssl headers since a while ago, as they'd like for you to go the full cocoa way and link against their Security.framework
<Rotonen>
with openssl you'll have to figure out what is lipo (man lipo on a mac) when you create the static library
<jafd>
Rotonen: headers are not the issue as long as the binary works subsequently
<Rotonen>
jafd: this is only a discussion since apple dropped shipping the headers and open source people do not like linking against apple cocoa frameworks the apple way, but rather make static builds of the libraries they are used to working which and for which they have the tooling already in place - in this context i'm saying libressl is easier to bundle as a static library - and i do consider it a software
<Rotonen>
mistake if something actually asserts on the name of the ssl implementation carried in the python implementation guts
<Rotonen>
jafd: btw. then those eggs would also break on the apple supplied python2.7 as ssl.OPENSSL_VERSION is libressl there as well
<jafd>
Rotonen: I wasn't thinking of names, but subtle API/ABI differences that may cause regressions (of course if you cannot start an interpreter at all it's a bigger one)
<Rotonen>
an issue only if you *need* to support older now-considered-deprecated-on-public-https kinds of stuff, but that is maybe not a stance a language implementation should take, that is true
<Rotonen>
but you lose those on openssl 1.1 as well, so you're saying openssl 1.0 is the way to roll for now?
<Rotonen>
then again if you are so far off the modern happy path one could just assume you can manage yourself and roll your own?
<jafd>
Rotonen: I have had a few work nightmares in trying to build some other things (revolving around Perl and Ruby) which ultimately turned out to be OpenSSL 1.0-only. No LibreSSL because a few "useless" bits were needed, no 1.1 because Ruby had to be stuck at 2.3.x. That's why I prefer to ask around when I hear about lib*ssl now. Just testing the shakiness of the ground.
<Rotonen>
jafd: people whom enter such corners can generally manage, even though if it's not pleasant, but sure, 1.0 can be worth the extra effort if that is a value to fight with its build system
<Rotonen>
if you build on macos 10.13, you have to get it to not use getentropy() if you want to support a macos a few versions back
<Rotonen>
and a couple of other syscalls, so you have to roll your own patches for the outputs of its Configure script for the static builds
<Rotonen>
vs. libressl understanding the xcode-compatible macos target environment variable in that regard
<Rotonen>
of course you can just build on the oldest system you want to support and that'll do it as things are generally way easier to do forwards-compatible
<Rotonen>
so grab a 10.8 and use that as the build machine?
<Rotonen>
last i took a closer look, cpython was compiling its mac builds on 10.6 to achieve the same
<arigato>
we don't need 10.8 or anything precise, just the oldest that you can reasonably provide. That's why we said that two versions should be enough
<Rotonen>
if one is to believe apple, one can only reasonably provide the very latest at any time, as they officially only support that now, but that's not fully sensible either
<jafd>
Rotonen: I was thinking of only supporting stuff on which Apple still bestows security updates. Those with custom setups should use Homebrew if they don't already.
<arigato>
+1
<Rotonen>
jafd: the issue is apple does not disclose which are actually still supported and for what apart from "we support the latest"
<jafd>
El Capitan/Sierra was a cutoff point for a lot of still-good hardware; security updates are provided as far back as Yosemite (I think until 10.14)
<Rotonen>
also building on an older one should not introduce you to any issues bar a compiler bug, but you do still get newer xcodes on the older ones
<Rotonen>
jafd: do you have a source on that? i've been looking this year on an official statement without finding one
<Rotonen>
jafd: i also *think* that's true, but cannot pin that on where and when it's been disclosed for the development communities at large
<jafd>
Rotonen: no official statement, but last year they had security updates for 10.10.5
<jafd>
So it's empirical observation
<Rotonen>
was it 10.11 which demands openg3 from the hardware and was the great divider accessibility-wise?
<jafd>
Rotonen: nope, 10.11 ran officially on my 2008 MBP
<jafd>
In 10.12 you don't get a few graphics-related things on laptops older than Ivy Bridge
<Rotonen>
with the global inclusiveness aspect of allowing people with a wider range of hand-me-down hardware to run things, that'd make 10.11 a sensible platform to build on
<Rotonen>
jafd: it's a mirror of 10.7 having dropped hardware which cannot do opengl2 (as uikit started assuming you can do that in hardware) and thus my old macbook2,1 was dropped (and uikit on 10.12 assumes opengl3 from the hardware thus your mbp was dropped)
<Rotonen>
so, building with a static openssl 1.0 on 10.11?
deathsparton_ has joined #pypy
<jafd>
Rotonen: as an old hackintosher, I assumed it was because they started demanding full 64-bit support since 10.8 :) / Yes, static (or shipped, whichever works) openssl 1.0 targeting 10.11 and of course checked on later versions
deathsparton_ has quit [Excess Flood]
deathsparton_ has joined #pypy
<jafd>
Rotonen: however I won't have time to assemble a build pipeline until the weekend, so if someone beats me to it, I'm good with that
<jafd>
Another thing is - what do we expect the standalone distribution to look like? A .pkg or something else?
demonimin has quit [Read error: Connection reset by peer]
chelz has joined #pypy
<arigato>
jafd: right now it's a tarball, but a .pkg would be better, I suppose
bachmann has joined #pypy
bachmann has quit [Client Quit]
<jafd>
ok, a final question - would we have a tracking issue for this? IRC has its disatvantages (hard to keep asynchronous focused conversation is my pet reason to dislike it a bit)
<jafd>
(also, inability to fix typos is my second pet reason :))
<fijal>
we are not very heavily issue tracker oriented
<Rotonen>
jafd: i had a 64bit core2duo on that thing, but no opengl2 on the gpu, i assumed it was opengl (but blindly based on the later uikit reason given - was not into apple dev back then, but was doing apple [ios] dev roughly during 10.7 .. 10.12 [and still some now here and there])
demonimin has joined #pypy
<fijal>
*but* if you feel like it, it's ok if you create an issue and we'll try to respond there
<fijal>
(bibucket issue tracker being not that good is one of the reasons though)
<Rotonen>
jafd: i'm available for questions / testing during the weekend if you hit snags or so
<arigato>
jafd: and don't worry if you can only progress next week-end, that's more than fine. I think the next release should be in 2-3 months
<mattip>
and thanks!
<arigato>
(though maybe we want to fix the current release with a proper Mac version)
<arigato>
yes!
<jafd>
arigato: I think if a reproducible build/release pipeline can be established at all, it could work with more codebase versions (although I envision tweaks for how we detect libraries presence/suitability)
<kenaan>
mattip cpyext-datetime2 4b6249f566e2 /pypy/module/cpyext/: move datetime type typedefs into parse, try to add a make_typedescr (for tzinfo)
<jafd>
Rotonen: On a slightly different note, I imagine your old MacBook had a 32-bit EFI and a 64-bit CPU, and support for that was gone since 10.8. Intel GMA950/GMAX1300 vanished at the same point, so projects like macosxbootloader wouldn't be helpful. So we're both right on this. :)
infinite has quit [Ping timeout: 268 seconds]
<mattip>
jafd: much of that already exists, see rpython/translator/platform/darwin.py
<Rotonen>
could not get 10.7 either (now it runs coreboot + openbsd, just because)
<mattip>
anyone feel like helping me through PyDateTime_Time, PyDateTime_DateTime, PyDateTime_Delta c-api design problem?
<mattip>
I cannot use make_typedescr to intercept the attach call when creating a pyobj from w_obj
chelz has quit [Ping timeout: 276 seconds]
<mattip>
since datetime is pure python and the make_typedescr() uses the layout.typedef which is `object`'s
<mattip>
see the XXX in 4b6249f566e2
nic9075 has quit [Ping timeout: 260 seconds]
infinite has joined #pypy
<arigato>
mattip: so currently, there are PyDateTime_DateStruct etc., but none of them have fields?
<mattip>
correct
<arigato>
I think one possible workaround would be to make a dummy class at interp-level and inherit from it in lib_pypy/datetime.py
<arigato>
or three of them
<mattip>
nice
<mattip>
so a builtin _datetime module with just those classes
<arigato>
"heeeew what a smelly hack" is more what I'd have expected :-)
<arigato>
but yes
<mattip>
maybe we can hide it deeper, like in __pypy__ or something
squeaky_pl has quit [Remote host closed the connection]
lasley has joined #pypy
<lasley>
Hi PyPy maintainers! Word on the street is you need someone to help with Mac builds. I may be able to help. Any information on what specifically you're looking for?
* nedbat
is pleased to see the system works :)
<lasley>
Indeed it was a Ned retweet that caught my eye. I'm reading the history now - ignore my lack of research on initial question
<nedbat>
lasley: no worries, this is what we hoped would happen.
oberstet has joined #pypy
yuyichao has quit [Ping timeout: 256 seconds]
raynold has joined #pypy
<mattip>
nedbat: impressive
<mattip>
when we are ready to ask for volunteers to revamp www.pypy.org can we get you to tweet it :)
<nedbat>
mattip: i did not expect this response, tbh :)
<nedbat>
mattip: any time
<KingParrot>
To the desktop of Python.... Ha Ha Ha
tbodt has joined #pypy
fryguybob has joined #pypy
eshigee has joined #pypy
eshigee has quit [Client Quit]
eshigee has joined #pypy
Rhy0lite has quit [Quit: Leaving]
<KingParrot>
I extracted something to my desktop and now the crazy thing wont delete.
<simpson>
I'm reading a whitepaper on machine learning. Normally they're very disappointing, but this one is pretty good: https://arxiv.org/pdf/1801.04016.pdf
amaury has joined #pypy
KhaledEz has joined #pypy
<KingParrot>
ok
adamholmberg has quit [Remote host closed the connection]
<simpson>
aolko: Presumbly PyCharm doesn't have the logic required to allow for the existence of a PyPy 3.x interpreter.
<simpson>
Not sure if that's a problem, but presumably you can file an upstream bug with them.
<aolko>
done
pilne has joined #pypy
<mattbillenstein>
hello all, reading up on the osx lib issues -- what is the concensus as to what we should do here? (I'm running the High Sierra buildbot btw)
<mattbillenstein>
fwiw, cpython is currently having some issues figuring out which openssl to support and how to link to it across platforms
<mattbillenstein>
hard to coordinate across so many timezones -- I'll check back 9am CET