cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
speeder39 has quit [Quit: Connection closed for inactivity]
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 268 seconds]
wallet42 has joined #pypy
parazydix has joined #pypy
parazydix has quit [K-Lined]
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 252 seconds]
moei has quit [Quit: Leaving...]
dddddd has quit [Remote host closed the connection]
mkaito_iz has joined #pypy
mkaito_iz has quit [Ping timeout: 252 seconds]
XenoirKC has joined #pypy
XenoirKC has quit [Remote host closed the connection]
moei has joined #pypy
_whitelogger has joined #pypy
Zaab1t has joined #pypy
Zaab1t has quit [Quit: bye bye friends]
BuckTurg- has joined #pypy
BuckTurg- has quit [K-Lined]
kori has joined #pypy
kori has quit [K-Lined]
MonoMonkeynU has joined #pypy
MonoMonkeynU has quit [K-Lined]
squidicuzzk has joined #pypy
squidicuzzk has quit [Remote host closed the connection]
jacob22__ has joined #pypy
swordsqr has joined #pypy
swordsqr has quit [K-Lined]
dddddd has joined #pypy
ASucAL has joined #pypy
ASucAL has quit [Remote host closed the connection]
oberstet has joined #pypy
oberstet has quit [Quit: Leaving]
antocuni has joined #pypy
jacob22__ has quit [Read error: Connection reset by peer]
__pv has quit [Remote host closed the connection]
lritter has joined #pypy
MetacityAT has joined #pypy
MetacityAT has quit [Remote host closed the connection]
__pv has joined #pypy
jacob22__ has joined #pypy
antocuni has quit [Ping timeout: 246 seconds]
oberstet has joined #pypy
_whitelogger has joined #pypy
ewookJQ has joined #pypy
ewookJQ has quit [Remote host closed the connection]
antocuni has joined #pypy
antocuni has quit [Ping timeout: 245 seconds]
witran has joined #pypy
WGH has quit [Ping timeout: 260 seconds]
oberstet has quit [Ping timeout: 252 seconds]
mattip has joined #pypy
<mattip> in light of win32 issue #2899 (we need to use LoadLibrary_Ex with LOAD_WITH_ALTERED_SEARCH_PATH), I will look at the locale-encoded dll name failures too
<mattip> so arigato was right, a few days ago he volunteered to look at those failures and I told him to wait for the unicode-utf8 branches. Oops
jphelps has joined #pypy
jphelps has quit [Remote host closed the connection]
tzafrirQN has joined #pypy
tzafrirQN has quit [Remote host closed the connection]
Gunde has joined #pypy
Gunde is now known as Guest55779
Guest55779 has quit [Remote host closed the connection]
dddddd has quit [Remote host closed the connection]
lritter has quit [Remote host closed the connection]
antocuni has joined #pypy
antocuni has quit [Ping timeout: 268 seconds]
themsay has joined #pypy
dddddd has joined #pypy
bbot2 has joined #pypy
sh4rm4^bnc has joined #pypy
<sh4rm4^bnc> hi, i've read the documentation about building pypy from source but am a bit appalled from the listed huge build times. isn't it possible to build it from already translated C sources ?
<sh4rm4^bnc> i.e. the maintainer does "make dist" which creates the rpython binary which is then used to translate the sources into C which are then stuffed into the source tarball
<nedbat> sh4rm4^bnc: i know almost nothing about this, but i'm curious: why are the build times a problem?
<sh4rm4^bnc> think gentoo
<sh4rm4^bnc> build from source distro
<sh4rm4^bnc> every user builds from source, not just the distro kingpin
<nedbat> sh4rm4^bnc: i see
<nedbat> sh4rm4^bnc: if they want to build from source, why would they be ok with pypy doing half the build for them?
<nedbat> sh4rm4^bnc: i guess i don't know why gentoo users build from source
<sh4rm4^bnc> C source is source too
<nedbat> sh4rm4^bnc: one man's source is another man's object. why do they value building from source?
rubdos has quit [Ping timeout: 260 seconds]
<sh4rm4^bnc> it's just like C projects that ship bison or flex or gperf input files and expect the user to have those tools installed during the build
<sh4rm4^bnc> whereas they could instead ship the already generated C files and spare the user the dependencies
<nedbat> sh4rm4^bnc: gentoo users run bison, or they compile pre-bison'd C source files?
<sh4rm4^bnc> most projects unfortunately DON'T ship the pre-genned bison files
<sh4rm4^bnc> but it would be preferable
<nedbat> bbiab
speeder39 has joined #pypy
<sh4rm4^bnc> why do ppl want to build from source ? maybe they're cautious and dont want to trust other ppl's binaries.. or they like changing / customizing stuff. or they have a special environment, ...
<sh4rm4^bnc> building from source is a lifestyle, i guess... just like some ppl prefer to cook their own food
<mattip> sh4rm4^bnc: by shipping the c files, we force people to trust our generation of them and make it much harder to customize or adapt to an environment
<mattip> for instance the target architecture is baked in
<mattip> so we would ship x86, arm, x64, ... ?
<sh4rm4^bnc> why would you bake in an arch ?
<mattip> the JIT is architecture specific
<sh4rm4^bnc> well, couldnt the build generate a x86_64/jit.s and a i386/jit.s and then the makefile selects the right file ?
<mattip> we also bake in the GC strategy, and many other options
<mattip> they all get mixed in via the different optimization stages, kind of like LTO
rubdos has joined #pypy
<mattip> we have had thoughts about separate compilation of independent parts, but it is alot of work
<sh4rm4^bnc> hmm also when you would ship the C sources, one could actually go directly from 0 to pypy, without having to have a python interpreter already
<sh4rm4^bnc> i could imagine many ppl would prefer running all their stuff with pypy
* mattip off, zzz
kbni has joined #pypy
<_aegis_> why ship source when you can't meaningfully change build flags? it would be an esoteric version of pypy
<_aegis_> makes more sense to speed up the build.
kbni has quit [Remote host closed the connection]
<sh4rm4^bnc> well of course you should be able to meaningfully change build flags
<sh4rm4^bnc> the thing could generate the .c and .s files, and a makefile that respects the conventions like CFLAGS etc
<sh4rm4^bnc> and then you can do make -j8 and all your 8 cores build pypy in a few seconds
<nedbat> sh4rm4^bnc: it seems like you are saying, "we are people that like control, so we build from source, but please remove a bunch of control so it can build faster"
<sh4rm4^bnc> why remove control ?
<nedbat> sh4rm4^bnc: the last 30 minutes of discussion have mentioned a few choices that are hard to change if you ship C sources
<sh4rm4^bnc> really ? is it much harder to write into your output file: #ifdef x86_64 #include "jit_x86_64.c" instead of jit_x86_64();
<sh4rm4^bnc> just a matter of structuring the output
<nedbat> sh4rm4^bnc: that sounds like the kind of thing someone would say if they weren't intimately familiar with the code generation process
<nedbat> sh4rm4^bnc: sorry for all the questions, but how often do you rebuild pypy?
<sh4rm4^bnc> i never did
<sh4rm4^bnc> i never used it, yet
<sh4rm4^bnc> was just trying to bundle it for my distro
<sh4rm4^bnc> and while studying the docs i thought "hmm why wouldnt they ship the C sources?!"
<nedbat> sh4rm4^bnc: wait a sec, you donn't know how long it will take to build?
<sh4rm4^bnc> i'm reading the docs and the makefile
<sh4rm4^bnc> @echo "Building a regular (jitting) version of PyPy, using CPython."
<sh4rm4^bnc> @echo "This takes around 2 hours and $(URAM) GB of RAM."
<nedbat> sh4rm4^bnc: people here have tried to explain the difficulty of shipping sources.
<sh4rm4^bnc> that, clearly sounds real bad
<sh4rm4^bnc> like, taking 2x as long as mozilla firefox
<sh4rm4^bnc> which is 150MB of compressed sources
<sh4rm4^bnc> and C++ which compiles real slow
<nedbat> sh4rm4^bnc: pypy is worth it :)
<_aegis_> you cannot meaningfully change the *pypy* build flags if you ship C
<_aegis_> because none of them are CFLAGS
<_aegis_> they apply pre-generation, and are kinda important
<sh4rm4^bnc> make CFLAGS="-DUSE_GENERATIONAL_GCC -DARCH_X86"
<nedbat> sh4rm4^bnc: it seems like you aren't listening to people's answers....
<_aegis_> also, you're just asking for like 2gb of C source lol
<sh4rm4^bnc> lol impossible
<sh4rm4^bnc> that would mean the pypy bin would have like 200MB
<_aegis_> you are way out of your depth
<_aegis_> I saved my last pypy build because I've been looking at making the pypy binary smaller and optimizing the build process
<sh4rm4^bnc> not really, i'm quite familiar with C, build systems, conditional compilation etc
<_aegis_> (because I build it often and ship it with a real world app, and don't like how big it is or how long it takes to build)
<nedbat> sh4rm4^bnc: it seems like you aren't familiar with how pypy is generated
<_aegis_> the C source for my x86_64 pypy build is 391MB
<_aegis_> if you added every possible combination of build flags it would be at minimum a few gigabytes
<sh4rm4^bnc> well, if you tar that up it'll be 25 MB ?
<sh4rm4^bnc> that's still much less than firefox
<_aegis_> the built pypy library is 35-50mb for me
<nedbat> sh4rm4^bnc: you keep overlooking the build options
<_aegis_> and that's with *one* set of build flags
<sh4rm4^bnc> could you enlighten me as to what build options you mean ?
<_aegis_> only if you promise to stop talking over us.
<sh4rm4^bnc> talking over ? i thought we were having a productive discussion here
<_aegis_> we say things, you ignore them or tell us they are impossible
<_aegis_> what you are suggesting would require large compromises or a lot of human labor
<_aegis_> I'm working on getting you the xz'd size of my pypy C source but it's still compressing.
<sh4rm4^bnc> thanks, that would be interesting
<nedbat> sh4rm4^bnc: when someone says the jit choice isn't a CFLAG, instead of insisting it could be a CFLAG, ask why it isn't.
<_aegis_> in the current rpython compiler, you would need to generate every combination of flags as a separate source tree to hoist those into CFLAGS
<sh4rm4^bnc> nedbat, i was showing how one could pass build options via CFLAGS
<_aegis_> yes, we know how to do that.
<_aegis_> that's not the problem.
<nedbat> sh4rm4^bnc: yes, but that won't work here.
<_aegis_> it happens before the C exists.
<_aegis_> the only way around that, would only work for some of the rpython flags
<_aegis_> and it would be to modularize the build a bit
<_aegis_> which probably requires coming up with abstraction layers / API layers that don't exist yet
<sh4rm4^bnc> right... that was what i thought
<_aegis_> and I'd guess at least a month of work by a couple people already very familiar with the process
<sh4rm4^bnc> a month ? you can build rome in a month :)
<nedbat> sh4rm4^bnc: you did it againn
<sh4rm4^bnc> what, being me ?
<nedbat> sh4rm4^bnc: an expert is telling you something, and you are telling them they are wrong.
<sh4rm4^bnc> no, i am indicating that i think their estimation is a bit exaggerated
<nedbat> sh4rm4^bnc: ie, wrong.
<sh4rm4^bnc> nedbat: i am not trying to be hostile here
<nedbat> sh4rm4^bnc: you have never even built the software, much less studied it, but you think you know enough to make that claim?
<nedbat> sh4rm4^bnc: if your goal is to learn things, you are doing a bad job
<_aegis_> I'm engaging you respectfully, you are repeatedly disagreeing with me about something you are not a subject matter expert on
<_aegis_> at this point I would encourage you to do more research and maybe try implementing some of the things you're talking about before continuing this discussion
<_aegis_> xz output is 23mb for my x86_64-only mac-only jit-only build, with no possible compile-time configuration of pypy itself, and none of the supporting python packages included
<sh4rm4^bnc> that would be acceptable, imo
<_aegis_> no, because you need one of those per architecture, and you cannot configure pypy itself at all, and you'd need to reimplement the tail-end rpython build system yourself
<sh4rm4^bnc> cpython is 12 mb sources
<nedbat> sh4rm4^bnc: why are you telling us that?
<sh4rm4^bnc> doh, why are you criticizing every word i say ?
<nedbat> sh4rm4^bnc: i'm nnot criticizing, i'm trying to understand your point.
<_aegis_> if you look at the actual intermediate C source it's barely more usable than assembly w/ debug symbols anyway
<sh4rm4^bnc> i think having the ability to build pypy from C sources would be a horrific thing
<nedbat> sh4rm4^bnc: it's a factoid that seems to have no bearing on the current discussion, so I must be misunnderstannding somethinng.
<nedbat> sh4rm4^bnc: horrific? I thought it was what you wanted?
<_aegis_> terrific.
<sh4rm4^bnc> the build would be much faster, and you could directly bootstrap pypy as your system's python interpreter
<_aegis_> it's already architecture and ABI specific
<sh4rm4^bnc> uh yeah
<nedbat> sh4rm4^bnc: "terrific" is really good. "horrific" is really bad.
<_aegis_> so the C sources are not going to be more useful than a binary in most places
<nedbat> sh4rm4^bnc: english is messed up, yo.
<sh4rm4^bnc> yeah i meant terrific :)
<nedbat> sh4rm4^bnc: but you are learning why it isn't done that way.
<_aegis_> much of pypy's perf is going to come from the JIT, which won't be affected by your host C compiler's options much
<_aegis_> so "but I want to funroll my loops" doesn't apply
<sh4rm4^bnc> i usually use -Os related optimizations
<sh4rm4^bnc> but it's also for usage of crosscompilers and hardening or debug options
<_aegis_> none of that applies to pypy intermediate source in a major way
<_aegis_> python itself is memory safe, the pypy JIT output will bypass any compiler hardening because it is itself a compiler
<_aegis_> that has full control of e.g. its own stack and memory mappings
tiagofalcaonP has joined #pypy
<sh4rm4^bnc> so how would i cross-compile pypy for say rasppi ?
tiagofalcaonP has quit [K-Lined]
<sh4rm4^bnc> or maybe even an arch who doesn't even have jit support yet ?
<nedbat> sh4rm4^bnc: sorry, i don't know
<sh4rm4^bnc> with a modular build of C sources it could be possible... make CC=powerpc64-linux-musl-gcc pypy-without-jit
<sh4rm4^bnc> even a configure script is thinkable... ./configure --without-jit --with-gc=generational
<nedbat> sh4rm4^bnc: it still seems like you are ignoring what you've been told.
<sh4rm4^bnc> > /src/build/pypy/pypy2-v6.0.0-src/rpython/rlib/clibffi.py(288)<module>()
<sh4rm4^bnc> -> assert libc_name is not None, "Cannot find C library, ctypes.util.find_library('c') returned None"
<sh4rm4^bnc> is it possible nobody using musl libc has ever built pypy from source ?
<nedbat> sh4rm4^bnc: i wouldn't know
<sh4rm4^bnc> alpine linux, for example
<sh4rm4^bnc> which is cool kids use in docker
<sh4rm4^bnc> *what
<sh4rm4^bnc> "There are a few workarounds for differences between Alpine Linux and the Debian-based Linux distributions that the PyPy team has thus far worked against."
<nedbat> sh4rm4^bnc: i'd want to find the discussion to see what they are talking about.
<sh4rm4^bnc> it's listed in the last paragraph
<sh4rm4^bnc> stuff like
<sh4rm4^bnc> RPython expects the stdin/stdout/stderr file handles in stdio.h of the standard libc to be of type FILE*. With musl these are of type FILE *const rather. We patch RPython with the correct type.
<nedbat> sh4rm4^bnc: that shows the stuff, it doesn't show what "worked against" means.
<sh4rm4^bnc> oh yeah i don't know either
<nedbat> sh4rm4^bnc: seems like this repo will have details about how to get it to build the way you want.