cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
speeder39 has quit [Quit: Connection closed for inactivity]
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 268 seconds]
wallet42 has joined #pypy
parazydix has joined #pypy
parazydix has quit [K-Lined]
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 252 seconds]
moei has quit [Quit: Leaving...]
dddddd has quit [Remote host closed the connection]
mkaito_iz has joined #pypy
mkaito_iz has quit [Ping timeout: 252 seconds]
XenoirKC has joined #pypy
XenoirKC has quit [Remote host closed the connection]
moei has joined #pypy
_whitelogger has joined #pypy
Zaab1t has joined #pypy
Zaab1t has quit [Quit: bye bye friends]
BuckTurg- has joined #pypy
BuckTurg- has quit [K-Lined]
kori has joined #pypy
kori has quit [K-Lined]
MonoMonkeynU has joined #pypy
MonoMonkeynU has quit [K-Lined]
squidicuzzk has joined #pypy
squidicuzzk has quit [Remote host closed the connection]
jacob22__ has joined #pypy
swordsqr has joined #pypy
swordsqr has quit [K-Lined]
dddddd has joined #pypy
ASucAL has joined #pypy
ASucAL has quit [Remote host closed the connection]
oberstet has joined #pypy
oberstet has quit [Quit: Leaving]
antocuni has joined #pypy
jacob22__ has quit [Read error: Connection reset by peer]
__pv has quit [Remote host closed the connection]
lritter has joined #pypy
MetacityAT has joined #pypy
MetacityAT has quit [Remote host closed the connection]
__pv has joined #pypy
jacob22__ has joined #pypy
antocuni has quit [Ping timeout: 246 seconds]
oberstet has joined #pypy
_whitelogger has joined #pypy
ewookJQ has joined #pypy
ewookJQ has quit [Remote host closed the connection]
antocuni has joined #pypy
antocuni has quit [Ping timeout: 245 seconds]
witran has joined #pypy
WGH has quit [Ping timeout: 260 seconds]
oberstet has quit [Ping timeout: 252 seconds]
mattip has joined #pypy
<mattip>
in light of win32 issue #2899 (we need to use LoadLibrary_Ex with LOAD_WITH_ALTERED_SEARCH_PATH), I will look at the locale-encoded dll name failures too
<mattip>
so arigato was right, a few days ago he volunteered to look at those failures and I told him to wait for the unicode-utf8 branches. Oops
jphelps has joined #pypy
jphelps has quit [Remote host closed the connection]
tzafrirQN has joined #pypy
tzafrirQN has quit [Remote host closed the connection]
Gunde has joined #pypy
Gunde is now known as Guest55779
Guest55779 has quit [Remote host closed the connection]
dddddd has quit [Remote host closed the connection]
lritter has quit [Remote host closed the connection]
antocuni has joined #pypy
antocuni has quit [Ping timeout: 268 seconds]
themsay has joined #pypy
dddddd has joined #pypy
bbot2 has joined #pypy
sh4rm4^bnc has joined #pypy
<sh4rm4^bnc>
hi, i've read the documentation about building pypy from source but am a bit appalled from the listed huge build times. isn't it possible to build it from already translated C sources ?
<sh4rm4^bnc>
i.e. the maintainer does "make dist" which creates the rpython binary which is then used to translate the sources into C which are then stuffed into the source tarball
<nedbat>
sh4rm4^bnc: i know almost nothing about this, but i'm curious: why are the build times a problem?
<sh4rm4^bnc>
think gentoo
<sh4rm4^bnc>
build from source distro
<sh4rm4^bnc>
every user builds from source, not just the distro kingpin
<nedbat>
sh4rm4^bnc: i see
<nedbat>
sh4rm4^bnc: if they want to build from source, why would they be ok with pypy doing half the build for them?
<nedbat>
sh4rm4^bnc: i guess i don't know why gentoo users build from source
<sh4rm4^bnc>
C source is source too
<nedbat>
sh4rm4^bnc: one man's source is another man's object. why do they value building from source?
rubdos has quit [Ping timeout: 260 seconds]
<sh4rm4^bnc>
it's just like C projects that ship bison or flex or gperf input files and expect the user to have those tools installed during the build
<sh4rm4^bnc>
whereas they could instead ship the already generated C files and spare the user the dependencies
<nedbat>
sh4rm4^bnc: gentoo users run bison, or they compile pre-bison'd C source files?
<sh4rm4^bnc>
most projects unfortunately DON'T ship the pre-genned bison files
<sh4rm4^bnc>
but it would be preferable
<nedbat>
bbiab
speeder39 has joined #pypy
<sh4rm4^bnc>
why do ppl want to build from source ? maybe they're cautious and dont want to trust other ppl's binaries.. or they like changing / customizing stuff. or they have a special environment, ...
<sh4rm4^bnc>
building from source is a lifestyle, i guess... just like some ppl prefer to cook their own food
<mattip>
sh4rm4^bnc: by shipping the c files, we force people to trust our generation of them and make it much harder to customize or adapt to an environment
<mattip>
for instance the target architecture is baked in
<mattip>
so we would ship x86, arm, x64, ... ?
<sh4rm4^bnc>
why would you bake in an arch ?
<mattip>
the JIT is architecture specific
<sh4rm4^bnc>
well, couldnt the build generate a x86_64/jit.s and a i386/jit.s and then the makefile selects the right file ?
<mattip>
we also bake in the GC strategy, and many other options
<mattip>
they all get mixed in via the different optimization stages, kind of like LTO
rubdos has joined #pypy
<mattip>
we have had thoughts about separate compilation of independent parts, but it is alot of work
<sh4rm4^bnc>
hmm also when you would ship the C sources, one could actually go directly from 0 to pypy, without having to have a python interpreter already
<sh4rm4^bnc>
i could imagine many ppl would prefer running all their stuff with pypy
* mattip
off, zzz
kbni has joined #pypy
<_aegis_>
why ship source when you can't meaningfully change build flags? it would be an esoteric version of pypy
<_aegis_>
makes more sense to speed up the build.
kbni has quit [Remote host closed the connection]
<sh4rm4^bnc>
well of course you should be able to meaningfully change build flags
<sh4rm4^bnc>
the thing could generate the .c and .s files, and a makefile that respects the conventions like CFLAGS etc
<sh4rm4^bnc>
and then you can do make -j8 and all your 8 cores build pypy in a few seconds
<nedbat>
sh4rm4^bnc: it seems like you are saying, "we are people that like control, so we build from source, but please remove a bunch of control so it can build faster"
<sh4rm4^bnc>
why remove control ?
<nedbat>
sh4rm4^bnc: the last 30 minutes of discussion have mentioned a few choices that are hard to change if you ship C sources
<sh4rm4^bnc>
really ? is it much harder to write into your output file: #ifdef x86_64 #include "jit_x86_64.c" instead of jit_x86_64();
<sh4rm4^bnc>
just a matter of structuring the output
<nedbat>
sh4rm4^bnc: that sounds like the kind of thing someone would say if they weren't intimately familiar with the code generation process
<nedbat>
sh4rm4^bnc: sorry for all the questions, but how often do you rebuild pypy?
<sh4rm4^bnc>
i never did
<sh4rm4^bnc>
i never used it, yet
<sh4rm4^bnc>
was just trying to bundle it for my distro
<sh4rm4^bnc>
and while studying the docs i thought "hmm why wouldnt they ship the C sources?!"
<nedbat>
sh4rm4^bnc: wait a sec, you donn't know how long it will take to build?
<sh4rm4^bnc>
i'm reading the docs and the makefile
<sh4rm4^bnc>
@echo "Building a regular (jitting) version of PyPy, using CPython."
<sh4rm4^bnc>
@echo "This takes around 2 hours and $(URAM) GB of RAM."
<nedbat>
sh4rm4^bnc: people here have tried to explain the difficulty of shipping sources.
<sh4rm4^bnc>
that, clearly sounds real bad
<sh4rm4^bnc>
like, taking 2x as long as mozilla firefox
<sh4rm4^bnc>
which is 150MB of compressed sources
<sh4rm4^bnc>
and C++ which compiles real slow
<nedbat>
sh4rm4^bnc: pypy is worth it :)
<_aegis_>
you cannot meaningfully change the *pypy* build flags if you ship C
<_aegis_>
because none of them are CFLAGS
<_aegis_>
they apply pre-generation, and are kinda important
<sh4rm4^bnc>
make CFLAGS="-DUSE_GENERATIONAL_GCC -DARCH_X86"
<nedbat>
sh4rm4^bnc: it seems like you aren't listening to people's answers....
<_aegis_>
also, you're just asking for like 2gb of C source lol
<sh4rm4^bnc>
lol impossible
<sh4rm4^bnc>
that would mean the pypy bin would have like 200MB
<_aegis_>
you are way out of your depth
<_aegis_>
I saved my last pypy build because I've been looking at making the pypy binary smaller and optimizing the build process
<sh4rm4^bnc>
not really, i'm quite familiar with C, build systems, conditional compilation etc
<_aegis_>
(because I build it often and ship it with a real world app, and don't like how big it is or how long it takes to build)
<nedbat>
sh4rm4^bnc: it seems like you aren't familiar with how pypy is generated
<_aegis_>
the C source for my x86_64 pypy build is 391MB
<_aegis_>
if you added every possible combination of build flags it would be at minimum a few gigabytes
<sh4rm4^bnc>
well, if you tar that up it'll be 25 MB ?
<sh4rm4^bnc>
that's still much less than firefox
<_aegis_>
the built pypy library is 35-50mb for me
<nedbat>
sh4rm4^bnc: you keep overlooking the build options
<_aegis_>
and that's with *one* set of build flags
<sh4rm4^bnc>
could you enlighten me as to what build options you mean ?
<_aegis_>
only if you promise to stop talking over us.
<sh4rm4^bnc>
talking over ? i thought we were having a productive discussion here
<_aegis_>
we say things, you ignore them or tell us they are impossible
<_aegis_>
what you are suggesting would require large compromises or a lot of human labor
<_aegis_>
I'm working on getting you the xz'd size of my pypy C source but it's still compressing.
<sh4rm4^bnc>
thanks, that would be interesting
<nedbat>
sh4rm4^bnc: when someone says the jit choice isn't a CFLAG, instead of insisting it could be a CFLAG, ask why it isn't.
<_aegis_>
in the current rpython compiler, you would need to generate every combination of flags as a separate source tree to hoist those into CFLAGS
<sh4rm4^bnc>
nedbat, i was showing how one could pass build options via CFLAGS
<_aegis_>
yes, we know how to do that.
<_aegis_>
that's not the problem.
<nedbat>
sh4rm4^bnc: yes, but that won't work here.
<_aegis_>
it happens before the C exists.
<_aegis_>
the only way around that, would only work for some of the rpython flags
<_aegis_>
and it would be to modularize the build a bit
<_aegis_>
which probably requires coming up with abstraction layers / API layers that don't exist yet
<sh4rm4^bnc>
right... that was what i thought
<_aegis_>
and I'd guess at least a month of work by a couple people already very familiar with the process
<sh4rm4^bnc>
a month ? you can build rome in a month :)
<nedbat>
sh4rm4^bnc: you did it againn
<sh4rm4^bnc>
what, being me ?
<nedbat>
sh4rm4^bnc: an expert is telling you something, and you are telling them they are wrong.
<sh4rm4^bnc>
no, i am indicating that i think their estimation is a bit exaggerated
<nedbat>
sh4rm4^bnc: ie, wrong.
<sh4rm4^bnc>
nedbat: i am not trying to be hostile here
<nedbat>
sh4rm4^bnc: you have never even built the software, much less studied it, but you think you know enough to make that claim?
<nedbat>
sh4rm4^bnc: if your goal is to learn things, you are doing a bad job
<_aegis_>
I'm engaging you respectfully, you are repeatedly disagreeing with me about something you are not a subject matter expert on
<_aegis_>
at this point I would encourage you to do more research and maybe try implementing some of the things you're talking about before continuing this discussion
<_aegis_>
xz output is 23mb for my x86_64-only mac-only jit-only build, with no possible compile-time configuration of pypy itself, and none of the supporting python packages included
<sh4rm4^bnc>
that would be acceptable, imo
<_aegis_>
no, because you need one of those per architecture, and you cannot configure pypy itself at all, and you'd need to reimplement the tail-end rpython build system yourself
<sh4rm4^bnc>
cpython is 12 mb sources
<nedbat>
sh4rm4^bnc: why are you telling us that?
<sh4rm4^bnc>
doh, why are you criticizing every word i say ?
<nedbat>
sh4rm4^bnc: i'm nnot criticizing, i'm trying to understand your point.
<_aegis_>
if you look at the actual intermediate C source it's barely more usable than assembly w/ debug symbols anyway
<sh4rm4^bnc>
i think having the ability to build pypy from C sources would be a horrific thing
<nedbat>
sh4rm4^bnc: it's a factoid that seems to have no bearing on the current discussion, so I must be misunnderstannding somethinng.
<nedbat>
sh4rm4^bnc: horrific? I thought it was what you wanted?
<_aegis_>
terrific.
<sh4rm4^bnc>
the build would be much faster, and you could directly bootstrap pypy as your system's python interpreter
<_aegis_>
it's already architecture and ABI specific
<sh4rm4^bnc>
uh yeah
<nedbat>
sh4rm4^bnc: "terrific" is really good. "horrific" is really bad.
<_aegis_>
so the C sources are not going to be more useful than a binary in most places
<nedbat>
sh4rm4^bnc: english is messed up, yo.
<sh4rm4^bnc>
yeah i meant terrific :)
<nedbat>
sh4rm4^bnc: but you are learning why it isn't done that way.
<_aegis_>
much of pypy's perf is going to come from the JIT, which won't be affected by your host C compiler's options much
<_aegis_>
so "but I want to funroll my loops" doesn't apply
<sh4rm4^bnc>
i usually use -Os related optimizations
<sh4rm4^bnc>
but it's also for usage of crosscompilers and hardening or debug options
<_aegis_>
none of that applies to pypy intermediate source in a major way
<_aegis_>
python itself is memory safe, the pypy JIT output will bypass any compiler hardening because it is itself a compiler
<_aegis_>
that has full control of e.g. its own stack and memory mappings
tiagofalcaonP has joined #pypy
<sh4rm4^bnc>
so how would i cross-compile pypy for say rasppi ?
tiagofalcaonP has quit [K-Lined]
<sh4rm4^bnc>
or maybe even an arch who doesn't even have jit support yet ?
<sh4rm4^bnc>
"There are a few workarounds for differences between Alpine Linux and the Debian-based Linux distributions that the PyPy team has thus far worked against."
<nedbat>
sh4rm4^bnc: i'd want to find the discussion to see what they are talking about.
<sh4rm4^bnc>
it's listed in the last paragraph
<sh4rm4^bnc>
stuff like
<sh4rm4^bnc>
RPython expects the stdin/stdout/stderr file handles in stdio.h of the standard libc to be of type FILE*. With musl these are of type FILE *const rather. We patch RPython with the correct type.
<nedbat>
sh4rm4^bnc: that shows the stuff, it doesn't show what "worked against" means.
<sh4rm4^bnc>
oh yeah i don't know either
<nedbat>
sh4rm4^bnc: seems like this repo will have details about how to get it to build the way you want.