vkirilichev has quit [Remote host closed the connection]
vkirilichev has joined #pypy
<mihaid>
hello, I have a question regarding enabling profile-guided optimizations. In order to get this to work on any target (including non-pypy targets), after the profile generation you would have to run a training script / workload. However, different targets would require different scripts with a different set of arguments. I see two possible solutions
<mihaid>
for this. Either, there would be a generic script, that by default would suit a PyPy target, which the user might change to suit any other target. Or there can be more profopt arguments added to the translation to include arguments regarding the training script. I'm not sure which these ideas is preferable, though?
vkirilichev has quit [Ping timeout: 240 seconds]
<LarstiQ>
are you doing this for pypy or for rpython?
<LarstiQ>
how much work is it to submit PRs for both?
antocuni has joined #pypy
amaury has joined #pypy
amaury has quit [Quit: Konversation terminated!]
amaury has joined #pypy
<mihaid>
right now I have a PR for pypy that is fully working with PGO and shared, but it does not work for other binaries such as the one in described by targetrpystonedalone.py (mainly because of the issue above)
oberstet has quit [Ping timeout: 240 seconds]
<LarstiQ>
mihaid: either you have something that works for pypy only, or you have something that works for all rpython users
<LarstiQ>
doing the middle of targetrpystonedalone.py I don't think is useful, except for benchmarking perhaps
jcea has joined #pypy
<mihaid>
LarstiQ: well, targetrpystonedalone.py was what cfbolz proposed on my PR to make it more generic. Do you have any advice about another target that is more suitable for a generic PGO, please?
<cfbolz>
mihaid: I think the main problem for me is not the argument to the training run, but the fact that the executable name is hard-coded
<cfbolz>
mihaid: basically it's fine if every language has to put into the target the default training run it wants to use
<cfbolz>
(like what is already in the PyPy target)
amaury has quit [Ping timeout: 255 seconds]
<mihaid>
oh, so just to make this clear. The target should be generic, but it is acceptable to expect that other target will have to modify the makefile for themselves when doing profile guided optimizations?
<cfbolz>
mihaid: not the makefile
<cfbolz>
Just the option
<cfbolz>
Sorry, with 'target' I mean the language specific target*.py files
adeohluwa has quit [Quit: Connection closed for inactivity]
<mihaid>
cfbolz: okay, i think I got it. One more thing, though. When I last update the PR, I added a profoptpath, as a path to the training script. However, I realized now that its arguments, regardless of the script, are the same as for regression testing, because they are hard coded. Should I also add a profoptargs option to the translation?
<mihaid>
so, in my PR, right now you can change the training test for the pgo, but it will run with the same arguments (--pgo -x test_asyncore test_gdb test_multiprocessing test_subprocess ) as the regression test. A different training script however, would need different arguments. Should I pass them in a new variable?
<mihaid>
I got the target*.py issue. This was about the training tests.
<arigato>
write_float_to_mem is called by two functions that we'll call OLD and NEW
<arigato>
you have old code in OLD, which calls write_float_to_mem(12371848189798L), where the value is a longlong-encoded float
<arigato>
because of the mere presence of NEW, then write_float_to_mem's arg is annotated as a float,
<arigato>
so the code in OLD will first convert the value to a float,
<arigato>
and call write_float_to_mem(1237E10)
<arigato>
which is nonsense
<antocuni>
yes, but the function NEW (which is, more concretely: llmodel.AbstractLLCPU.bh_gc_store_indexed_f) should be annotated even if the codewriter never sees the llop
<antocuni>
this is what I'd expect, and it's the part I don't understand :)
<arigato>
parts of the jit are not annotated at all if some jitcodes don't show up
<arigato>
notably, pyjitpl's and blackhole's implementations of the corresponding jitcodes
<antocuni>
uhm, probably this explains it
<antocuni>
who annotates e.g. executor.do_gc_store_indexed?
<arigato>
who knows? maybe only the blackhole's version of the jitcode
<antocuni>
we reached the point in which our JIT is self-conscious and decided what to annotate by itself :)
amaury has joined #pypy
<arigato>
obviously :-)
<arigato>
note that you had a failing test on 32-bit
<antocuni>
yes, that's how I found the bug
<arigato>
I *think* that there are hacks for non-translated runs where mixture of longlong-for-storage and floats are detected
<arigato>
so it's basically your fault for not running tests :-)
<antocuni>
well no, I *did* run the tests (on buildbot because I didn't have a 32 bit chroot locally)
<antocuni>
my fault was to dismiss the failure as unrelated
<antocuni>
because the bug showed up also in places in which the llop was not called at all
<antocuni>
anyway, I suppose I could add some _annenforceargs_ or similar to write_float_at_mem, to catch this kind of error earlier next time
<arigato>
ah righ
<arigato>
yes, possibly
<arigato>
normally, a function that receives a float unexpectedly will crash because of what it tries to do with it
<arigato>
so yes, if write_float_to_mem is an exception, better make it crash
<antocuni>
yes
<antocuni>
also, I did not expect it is possible to union r_longlong and float
<antocuni>
but probably it is too late to forbid it
<cfbolz>
we should try to forbid it anyway
<cfbolz>
it's clearly nonsense
tilgovi has quit [Ping timeout: 272 seconds]
oberstet has joined #pypy
inad922 has joined #pypy
<arigato>
it's forbidden before translation
<arigato>
in the case of longlong-for-storage
<arigato>
after translation, I think I tried but gave up
<cfbolz>
ah :-(
inad922 has quit [Ping timeout: 268 seconds]
<kenaan>
antocuni faster-rstruct-2 65b6be2c1b4f /rpython/jit/backend/llsupport/llmodel.py: enforce the annotation of write_float_at_mem: the problem fixed by 0e91701e92f0 was that we mixed a r_f...
DragonSA has joined #pypy
DragonSA has joined #pypy
DragonSA has quit [Changing host]
amaury has quit [Ping timeout: 246 seconds]
antocuni has quit [Ping timeout: 268 seconds]
jacob22_ has quit [Quit: Konversation terminated!]
<mihaid>
hello, I have another question regarding profile-guided optimizations. Allowing the profopt for other targets will also change the goal directory, and this is a problem because the resulting pypy-c binary needs to be in the goal directory in order to be trained. Should I add the goal directory as a parameter, or should I assume that the translation
<mihaid>
is run from the goal directory and get an absolute path to it?
<arigato>
should we issue a 5.7.2 release? because of b2569cf1bd55, which I suspect is causing other issues too that we see
<cfbolz>
mihaid: you should be able to find the correct output filename as an absolute path, somehow
<cfbolz>
mihaid: but anyway, I think the makefile works like this: it builds everything in /tmp/usession*/testing_1/ and then later the binary is copied
<cfbolz>
the Makefile can work completely in the testing_1 directory
<antocuni>
the only drawback is that if you run pypy-c from the testing_1 directory, it might not be able to import the stdlib correctly, I think
<cfbolz>
it will use the compiled in path
<arigato>
which should always be the right one in this precise case and likely no other
<cfbolz>
yes
<mihaid>
I tried running the pypy-c binary in the /tmp/.../testing and I got an import error, unfortunately
<arigato>
it prints big warnings, but it should work
<arigato>
read carefully the CPython doc about "sys.prefix"
<arigato>
so far we don't set it at all if we don't find the library paths
<arigato>
maybe we should
<arigato>
or, the alternative is to fix sysconfig.py
<arigato>
to not crash if sys.prefix is not set
<arigato>
...no, this sysconfig.py is straight from CPython and it reads sys.prefix there too
<arigato>
so the problem is really that nobody is expecting sys.prefix to be undefined, also in CPython
<kenaan>
ltratt default 008c6f3a2f8f /rpython/translator/platform/: Build main binaries on OpenBSD with wxallowed. On RPython, JIT compilers that need to read and write to the same ...
vkirilichev has quit [Read error: Connection reset by peer]
<kenaan>
cfbolz default 1d471fdc1963 /rpython/annotator/: fix issue #1877: replace bare Exceptions with AnnotatorError in the annotator
vkirilic_ has quit [Ping timeout: 246 seconds]
tbodt has joined #pypy
<mihaid>
so I don't know how to fix the import site issue, but the argument that I added would only need to be used when profopt-ing something other than PyPy. Basically, right now if you need to translate with profopt for PyPy, you just specify --profopt.
<cfbolz>
mihaid: it's a bit weird that the Makefile hard-codes the goal directory, though. in theory it would be cool if all the building happened in testing_1/
<mihaid>
yes, but I see no other way out. Maybe I can fix this in another issue / pull request?
<cfbolz>
agree that it's annoying to have the pgo work blocked by the prefix issue
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<kenaan>
Alecsandru Patrascu sockopt_zero 74c6874f803f /pypy/module/_socket/: added the possibility to have zero sized buffer length
<kenaan>
Alecsandru Patrascu sockopt_zero 4c2ff9e6a959 /pypy/module/_socket/: proper fix after review
<kenaan>
cfbolz default 965ffd1ed1de /pypy/module/_socket/: Merged in palecsandru/pypy2_sockopt_zero/sockopt_zero (pull request #548) Passing a buffersize of 0 to socket.get...
vkirilichev has joined #pypy
<kenaan>
rlamy py3.5 619bc1ea9e6d /pypy/module/cpyext/test/test_listobject.py: fix test
tbodt has joined #pypy
amaury has joined #pypy
jacob22_ has joined #pypy
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tbodt has joined #pypy
vkirilic_ has joined #pypy
<mihaid>
I think I have found a quick fix. The pypy-c binary is looking for lib-python and lib_pypy to use pypy_find_stdlib in the current directory. If I copy them in /tmp/.../testing then I have no more import site errors.
<mihaid>
is this ok?
Remi_M has quit [Quit: See you!]
vkirilichev has quit [Ping timeout: 260 seconds]
<kenaan>
rlamy py3.5 e8a4ea639258 /pypy/module/cpyext/test/issue2482.c: Adapt test to py3
cstratak has quit [Quit: Leaving]
cstratak has joined #pypy
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<cfbolz>
mihaid: bit heavy-handed. arigato, any other clue?
<arigato>
I'd say, fix sys.prefix
tbodt has joined #pypy
tbodt has quit [Client Quit]
tbodt has joined #pypy
* arigato
looks more carefully at clean-exported-state, the merging of that branch falls inside the (large) window for a rare unrolling crash
<arigato>
mattip: I suppose that if we back out PR #490 and don't see a different crash, we should release 5.7.2 now, because we got several reports about that
<mattip>
ok
<mattip>
so we need to cherry pick the commits that are bug fixes into the release branch
<mattip>
it might make sense to just go ahead with 5.8 and get the cpyext improvements (latest numpy requires them) and the PyBuffer improvements
<arigato>
either that if you feel like it, otherwise we make 5.8 now, yes
<antocuni>
I'd like to merge faster-rstruct-2 before the next release, if it's possible :)
<arigato>
depends on you :-) but I don't expect the next release to occur tomorrow
<antocuni>
perfect
<arigato>
mattip: are you still ok to be release manager? you can opt out if you don't want it any more
<antocuni>
the "now" in "we make 5.8 now" worried me :)
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tbodt has joined #pypy
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tbodt has joined #pypy
tbodt has quit [Client Quit]
jamescampbell has quit [Remote host closed the connection]
pilne has joined #pypy
arigato has quit [Quit: Leaving]
<kenaan>
antocuni faster-rstruct-2 356dacc477eb /rpython/translator/backendopt/: backout 85d3ab6fe80b, which was broken, and start to rewrite the logic in a more generic way. Also, wri...
<kenaan>
antocuni faster-rstruct-2 d5be1a60c5a3 /rpython/translator/backendopt/: add support & test for gc_load_indexed of a list of chars
<kenaan>
antocuni faster-rstruct-2 1e87cd8c5f6c /rpython/translator/backendopt/: implement the analyzer for gc_store_indexed
<kenaan>
antocuni faster-rstruct-2 64c71c919e55 /rpython/translator/backendopt/test/test_writeanalyze.py: add a passing test for gc_store_indexed on a list of chars
<kenaan>
antocuni faster-rstruct-2 616c5957da8d /rpython/jit/backend/llsupport/llmodel.py: @enforceargs does not play well with @specialize, use _annenforceargs_ directly