cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
KindOne has joined #pypy
jcea has quit [Quit: jcea]
altendky has joined #pypy
dddddd has quit [Ping timeout: 265 seconds]
andi- has quit [Remote host closed the connection]
andi- has joined #pypy
oberstet has joined #pypy
lritter has joined #pypy
_whitelogger has joined #pypy
BPL has joined #pypy
jvesely has joined #pypy
BPL has quit [Quit: Leaving]
marvin has quit [Remote host closed the connection]
07EAAWNFI has joined #pypy
lazka has quit [Quit: bye]
07EAAWNFI has quit [Remote host closed the connection]
marvin has joined #pypy
lazka has joined #pypy
omry has quit [Ping timeout: 264 seconds]
omry has joined #pypy
lazka has quit [Quit: bye]
marvin has quit [Remote host closed the connection]
marvin has joined #pypy
lazka has joined #pypy
tsaka__ has joined #pypy
lazka has quit [Quit: bye]
marvin has quit [Remote host closed the connection]
marvin_ has joined #pypy
lazka has joined #pypy
dddddd has joined #pypy
YannickJadoul has joined #pypy
oberstet has quit [Remote host closed the connection]
oberstet has joined #pypy
holdsworth has quit [Read error: Connection reset by peer]
holdsworth has joined #pypy
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
yajadoul has joined #pypy
yajadoul has quit [Client Quit]
yajadoul has joined #pypy
YannickJadoul has quit [Ping timeout: 250 seconds]
Ai9zO5AP has quit [Ping timeout: 240 seconds]
epony has quit [Read error: Connection reset by peer]
<mattip>
ok, but there is no cpython2 or cpython3 either, so why do we need one for pypy?
<tos9>
right :)
<tos9>
that's the answer to his question
<tos9>
the way you declare CPython 2 is you declare 2 classifiers, 1 for CPython being supported, 1 for Py2 being supported
<tos9>
It's the same for PyPy
<arigo>
what are the .debug files in the binaries of pypy and how can I use them with gdb?
<arigo>
ah OK, it seems to be used automatically, but of course I don't have the .c sources, so it has only limited effects
jacob22 has quit [Read error: Connection reset by peer]
jacob22 has joined #pypy
<antocuni>
arigo: they contain the debug symbols which are stripped from the main executable. It's still useful because it shows you the names of non-exported functions
<antocuni>
else the C tracebacks would contain mostly addresses
jcea has joined #pypy
<arigo>
OK
<arigo>
I'm still a bit stuck because on random issue #449 because it works with a pypy3-c that I compile myself, but fails with the official one
<antocuni>
does it work with a nightly build?
<arigo>
I guess?
<arigo>
that's where the official one comes
<arigo>
also, the Arch Linux version also fails
<arigo>
I mean, no, sorry
<arigo>
I guess that with a nightly build it's the same as with the official release build
<arigo>
i.e. it fails
<antocuni>
well, you said the official one doesn't work. So if the bug is present in a nightly build, you may be able to recover the C sources from the usession directory
<arigo>
ah
<arigo>
thanks
<antocuni>
also, it might be a good idea to tweak our release process and always publish the C sources... it might be useful for cases like this
<arigo>
it's a major size issue, though
<antocuni>
although we would need to release different sources for different platforms, which is a bit weird
<mattip>
fwiw, the linux buildbots are running inside manylinux2010-based docker, so maybe some kind of system call or support library call?
<antocuni>
arigo: the binary tarball for linux64 is ~30MB, the C sources are ~17MB
<arigo>
ah, only? OK
<antocuni>
just tried to create a tarball out of the c-rXXX directory produced by your "done" script, after doing "make clean"
<arigo>
OK
<arigo>
OK, I cannot access bencher7's buildslave user
<antocuni>
also, it would make it easy to compile an lldebug version of an officially released version, if you need it
<arigo>
or use "sudo" on bencher7 generally
<arigo>
bah OK, I understood the problem
<arigo>
OK that *is* really obscure
<mattip>
arigo: you are now in the sudo group
<arigo>
mattip: thanks
<arigo>
it works with the locally-compiled pypy3-c because this one has a built-in sys.path that makes sense for my system, and this built-in sys.path is used for exactly one import statement:
<arigo>
import os, sys
<arigo>
sys.path[:] = path
<arigo>
the bug is "import os"... before setting up sys.path
<mattip>
heh
<mattip>
nice catch
<arigo>
it was indeed added very recently, it's just in the 7.3.1 I think
<antocuni>
do we have the equivalent of we_are_translated() if we write C code?
<cfbolz>
antocuni: how can you have untranslated C code?
<antocuni>
indeed, it's a bogus question. What I want is to know whether my C code is being called inside a translated pypy or during tests
<mattip>
we go through all that effort to ensure translated and untranslated appear the same, and now you want to undo that? :)
<antocuni>
I think it works, but as the comment says, it's a delicate piece of code because if you call any RPython function, things are getting confused
<antocuni>
I was wondering whether it was better to reverse the logic, i.e. write HPyErr_Occurred directly in C, and call a python helper for the non translated case
xcm has joined #pypy
<arigo>
maybe
<antocuni>
opinions? The current solution has the advantage of working already, at the cost of the big disclaimer in the comment
<antocuni>
uhm, now that I think of it, other similar functions will be harder to write in C; e.g. for HPyErr_SetString, we need to convert the char* into a w_string
dddddd has quit [Ping timeout: 256 seconds]
<arigo>
well you can split it into the place that builds the w_string, and the actual setting of the exception
<antocuni>
yes, that's my point; I need to split it anyway, so I'm not sure whether "1st half in C e 2nd half in RPython" is better than "1st half in RPython and 2nd half in C"
<arigo>
I guess the new proposed solution is clearer
<arigo>
if it works, at least
<antocuni>
uhm, do we have a way to set an RPython exception from C? In translator/c/src/exception.h I see only _RPyRaiseSimpleException, which I'm not sure it works for OperationError
<arigo>
you just want to move the initial Err_Clear() to C
<arigo>
it's even unclear if we want Err_Clear() here
<arigo>
I guess we do if we want to list HPyErr_SetString() in the list of functions that can be called if there is already another exception
<arigo>
that's not very clear, but why not
<antocuni>
yes exactly, and I think it's a common pattern in CPython, isn't it?
<arigo>
to "overwrite" an exception? I don't know if it's common
<antocuni>
well, for example it is used by our own HPy test :) test_basic.test_exception_occurred
<arigo>
yes, it's certainly not common to do PyErr_Clear() followed by PyErr_SetString()
<antocuni>
I think it's fine to decide either way, for pypy it doesn't change much. I wonder if it changes anything for e.g. graalpython
<antocuni>
phlebas, fangerer_: do you have opinions about this?
<arigo>
I think it's fine. Anyway we'll need a list of functions that are allowed to be called with an exception
<antocuni>
for more context, the question is whether we want to allow the pattern "if (HPyErr_Occurred()) HPyErr_SetString(...)", or whether we want to require the user to call HPyErr_Clear before the SetString
<arigo>
and yes, for sanity these functions should be written in C, at least their first bit
<antocuni>
right, I tend to agree
<antocuni>
other question: do we have an easy way to call RPython code from C in a way which works both in tests and after translation?
<arigo>
(and remember, in the future debug mode, we need to check in all other cases)
<antocuni>
arigo: yes, I wrote it in a big comment at the beginning of interp_err.py :)
<phlebas>
antocuni: I think overwriting is fine for us, without having to clear. Just to be sure I understand correctly, this won't chain the exceptions, it's really for replacing, right?
<antocuni>
phlebas: good question. Looking at CPython's source for PyErr_SetObject, I see they have an if for "implicit exception chaining"
<antocuni>
which I think was added recently
<phlebas>
I mean, we might have to support that, too, but maybe this is an opportunity to be clearer about it?
<antocuni>
yes, I am wondering the same
<antocuni>
admittedly, I don't know enough about how CPython works to have an opinion right now
<arigo>
antocuni: re your question, I don't know, but I think there is a simple hack to set the name of the rpython-generated function in C
<antocuni>
but in principle I am fine to have simpler API in HPy, if they allow for a cleaner implementation
<antocuni>
arigo: yes, but it doesn't work in tests, does it?
<antocuni>
arigo: I am leaning towards thinking that the current solution is just simpler :)
<arigo>
no, cpyext/api.py goes through hoops to make it work
<arigo>
well, we're very likely to need it at some point
<antocuni>
for what?
<arigo>
otherwise, you can't test directly any code written in C that calls back to some RPython
<arigo>
you can hope that such code would be the same C code as needed for hpy on CPython, for example, and it's already tested there
<antocuni>
I'm not sure to follow. In the PyPy tests, we are already calling RPython from C, through the context
<arigo>
right, OK
<antocuni>
but here I was searching for a way to call RPython from C without the context (unless we want to put in the context all the helpers functions that we need, which looks obscure)
<arigo>
precisely
<arigo>
you could use more #ifdefs and call them through the context only when testing untranslated...
<antocuni>
true... but I'm no longer sure whether the end result will look cleaner than the current one,
<arigo>
after translation, I think so, yes
<arigo>
another alternative, which is more explicit and maybe better:
<arigo>
the @API.func decorator should have support for functions that can explicitly be called with an exception set, and in that case, it should grab the exception and give it to you as an extra argument
<arigo>
so we are explicit about marking, and we remain "safe" by not needing walls of comments
<arigo>
or only once, in the implementation of the decorator
<antocuni>
ok, but how do we implement e.g. Err_Clear()?
<arigo>
@API.func(grab_exception=True)
<arigo>
def Err_Clear():
<arigo>
pass
<arigo>
def Err_Clear(old_exception):
<arigo>
pass
<arigo>
if you want the exception to be preserved instead, then you need to re-raise it
<antocuni>
this would make the implementation of HPyErr_Occurred a bit weird, and probably unnecessarily inefficient
<arigo>
I'm fine if for HPyErr_Occurred() we add another hack: we could have the C version and use it only if we're translated, maybe with #ifdefs and a bit of logic for running tests
<antocuni>
I claim that we don't have an easy way to do that (or, if we have, I haven't found it). That's why I needed to split it into two parts with "if we_are_translated()" :)
<arigo>
ah no
<antocuni>
ah no wait, maybe we can
<arigo>
I claim that gcc has a good chance to optimize the function completely
<antocuni>
I assumed that the context is a prebuilt struct, but actually it seems to be populated by State.setup, which is executed at runtime. So we can maybe use we_are_translated() there?
<arigo>
in module.cpyext there is really different top-level setup() logic for the two cases
<antocuni>
which ones?
<arigo>
but yes, first let's check if HPyErr_Occurred() written in this way would really be any different, because I'm not sure
<arigo>
module.cpyext.api.setup_library() versus module.cpyext.api.build_bridge()
<arigo>
basically just do "if not self.space.config.translating:"
<antocuni>
I see
<arigo>
(yes, HPyErr_Occurred() would probably still be unnecessarily complicated, with the re-raising that records things like the current RPython traceback entry)
<antocuni>
well, if we have an easy way to write two different functions for before and after translation, we probably no longer need the grab_exception hack
<arigo>
(also, you can't write an RPython function that both returns a value *and* (re-)sets an exception)
<arigo>
yes, then all functions written in RPython must not be called with an exception set, again, and that's a clear rule
<antocuni>
with some logic in State.setup which automatically select one or the other depending on the space.config.translating
<arigo>
about the type-building support, there is still nothing that was done, right?
<antocuni>
nope, my plan was to switch to it after I fixed the exception tests in PyPy
<antocuni>
arigo: uhm, but I still don't see an easy way to implement HPyErr_SetString with the "new" approach
<antocuni>
ah, maybe it's enough to add a "clear_exception" flag to API.func
<arigo>
that's obscure, and won't work if we need to chain exceptions
<antocuni>
so, back to grab_exception again?
<arigo>
maybe?
<antocuni>
I admit I don't like the direction in which we are going. Once nice property of the current code is precisely that we do NOT generate obscure wrappers around functions
<antocuni>
the code that you see is exactly we you put in the context, without all the magic which is done by cpyext
<antocuni>
if we start by adding grab_exception, then later we will add another flag, and the soon we will end up with monstrosities like make_wrapper_second_level & co.
<arigo>
agreed
<arigo>
I think it's better to find a reasonable way to call RPython <-> C and use explicitly C wrappers for anything nonstandard
<antocuni>
right. And the "reasonable way" could be to put a well defined C name on the rpython function for the translated case, and add it to the context for the untranslated case
<antocuni>
unless we can think of something better
* antocuni
looks at cpyext's build_bridge
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
<arigo>
also, it's a question of priorities, but I think that at the moment we'd get people to use HPy for CPython only on the promise of future fast PyPy support, if it worked at all
<arigo>
missing a comma for clarity there
<arigo>
also, it's a question of priorities, but I think that at the moment we'd get people to use HPy for CPython only, if it worked at all (together with the promise of future fast PyPy support)
<arigo>
the reference tracking and debug mode alone would be welcome even if you only care about CPython, and it simplifies some parts of the API
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
Dejan has joined #pypy
<antocuni>
arigo: sorry, not sure to understand what you want to say
<antocuni>
and how it is related to the pypy-only implementation details which we were talking about 😅
Ai9zO5AP has quit [Ping timeout: 240 seconds]
Ai9zO5AP has joined #pypy
Ai9zO5AP has quit [Excess Flood]
Ai9zO5AP has joined #pypy
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
jacob22 has quit [Read error: Connection reset by peer]
<ronan>
For the recvmesg_into topic, I'm wondering whether I should keep making changes on py3.6 and backport later, or rebase to default
<ronan>
hrm, in order to build topic branches, buildslaves need to have evolve enabled
<ronan>
mattip: can you install evolve on "bencher4"?
dmalcolm__ is now known as dmalcolm
YannickJadoul has quit [Remote host closed the connection]
YannickJadoul has joined #pypy
<arigo>
ronan: I can. I see it has got hg version 4.8.2, that's recent enough, right?
YannickJadoul has quit [Quit: Leaving]
<mattip>
anything done inside the docker image should be committed to the Dockerfile.
<arigo>
ah sorry, forgot it's inside a docker image
<arigo>
go ahead then
<mattip>
offline for a bit, will do it soonish
<mattip>
Rohan:
<mattip>
ronan: just make it a long lived branch.
* mattip
still don’t understand why we are using git workflow with mercurial
xcm has quit [Killed (card.freenode.net (Nickname regained by services))]
xcm has joined #pypy
oberstet has quit [Remote host closed the connection]
<ronan>
mattip: that's what I'm trying to do now, but the interaction of topics with rebase is a bit hairy
<arigo>
you can make a regular branch commit on top a topic commit, or does stuff explode if you were to attempt that?
<arigo>
maybe you get "branch 'default' has multiple heads" when you try to push?
<arigo>
maybe you need to make and push a new branch out of "default", and then use the web interface to merge topic/default/recvmsg_into => your new branch?
<ronan>
arigo: IIRC the web UI doesn't allow topics to move across branches
<arigo>
OK
<arigo>
the topics feel like an incomplete attempt at getting something that is not very clear to me, but maybe they'll continue improving in the future
<ronan>
I found a way: create a branch off default, then rebase --keep the topic onto it, then manually strip the topic
<arigo>
"ah"
<arigo>
manually stripping the topic doesn't mean its old commits will disappear from, say, my own working copy, I guess
<arigo>
my own local repository
<ronan>
arigo: one good thing I just found is that it's more convenient than hg histedit to e.g. fixup commits after rebasing
<ronan>
the topic is additional metadata outside the commit
<ronan>
you can change or clear it without altering the commit
SashaRose has joined #pypy
<arigo>
ah, and also commits can be marked obsolete and they effectively disappear
<arigo>
so you should not strip but obsolete them
<ronan>
yes, they effectively disappear but you can easily restore them
SashaRose has quit [Client Quit]
<ronan>
it's a better model than git-style random history rewriting, but it takes time to get used to it
<arigo>
OK. just saying, "strip" in hg terms means kill the changeset from your repo
<arigo>
which is pointless if the changeset is already pushed
<ronan>
indeed
<ronan>
the evolve-friendly alternative is hg prune
<arigo>
ah right
<arigo>
ah, that's an alias to "hg obsolete", OK
<arigo>
yes, it's a nice model in theory. I just hope that in practice it doesn't add too much of Yet Another Layer of things to learn before being able to use hg