<kenaan>
Raemi extradoc f800f7a9c095 /blog/draft/2019-07-arm64.rst: final fixes..
inhahe has quit [Ping timeout: 246 seconds]
inhahe has joined #pypy
antocuni has joined #pypy
kenaan has quit [Read error: Connection reset by peer]
xcm has quit [Read error: Connection reset by peer]
xcm has joined #pypy
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
<fijal>
arigato: I looked a bit into dynamic jitcodes
<fijal>
it's a bit of a mess with jitdrivers and stuff like that
<fijal>
but maybe we can do some hack in between, with a level of indirection. That's not ideal for most cases though
<fijal>
since we can't get the correct call_assembler
<fijal>
or at least some cases
<fijal>
but if you do something like "this is a jitdriver which accepts three arrays of int, float, ptr" and then dynamically call new jitcodes, that's very doable
<arigato>
maybe we can start with not adding any jitdriver at all
<arigato>
adding new jitcodes for functions that are actually implemented in C
<arigato>
but which don't loop
<arigato>
or don't loop before residual calls, more precisely
<fijal>
why "don't loop"?
<arigato>
so not jitdriver needed
<fijal>
so essentially replacing a call to a C function with a call (maybe inlined) to a jitcode?
<arigato>
think about the first use case: adding a jitcode for the numpy array index
<arigato>
yes
<fijal>
right, that makes sense
<fijal>
so you already have jitdriver_sd and the whole metainterp set up, you just replace a call to C function
<fijal>
makes sense
<arigato>
(...adding a jitcode for the numpy array indexing operation)
<fijal>
and then if you fail a guard you blackhole until you get out, essentially
<fijal>
(which if you loop can be a minor disaster I guess)
<antocuni>
what are dynamic jitcodes?
<fijal>
antocuni: dynamically loaded jitcodes for jitting inside C functions
<arigato>
blackhole: yes, which should just work like it does now
<fijal>
we *are* going to have something like "this is a C function, this is a jitcode, I promise they do the same thing"
<antocuni>
whooo, I'm undecided whether to be excited or scared
<antocuni>
how would you *produce* this jitcode in the first place? Using llvm or something that?
<fijal>
I think let's start with "write it by hand"
<fijal>
"compile from Cython" would be a good idea too
<arigato>
woohoo, we need an assembler in the traditional sense
<antocuni>
and to test the jitcodes, we write a jitcode interpreter in rpython
<fijal>
we *have* a jitcode interpreter
<fijal>
it's called metainterp
<arigato>
yes, and it's possible in theory to implement the whole thing without any hack in our current jit, by jitting this rpython interpreter *cough*
<arigato>
meta-meta-tracing
<fijal>
I really don't understand what you're talking about tbh
<fijal>
we have the interpreter - what we need is to be able to load jitcodes from disk/memory
<fijal>
and be able to say "please replace this C call with an inlined tracing call"
<fijal>
(and a ton of issues related to non-prebuilt things)
<arigato>
anto's point is that if we write jitcodes manually, we need a way to test them directly
<fijal>
you mean like run them with blackhole?
<fijal>
or executor?
<arigato>
yes
<fijal>
well, so we have that
<antocuni>
my last comment was almost a joke to respond to "we need an assembler in the traditional sense"
<fijal>
yes, we need assembler
<fijal>
as in something that will read it from some representation and then produce an instance of JitCode
<arigato>
right, the equivalent of self.interp_operations() from the tests
<fijal>
it's *mostly* just that
<antocuni>
my first concern was "how do you produce a jitcode which is equivalent to a given piece of C code"; I think the answer is "at first, we will just write it by hand", which is fine for me
<antocuni>
so I think all my questions have been answered :)
<arigato>
sorry, I've been dumping my thought here and on #pypy-sync instead of thinking it through first :-)
<mattip_>
NameError: global name 'get_nonmoving_unicodebuffer' is not defined
<krono>
mattip_: thats probably related to a change exactly there yesterday
<mattip_>
also happens on default
<mattip_>
yup
<krono>
get_nonmoving_buffer ist now get_nonmoving_buffer_ll or so
<krono>
so probably simliarily for unicode
mattip_ has quit [Remote host closed the connection]
<arigato>
oops, I'll fix that
<arigato>
krono: if a class has got a _freeze_ method returning True, then it's instances are translated as never-changing, never-created-dynamically
<arigato>
s/it's/its
<krono>
hmm, yea, i just read around
<krono>
it's hidden in 2..3 places and I think I got the idea
antocuni has joined #pypy
<krono>
Am I right that if I have an abstract superclass and one concrete class is _freeze_ and the other not, that is not supposed to work?
<arigato>
yes, that's unlikely
<krono>
interesting, thanks
<arigato>
if a class has got a _freeze_ method then it's not considered a class at all, for the purpose of inheritance and typing
<arigato>
instead, its prebuilt instances are considered as individuals
<arigato>
I'm not sure that isinstance() would work, for example
<krono>
aaah funny
<krono>
thanks
<arigato>
in pypy the most important example of that is the object space, so that all the 'space' variables we have are found to contain a single prebuilt instance and they get typed as a "void" (because no data is needed)
kenaan has joined #pypy
<kenaan>
arigo default 2315521d2c5d /rpython/rtyper/lltypesystem/: Fix for 19e211d4c76b (oops, sorry)
<krono>
arigato: I saw :) maybe add a hint to the docs? Just when someone stumbles over it that they have an idea of what's going on
<arigato>
so PyPy_BuildValue is called at a random unexpected moment
<antocuni>
another advantage of HPy is that since you need a context, you will no longer be able to do this kind of stuff
<arigato>
right
<arigato>
so in this case, it's called when we dlopen() the library, and this occurs with the GIL released in PyPy, naturally
<arigato>
(it may be the same in CPython and there is a latent bug, or not)
<arigato>
(fwiw, the crash really occurs not because of the GIL, but because we initialize the GC's rawrefcount hack only after we figure out that a module is a cpyext module, and these static variables are initialized before)
<antocuni>
what about detecting the case and aborting early, saying that it's not implemented/supported by PyPy?
<antocuni>
I don't think it's officially supported even by CPython, it looks more it works by accident
<arigato>
well it's easy to detect or fix, but it comes as the cost of one extra check for every single from_ref() or similar
<antocuni>
ah I see, because obviously the only place where we can detect it is inside the Py* calls
<arigato>
or we put the check in the generic code for all PyPyXxx() functions
<arigato>
yes
<antocuni>
yes, but it's still costly
<antocuni>
or we handle the segfault and somehow detect the case in the fault handler :)
<arigato>
sure :-)
<antocuni>
can we dlopen it twice, linking it to different versions of PyPy_*?
<arigato>
please no
<arigato>
there's already a hack in wrapper_second_level, to detect when we call a PyPy_* without the GIL when the documentation says we need the GIL, and to metaphorically shrug and acquire/release the GIL anyway
<antocuni>
no ok, it's not the right think probably. But I'm curious if it is theoretically possible or not
<arigato>
that sounds like a good place to put this extra check
marky1991 has quit [Ping timeout: 248 seconds]
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
<fijal>
arigato: feels like the C call is the wrong level, in a sense
<fijal>
(for calling dynamic jitcodes)
<fijal>
maybe not
<fijal>
I'm trying to think how that would work
<kenaan>
arigo default c89821342184 /pypy/module/cpyext/: cpyext: add a workaround that should make one case work (typically seen only with C++ code)
<fijal>
the problem is that a typical cython function has (essentially) a signature PyObject*, PyObject* -> PyObject*
<arigato>
if we want to JIT some code that manipulates PyHandle, then it's possible, with a few special cases in the JIT
<arigato>
maybe even PyObject*, if we really want
<arigato>
no, what am I say?
<arigato>
if you have C code that manipulates PyObject* or PyHandles, turn it into jitcode that looks like it comes from RPython, manipulating GC objects directly
<arigato>
maybe we need both approaches, e.g. we still need to manipulate PyObject* or PyHandles directly if they are read from or written to fields of C structures
<antocuni>
fijal: I have some rough ideas for HPy so that C function will be able to expose different calling conventions
<antocuni>
so a "max" function will have two signatures: (HPy, HPy)->HPy but also (long, long)->long
<arigato>
right, so that might fix half of the problem, and we're left with the other half of really needing HPy or PyObject* sometimes anyway
<antocuni>
and another possible "calling convention" is rpython-like
<antocuni>
so for example you could decide whether to pass pyhandles (if you call from C) or directly W_Root (if you call from rpython)
<fijal>
right, but I don't think you would ever call it from RPython
<fijal>
at the end of the day you have a python call to a C function
<fijal>
that can come from Cython or somewhere else
<fijal>
I *think* you really need to support PyObject* (or PyHandle)
<fijal>
because someone inside the C function will call some part of C API
<fijal>
heh so we're back at square one - if the C code calls PyList_GET_ITEM, we want the *RPython* implementation of PyList_GET_ITEM not the C one
<antocuni>
yes but e.g. it would be useful in the opposite direction: I am inside C and doing HPy_Call(my_hpy_callable, my_hpy_arg)
<fijal>
which we're in the process of moving back
<antocuni>
depending on whether my_hpy_callable is implemented in C or RPython, you want to convert my_hpy_arg to W_Root, or keep it a HPy handle
<antocuni>
fijal: why do you want to call the rpython GET_ITEM instead of the C one?
<fijal>
antocuni: to trace it in the JIT
<antocuni>
so, the jitcode wants to call the rpython version, but the C code wants to call the C version
<fijal>
... which means we again land with 2 versions
<antocuni>
well, in the pyhandle world, HPy_GetItem is a thing wrapper around space.getitem
<antocuni>
s/thing/thin
<fijal>
right
<fijal>
I think it's ok to consider pyhandle world only
<fijal>
but not pure-C world only
<antocuni>
yes, I think it is fine to say that if you want your C extension to be fast on PyPy, you need to use hpy
<fijal>
yeah, as long as someone like cython can use hpy automatically
<antocuni>
yes, this is the plan
<arigato>
makes sense to me too
* arigato
dinner
* fijal
ponders what would be a good start then
<fijal>
maybe I'll start with dynamic jitcodes that really just go signed, signed -> signed and that's it
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
<arigato>
Yes, and call that from a regular rpython loop with a jitdriver, maybe using a custom llop
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
marky1991 has quit [Ping timeout: 245 seconds]
jcea has joined #pypy
kanaka has joined #pypy
kanaka has quit [Changing host]
kanaka has joined #pypy
antocuni has quit [Ping timeout: 248 seconds]
Rhy0lite has quit [Quit: Leaving]
<kenaan>
rlamy apptest-file d7026dea7f27 /pypy/: Use a separate flag for new-style apptests
oberstet has quit [Ping timeout: 268 seconds]
oberstet has joined #pypy
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]