antocuni changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "PyPy: the Gradual Reduction of Magic (tm)"
<kenaan_>
antocuni vmprof-enable-kwargs ef1507afe550 /: a branch where to implement the same _vmprof logic as in vmprof's enable-kwargs branch
<kenaan_>
antocuni default b7758cca88a3 /pypy/module/_continuation/test/test_stacklet.py: improve the test by also checking the full call-stack at various points. Probably it does not test anything more...
<kenaan_>
antocuni continulet-no-frame-loop f7a1a6eb6908 /pypy/module/_continuation/: temporary checkin with some debugging stuff + a new logic to avoid building cycles of frames
<kenaan_>
antocuni continulet-no-frame-loop ba1ff85734b9 /pypy/module/_continuation/interp_continuation.py: WIP: add two temporary flags which allow to select: 1) the old switch logic vs the new one and ...
<kenaan_>
antocuni continulet-no-frame-loop 3e4c6ca55d1d /pypy/module/_continuation/test/test_stacklet.py: WIP: refactor test_xxx into a proper failing tests, with a decent name and real asserts
<kenaan_>
antocuni continulet-no-frame-loop 82b54bb1e271 /pypy/module/_continuation/test/test_stacklet.py: add a passing test to check that we stick the continulet stack at the right position of the f_b...
<kenaan_>
antocuni continulet-no-frame-loop d5212118820d /pypy/module/_continuation/: introduce the concept of running/paused continulet, depending on bottomframe.f_backref; fix the...
<njs>
I suspect keeping python syntax compatibility is going to be tough for the ->cpyext route, given that you're going to want things like FFI calls to be statically visible
<njs>
but sure, you're saying you plan to switch the input lang, so whatever :-)
<njs>
I'd prefer types like int8, uint16, ..., float32, float64, + intptr_t, uintptr_t, C long
<antocuni>
fijal: I am unsure whether it's a good idea to keep the distinction between "basic int types" and "low level int types"
<njs>
well, meh, I guess you do probably want some more C types for FFI purposes
<fijal>
njs: ok
<fijal>
njs: I'm happy to expose the whole C, do you think the ll.<> is the right prefix?
<fijal>
or C.foo
<fijal>
right now this is just about declaring the input and result types
<fijal>
and classes and it's attributes
<njs>
I don't really know what the ll.<> thing is about. I don't know rpython :-)
<fijal>
njs: the C-level types
<fijal>
njs: or ok, no, the ll.XXX is something I don't want you to care about
<njs>
I can see why you might want to start with rpython for a POC, but it might be useful to give like a cython translation phrasebook for those of us who are more familiar with it :-)
<fijal>
hehe, indeed, but that's what I'm asking about :)
<fijal>
"it's nonsense" is an answer
<fijal>
let me rework this to be more palatable to someone with cython knowledge
<antocuni>
fijal: also, about high-level vs low-level types: I agree that it's nice in theory that a single high-level type can have multiple ll implementations, but I also think it makes things terribly more complicated when you want to interface to C
<antocuni>
so, I am thinking of something more similar to e.g. C++ templates
<njs>
it is not immediately obvious what you're intending the "Classes:", "PBCs:", and "Examples of a signature:" sections to mean, if that's the question
<fijal>
njs: ok, so I'll prepare some more materials for people not intimately familiar with RPython maybe ;-)
<njs>
"object spaces" are not a thing that means much to non-pypy folks :-)
<fijal>
njs: this is indeed for "internal dissemination, top-secret"
<antocuni>
where you say: List(T) is "GcStruct { int length; T* items}"
<fijal>
antocuni: it's not T*
<fijal>
but I'm not sure what do you want tbh?
<antocuni>
to make the relationship between HL vs LL types a core part of the typesystem
<antocuni>
instead of as it is now where you need hacks like hlstr and llstr, for example
<fijal>
but surely you want to call l.append and do x = [1, 2, 3] no?
<antocuni>
yes
<fijal>
so....?
<fijal>
how do you propose it done differently than array(T)?
<antocuni>
right, it might be that my comments are off topic wrt the current discussion
kolko has quit [Ping timeout: 250 seconds]
<antocuni>
basically, I'm saying that I'd like a standard/official way to go from array(T) to the ll representation and back
<fijal>
I can't tell, because I can't see what are you proposing
<antocuni>
but it might not be related to this typesystem thing you are describing
<fijal>
yes, ok, maybe, but it's a) not clear how to do it nicely and b) irrelevant to the typing part
<antocuni>
yes exactly
<antocuni>
sorry
<fijal>
ok, so what do you think?
<antocuni>
it seems like a reasonable starting point
<antocuni>
is the @rpython optional or always needed?
<fijal>
I think the plan is to make it always needed
<fijal>
(but start optional)
<fijal>
the key here being that IF @rpython is present, then the annotations are a bit more serious
<fijal>
they would not let you union for example
<antocuni>
one possibility is to make it needed only if you want to export a function to other modules
<antocuni>
this would be a good step in the direction of separate compilation
<fijal>
why not make it always needed?
<antocuni>
because it's tedious; also, it would clearly mark the boundary between the public API of a module and its private part
<antocuni>
also, regarding the typesystem: we probably want something similar to typedef
<fijal>
yes, I wrote it down, no?
<njs>
are you talking about the same thing?
<fijal>
njs: forget that I showed you anything please :)
<fijal>
njs: I'll show you something else once we can have an internal stuff working
<antocuni>
fijal: the typedef stuff? I cannot see it
<njs>
my guess is that fijal's goal here (though I could be wrong, he didn't say) is to write down some syntax for taking stuff that currently has to be written as C extensions and writing it in... something else
<fijal>
njs: yes, among other things
<njs>
fijal: heh
<fijal>
njs: the SHORT term goal here is to make rpython more palatable
<fijal>
njs: I think what I would do, is I would write a subset of what I just did for extension modules
<njs>
making rpython more palatable seems like a reasonable goal, but yeah it's not immediately relevant to me :-)
<fijal>
antocuni: line 29
<fijal>
antocuni: btw, I'm ok with "tedious" as opposed to "fucking magic that works 99% of the time and explodes remaining 1% with assertion errors"
<fijal>
it also clearly marks what's rpython whats' not
<antocuni>
fijal: line 29 says "Classes::"; no mention of typedefs
<fijal>
yes, so this is how you define an instance
<fijal>
and its attributes
<fijal>
does it have to specifically say "typedef" or are you trying to be difficult?
<fijal>
antocuni: ^
<antocuni>
well no, it's just something different. I was thinking of C-like typedefs, i.e. you give a short name to an arbitrary type
<fijal>
ah
<fijal>
like foo = GcStruct(...)
<fijal>
that stuff?
<antocuni>
yes
<fijal>
I would say we keep the current thing, but we have a way to register it as a name in some namespace
<antocuni>
yes, exactly
<antocuni>
a typedef :)
<fijal>
yes ok
<fijal>
having ronan around would be good
<njs>
maybe you should rewrite everything in rust, I hear it has good types
<fijal>
njs: go away
<njs>
:sniff:
slacky__ has joined #pypy
<ronan>
hi
raynold has quit [Quit: Connection closed for inactivity]
jcea has joined #pypy
songww has quit [Quit: songww]
Rhy0lite has joined #pypy
oberstet has joined #pypy
raynold has joined #pypy
<ronan>
fijal: new syntax is a distraction, and a bad idea IMO
<fijal>
ronan: how else do you want to do it?
<ronan>
we can just use Python
<ronan>
as in @rpython(ObjSpace, int, float, returns_=str)
<fijal>
that is hard to get working because of circular imports
<antocuni>
also, a DSL has good advantages, like the possibility of using things like "bytes?"
<fijal>
besides, it's not ObjSpace - it's a special PBC
<ronan>
fijal: we can accept strings for those special cases
<antocuni>
ronan: compare e.g. rffi way of specifying signatures vs cffi
<fijal>
ronan: having a mix is even worse
<ronan>
antocuni: cffi didn't invent a language, it just uses C
<ronan>
fijal: no, having a parser is worse
<fijal>
why?
<ronan>
languages are hard and parsers are always buggy
<fijal>
we can't have a constructive conversation with just absolute statements ;-)
<antocuni>
I'd assume that a group of people who write compilers should be able to write a parser for a small DSL :)
<fijal>
or not
<ronan>
I wouldn't ;-)
<fijal>
that's a bit beyond the point
<fijal>
ronan: how would you sort out a) circular imports, b) things like str?
<cfbolz>
we all hate writing parsers ;-)
* fijal
really does not mind
<ronan>
fijal: a) use a string (like typing/mypy does), b) just write str
<fijal>
so @rpython("int", "int", returns_="int")
<fijal>
?
<fijal>
for example
<fijal>
typing recently moved to some parser hacks precisely for those reasons :)
<fijal>
FYI
<fijal>
I meant specifically "str?" not "str"
<ronan>
fijal: no, that would just be '@rpython(int, int, returns_=int)'
<fijal>
but sometimes you would put things in "" because of circular imports?
<ronan>
yes
<fijal>
IMO @rpython("int, int -> int") is far more concise
<ronan>
but you need to parse it...
<fijal>
yes, that's fine
<antocuni>
ronan: really, it's not THAT hard ;)
<fijal>
famous last words ;-)
<cfbolz>
the int example is boring. I'd be more interested in seeing what a real objspace method looks like with syntax
<fijal>
ronan: the attributes become a lot harder btw
<fijal>
cfbolz: like @rpython("pypy.space, pypy.W_Root, list(pypy.W_Root)->pypy.W_Root")
<fijal>
for example
<cfbolz>
yes, in your version
<cfbolz>
but in ronan's?
<fijal>
or @rpython("space, W_Root, list(W_Root)->W_Root", namespace="pypy")
<fijal>
ronan: ?
<antocuni>
since we are talking about syntax, I vote for "list[W_Root]" or "list<W_Root>"
<fijal>
@rpython("space", str, [W_Root], {str: W_Root}, TypeDef, bool, bool) looks very similar
<fijal>
but no, because it's all of a sudden List(W_Root, resizable=False) or something
<fijal>
and "str(non-null)" or what?
<fijal>
I don't like how something like a nullable and non-nullable are dissimilar by a large degree
<ronan>
the latter is a feature for me
<ronan>
how do you define str(non-null) in your version, though?
<ronan>
or a non-resizable list?
<fijal>
str?
<fijal>
array(foo)
<ronan>
I mean a no_nul str
<fijal>
str?
<fijal>
with "?"
<cfbolz>
no, wait
<fijal>
ah, no nul
<fijal>
maybe str[something-here]
<fijal>
then <> is for templates [] is for parameters
<cfbolz>
I must say the more syntax you invent the more I believe ronan ;-)
<ronan>
indeed, we need something quite complex to define all the types we use, and I'd rather use Python for that than a half-baked DSL
<antocuni>
here is where my typedef idea comes handy; "typedef str[no_nul=True] str_no_nul"
<fijal>
I must say I'm not a huge fan of DSL either, but noone has provided a reasonable answer to circular imports
<fijal>
any thoughts?
<fijal>
ronan: ?
<ronan>
fijal: my answer is still strings
<cfbolz>
sorry, afk
<cfbolz>
imo we should make this more concrete by looking at a real recursive import problem in interpreter/ or so
<ronan>
if you get one in the signature, you do the lookup at annotation time in the module globals
<fijal>
ronan: no
<fijal>
which module?
<fijal>
you might not have it ever in the module globals
<ronan>
but how would it work in that case in your version?
<cfbolz>
he wants a new namespace where all the types live
<ronan>
but then you need to make sure that everything is defined before using it
<ronan>
it's like a circular import problem, except different
<cfbolz>
well, you do what you suggested, do the lookup only at annotation time
<cfbolz>
but yes
<antocuni>
I think we are looking at this problem somewhat from the wrong perspective
<antocuni>
if the goal is to make RPython a "saner" a more usable language, we should approach it first from the user point of view
<fijal>
IMO having "you write W_Root or 'W_Root' depending on the phase of the moon, but both work" is a bit user unfriendlt
camara has joined #pypy
<antocuni>
i.e., to find a way which is ideally: 1) easy to read; 2) easy to write; 3) consistent; 4) easy to implement
<camara>
str+ for the no nul case
<camara>
taking inspiration from regular expressions
<fijal>
camara: what do you find more readable/writable?
<fijal>
IMO ronans strategy fails the 3)
<antocuni>
also the 2, IMHO
<fijal>
mitsuhiko: ping?
<antocuni>
about no_nul: I would say that str and str(no_nul=True) should just be two different types (even if they end up being implemented by the same annotation with a different flag)
<antocuni>
so, for example: "str" and "str0"
<ronan>
I think fijal's solution fails 1, 3 and 4
<fijal>
what's inconsistent?
<ronan>
1 and 4 because of the DSL aspect
<ronan>
3 is a general property of RPython
<antocuni>
ronan: why is it hard to read, assuming that we design a good DSL?
<antocuni>
also, note that your proposal is ALSO a DSL, just baked by python's parser instead of our own
<ronan>
antocuni: because it's a new language
<antocuni>
I didn't include "easy to learn" :-P
<antocuni>
I'm serious: I think it's fine to trade a bit of learning curve (in the form of having to learn a DSL) in exchange for 1 and 2
<ronan>
antocuni: and yes, I'm also propsing a DSL, but one that's easier to implement and learn
slacky__ has quit [Read error: Connection reset by peer]
<antocuni>
yes, and which is harder to read and write
<antocuni>
ronan: e.g., take the example of test_optimizeopt
<antocuni>
we used to write things like: oplist=[ResOperation('add', v0, Const(1)), ...]
<fijal>
antocuni: no, we didn't write them ;-)
<antocuni>
then we decided to switch to a DSL
<cfbolz>
actually, that DSL shows most of ronan's points
<antocuni>
fijal: I'm quite sure that we did at some point, but well
<antocuni>
cfbolz: does it?
<cfbolz>
yes
<ronan>
antocuni: that's a different case
<cfbolz>
eg it's completely bizarre to have to figure out where descrs come from
<fijal>
antocuni: well, my point was more "we did not write enough of them"
<fijal>
cfbolz: yes, we could have done a better job
<antocuni>
ah yes
<ronan>
that DSL allows switching from a declarative representation to an imperative one
<antocuni>
ronan: I find it very similar to your proposal
<fijal>
ronan: I think I'll implement a prototype and can we take discussion from there?
<fijal>
more likely, I'll start with a way of fixing annotations so they don't change any more
<fijal>
ronan: can we have a 3.5 recap on #pypy-sync though?
<ronan>
my issue with the typing DSL is that I'm fairly sure that some things will be impossible to write
<cfbolz>
note that we have several such mechanisms around already, and they are all not used much (rlib.signature, enforceargs, etc)
<ronan>
fijal: as I said at first, syntax is a distraction, we need to make things work first
<fijal>
yeah
<fijal>
fair
<fijal>
I'll show you my changes, hopefully later today
<camara>
Why not use a similar syntax as type hints and get rid of the decorator. That way the typing is near the argument names. Maybe even change def to rdef to distingish rpython defs and aid the parser.
adamholmberg has joined #pypy
<fijal>
camara: because we don't preprocess rpython
<antocuni>
we could modify our pypy27 to accept type annotations, and use those :)
<antocuni>
then, we write a module which hijacks the parser to support them also on cpython 2.7
<antocuni>
and we call it "python28", so that people can do "pip install python28" :)
<arigato>
# -*- coding: python-with-hack -*-
<ronan>
antocuni: yuck! I'd rather port rpython to Python 3
<camara>
why would preprossing be necessary. You just change the language.
<antocuni>
arigato: this looks like an excellent idea!
<fijal>
camara: because our language starts from bytecode-level
<fijal>
so once it's imported
<fijal>
arigato: do you want to have opinions?
<arigato>
not really
<arigato>
I am honestly of the opinion that RPython is a bit hopeless for end users
<fijal>
there is a chance of that :)
<fijal>
but I meant more for "more interpreter authors"
yuyichao has quit [Ping timeout: 248 seconds]
<Rotonen>
ronan: the zope/plone stack is actually going for rpython to python3 porting, be welcome to help there
<fijal>
Rotonen: it's a different rpython right?
<arigato>
yes, unrelated
<Rotonen>
fijal: that's where my superficial understanding falls apart
<kenaan_>
antocuni fix-vmprof-stacklet-switch 3e4e9ff62be1 /rpython/rlib/: make sure that vmprof don't sample the stack in the middle of stacklet switching, else it rea...
<kenaan_>
antocuni fix-vmprof-stacklet-switch 52a6650ba479 /pypy/module/_continuation/test/test_translated.py: add a test which is failing (i.e., segfaulting) on default, and that it seems to be fixed on ...
<kenaan_>
antocuni fix-vmprof-stacklet-switch 46ee55287ed4 /: remove the space param from rvmprof.{start,stop}_sampling: it is not used and also it does no...
<kenaan_>
antocuni fix-vmprof-stacklet-switch 2c8aa32187c0 /: close merged branch
<kenaan_>
antocuni default 927cc69f4d52 /: merge the fix-vmprof-stacklet-switch: make sure that vmprof does not segfault in presence of continuation.switch...
adamholmberg has quit [Ping timeout: 255 seconds]
yuyichao has joined #pypy
<camara>
fijal, for a crazy idea. I'm sure a hack could be done to import a module called rpython or rdef which does nothing under pypy and under CPython it preprocesses the module converting type annotations to a decorator. That way the user could use the cleaner type annotation format.
<fijal>
I'm not sure if I want such hacks ;-)
infinite has quit [Ping timeout: 260 seconds]
<camara>
Well the hack would only be under CPython.
<kenaan_>
rlamy py3.5 dfff7758834e /pypy/module/posix/: Merged in thisch/pypy/py3.5 (pull request #584) Change return type of os.times to posix.times_result
<kenaan_>
thisch py3.5 852c26ea2a1c /pypy/module/posix/: Change return type of os.times to posix.times_result The return type was changed in CPython3.3. Related: #2375
<kenaan_>
thisch py3.5 898194d1902c /pypy/module/posix/test/test_posix2.py: Improve test for posix.times() Test that posix.times() returns a times_result object.
infinite has joined #pypy
<antocuni>
arigato: when you feel like, please have a look at my continulet-no-frame-loop branch. It seems to work and the tests pass, but since the logic is subtle, I'd like some opinions/reviews about it
<arigato>
ok
<antocuni>
thanks
<antocuni>
basically, the general idea is to remember also the topframe of the continulet, and to set the bottomframe.f_back to None when it's not running
camara has quit [Remote host closed the connection]
<kenaan_>
rlamy py3.5 12f1fb4860ac /lib-python/3/test/test_inspect.py: Fix test to work on PyPy
<fijal>
that was a long day
<kenaan_>
fijal unicode-utf8 1d6d78e72d50 /: general progress towards moving more of the infrastructure from runicode towards unicodehelper, which helps us...
adamholm_ is now known as adamholmberg
zmt00 has joined #pypy
marky1991 has joined #pypy
tbodt has joined #pypy
<cfbolz>
antocuni: would you unbreak test_whatsnew please?
<antocuni>
cfbolz: sure
ceridwen has quit [Quit: Ex-Chat]
<antocuni>
we should have an hg hook which reminds you
<arigato>
after the merge, it is SomeListlistdef=ListDef2)
mattip has joined #pypy
<arigato>
these don't compare equal at the moment, and making them compare equal is full of issues
<arigato>
e.g. they can be equal for a moment, and no longer equal when one of them is later generalized, etc.
<fijal>
right
<fijal>
so I want to have also a flag saying "don't ever generalize", which would spread
adamholm_ has joined #pypy
<arigato>
I see what you're trying to do in general, but I don't think that rpython/annotation is a good start for that
<fijal>
what is?
<arigato>
I mean, I think I see what you're trying to do, but in order to do this cleanly, we shouldn't start with the code in rpython/annotation/
<fijal>
but write it from scratch?
<arigato>
yes, differently
<fijal>
but it's not gonna give us any direct way to improve the setup in between
<fijal>
which means it won't happen
<arigato>
likely
<fijal>
so what I'm trying to do is to get us somehow started to be able to say add annotation to W_IntObject or W_ListObject and be able to say what types are on attributes
adamholmberg has quit [Ping timeout: 240 seconds]
<fijal>
and if they try to be merged we explode
<fijal>
sure, in the ideal world
<arigato>
ok, that looks fine
<arigato>
and not too hard
<ronan>
fijal: that's basically just a type signature
<fijal>
ronan: no, because we need to fight things like list merges, dict merges and all the other crap
<arigato>
fijal: look at classdef.py where the SomeXxx get assigned and re-assigned to attribute names
<fijal>
arigato: but that does not help, for the reasons above
<fijal>
because we merge stuff
<fijal>
so it's all connected
<ronan>
fijal: we need to fight that to get proper type signatures anyway
<fijal>
ronan: fight what?
<ronan>
list merges, dict merges and all the other crap
<fijal>
yeah
<fijal>
which is why I started there
<fijal>
but instead of actually trying to get anywhere, I'm arguing on IRC
<ronan>
but unions aren't the problem
<fijal>
less so
<fijal>
but they're still a bit of a problem
<arigato>
can't you find a way to set 'too_late_for_change' on listdefs and dictdefs?
<ronan>
the issue is that we can't really say "any list of ints", we can only say "this list here contains ints"
<fijal>
ronan: sorry?
<arigato>
...or 'dont_change_any_more', as it is called
<ronan>
fijal: well, SomeList keeps track of where the list comes from, which is why we get weird merges on unions
<fijal>
is there an equivalent on classes and instances?
<arigato>
I think we should set 'dont_change_any_more' in rlib/types.py, too
<arigato>
there is no equivalent for classes and instances
<arigato>
but you don't need one
* fijal
gives up for today
<arigato>
I think? can you give an example where you need that for instances? merging two instances of subclasses gives SomeInstance(base_class), but without mutating anything
<fijal>
I don't want new attributes
<fijal>
I don't want attributes moving up the chain
<arigato>
right, but that doesn't occur at the merge
<fijal>
it can, no?
<fijal>
it can at least add new stuff to base class
<arigato>
no?
<arigato>
I'm saying that if you merge two
<arigato>
...SomeIntance(), then nothing is mutated for the attributes
<arigato>
if later that SomeInstance() goes into a getattr() or setattr() operation, then yes
<arigato>
but you can detect that
<arigato>
there is "_slots_ = []" already doing this
<arigato>
iow, the model is that SomeXxx are immutable, and a few of them point to a ListDef/DictDef/ClassDef which is itself mutable
<arigato>
these three places already try to have ways to stop mutation from occurring, but not done very systemat9cally
<mattip>
about failing tests - could someone with a macosx try to replicate the failing tests in pypy-c-jit-macosx-x86-64 pypy.module.posix.test.test_posix2
<mattip>
and the failing test_lock own test passes after translation, hg bisect says it started after 167b802baf3b