<mattip>
on line 45 there is self.optimize(block), there should be a graph there too
<mattip>
right
<petronny>
I see. I'll try this when it show up again.
darkman66 has joined #pypy
oberstet has joined #pypy
arigato has quit [Read error: Connection reset by peer]
bbot2 has quit [Read error: Connection reset by peer]
bbot2 has joined #pypy
forgottenone has joined #pypy
antocuni has joined #pypy
forgottenone has quit [Ping timeout: 255 seconds]
forgottenone has joined #pypy
Mutter has joined #pypy
Mutter has quit [Ping timeout: 250 seconds]
leshaste has joined #pypy
jacob22 has joined #pypy
jacob22_ has quit [Read error: Connection reset by peer]
arigo has joined #pypy
<arigo>
ugh
arigo is now known as arigato
lritter has joined #pypy
<arigato>
mattip: the invalid string detection in W_UnicodeObject.__init__ is still bogus
<arigato>
if the utf8str is invalid nonsense, it will raise an exception, which will be ignored
<arigato>
fwiw, if utf8str is invalid nonsense, various other functions will segfault, which I think is what occurs on issue #2979
<kenaan>
guil...@Guillaumes-MacBook-Pro.local cffi/cffi c0fbaf8dc825 /: fix #407 add support for u/U suffix in integer constants (eg. 0xABu, or 0xCDU).
<kenaan>
guil...@Guillaumes-MacBook-Pro.local cffi/cffi d1bf39c55881 /: add support for long/long long C integer constant suffixes, and support for base 2...
<kenaan>
guil...@Guillaumes-MacBook-Pro.local cffi/cffi fa1ef05714de /cffi/cparser.py: remove extra comments.
<mattip>
arigato, so we should always call rutf8.check_utf8 ?
<arigato>
mattip: can't we just do assert length == len(utf8str.decode('utf8'))?
<arigato>
ah no, not on platforms with a cpython with 16-bit unicodes
antocuni has quit [Ping timeout: 250 seconds]
jacob22 has quit [Ping timeout: 240 seconds]
<mattip>
they should fail to be converted to utf8 before calling W_UnicodeObject()
Zaab1t has joined #pypy
<arigato>
well, yes, but my current guess about issue #2979 is that we have one place that still calls W_UnicodeObject with something invalid
<kenaan>
arigo default a59c3b47eec9 /: Unicode characters out of range(0x11000): fix a few docstrings, and try to more systematically test (and fix) vario...
<arigato>
although it seems the crash is caused by a unicode object that pretends to be of length 0
<mattip>
it crashes using pyinteractive
<arigato>
so the only way would be ._length == 0 but ._utf8 != ''
<oberstet>
should that be of interest, we recently compared a bunch of serializers (JSON, CBOR, etc for use with Autobahn and Crossbar.io) on CPy and PyPy, including vmprof cpu profiles, pls find results here https://crossbario.com/docs/benchmarks/serialization/index.html
<mattip>
in pyinteractive, somehow we are calling objspace.std.unicode_from_object(w_obj) where w_obj is
<mattip>
<pypy.module._weakref.interp__weakref.WeakrefLifeline object at 0x7f2c5b909950>
<mattip>
which fails space.type(w_obj)
<oberstet>
summary ^: pypy is awesome (well, we knew that before;) and is able to serialize WAMP messages on a single core with up to: 960k msgs/sec or 5.3GB/sec. that's quite something
<kenaan>
arigo default fa16b2515a57 /pypy/module/array/: Be more informative, just because we can
<arigato>
mattip: OK, that's likely something different
<arigato>
space.type() may fail on unexpected internal objects, but well, I guess it shouldn't crash
<arigato>
it crashes because WeakrefLifeline has got 'typedef = None'
<mattip>
+1
Zaab1t has quit [Ping timeout: 250 seconds]
<arigato>
normally we shouldn't see these objects around, but we can if we fish them in non-conventional ways, like gc.get_objects()
<oberstet>
rgd vmprof: fwiw, crossbar has that built-in now in an easy to use way to profile in production: "crossbar start --vmprof" will run all crossbar workers under vmprof and produce 1 profile per worker
<arigato>
oberstet: cool
Zaab1t has joined #pypy
<mattip>
oberstet: seems like there is still room for improvement on larger payloads
forgottenone has quit [Quit: Konversation terminated!]
<oberstet>
mattip: latest cpy3 and latest pypy3
<oberstet>
yes, cpython/cbor is a bit faster than pypy on large payloads
<arigato>
mattip: I guess we should remove the 'typedef = None' (it seems only used in WeakrefLifeline), and replace it with something real, maybe put directly in W_Root, that would give it a valid app-level type (one that says 'internal' or something)
<oberstet>
rgd the full WAMP RPC test, to sum up: pypy is 4-6x faster than cpy. this benchmark is "real world" load (not synthetic benchmark like the serializer stuff)
<arigato>
mattip: ah, also remove "typedef = None" from setobject.W_BaseSetObject
<arigato>
mattip: the crash of issue #2979 is caused by an expected null pointer, actually. tracking...
<arigato>
OK found the cause, it's quite indirect
<arigato>
it's also a general problem
<arigato>
we have dicts whose keys are known to be W_UnicodeObject
<arigato>
or None
<arigato>
so our rpython logic creates a dummy value of type W_UnicodeObject to represent "deleted key"
<arigato>
it's a dummy instance of W_UnicodeObject with all fields zero
<arigato>
normally it's never seen, but I guess with gc.get_objects() you'll see it
<mattip>
should we protect access to _utf8 by checking _length first?
<arigato>
of course the actual crash is that the code in our 'unicode.__str__' method assumes the W_UnicodeObject is valid, but instead it gets one with "._utf8" being none
<arigato>
it's completely at a different level
<arigato>
if we had dictionaries whose keys are W_ListObject, we'd get the same crash because something must never be None inside W_ListObject
<arigato>
(OK bad example, but W_TupleObject, who must have a ".items" that is an rpython list, not None)
<mattip>
:)
<mattip>
so any hashable W_Root could have a classmethod to return a default object
<arigato>
it's an RPython issue, so it's not only about W_Root
<arigato>
I think it cannot show up for non-W_Roots because in pypy the gc.get_objects() function lists all RPython instances, but then only keeps the ones that are W_Root
<mattip>
try_cast_gcref_to_w_root
dddddd has joined #pypy
Zaab1t has quit [Ping timeout: 272 seconds]
<arigato>
right, not sure why you're still seeing a WeakrefLifeline then
<arigato>
mattip: I have some minimal hacks in mind, I'm not sure they are a good idea though. That would involve writing the GCFLAG_EXTRA as set in these few special prebuilt objects
<mattip>
arigato: something somewhere has to have additional knowlege/state
<arigato>
if we set GCFLAG_EXTRA on these few prebuilt objects created in rmodel.py, then nothing change, except that the gc.get_objects() algorithms in rlib/rgc.py will skip them
<mattip>
ok
<mattip>
but how do you know which ones need GCFLAG_EXTRA?
<arigato>
the ones made in rmodel.py:454
<arigato>
we'd just add a flag to lltype.malloc()
<arigato>
and propagate this flag all the way to the gc
<mattip>
heh
<mattip>
would that be equivalent to returning None from get_ll_dummyval_obj?
<arigato>
well, we can kill the whole dummyval thing and fix the problem, too, but the cost is a larger dictionary table for many of them
<arigato>
what I'm suggesting would not be equivalent to that, no
<mattip>
this used to work in 7.0, even with a dummyval for W_TupleObject
<arigato>
yes, because we don't actually have the dictionary type {W_TupleObject: xxx}, I guess
<arigato>
only {W_Root: xxx}
<arigato>
i.e. it worked so far by chance
<arigato>
(we always had {W_UnicodeObject: xxx}, because that's used for the app-level "dictionary-with-unicode-keys" optimization
<arigato>
(either I'm wrong and it was only added by the unicode-utf8 branch, or else, evne if we had this dictionary type before, then before unicode-utf8 it was not causing crashes)
adamholmberg has joined #pypy
<mattip>
maybe we could copy the behaviour that avoided the crash (before the branch) to avoid any deep surgery in rpython
<mattip>
unless you feel it is something that needs fixing anyway
<arigato>
yes, it is
<arigato>
it's probably easy to work around it, using the is_valid() solution you proposed
<arigato>
and implementing is_valid() only on W_UnicodeObject is probably fine
<arigato>
until next time
<mattip>
whatever is esaiest. It seems propagating the flag all the way to the GC would also make the dictionary table larger, but probably less-so than removing the dummyvals
<arigato>
? no
<arigato>
the end result is that a few constants in the C sources have an extra flag in their gc header
<mattip>
ahh, right. singletons
<arigato>
ah, I found another solution that doesn't involve the GC that much
<arigato>
we can store all the dummy objects together (using the rarely-used but existing lltypesystem/llgroup.py), and then the check in rlib/rgc.py should be "if the address of the object is exactly between these bounds, then skip it"
antocuni has joined #pypy
<arigato>
ah! there is already special code in rgc.try_cast_gcref_to_instance
<arigato>
for the rdict's dummyobjs
<mattip>
:)
<arigato>
this test only works on 32-bit I think
<arigato>
can you easily check if pypy 7.1.0 on linux32 has got the same problem or not?
marky1991 has joined #pypy
<mattip>
yes
<mattip>
no crash on the release Python 3.6.1 (de061d87e39c, Mar 24 2019, 22:18:06) PyPy 7.1.0-beta0 with GCC 5.3.1 20160413 on linux
antocuni has quit [Remote host closed the connection]
<mattip>
when running [str(x) for x in gc.get_objects()]
antocuni has joined #pypy
<arigato>
OK
Zaab1t has quit [Ping timeout: 255 seconds]
jcea has joined #pypy
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 255 seconds]
adamholmberg has joined #pypy
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
<kenaan>
cfbolz default c46e92cbc2f9 /pypy/interpreter/astcompiler/: issue2980: stop using arbitrarily much stack for building constant lists and sets. do it element by element instea...
<kenaan>
cfbolz default 9f6768bbafd6 /pypy/module/array/interp_array.py: don't use space.str_w
adamholmberg has quit [Read error: Connection reset by peer]
<tos9>
This is (as usual) macOS, and the relevant line seems like it's `ld: library not found for -lgcc_s.10.4` -- but that doesn't seem to happen with CPython (even when compiling from source via --no-binary)
<tos9>
Any hints?
<mattip>
rings a bell, doesn't google says anything about clang library not found for -lgcc_s.10.4 ?
<tos9>
nothing so far that looked relevant
<tos9>
basically some posts that told me to install gcc4, which I already have installed
<antocuni>
it might be that pypy's distutils is broken
<antocuni>
I think I already had an issue on linux which was very vaguely related to this, let me search the issue tracker
<tos9>
antocuni: I mean numpy is also being super annoying
<tos9>
like, give me whatever "simple C program" you claim can't be compiled
<tos9>
so I can run that command and see for myself
forgottenone has quit [Quit: Konversation terminated!]
<tos9>
Yeah there's also a line in the original log that I noticed said UserWarning: Env. variable MACOSX_DEPLOYMENT_TARGET set to 10.3
<tos9>
Which looked suspicious
<tos9>
Running MACOSX_DEPLOYMENT_TARGET=10.14 does indeed give me... different errors
<mattip>
no idea. Just relaying information I see in random issues that were closed
<tos9>
mattip: yeah thanks! def helpful
<mattip>
can you try MACOSX_DEPLOYMENT_TARGET=10.6 ?
<tos9>
(something is indeed happening now, though still no idea what would be different from cpython)
<tos9>
mattip: the install with MACOSX_DEPLOYMENT_TARGET=10.14 is still running!
<tos9>
so waiting to see if it errros
<mattip>
ahh, then go for it
<tos9>
Yeah that ... at least installed when running pip install now
<mattip>
does a build on cpython use different flags that come from some sysconfig magic?
<mattip>
you can add -v to see the actual build output
<tos9>
trying
<tos9>
mattip: what's a good thing to run on the installed numpy to see how likely it is to be fully working
<mattip>
numpy.test()
<tos9>
ok yeah that failed
<mattip>
that was quick
<tos9>
hah well -- I suspected it would, because when I ran `python setup.py build_clib` in the repo that showed me obvious errors from clang
<tos9>
but when I did `pip install numpy`, which must have done the same thing, no output was shown
<tos9>
so something silently failed, and chose not to complain
<tos9>
which yeah ugh
<mattip>
what happenned with numpy.test() ?
<tos9>
Original error was: dlopen(/Users/jberman/Desktop/venv/lib/python2.7/site-packages/numpy/core/_multiarray_umath.so, 2): Symbol not found: _aheapsort_bool
<tos9>
(complaints about multiarray not working)
iomari has joined #pypy
iomari has quit [Client Quit]
<mattip>
so it is linking with a library but loading a different one
<mattip>
maybe
<mattip>
would be good to see the cpython flags
<mattip>
something like distutils.sysconfig.get_config_vars('CC', 'CXX', 'OPT', 'BASECFLAGS', 'CCSHARED', 'LDSHARED', 'SO')
<mattip>
on both cpython3 and pypy3
<tos9>
mattip: yeah those are definitely very different
<tos9>
mattip: good question, it's at least possible... IIRC the homebrew formula will tell me, /me checks
<mattip>
sys.version should also tell you
<tos9>
[PyPy 7.1.0 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)]
<tos9>
(Thanks)
<tos9>
so, clang then
<tos9>
Also FWIW I get working numpy from source even with homebrew CPython
<tos9>
Didn't actually get a second to step through the numpy.distutils stuff... will try to do that now
<mattip>
the sysconfig CC is wrong then, it should be clang not cc
<mattip>
somehow that should be fixed to reflect what was really used, maybe by the same machinery that sys.version uses
<tos9>
mattip: /me probably has lost you -- wasn't anto saying that PyPy does something intentionally different there and doesn't just use the compiler that was used when the binary was originally compiled?
<tos9>
are you saying basically to change that or did I miss the point?
<mattip>
pypy hardcodes values in lib_pypy/_sysconfigdata.py, without regard to what is used to build pypy
<mattip>
what happens if you edit that file to match cpython?
<mattip>
our buildslave is a favor from the cpython buildslave guy, it is suprising the build even succeeds
adamholmberg has quit [Remote host closed the connection]
<mattip>
tos9: still the same problem: ld: library not found for -lgcc_s.10.4
<mattip>
where is that coming from?
<tos9>
mattip: yeah -- but I'm not lying right? It's definitely now using clang
<mattip>
ahh, still seeing UserWarning: Env. variable MACOSX_DEPLOYMENT_TARGET set to 10.3
<mattip>
so that would be a good place to stop in a debugger and figure out why
<tos9>
yeah I didn't set that envvar in this shell
<tos9>
should I?
<mattip>
couldn't hurt, nothing works anyway
<tos9>
:)
<mattip>
the code in numpy/distutils/fcompiler/gnu.py explicitly sets MACOSX_DEPLOYMENT_TARGET if not cpython
<mattip>
so that is bad
<mattip>
(unless it is set in the environment)
<tos9>
so by the way I am happy to find out if I can offer donating an osx buildslave in exchange for numpy distributing osx wheels for pypy if that's worth exploring later
<tos9>
might be able to
<tos9>
though I only (and specifically) care about py2 I guess which I know numpy dropped support for but that probably I can figure out
<mattip>
might be easier to just build-and-upload to antocuni's repo
<tos9>
oh right, I always forget about that
<tos9>
yeah ok, lemme get this working first
kipras has joined #pypy
<mattip>
there is another stage to make good wheels - you need to use OpenBLAS not Accelerate
<tos9>
I have openblas installed
<tos9>
Oh, apparently it's keg only
<tos9>
Well, that's easy enough to fix
oberstet has quit [Quit: Leaving]
<mattip>
I had a random thought about using the packages.pypy.org script to not only *check* the packages,
<tos9>
mattip: yeah still failing with the multiarray nonsense :/
<tos9>
This is with MACOS_DEPLOYMENT_TARGET=10.6 and sysconfig_pypy with values from cpython
<tos9>
mattip: my C knowledge sucks but I think the equivalent is `otool -L`?
<tos9>
ldd tells you what stuff a binary wants to dynamically link?
<tos9>
(If so yeah that's otool -L)
<mattip>
another idea
<tos9>
Aright I give up for now :/ Lunch time
<tos9>
mattip: (Really appreciated of course, will prob come back later)
<mattip>
what does cpython -c"import distutils.sysconfig as sc; f = sc.get_makefile_name(); g = {}; sc.parse_makefile(filename, g); print(g.get('MACOSX_DEPLOYMENT_TARGET', '10.3'))"
<mattip>
print?
<arigato>
may I randomly complain that the Python 3.6 syntax:
<arigato>
>>> (a, *b) = (2,3,4)
<arigato>
>>> b
<arigato>
[3, 4]
<mattip>
tos9: that is the value numpy uses
<arigato>
obscurely behaves differently than a function call, by putting a list (not a tuple) in b?
<tos9>
mattip: nameerror for get_makefile_name
<mattip>
grr. get_makefile_filename
<tos9>
mattip: prints 10.14
<mattip>
cpython -c"import distutils.sysconfig as sc; f = sc.get_makefile_filename(); g = {}; sc.parse_makefile(f, g); print(g.get('MACOSX_DEPLOYMENT_TARGET', '10.3'))"
<mattip>
right, so that is what you want to use
<mattip>
I guess we could hard-code that :)
* mattip
off, bbl
<mattip>
arigato: your complaint was registered and processes. thank you for your call.
<arigato>
:-)
<cfbolz>
arigato: yes, some of that is quite baroque too
<cfbolz>
arigato: I had no clue you could build a list like [a, b, *c, *d, f]
<arigato>
it's even done using strange bytecodes
<arigato>
basically there is a bytecode for "concatenate these N lists", and then your example becomes "CONCATENATE([a, b], c, d, [f])"
<arigato>
or maybe "CONCATENATE((a, b), c, d, (f,))"
<cfbolz>
The latter
<arigato>
CONCATENATE is called BUILD_LIST_UNPACK or BUILD_TUPLE_UNPACK depending on whether you want to build a final list or tuple
<cfbolz>
So you can do that for dicts too? {**a, **b}
<arigato>
yes, there is BUILD_MAP_UNPACK
<cfbolz>
Advanced
<Alex_Gaynor>
why not `BUILD_DICT_UNPACK` (not that it matters)
<arigato>
even two variants of BUILD_MAP_UNPACK, one which detects duplicates and complains, the other which doesn't
<arigato>
Alex_Gaynor: I think it can be traced back to the BUILD_MAP opcode that existed since forever to make an empty dictionary
<Alex_Gaynor>
I wonder what the origin of that name is, was `dict` called `map` once upon a time?
<cfbolz>
arigato: so, do you have an opinion what bytecode to use in PyPy2 to make a huge tuple?
<arigato>
"tp_as_mapping" and PyMapping_Keys() & co.
<cfbolz>
LOAD_GLOBAL 'tuple' + Call?
darkman66 has joined #pypy
<arigato>
cfbolz: no, as I said in the issue, you can't use LOAD_GLOBAL
<cfbolz>
Ah, sorry, didn't see your reply yet
<cfbolz>
Reading
<cfbolz>
Yes, build_tuple_unpack is what I need, but it's not there in Python2
<arigato>
so one reasonable solution is to add it in python2, maybe only with argument == 1
<cfbolz>
Sounds like a plan
<cfbolz>
Then it's at least a bytecode with a known and googlable semantics
<arigato>
one complete hack would be to MAKE_FUNCTION; CALL_FUNCTION_VAR, with a code object that corresponds to a sub-function that would look like "lambda *x: x"
darkman66 has quit [Ping timeout: 268 seconds]
<cfbolz>
arigato: hah!
<arigato>
just saying it *is* possible :-)
<cfbolz>
I like that, in a weird way
<arigato>
or:
<arigato>
make bytecode equivalent to ().__class__(list)
<cfbolz>
Will try that
<cfbolz>
(this is a super bizarre case anyway, of course)
darkman66 has joined #pypy
<cfbolz>
BTW, I also noticed a case where CPython's bytecode is better than ours: compare (1,3,3,*a,5,6,5)
<arigato>
yes, probably not worth a bytecode, from our point of view (but don't tell that to CPython)
<cfbolz>
Yes
<arigato>
ah, it's because we have optimize.py that works at the slightly higher level of ast trees
<arigato>
so it's not able to turn any sequence of LOAD_CONST(*3)+BUILD_TUPLE into a LOAD_CONST(tuple)
<cfbolz>
arigato: yes, but it's not hard to support on the ast-tree level either
<arigato>
yes
<cfbolz>
I might do that, while I am thinking about bytecode ;-)
<arigato>
:-)
darkman66 has quit [Ping timeout: 255 seconds]
demonimin has joined #pypy
Zaabtop has joined #pypy
Zaab1t has quit [Ping timeout: 245 seconds]
Zaabtop is now known as Zaab1t
<cfbolz>
arigato: I refuse to fix the case where people write (*a, 1, 2, 3, ..., 1000)
<cfbolz>
or worse, (*a1, *a2, *a3, ..., *a1000) ;-)
<kenaan>
cfbolz py3.6 5521ea7a9241 /: merge default (including a re-implementation of c46e92cbc2f9 for Python3)
darkman66 has quit [Remote host closed the connection]
darkman66 has joined #pypy
darkman66 has quit [Ping timeout: 250 seconds]
darkman66 has joined #pypy
darkman66 has quit [Remote host closed the connection]
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
marky1991 has quit [Read error: Connection reset by peer]
Rhy0lite has quit [Quit: Leaving]
brent has quit [Ping timeout: 268 seconds]
marky1991 has joined #pypy
darkman66 has joined #pypy
darkman66 has quit [Remote host closed the connection]
darkman66 has joined #pypy
<kenaan>
cfbolz py3.6 e41b9e56b6cc /pypy/interpreter/astcompiler/: do the same optimization that CPython does for (1, 2, 3, *a) (but on the AST level)
darkman66 has quit [Remote host closed the connection]
speeder39_ has joined #pypy
darkman66 has joined #pypy
darkman66 has quit [Remote host closed the connection]
darkman66 has joined #pypy
marky1991 has quit [Read error: Connection reset by peer]
darkman66 has quit [Remote host closed the connection]
<kenaan>
cfbolz default f6a9c3de9c7d /pypy/interpreter/astcompiler/: (cfbolz, arigo giving the idea) to construct gigantic tuples in constant stack space, first build a list, then use...
<kenaan>
cfbolz py3.6 1790f4267a4b /pypy/interpreter/astcompiler/: merge default (no changes, the 3.6 version already worked)