cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
<N8Dawg>
Hi new to channel, I'd like to experiment with some ideas on Pypy, which means hacking the interpreter,problem is the 2hr build time for Pypy will make this infeasible
<mattip>
arigato: upstream cffi doesn't vendor pycparser, or did you mean lib_pypy/cffi ?
N8Dawg has joined #pypy
<N8Dawg>
.
<arigato>
...you're right
<N8Dawg>
Hi having issues with web irc client, not sure what people use, connection get timed out, anyway to view last 30 minutes history and see any messages I may have missed?
<mattip>
look at the group topic :)
<fijal>
I use irc cloud
Ai9zO5AP has joined #pypy
Ai9zO5AP has quit [Client Quit]
<N8Dawg>
mattip thanks!
ajlawrence has quit [Remote host closed the connection]
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
<kenaan>
mattip default bcb0c566c840 /extra_tests/cffi_tests/: test that pycparser is latest version
<kenaan>
mattip default af4a8595c924 /lib_pypy/cffi/_pycparser/: update pycparser to latest version
tsaka__ has quit [Ping timeout: 258 seconds]
<cfbolz>
antocuni: I replied to your comment about the dictionary order
<antocuni>
where?
<cfbolz>
just as a commit ;-)
<cfbolz>
basically dict order is always a bit wonky
<cfbolz>
for jsondicts it's just reverse the one in the file
<cfbolz>
do we care?
<cfbolz>
or just about the strange half-half
<antocuni>
ah true, I didn't realize that they are always out of order
<cfbolz>
fully reverse is at least pretty consistent
<antocuni>
I am really unsure whether we are or not
<antocuni>
*we care
<antocuni>
I mean, in theory you don't care
<antocuni>
but then it's a very visible difference to cpython
<arigato>
what occurs if you load and re-save a json file?
<cfbolz>
arigato: on saving it's sorted
<arigato>
ah ok
<cfbolz>
... i think
<cfbolz>
let me check
<cfbolz>
no
<arigato>
sort_keys=False by default
<kenaan>
cfbolz default 648820f106b9 /pypy/module/pypyjit/test_pypy_c/test_string.py: should fix the encoding errors
<cfbolz>
arigato: right
<antocuni>
so the solution it's easy: we just load/save/load and we have the keys in the correct order :)
<cfbolz>
arigato: but I don't think there's a guarantee for roundtripping that keeps the strings identical
<cfbolz>
anyway, if there's consensus that I should care, I can probably try harder
<arigato>
I would say yes, it's strange if loading and saving a json file reverses all keys
<antocuni>
especially now that on CPython the order is guaranteed
<cfbolz>
ok. that means we merge after the release, though. will take me a bit to fix the order
<antocuni>
ideally, it would be nice to have an option to disable the sorting in case it costs performance. But then it is hard to do it in a way which is compatible with CPython
<cfbolz>
antocuni: I think it can be made to not cost very much
<antocuni>
cfbolz: ah true, you can make it a double-linked list and traverse from start
<cfbolz>
antocuni: no, I keep arrays of keys anyway, I can traverse those
<cfbolz>
it's all not too bad, I think
<cfbolz>
arigato: if at any point you could take a look at just the file simd.py on the json-decoder-maps branch, that would be useful
<arigato>
OK
<antocuni>
cfbolz: anyway, the code in general looks very good. I liked many of the tricks you did which I'd never have thought about :)
<cfbolz>
antocuni: I had three hours for days a week for five weeks in a café, without internet
<cfbolz>
that helped :-)
<antocuni>
:)
tsaka__ has joined #pypy
<mattip>
cfbolz: could you merge default into py.36? interpreter/pyparser/error.py changed and does not pick up the wrap_info change properly
<cfbolz>
mattip: ah, sorry, yes
xcm has quit [Remote host closed the connection]
<mattip>
ronan: I wonder why the extra_tests of cffi did not pick up the DeprecationWarning in the nightly run of c61c6e7c5d66 (which included your change)
xcm has joined #pypy
<mattip>
in any case, we should be able to reinstate abe60bf6dc7b after updating pycparser
<arigato>
cfbolz: I'm not getting the "if shift == WORD_SIZE * WORD_SIZE" line: why is the limit 64 on 64-bit but only 16 on 32-bit platforms?
<cfbolz>
looking...
<cfbolz>
arigato: note that so far this code is only enabled on 64 bit :-(
<arigato>
no, it's in find_end_of_string_slow()
<cfbolz>
ah, hum
<arigato>
which I'm trying to understand before looking at the rest
<cfbolz>
yes, that sounds wrong indeed
<cfbolz>
should be just * 8?
<arigato>
if we put the limit at 32, then the "word |= ord(ch) << shift" above will be called once with shift==28
<arigato>
discarding 4 bits of ord(ch) for every 8th ch...
antocuni has quit [Ping timeout: 264 seconds]
<cfbolz>
arigato: yes, that sounds wrong too :-(
<arigato>
the point of strhash is to return some arbitrary fast-to-compute hash value, right?
<cfbolz>
arigato: not completely arbitrary, it should be one that can be computed based on full words
<cfbolz>
but the slow and the fast version need to be compatible
<arigato>
OK
<cfbolz>
arigato: and I think that means that the problem is shift += WORD_SIZE, which should be just shift += 8
<arigato>
ah
<cfbolz>
because we really want to do this byte by byte, it's not part of the hashing
<arigato>
OK I see
<cfbolz>
arigato: I apologize for the confusion, I think what happened is that I confused '8' with WORD_SIZE, when the two 8s were just 'number of bits in byte'
<arigato>
sure
<arigato>
any_char_in_words_zero() is slightly scary because it returns more than a yes/no answer
<cfbolz>
arigato: you mean the name? or the concept in general
<arigato>
the implementation (as far as I can tell)
<cfbolz>
yes, that's in some way the hairiest part
<arigato>
it looks where the first null char is, in little-endian order, and return the byte 0x80 in the same position, with lower positions 0x00 and higher position undefined
<arigato>
is that right?
<cfbolz>
yes
<cfbolz>
hm, I had a blog post that explained the bit-twiddling tricks used here, I think I should put it in a comment
<cfbolz>
this is of course something much simpler with real sse instructions
<arigato>
yes I remember reading or hearing about it
<arigato>
I'm making the changes that we're discussing
<arigato>
if that's fine with you
<kenaan>
cfbolz json-decoder-maps 42f36c45bad2 /pypy/module/_pypyjson/: remove usage of get_nonmovingbuffer_final_null
<cfbolz>
arigato: sorry, totally, I just noticed that the tests were broken
<cfbolz>
because of get_nonmovingbuffer_final_null
<cfbolz>
fixed now though
<arigato>
OK
<arigato>
also, splice_words()?
<arigato>
what is it supposed to do and why is it always called with word==r_uint(0)?
inhahe has quit [Ping timeout: 268 seconds]
<cfbolz>
arigato: ah, I see. I was using it in a more general implementation, that I later killed. I suppose it should be simplified?
<cfbolz>
and given a better name
<arigato>
(maybe I'm just confused because it is called with an argument "word" that doesn't get into the parameter called "word")
<arigato>
OK
<cfbolz>
something like "set_bits_to_zero" or so?
<arigato>
"mask_off_high_bytes"?
<arigato>
"set_high_bytes_to_zero"?
<cfbolz>
yes, sounds great
<cfbolz>
(I think in general I should do a round of commenting on that file, the decoder proper is commented much better)
xcm has quit [Remote host closed the connection]
<arigato>
hum something seems off in find_end_of_string_slow() with 'i'
<cfbolz>
how so?
<arigato>
why do you initialize strhash with ll_chars[i], but then the same ll_chars[i] will also be part of the main loop's strhash computation, no?
xcm has joined #pypy
<cfbolz>
arigato: that's the same as objectmodel._hash_string
<arigato>
uh
<cfbolz>
(ie the same as cpython)
<arigato>
fine
<cfbolz>
arigato: it's possible that I should indeed use a more well understood hash. Don't think CPython's is that obviously well thought out. Otoh, collisions don't matter much
<arigato>
find_end_of_string_slow() ignores 'length', so does it assumes that the string is well-formed, or do you rely on ll_chars[length] == \x00?
<cfbolz>
arigato: the latter
<cfbolz>
I use the null terminating variant
<arigato>
OK, then find_end_of_string_simd_unaligned() could also ignore 'length' in its "remaining chars" loop
<cfbolz>
Yes good point
dddddd has joined #pypy
<kenaan>
arigo json-decoder-maps a6922067f4ef /pypy/module/_pypyjson/: Review, minor tweaks
<arigato>
cfbolz: done
<cfbolz>
arigato: thanks a lot!
<cfbolz>
General verdict?
N8Dawg has quit [Remote host closed the connection]
<arigato>
cool trick
<cfbolz>
:-)
<cfbolz>
arigato: we could do the same things for string search, particularly char search
<mattip>
would a docstring like "Display the source snippet in the error message" be appropriate?
<cfbolz>
mattip: I suppose we should even rename the method, it now does a lot more than "wrap the info"
<cfbolz>
(with and without my change)
<cfbolz>
find_sourceline_and_wrap_info ?
<cfbolz>
it's not so much about displaying, more finding the line and then storing it into the wrapped tuple that is used to produce a wrapped syntax error
BPL has joined #pypy
<cfbolz>
antocuni: it seems I forgot that I implemented it that way, the jsondicts (that do use the maps) are already using the right order
<mattip>
ahh. So no wonder I was confused, it is complicated. I think the name is fine, but a docstring would help
<cfbolz>
mattip: it has one or two callers, I think fixing would be better
<cfbolz>
antocuni: so only the dicts that switch from map to regular are wrong, those can be fixed
<antocuni>
cfbolz: cool 👍
<cfbolz>
mattip: thanks for pushing me to make this clearer, running tests, then pushing
circ-user-KIvj2 has quit [Read error: Connection reset by peer]
circ-user-KIvj2 has joined #pypy
<kenaan>
cfbolz json-decoder-maps 22fe436a1585 /pypy/objspace/std/test/test_jsondict.py: a test that devolving doesn't change the order
<kenaan>
cfbolz json-decoder-maps 06effcd1097c /pypy/module/_pypyjson/: fix remaining anto comment: when hitting a blocked map, insert keys in the correct order into the real d...
<cfbolz>
mattip: I addressed all review comments, should I merge?
<mattip>
+1
<mattip>
but let's wait to merge to py3.6 till it passes a nightly
<mattip>
I haven't started the 3.6 release yet
<cfbolz>
yes, will do it tomorrow. I need to look into the 3.6 variant anyway
mental32 has quit [Remote host closed the connection]
fryguybob has quit [Ping timeout: 240 seconds]
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
mental32 has joined #pypy
mental32 has left #pypy [#pypy]
mental32 has joined #pypy
antocuni has quit [Ping timeout: 240 seconds]
<kenaan>
arigo cffi/cffi[release-1.13] e295aee5c3ab /cffi/backend_ctypes.py: Don't return a bool from __int__(), as it raises a (Pending?)DeprecationWarning
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
fryguybob has joined #pypy
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
jcea has quit [Remote host closed the connection]
jcea has joined #pypy
marky1991 has quit [Ping timeout: 240 seconds]
ceridwen has joined #pypy
fryguybob has quit [Read error: Connection reset by peer]
fryguybob has joined #pypy
glyph has quit [Ping timeout: 245 seconds]
glyph has joined #pypy
Rhy0lite has quit [Quit: Leaving]
Dejan has quit [Quit: Leaving]
inhahe has joined #pypy
inhahe_ has quit [Ping timeout: 246 seconds]
astronavt has quit [Quit: ...]
astronavt has joined #pypy
marky1991 has joined #pypy
Ai9zO5AP has joined #pypy
xcm has quit [Read error: Connection reset by peer]
marky1991 has quit [Read error: Connection reset by peer]