bendlas has quit [Remote host closed the connection]
agates has quit [Remote host closed the connection]
kurtbrose[m] has quit [Remote host closed the connection]
<kenaan>
antocuni faster-rstruct-2 6878ac6c89b8 /pypy/module/pypyjit/test_pypy_c/test_buffers.py: this this test_pypy_c test: now struct.unpack generates a fast raw_load_i; use 'i' instead of '<i' to m...
agates has joined #pypy
<antocuni>
plan_rich_: ping
jcea has joined #pypy
oberstet2 has joined #pypy
<kenaan>
antocuni faster-rstruct-2 18131865f52d /rpython/rlib/test/test_buffer.py: make this test endianess-independent: hopefully this fixed the failure on S390X
<antocuni>
plan_rich_: never mind, I wanted to ask whether s390x is big-endian but google told me so :)
<antocuni>
ouch, yesterday I started win32 buildbots for my branch but they have not started yet because the buildslave is building running the default ones
<antocuni>
own tests take ~8hr, pypy-c-jit ~7.5hr
<antocuni>
so, I suppose I'll see the results only tomorrow :(
jamescampbell has quit [Ping timeout: 255 seconds]
jamescampbell has joined #pypy
inad922 has joined #pypy
jamesca__ has quit [Ping timeout: 255 seconds]
jamescam_ has quit [Ping timeout: 272 seconds]
inad922 has quit [Ping timeout: 260 seconds]
jamescam_ has joined #pypy
nimaje1 has joined #pypy
nimaje is now known as Guest82920
Guest82920 has quit [Killed (card.freenode.net (Nickname regained by services))]
nimaje1 is now known as nimaje
jamescampbell has quit [Ping timeout: 245 seconds]
palecsan has joined #pypy
jamescampbell has joined #pypy
<palecsan>
hello
jamescam_ has quit [Ping timeout: 260 seconds]
amaury has quit [Ping timeout: 240 seconds]
<arigato>
hi
<palecsan>
arigato: I've tested your idea in multiprocessing. I've put a delay *100 in thread_pthread.c:mutex2_lock_timeout(), but the issue is the same, it does not make my usecase 100 times slower
<kenaan>
antocuni default a8f0d06d39a8 /pypy/module/pypyjit/test_pypy_c/test_ffi.py: kill this test: originally it was meant to check that we take the fastpath when doing a ctypes call, but now the...
jamescampbell has joined #pypy
girish946 has joined #pypy
jamescam_ has joined #pypy
adeohluwa has quit [Quit: Connection closed for inactivity]
jamesca__ has joined #pypy
jamesca__ has quit [Read error: Connection reset by peer]
jamescampbell has quit [Ping timeout: 246 seconds]
arigato has quit [Read error: Connection reset by peer]
jamescam_ has quit [Read error: Connection reset by peer]
jamescampbell has joined #pypy
Tiberium has quit [Remote host closed the connection]
jamescam_ has joined #pypy
mattip has quit [Ping timeout: 240 seconds]
mattip has joined #pypy
jamescampbell has quit [Ping timeout: 255 seconds]
Tiberium has joined #pypy
jamescampbell has joined #pypy
Tiberium has quit [Remote host closed the connection]
jamescam_ has quit [Ping timeout: 240 seconds]
Tiberium has joined #pypy
agates has quit [Remote host closed the connection]
bendlas has quit [Remote host closed the connection]
kurtbrose[m] has quit [Read error: Connection reset by peer]
glyph has quit [Ping timeout: 240 seconds]
pepesza has quit [Ping timeout: 260 seconds]
pepesza has joined #pypy
glyph has joined #pypy
agates has joined #pypy
oberstet2 is now known as oberstet
<cfbolz>
antocuni: I'm reading the diff, but will take me some time
<antocuni>
cfbolz: sure, it's a big one
<antocuni>
thanks
<cfbolz>
(and I am already looking forward to the issues I can close if this gets merged ;-) )
<antocuni>
I think that the changes in rstruct/struct should be robust and very well tested (I wrote additional tests for pack and ran buildbots on 64/32 bits and little/big endian machines)
<cfbolz>
antocuni: we could add some hypothesis tests, at least for roundtripping
<antocuni>
but for the other changes I highlighted in the PR message, review is very welcome :)
<antocuni>
uhm, I never used hypothesis
<antocuni>
but it sounds like a good idea
<antocuni>
and, what are the issues that you can close after this merge?
jamescampbell has quit [Ping timeout: 260 seconds]
<antocuni>
ah, cool :)
tav has joined #pypy
jamescampbell has joined #pypy
jamescam_ has joined #pypy
tav has quit [Quit: tav]
jamescampbell has quit [Ping timeout: 255 seconds]
tav has joined #pypy
jamescampbell has joined #pypy
<cfbolz>
antocuni: ah, I see about MutableStringBuffer. scary ;-)
<antocuni>
yeah, it looks like a bit of black magic. But it's more or less the same logic used by StringBuilder
kurtbrose[m] has joined #pypy
bendlas has joined #pypy
jamescampbell has quit [Read error: Connection reset by peer]
jamesca__ has joined #pypy
jamescam_ has quit [Ping timeout: 245 seconds]
<cfbolz>
antocuni: side question: does rlib/rstruct still only support fixed formats?
jamesca__ has quit [Read error: Connection reset by peer]
<antocuni>
cfbolz: you mean, compile-time constants?
<kenaan>
antocuni better-test-whatsnew c96a110a73d2 /: a branch where to improve test_whatsnew:
<cfbolz>
yes
<kenaan>
antocuni better-test-whatsnew ae2ab7014840 /pypy/doc/test/test_whatsnew.py: improve this test and make sure that the CURRENT branch is documented: this way we get a failure ea...
<kenaan>
antocuni better-test-whatsnew 95360dc76049 /pypy/doc/whatsnew-head.rst: add this branch to the documented ones, and make the test passing again
<kenaan>
antocuni better-test-whatsnew 57efe071e53a /: close branch about to be merged
<kenaan>
antocuni default b7531baf47cf /pypy/doc/: merge the branch better-test-whatsnew: now you must document your branch in whatsnew-HEAD.rst BEFORE merging it
<antocuni>
cfbolz: yes, I didn't touch anything of that logic
<cfbolz>
ok
<antocuni>
and, as before, only unpacking is supported
Tiberium has quit [Remote host closed the connection]
<tos9>
I should probably know this but I forget -- are there caveats to using buffer()
<tos9>
I want to slice some strs -- it's gonna give me read-only views of them without copying right?
<tos9>
If I had time to write some benchmarks I would, but I don't :/, so I have to just cowboy code out what I suspect to be good enough for a thing I am somewhat sure should end up being CPU bound (https://en.wikipedia.org/wiki/LEB128#Decode_signed_integer)
<mattip>
I'm not sure what hashlib.md5(x) is doing on pypy, but it is not going through space.strbuffer_w AFAICT
<mattip>
bufferstr_w
jkilpatr has quit [Remote host closed the connection]
<mattip>
this is from a failing test on numpy, that if I remember correctly used to pass
<mattip>
indeed, on 5.7.0 that I have laying around it prints True
<mattip>
I tried using our array.c test class and it works correctly on untranslated tests, numpy must be triggering a different code path
oberstet has quit [Ping timeout: 260 seconds]
<ronan>
hmm, hashlib.md5(x) should use space.strbuffer_w()
<mattip>
do you have a suggestion how to verify that, other than building a complete pypy w/ a print there?
<mattip>
I guess I could try an lldebug, under gdb
amaury has quit [Ping timeout: 240 seconds]
<ronan>
mattip: maybe look at something else that uses bufferstr
vkirilichev has joined #pypy
<ronan>
mattip: progress! buffer('') + x == '\0' * 8
<mattip>
huh
<mattip>
on cpython that raises a ValueError : operands could not be broadcast together with shapes (0,) (3,)
ESphynx has joined #pypy
jwhisnant has joined #pypy
<ESphynx>
hi guys, when using CFFI, how do you handle multiple modules that include one another? (a header file that include another header ? all in my modules being wrapped up by CFFI, that is)
<ronan>
I guess that's a different issue though
<ESphynx>
How to let CFFI know about the data types without generating duplicate bindings again?
<mattip>
ESphynx: if I understand you correctly, you need to simulate the #include macro preprocessing in C by doing two things,
<ESphynx>
mattip: I want to avoid having bindings duplicated
<ronan>
mattip: your issue probably is that space.bufferstr_w(x) returns '\0'*8 instead of '\0'*24
<mattip>
one, use some kind of replacing mechanism to get rid of the "#include" lines, and
<ESphynx>
so just copying the whole content of my included header in the including header wouldn’t cut it, that would result in duplicate stuff
<ESphynx>
two is the key?
<mattip>
two, call ffi.cdef(string_of_declarations) for each header file
<mattip>
reusing the same ffi, in the same order as the #include macros
<ESphynx>
what does ffi.cdef(string_of_declarations) do ?
<ESphynx>
you mean it has to be done as one whole process?
<ESphynx>
one -- we have custom files already for this
<mattip>
so if you have in header1.h "#include <header2.h>\n typedef struct_from1 struct_from2;"
<ESphynx>
so if I understand correctly, it has to be done with the same cffi object and that is the key
<mattip>
right, multiple calls to ffi.cdef(".....")
<mattip>
where ... is the content of the headers, one after the other
<ESphynx>
mattip: but would it then be possible to generate multiple python modules out of that?
<ESphynx>
for each?
<ESphynx>
or the more including ones would need to include all dependencies always
<mattip>
a single python module can consume multiple headers, multiple libraries/shared objects
marr has joined #pypy
<mattip>
so the more including ones would need to declare all structs/typedefs exposed by the previous ones, and that are used in the external interfaces
<ESphynx>
mattip: but i mean if I have module2 including module1, and I want to provide both module1 and module2 as CFFI/Python modules
<ESphynx>
struct/typdefs is one thing
<ESphynx>
but the actual symbols?
<ESphynx>
I’ll experiment a bit
<mattip>
so module1 is easy, and module 2 needs to include the c bits from module one
<ESphynx>
but will it need to include the c file generated by CFFI for module1 stuff?
<ESphynx>
well I actuall thought this was the issue (might have been later), but seems there’s a conflict with ‘Thread’ :P
<ESphynx>
or not, might still be that issue :S
<mattip>
if module1 exports the C functions from the shared object, i.e. you use __attribute__ ((visibility ("default"))
vkirilichev has quit [Remote host closed the connection]
Tiberium has quit [Remote host closed the connection]
vkirilichev has joined #pypy
<mattip>
then module2 can use the shared object
<mattip>
but it may be more correct to compile-and-link module2 from c code
<mattip>
depending what you are trying to do
<ronan>
mattip: buffer('') + x also returns a string on pypy 5.7.1, but the right one, so that's clearly a separate bug
<mattip>
ronan: md5(x) == md5('\0'*8) on default, so your guess was right
<ESphynx>
mattip: well I was talking more about the symbols that CFFI defines
<mattip>
ESphynx: so I would need to see the motivation, IMO it is easier not to reuse CFFI symbols from module1 in module2
<ESphynx>
mattip: well we have a number of libraries using another
<ESphynx>
and automatically generating CFFI bindings for Python for all of them
<ESphynx>
and they have interdependencies
<ESphynx>
sounds wasteful to duplicate all that CFFI stuff
<mattip>
try a naive implementation, then try to measure
<mattip>
you may be suprised
<mattip>
not that I have any preknowlege of course, it may turn out the other way
<mattip>
personally I have found that the build stage (when using ffi.compile) is really fast, and anyway happens once per setup
<ESphynx>
mattip: this tiny module here generated a 415kb .c file
<ESphynx>
we have bigger module on topof that
<ESphynx>
I can’t imagine how this could possibly be a good thing.
<mattip>
ESphynx: but how much of that is cffi-generated and how much is your c-code pasted from ffi.set_source()?
vkirilichev has quit [Read error: Connection reset by peer]
vkirilichev has joined #pypy
<mattip>
if much of it is cffi, are you sure all the things in ffi.cdef() are your minimal external interfaces?
<ESphynx>
mattip: not sure whether that matters because it’s all new symbol stuff? in comparison, for _pyeC.c 415 k .c, the source cffi-ec.h is 38kb
<ESphynx>
mattip: and for the 2237kb _pyecere.c which also includes cffi-ec.h but also cffi-ecere.h, the cffi-ecere.h is 192kb
DragonSA has quit [Quit: Konversation terminated!]
<ESphynx>
and then I will have another library which include both of these and 2 more etc.
vkirilichev has quit [Client Quit]
<ESphynx>
that is the library external interfaces yes.
<mattip>
huh
<ESphynx>
mattip: but ideally all these modules would be able to use the cffi constructs defined in another?
<mattip>
not sure how much that would save, in the end the compiler needs to see all the declarations
<ESphynx>
mattip: it would save a tremendous amount.
<ESphynx>
mattip: it is not the declarations that I’m talking about , it is all those accessors to convert to Python etc.
<ESphynx>
the code generated by cffi, the actual headers are the real headers included by those C files (that’s not even included in those file sizes)
<mattip>
sorry, I'm not helping. Maybe try again when arigato/fijal are around, and definitely would be easier if the code is visible
<ESphynx>
hehe thanks for your input :)
<ESphynx>
but to sum it up, multi-module inheritance is not currently supported :)
<ESphynx>
I think it would be a critical feature to implement :P
<ESphynx>
multi-module includes* rather :)
<mattip>
maybe file an issue, a test case would make it more likely to get looked at
<ESphynx>
mattip: basically what I would be hoping for is to mark the previous parts of the cdef as ‘for reference only’, as opposed to ‘for generation of CFFI stuff’
<ESphynx>
let CFFI know these things exist and are available in the actual C headers that will be included the generated .c, but don’t define any Pythonish constructs for them
<njs>
tos9: they were extremely annoying to write, so if someone else can take advantage I'm happy :-) let me know if you need a relicense (or more specifically, what license you need since, well, https://github.com/njsmith/machomachomangler#license)
<tos9>
njs: I would probably need something MIT-ish if that was close enough, but lemme check first on how close it gets me
<tos9>
(Which I doubt will happen till tomorrow morning since I am le tired)
<tos9>
Definitely appreciate the pointer!
amaury has quit [Ping timeout: 258 seconds]
<njs>
tos9: hmm, that documentation is really unclear, but it sounds like the difference is just that it "knows" that it's 64 bits max, which allows the flag bit in byte 9 to be skipped and treated as part of the value instead? I guess that shouldn't be too hard to implement that special case (though I'm not going to do it :-)).
amaury has joined #pypy
<njs>
tos9: good luck in any case :-)
<njs>
I definitely remember the frustration of realizing every single description of leb128 en/decoding logic on the internet was written as a chunk of C code relying heavily on quirky C semantics and impossible to directly port to Python :-)