cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | if a pep adds a mere 25-30 [C-API] functions or so, it's a drop in the ocean (cough) - Armin
<rjarry>
in the library I am writing a cffi wrapper for, there is #include <time.h>
<rjarry>
which provides time_t
rubdos has joined #pypy
<wleslie>
ok, I'm not sure, I don't have any of these in my library
jcea has joined #pypy
adamholmberg has joined #pypy
dddddd has joined #pypy
<arigato>
rjarry: you can often use "typedef int... time_t;"
<arigato>
(literally)
<rjarry>
arigato: oh, is that a special syntax?
<rjarry>
thanks
<arigato>
yes
<rjarry>
arigato: when facing with a library function such as: int sr_connect(int options, sr_connection_t **);
lritter has joined #pypy
<rjarry>
in C, the use is: on the stack, sr_connection *conn = NULL;
<rjarry>
then, sr_connect(opts, &conn);
<rjarry>
however, with cffi, it seems more complex
<rjarry>
as addressof seems not to cut it
<arigato>
the cffi version is similar to what you could also write in C as:
<arigato>
conn = malloc()
<arigato>
sr_connect(opts, conn);
<rjarry>
the malloc is additional :)
<arigato>
now with sr_connection **conn;
MithrasX2 has quit [Ping timeout: 240 seconds]
<arigato>
yes, so maybe let's call it "sr_connection **pconn = malloc(); sr_connect(opts, pconn);"
<arigato>
and then that translates to cffi: pconn = ffi.new("sr_connection **"); sr_connect(opts, pconn)
<rjarry>
that's what I have come up with until now
<rjarry>
but I need to explicitly access to pconn[0]
<arigato>
yes
<rjarry>
for other functions that require sr_connection * args
<arigato>
the problem with addressof() is that while it could potentially work in this case, in general it can't, if say we're talking about a function "foo(int *arg)": because it doesn't make sense to take ffi.addressof(n) if n is a Python integer object
<rjarry>
should I always keep a reference to pconn or only to pconn[0] ?
<rjarry>
as all other functions want a 'sr_connection *' value, not 'sr_connection **'
<arigato>
it doesn't matter. if sr_connect() writes a result to pconn[0], then just after the call you can do "conn = pconn[0]" and forget pconn
<rjarry>
ok, I do not risk pconn freeing too much then
<rjarry>
when it goes out of scope
<arigato>
no, pconn is only holding the "sr_connection *", i.e. the pointer itself
<arigato>
not a complete "sr_connection"
<rjarry>
got it
<rjarry>
I had similar a problem with functions that allocate 'char *'
<rjarry>
I had to do pbuf = ffi.new('char **')
<rjarry>
call the fuinction with pbuff
<rjarry>
pbuf
<rjarry>
and the do not forget to explicitly call lib.free(pbuf[0])
<rjarry>
is there a cleaner way to do this?
<arigato>
yes, that's correct, but also you can say "buf = pbuf[0]" after the call to the function, and only use "buf" and forget pbuf
<arigato>
(including lib.free(buf))
<rjarry>
does that properly handle subtleties such as 'const char *' != 'char *' ?
<arigato>
OK. that's right. maybe better (on pypy, where ffi.gc() doesn't free immediately when doing out of scope) would be to use more explicitly:
<arigato>
p = lib.lys_path()
<arigato>
try:
<arigato>
return c2str(p)
<arigato>
finally:
<arigato>
lib.free(p)
<rjarry>
better than ffi.gc ?
<arigato>
yes, because it calls lib.free(p) here, as opposed to "some unknown time later"
<arigato>
you can also write it like this if you prefer:
<arigato>
with lib.gc(lib.lys_path(), lib.free) as p:
<arigato>
return c2str(p)
<rjarry>
I see
<arigato>
it's equivalent (shorter, but uses more concepts)
MithrasX has joined #pypy
<rjarry>
but it requires that the libc free() function is declared somewhere in the cffi object
<arigato>
yes
<rjarry>
there is no "builtin" way to achieve this?
<arigato>
that part is correct and unavoidable: you're calling some API that requires you to call free() afterwards
<arigato>
no
<rjarry>
ok
<arigato>
msotly because, why? it's just one line to add, like you did, to the cdef()
<rjarry>
that's easily overlooked and leads to memory leaks
<arigato>
yes :-( welcome to C
<rjarry>
hehe, I am used to it, thankfully
<rjarry>
thanks for all the clarifications, arigato :)
<arigato>
if you're calling the same function lys_path() from many places, then you should turn it into a helper function so you don't need to worry about memory management outside
<arigato>
welcome!
dmalcolm has joined #pypy
<arigato>
it's good practice in general to put all this memory-management functions into their own module
<arigato>
but maybe this libyang/schema.py is already a thin later around the lib functions in C, so that's enough
<rjarry>
coming back to sr_connect(), with that pconn
<rjarry>
pconn = ffi.new('sr_connection_t **')
<rjarry>
sr_connect(options, pconn)
<rjarry>
conn = pconn[0]
<rjarry>
is there a clean way to call sr_disconnect(conn) when conn goes out of scope?
<arigato>
we've seen them all: if you want to call sr_disconnect() at a known point, use try: finally: or "with lib.gc()". Otherwise, just say "conn = lib.gc(pconn[0], lib.sr_disconnect)" and it will call sr_disconnect at some not-very-well-specified point later
<arigato>
(not-well-specified but after the last reference to 'conn' goes away)
svanheulen has joined #pypy
MithrasX1 has joined #pypy
MithrasX has quit [Ping timeout: 250 seconds]
mattip has quit [Quit: ZNC 1.6.6+deb1ubuntu0.2 - http://znc.in]
mattip has joined #pypy
<mattip>
arigato: if you have a second, the win32 extra_test/test/test_semlock is hanging
<vstinner>
arigato: i understood that the psyco project is dead (superseded by PyPy obviously). but just in case, can you confirm that you don't use _PyThreadState_GetFrame anymore?
MithrasX2 has quit [Ping timeout: 240 seconds]
jacob22 has quit [Quit: Konversation terminated!]
<arigato>
no, we don't. And yes, psyco is python <= 2.6
<vstinner>
"python <= 2.6" oh wow. hello time travelers :)
<vstinner>
arigato: FYI i discovered (!) _PyThreadState_GetFrame while attempting to make the PyThreadState structure opaque in CPython
<vstinner>
especially the "Changes in the limited C API" part
<arigato>
cool. maybe one day you'll have made it opaque and generalized a little bit refcounting, and then we'll be able to remove the H in HPy_Xxx functions and be done :-)
<vstinner>
"remove the H in HPy_Xxx functions and be done" that would be nice, yep ;)
<vstinner>
arigato: the work i'm doing in CPython is the prepare the C API to move it towards HPy. hide as much implementation details as I can
<vstinner>
many implementation details are only exposed "by mistake" (aka "for efficiency") and can easily be hidden without touching to the C API
<vstinner>
the usual fix is to replace a macro (or a static inline function) with a regular “opaque” function call
<vstinner>
arigato: HPy and "make the C API opaque" projects are complementary and have the same goals ;)
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
WGH has joined #pypy
pyusr has joined #pypy
<pyusr>
cffi should add support for PyUnicode_Decode at the cffi.unpack level
<pyusr>
so a char* could be directly transformed to a string, without an .decode() step
<arigato>
yes, or if you have to use the library and can't reasonably write everything in Python, you could try to use a bit more C code (inside ffibuilder.set_source())
<arigato>
some two-steps process where all the functions in the function_table are from C, and build a C data structure,
<arigato>
which is then transformed (lazily or not lazily) into Python afterwards
<arigato>
if you're going for PyPy then the fastest would be to transform lazily
<arigato>
it would be a bit more involved to write than what you have now, but not too much
<arigato>
at least if you're comfortable writing C, with some malloc() and stuff
<pyusr>
it's ok, there are pure python parsers (the protocol is easy), but soon there will be a new vresion of protocol
<pyusr>
so I wanted to supported it via the C parser that is already written, but no biggie
<arigato>
OK
<arigato>
then yes, you can rewrite the parser in Python---and the same applies actually, if you can, make it lazy for better performance
<pyusr>
the reason I wanted to modify hiredis-py is to make it possible to set per read the encoding / bytes
<pyusr>
and not per object, so I thought let's try cffi
<pyusr>
it was a nice learnign experience
<pyusr>
arigato: I am actually trying a new model in python, where it's much more lazy than normal
<arigato>
cool!
<pyusr>
i.e. I'm reading the data from the I/O stream, as an memory-backed pool of small bytearrays
<arigato>
nice
<pyusr>
and then I parse that as a non-contingous memory with normal operations of read, readuntil, etc...
<pyusr>
(I also support recv_into both sync socket, and async socket)
<pyusr>
so minimizing allocations and copying
<arigato>
messy but nice
<pyusr>
yep, I'm trying to write now a benchmark to show how good or bad the idea is :)
<arigato>
just to be clear, the "lazy" I had in mind was in the returned data structures, e.g. you wouldn't build a list with all the objects at once (but maybe it doesn't make a difference if individual messages are not large)