arigato changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://quodlibet.duckdns.org/irc/pypy/latest.log.html#irc-end ) | use cffi for calling C | mac OS and Fedora are not Windows
speeder39_ has joined #pypy
dddddd has quit [Remote host closed the connection]
BPL has quit [Quit: Leaving]
<kenaan> mattip buildbot 89139fc90e06 /bot2/pypybuildbot/master.py: add platform-specific factory for aarch64 so that package.py gets the right archive-name
<mattip> 89139fc90e06 seems really fragile - right now the aarch64 packages are uploaded as linux64
<mattip> fixed the linux64 pypy2 download by hand, canceled the aarch64 py3.6 build
<mattip> ronan: the extra_tests are using a new pytest, which is emitting this warning
<mattip> "RemovedInPytest4Warning: --result-log is deprecated and scheduled for removal in pytest 4.0"
RoadrunnerWMC has joined #pypy
<RoadrunnerWMC> Hello! I'm using cffi to make bindings for an existing C library, but I've run into an issue with callbacks. Hope this is the right place to ask about it
<RoadrunnerWMC> I've made a minimal test case here: https://gist.github.com/RoadrunnerWMC/fb434e7ebaa12f533a2ce1df8d094579 It fails with an error as-is, but if you comment out the "unused_function" in test.py, it starts working
<mattip> RoadrunnerWMC: this is the right place but most of the "right people" are on European time, so your question may linger a while
<RoadrunnerWMC> All right, thank you; I'm not in a hurry :)
<RoadrunnerWMC> You run the test case by running build_cffi.py to build and then test.py to run
<RoadrunnerWMC> The error I get when I do so is "extern "Python": function _c_code._py_destroy_callback_function_impl() called, but @ffi.def_extern() was not called in the current subinterpreter. Returning 0."
<RoadrunnerWMC> Environment is Python 3.6 on KDE Neon (which's based on Ubuntu).
<RoadrunnerWMC> and latest cffi (1.12.3)
<RoadrunnerWMC> Expected result (which is what happens if you comment out that function) is for it to print "End of test.py" followed by "Destroy callback"
lritter has quit [Ping timeout: 244 seconds]
lritter has joined #pypy
<mattip> RoadrunnerWMC: you need to make sure obj.__del__ is being called before the interpreter exits
<mattip> for instance, if I put the call to "obj = wrapper.make_class()" in a function, all works as expected (on cpython)
<mattip> on PyPy, you need to explicity trigger the deletion since the gc works differently
<mattip> with something like "del obj"
<mattip> ronan: something is wrong with collecting tests and -D on windows. If I call
<RoadrunnerWMC> mattip: hm, I guess that makes sense, but I can't really dictate exactly how people use the bindings, can I?
<mattip> RoadrunnerWMC: yeah, maybe there is a problem that the callback or a part of it is being gc collected before the object
<mattip> ronan: pypy-venv\bin\pypy.exe -mpytest -D ..\build\pypy
<mattip> then nothing is collected, but if I call
<mattip> ronan: pypy-venv\bin\pypy.exe -mpytest -D ..\build\pypy\interpreter
<mattip> the tests are found
<RoadrunnerWMC> mattip: That makes a lot of sense, and also explains why seemingly unrelated changes affect the behavior. I wonder if there's a way to keep the callback alive long enough, then...
cfbolz has quit [Write error: Broken pipe]
samth has quit [Write error: Broken pipe]
kcr has quit [Write error: Broken pipe]
Alex_Gaynor has quit [Write error: Broken pipe]
wallet42 has quit [Write error: Broken pipe]
_whitelogger has joined #pypy
fling is now known as bedflinger
bedflinger is now known as fling
jcea has quit [Remote host closed the connection]
_whitelogger has joined #pypy
<RoadrunnerWMC> My current ideas are to either make my wrapper Python class into a context manager, or to entirely bypass the C library for this particular destructor callback and implement it in pure Python instead. Or maybe I could just not expose that callback in Python at all, since I don't think it's very necessary anyway
<RoadrunnerWMC> I'll look into it further tomorrow. Thanks for the help so far, mattip!
_whitelogger has joined #pypy
lritter has quit [Quit: Leaving]
jcea has joined #pypy
_whitelogger has joined #pypy
speeder39_ has quit [Quit: Connection closed for inactivity]
speeder39_ has joined #pypy
<arigato> RoadrunnerWMC: right, not too surprized that we get strange errors
<arigato> the object returned by ffi.gc() is freed by CPython at a random time late during interpreter shut-down
<arigato> at that point, the cffi logic to support ffi.def_extern() is not registered any more, but anyway it's unclear that we can really call arbitrary Python code at that point
<arigato> if you're using ffi.gc(), you can also try a more direct approach: right now, the ffi.gc() object calls c_class_destroy at the end, which itself calls back pure Python code
<arigato> the more direct approach would be to use ffi.gc() with a lambda that both calls c_class_destroy and then (before or after) calls some custom code that you pass as argument to make_class(), for example, if it is not None
<arigato> of course, that's assuming you need that at all; if you don't really need it then it's even better. The exact time destructors are called is slightly undefined on CPython and very undefined on PyPy
<arigato> (note that recent releases of cffi also support using an ffi.gc() object in a "with" statement, and force the destructor to run at the end)
travis-ci has joined #pypy
travis-ci has left #pypy [#pypy]
<travis-ci> vmprof/vmprof-python@master: The build was broken. (https://travis-ci.org/vmprof/vmprof-python/builds/570416030)
Taggnostr3 has quit [Remote host closed the connection]
Taggnostr has joined #pypy
_whitelogger has joined #pypy
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
agronholm has joined #pypy
jcea has quit [Ping timeout: 250 seconds]
bbot2 has quit [Quit: buildmaster reconfigured: bot disconnecting]
bbot2 has joined #pypy
jcea has joined #pypy
ekaOlogik has joined #pypy
jcea has quit [Ping timeout: 264 seconds]
_whitelogger has joined #pypy
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
speeder39_ has quit [Quit: Connection closed for inactivity]
agronholm has quit [Ping timeout: 248 seconds]
agronholm has joined #pypy
ekaOlogik has quit [Quit: http://quassel-irc.org - Chat comfortably. Anywhere.]
jcea has joined #pypy
dddddd has joined #pypy
jcea has quit [Remote host closed the connection]
<kenaan> arigo default 381752e5cd15 /rpython/jit/backend/: aarch64: fix test_gc_integration.py
<kenaan> arigo default a6010dc54fc4 /rpython/jit/backend/aarch64/runner.py: aarch64: fix test_regalloc_integration.py
ajlawrence has joined #pypy
<ronan> mattip: the warning is annoying but rather meaningless, --result-log is still here in pytest 5
jcea has joined #pypy
<RoadrunnerWMC> arigato: thanks for all that info! I think I'll just not support that callback, since it doesn't translate well at all to Python and it's not really necessary. (If somebody wants to run some code after they're done with that object, they can easily do so manually; the C library I'm wrapping is completely synchronous)
<arigato> ok!
<RoadrunnerWMC> While I'm here, I'll ask another question that Google wasn't too helpful about: how do people generally compile their package for a variety of OSes? Is there some standard relatively automated way to do that? CI services, maybe?
<arigato> I think so, but I don't really know myself
<arigato> for cffi, I use the automatic job of https://ci.cryptography.io/job/external-projects/job/cffi-wheel-builder/ but that might be for a specific project. But I'm sure the same can be done more generally elsewhere
<arigato> (cryptography is a big user of cffi)
BPL has joined #pypy
<RoadrunnerWMC> That link doesn't DNS-resolve for me, but it looks like they have their own CI server going, which would imply that existing free CI services are insufficient for this. That's a bit unfortunate if so, but thanks
<arigato> cryptography might be a bit of a special case; maybe others can point to some free CI service
<RoadrunnerWMC> That's true; I should probably ask people involved with some other packages that use C code (like Pillow). Thanks again
<mattip> RoadrunnerWMC: azure pipelines has linux, windows, macos
<mattip> soon it will be integrated more tightly with github as microsoft owns both
<mattip> ronan: ok, more worrying is the failure to collect any tests on win32 and -D
<ronan> mattip: I'm looking into it
<ronan> the conftest for _vmprof is doing something wrong
<mattip> cool, let me know if you need anything
<RoadrunnerWMC> mattip: looks nice at a glance; I'll probably give it a try. Thanks
<mattip> ronan: perhaps because the testrunner/app_level_tests.py and testrunner/runner.py collect per-directory,
<mattip> so the _vmprof pytest_collect_directory function is only specific to _vmprof,
<mattip> where the -D uses pytest to do all the work, so it collects _vmprof together with the other directories
ekaOlogik has joined #pypy
<ronan> mattip: the issue is that _vmprof/conftest.py silently crashes test collection
<ronan> but I don't know what is the right way to do what it tries to do
<mattip> could ask on #pytest how to get conftest.py to skip only its directory
ekaOlogik has quit [Quit: https://quassel-irc.org - Komfortabler Chat. Überall.]
ekaOlogik has joined #pypy
ekaOlogik has quit [Client Quit]
ekaOlogik has joined #pypy
<mattip> maybe by setting collect_ignore_glob=['_vmprof/test/*.py] in the conftest.py ?
jcea has quit [Ping timeout: 264 seconds]
jcea has joined #pypy
<kenaan> rlamy default 597f4be1ae97 /pypy/module/: Fix test collection on windows and s390x
lritter has joined #pypy
<kenaan> mattip buildbot 7ea68e40c299 /master/templates/layout.html: add aarch64 to rpython page
bbot2 has quit [Quit: buildmaster reconfigured: bot disconnecting]
bbot2 has joined #pypy
<mattip> aarch64 rpython tests look much better now
<mattip> there is a timeout in jit/backend/aarch64/test/test_ztranslation_basic that is masking a few other failures
ekaOlogik has quit [Read error: Connection reset by peer]
ekaOlogik has joined #pypy
ekaOlogik has quit [Client Quit]
ekaOlogik has joined #pypy
ekaOlogik has quit [Client Quit]
ekaOlogik has joined #pypy
<arigato> mattip: ah, these tests should be run one-at-a-time, not the whole directory
<arigato> there's a conftest doing that for the other dirs
<arigato> sorry, not a conftest. grep for cherrypick in bot2/pypybuildbot/*.py
<arigato> uh no
<arigato> confused
<arigato> ah found it
<kenaan> arigo default 872b51a36497 /pypy/testrunner_cfg.py: Try to split the run of individual test files in this directory too
<mattip> :)
<arigato> I think that all the tests in backend/aarch64 were passing, at some point
<mattip> before we kick off another rpython run, we should merge upstream vmprof in
<kenaan> arigo py3.6-sandbox-2 67130f4a3a0c /: hg merge sandbox-2
ekaOlogik has quit [Ping timeout: 245 seconds]
ekaOlogik has joined #pypy
ekaOlogik has quit [Quit: https://quassel-irc.org - Komfortabler Chat. Überall.]
<kenaan> mattip default 8cb85ca95940 /rpython/rlib/rvmprof/src/shared/: sync with upstream vmprof
<kenaan> mattip py3.6 17f20f51c2cb /: merge default into branch
xorAxAx has quit [Quit: Gateway shutdown]
xorAxAx has joined #pypy
<kenaan> arigo py3.6-sandbox-2 5fadf669fc02 /: fixes
<kenaan> arigo py3.6 4006ceea6169 /pypy/module/imp/importing.py: Remove the only usage of open() in the py3.6 core parts (as found in the branch py3.6-sandbox-2). Also wrap the raw ...
<kenaan> arigo py3.6-sandbox-2 f19d31ff52b6 /pypy/module/imp/importing.py: hg merge py3.6
<kenaan> arigo py3.6-sandbox-2 7f7d17b89d46 /pypy/module/posix/interp_scandir.py: comment
<kenaan> arigo py3.6 d21e9a6b4037 /: merge heads
Hasimir has joined #pypy
dansan_ has joined #pypy
dansan_ is now known as dansan
alexge50 has quit [Read error: Connection reset by peer]
alexge50 has joined #pypy
<kenaan> andrewjlawrence winconsoleio d892abac55ac /: Added a few more functions
Taggnostr has quit [Ping timeout: 250 seconds]
ajlawrence has quit [Remote host closed the connection]
xorAxAx has quit [Quit: Gateway shutdown]
xorAxAx has joined #pypy
agronholm has quit [Ping timeout: 250 seconds]
<kenaan> stefanor default 7fc3484b9993 /rpython/rlib/rposix.py: On GNU/Hurd use the sysconf() method (some day, it'll have SMP...)