<arigato>
zzarr: cffi is meant to work in a way that lets you ignore the GIL completely (it acquires and releases it as needed internally)
<mattip>
also documented the whole thing at /root/README-CHROOT-32
<arigato>
if you are having troubles like the GIL not being acquired in some situation, I'd be interested in hearing about it
<zzarr>
arigato, yes, I have callbacks to python that run in different threads that are started in C, but if I have an MongoClient in a callback it crashes the project
<zzarr>
I get Illegal instruction
<arigato>
OK, and why do you think the GIL is the problem?
<zzarr>
the MongoClient class should from what I have read be thread safe
<zzarr>
so I think that python don't know about the new thread (but I could be wrong)
<arigato>
I fear you need gdb to know where the Illegal instruction is coming from
<zzarr>
yea
<arigato>
in principle, cffi tries hard to be multithread-safe
<zzarr>
that's good to hear
<zzarr>
is there a way to test if it's thread safe in my environment?
<zzarr>
like a "is the GIL aware" function?
<arigato>
mattip: when I log to bencher4 I get the prompt "arigo@bnecher4-stretch32:~$", which is completely bogus but also I can't find where it comes from
<mattip>
huh. Sorry, they must be sharing hostnames somehow
<zzarr>
if it's not the GIL it has to have something to do with the Mongo connection I guess
<mattip>
fixed
<arigato>
zzarr: the GIL should really not be the problem but there is no way to know "do we have the GIL right now?"
<arigato>
zzarr: there are many potential issues in multithreaded senarios, with cffi I'd first assume cffi does its job correctly (it's tested) and check the other parts
<zzarr>
arigato, I did not mean to complain, I read in a stack thread that it was not thread safe
<arigato>
(it's tested as much as we can; it's always delicate, admittedly)
<zzarr>
arigato, in that case something else is the cause, so thanks for your time :)
lritter has joined #pypy
<zzarr>
I have built a small python module that connects the C part (I'm new to python)
<arigato>
did you make something using the C API of CPython? you shouldn't go there and use cffi at the same time
<zzarr>
arigato, no, I use cffi's ABI method
<zzarr>
I have not touched any other API/ABI at the same time
<arigato>
ok
<zzarr>
I don't need to import something to my C code and run any macro or something to access the GIL?
<arigato>
you can ignore the GIL completely
<zzarr>
in the background I use C++/Qt with Qt's thread implementation QThread, it's a wrapper of pthread
<zzarr>
I see, that's nice
<arigato>
but when C code is running, you must not assume it's protected by any GIL
<zzarr>
no I use QMutex to protect the C/C++ code
<arigato>
then everything "should" work
<zzarr>
cffi is a very nice implementation
<zzarr>
I know this is a little off topic, but which languages do cffi exist for?
dustinm has quit [Quit: Leaving]
dustinm has joined #pypy
dustinm has quit [Remote host closed the connection]
<arigato>
only for Python, although it was inspired by LuaJIT's ffi originally
<arigato>
LuaJIT's ffi is an ABI-mode-only version of cffi, if you want
<zzarr>
okey, thanks :)
dustinm has joined #pypy
<arigato>
but yes, I would see the point in implementing cffi for other languages
<arigato>
e.g. I had to learn very obscure ways to use several limited equivalents for Ruby
<zzarr>
I would like to have if for ruby, javascript, php and other languages
<arigato>
I can consider implementing it for $5000 per language :-/
<arigato>
in other words, it's probably unlikely that someone will come along and just implement it
<arigato>
because there are (somewhat artifical) barriers between these languages
<kenaan>
stevie_92 cpyext-gc-cycle a3d95a9939f4 /rpython/memory/gc/test/: Added more tests for "modern" rawrefcount finalizers
<zzarr>
arigato, I meant that the implementation would be a C API/ABI for the languages, not that you could import a python module in ruby or something like that
<arigato>
zzarr: yes, I understood it this way too
<zzarr>
:)
<arigato>
I meant there is a barrier between languages in that people from Python don't often look at what's going on in the Ruby world, or any combination
<arigato>
I'm the first guilty of that
<zzarr>
arigato, I see
<arigato>
it was a bit by chance that we found out LuaJIT's ffi, and decided it was a good idea to have a Python version of it
<arigato>
the original motivation was related to pypy in the sense that "ctypes", the standard-library equivalent in Python, is a large mess and we wanted something that is easier to implement and optimize well on pypy
<zzarr>
It's always nice when one project inspires the next
<arigato>
("ctypes" is more like ruby's "ffi" in that it is an ABI-only version with a complicated way of declaring and using things)
<zzarr>
arigato, having used ctypes and realized that it was not a good ide
<vstinner>
it would be great if you (PyPy devs) could review it
<vstinner>
i didn't read it yet. i planned to write such PEP, but Steve was faster than me :)
<simpson>
Cool stuff.
<simpson>
The pattern of constructing a formal hierarchy which represents an ideal rather than reality is possibly an anti-pattern, but I guess it's progress. I'd like to see formal demonstrations of the closure properties for rings and layers.
xcm has quit [Read error: Connection reset by peer]
<fijal>
vstinner: so I think our view has been that it's not possible to have "python ring" because it would need to contain pyincref
<fijal>
cfbolz: did you show Victor what we wrote for docs?
<cfbolz>
nope
<cfbolz>
sorry, on my todo list somewhere ;-)
<fijal>
vstinner: I think a discussion should start with "where does pyincref belong"
<fijal>
cfbolz: it takes less than 5 minutes, just post it
<fijal>
Here
<fijal>
Having todos of items that take less than 5 min is counterproductive
<fijal>
cfbolz: should I just do it then?
danieljabailey has joined #pypy
<cfbolz>
fijal: feel free, I can't even find the link right now :-(
<fijal>
cfbolz: ok, so please do things quickly like that. Trying to remember took me already more time than actual writing
<fijal>
(or not at all)
<cfbolz>
not at all is probably more realistic right now :P
<fijal>
Fine
<fijal>
vstinner: essentially we rewrote intro to the c doc, I'll do it again once I have access to computer - most people already use cffi or cython for new stuff. Can we recommend that from python docs?
<arigato>
vstinner: this document is highly pointless imho
<arigato>
I'm sure that over time, a fraction of new C extensions modules will be written using only the outer ring. Thanks, that doesn't help at all any implementation that cares about all other modules
<arigato>
so deciding how many layers and how many rings there should be in CPython looks like a waste of time to me, sorry
<simpson>
I like the idea but only if eventually some of those layers and rings will be made completely off-limits.
<simpson>
But the root of the problem still seems to be that people think that writing C is an acceptable way to write Python. That's a cultural issue not at all addressed by this document, except at the top where it says that this issue won't be addressed.
<arigato>
I'll stand on my position that this doesn't help either pypy nor a future pypy-scale implementation of python, so if anything it's just a clean-up internal to CPython---which might not be a bad idea to do, fwiw
<arigato>
I think that any pypy-scale implementation will take the same path than pypy, i.e. a large mess to emulate the CPython objects and then incrementally supporting more and more of the C API, as needed and as requested by users
<vstinner>
arigato: "this document is highly pointless imho" you should reply to the email in this case :)
<arigato>
vstinner: feel free to quote me. Here I'm answering to you in this channel :-)
<arigato>
but note that my "highly pointless" initial comment was tempered by "if anything it's just a clean-up internal to CPython---which might not be a bad idea to do, fwiw" :-)
<vstinner>
fijal: "Can we recommend that from python docs? " it's up to you, not to me. please propose a change to the Python doc
<vstinner>
arigato: i don't understand what is a ring... i guess that i should read Steve's PEP
<arigato>
sorry, I was assuming that you did read the PEP before you asked us for comments on it
<vstinner>
cfbolz: in short, i'm doing dozens things per day, i fail to keep track of everything. i'm supporting changes for the C API or other solutions to not use the C API, but I cannot write the PR for you, sorry :(
<cfbolz>
vstinner: well, this is more the question "do you think such a PR has a chance?'
<cfbolz>
vstinner: if not even you like it, I am not going to open it
<vstinner>
arigato: no sorry, i just wanted to make sure that this channel is aware, but i'm doing other stuff this afternoon. i'm fighting the messy CPython initializaton code
<cfbolz>
if you think it does, I'll put it on my todo list
<vstinner>
"do you think such a PR has a chance?' i don't know why you (PyPy devs) don't want to contribute to Python. I understand that you had a few very bad experience in the past. sorry, i'm not aware of that
<vstinner>
i'm trying to help so CPython and PyPy can collaborate better
<vstinner>
what I can say that that I'm repeating that the C API is a problem for Python for 1 year, and slowly more and more core devs are aware of it
<vstinner>
cfbolz: things are changing, right now
<arigato>
that's cool. you need to push us forward though, until we see concrete changes for ourselves
<vstinner>
but i would prefer that discussions not only happen on this IRC channel, but on mailing lists which have a wider audience
<vstinner>
arigato: i already made some functions/structures opaque/private in Python 3.8. you now have to opt-in to get them: define Py_BUILD_CORE and adjust the C compiler include path, to get these internal headers
<vstinner>
I added _PyTuple_ITEMS() macro to get op->ob_item, this macro is "internal". I did that to better identify code which directly rely on PyObject**
<vstinner>
i know that it's a very small step, but well, we cannot change the world in 1 day (i cannot)
<cfbolz>
vstinner: that's not really an answer to my question though
<cfbolz>
also "i don't know why you (PyPy devs) don't want to contribute to Python" is quite badly worded, since obviously working on PyPy *is* contributing to the wider Python language
<vstinner>
cfbolz: by contributing to Python, i mean be involved in discussions to change the C API
<cfbolz>
just saying
<fijal>
vstinner: I will do a Pr, but if it ends up like last time I will refrain for another 3 years
<arigato>
I already wrote my position in one mail in the long thread to python-dev. I think it contains everything I wanted to say about that
jcea has joined #pypy
<arigato>
if refusing to participate in long threads about "just" cleaning up a bit the C API means "not contributing to Python", then too bad, I'm not contributing to Python
<arigato>
though I would argue that this reall means "not contributing to CPython"
<vstinner>
arigato: are you talking about PyHandle?
<vstinner>
fijal: i cannot promise that your PRs will be merged :) but i can promise to have a look if you ping me :)
<vstinner>
fijal: i think that most people are that Cython and cffi are good alternatives to C extensions written directly with the C API
<arigato>
and the long thread on python-dev just before
<vstinner>
fixing C extension doc, fixing the C API (opaque structure, remove functions, avoid borrowed references), use cffi/Cython, etc. are different solutions, but IMHO they are not exclusive but complementary
<arigato>
to be clear, I think that you're doing a good job about this whole C API clean-up
<arigato>
I'm not interested myself as long as I don't see anything useful for other implementations than CPython
<vstinner>
arigato: sorry, i'm not sure of what you are proposing. do you see any option which would make PyPy's life easier?
<vstinner>
if the plan is completely useless, it should be said, so we will not waste 1 or 2 years
<arigato>
the plan is probably useful as a clean-up of CPython
<vstinner>
arigato: ok but the question is if this plan can prepare a brigher future for a second plan to make PyPy life easier, for example
<arigato>
I think it's good if, as a result, in the future people think more carefully before adding many APIs
<arigato>
they are all of the kind "not make life worse" rather than "make life better"
<simpson>
vstinner: I am spitballing, but I suspect that in the future, people who haven't had our experiences will foolishly decide to write C when they should be writing Python. I think that a second plan should include a direction towards ending that practice.
<vstinner>
well, i started to ask to stop adding new C API like https://github.com/python/cpython/pull/11954 "Add PyTuple_FromArray() function": the new function became an internal function during the review
<simpson>
vstinner: Sure. The first category shouldn't exist and I disapprove of people who take this path. (I know that this is a harsh opinion, but bad things like Numpy and Scipy and Conda exist as a result of this mindset.)
<simpson>
It does suck that Python wasn't designed for zero-copy and that there's a lot of mutable/immutable mismatch.
<vstinner>
hum. so I finally read the draft PEP.... i don't think that it's doable to isolate perfectelly code as Steve wants. right now, encodings, memory allocations, etc. : everything is mixed. it's not possible to isolate memory allocation from the rest of the code
<vstinner>
simpson: "It does suck that Python wasn't designed for ..." well, Python is what is is :) we have to deal with it. things are changing, but slowly
<vstinner>
simpson: antoine pitrou is working on Arrow which is designed for 0-copy and he wrote a PEP to adapt pickle for Arrow
<vstinner>
simpson: the Python 3 protocol becomes more and more usable to reduce memory copies
<simpson>
Sure. But will it convince people to stop writing C? Probably only if it says clearly so on the label!
<simpson>
Also, to me, "adopt pickle for Arrow" is not good news. It's yet more usage of a terrible serialization format that should have died a decade ago.
<simpson>
Sometimes I wonder whether I'm a photo negative, living in an opposite world.
Ai9zO5AP has joined #pypy
<arigato>
I'm with simpson on that one, I wonder what "adopt pickle for Arrow" means but to me it sounds like "invent some new C API so that Arrow and pickle can communicate with each other" and oh btw that's likely cPickle you're talking about and the pure python pickle is lost and oh btw pypy has got only the pure python pickle, new mess ahead
<vstinner>
simpson: the PEP is not specific to Arrow. it's to modify the API to avoid memory copies... sorry, I didn't read the PEP, I cannot tell you more :-(
<arigato>
it strikes me as strange that the python world is now talking more about buffers, but doesn't have a pure python way to access or define buffers yet
<simpson>
All this talk of speed would be less irritating if Python were still slow. The fact is that, as a community, several different speed-oriented fast runtimes have been built. PyPy's the only one that has remained fast and remained maintainable.
forgottenone has quit [Quit: Konversation terminated!]
<simpson>
So perhaps the speed junkies would do well to sit down and study PyPy for a moment before claiming that they need their C.
<arigato>
CPython is not slow if all the speed is in C and you don't actually write much Python code!
<vstinner>
arigato: "but doesn't have a pure python way to access or define buffers yet" did you open an issue or write an email to propose to add such API?
* arigato
grumpy sorry
<vstinner>
arigato: i don't know well this type which scares me :)
<simpson>
vstinner: I've only skimmed this PEP, sorry. It has zero concern for security, which is disappointing.
<vstinner>
simpson: this PEP is awaiting your review for 6 months
<vstinner>
simpson: if you don't comment it, there is a risk that it's accepted as it is ;)
<vstinner>
i don't care much about pickle :)
<simpson>
vstinner: Higher up on my priority list: Getting NixOS's Python 2.7 tree compatible with PyPy 2.7 because, for some strange reason, people keep running around and bleating about the sky falling and creating websites like https://pythonclock.org/
<simpson>
Frankly, that core devs haven't already internalized as a community that pickles are a massive security risk is shocking to me.
<vstinner>
i don't understand why you exclude yourself from "core devs" and "the community"
marky1991 has quit [Ping timeout: 240 seconds]
<simpson>
Because like half a decade ago, I got tired of having conversations with core devs at PyCon where I ask "why not <obvious improvement to PEP>" and they say "because nobody understood why that would be useful".
<simpson>
So now I'm a Monte core developer, and I only write Python as needed, begrudgingly. I seem to have wandered into being one of the NixOS Python people.
<kenaan>
cfbolz py3.6 18e9ae494869 /pypy/interpreter/pyframe.py: kill dead global
<kenaan>
cfbolz py3.6 aebd4e263fc5 /lib-python/3/test/test_traceback.py: pypy's recursion limit is not precise
<kenaan>
cfbolz py3.6 905b81e90065 /pypy/module/thread/os_local.py: fix changed module name
<kenaan>
cfbolz py3.6 1db0679fa78a /lib-python/3/idlelib/idle_test/test_widgetredir.py: this was actually a rename to test_redirector.py (which we have)
<kenaan>
cfbolz py3.6 438c53ddd510 /pypy/interpreter/: follow more closely the complicated logic that CPython uses for code object equality
<arigato>
vstinner: ``why you exclude yourself from "core devs" and "the community"``: for me, the answer is that python-dev is not working for me. I feel much more useful to the larger python community by writing pypy and cffi than by discussing things in endless threads with no conclusion at the end
<simpson>
vstinner: There's a bigger pattern here. Python 3 was such a massive letdown that folks decided to build languages like Monte and Julia and Hy instead. A decade's been spent pushing Python 3 instead of supporting PyPy and now we're stuck with lots of holes to fill in.
lesshaste has quit [Quit: Leaving]
<simpson>
On NixOS, we run test suites for many FLOSS packages. We run PyPy's suite, disabling about a dozen tests that need network or filesystem or etc. We don't even *bother* with CPython's test suite; it is so buggy that we skip tests entirely for CPython.
<vstinner>
simpson: "it is so buggy that we skip tests entirely for CPython" which kind of issue do you have? i know better the Python 3 test suite, but on buildbots, tests became *very stable* since 2 years. I spent a significant time to make them more reliable
<cfbolz>
simpson: this is getting quite into general "gripes with cpython" territory, maybe we can stay a bit more pypy relevant again or take it out of the channel
<simpson>
cfbolz: Agreed. vstinner, we can use #nixos or #nixos-dev if you're interested in what happens if I enable the test suite. I think it's too many errors to count after 30min of CPU time.
marky1991 has joined #pypy
<simpson>
vstinner: I want to point out one meta-thing: Aside from arigato's recent email to python-dev and Steve's proposed PEP, I think that all of the same participants in this conversation brought up all of the same points last time that we chatted. We really *are* stuck here.
<vstinner>
simpson: sorry, i'm not really interested to dig into NixOS issues, but if they are issues, don't expect fixes in 2.7 branch which gets less and less fixes. we fixed some tests for reproducible builds recently.
<vstinner>
sorry for the noise :)
<vstinner>
arigato: "endless threads with no conclusion at the end" is an issue, but most threads get a solution. when no solution is found, it's usually because it's a hard issue only with drawbacks, and someone brave has to take the least worst option :)
<kenaan>
cfbolz py3.6 8e1f12b21406 /: try to update _ctypes_test.c
<cfbolz>
vstinner: let's just say that some people deal better with long discussion threads than others. and, given that even guido reached his limit, I'd argue that there is indeed a problem here
<marmoute>
Should we start using "Guido" as a unity for a discussion length ?
<marmoute>
"This thread is 0.1 Guido long", or "This persone can withold 3 Guido of discussion" ?
<kenaan>
cfbolz py3.6 40c7bca434b8 /: try to update testcapimodule.c
<arigato>
marmoute: .-)
kipras has joined #pypy
<vstinner>
arigato: ok. i read Steve's proposal and ... I don't really understand the purpose of these changes :-( it seems like his main concern is embedding Python. not really try to fix the C API
<cfbolz>
yay, bug in pypy's f-strings :-)
dante has quit [Quit: bye]
marky1991 has quit [Ping timeout: 250 seconds]
dmalcolm has quit [Remote host closed the connection]
<arigato>
cfbolz: heh, where? :-)
<cfbolz>
arigato: it's actually newformat.py
<cfbolz>
arigato: the padding computation uses number of bytes, instead of string length
<cfbolz>
so a utf-8 bug
dmalcolm has joined #pypy
<arigato>
so not my Perfectly Correct And Complete implementation of f-strings? :-) (I'm still impressed by CPython's really good tests for f-strings, I guess)
<cfbolz>
arigato: yes, it's a bug even on pypy2
<arigato>
(I'd have expected that by now we'd have gotten various bug reports about problems in corner cases, but no)
<cfbolz>
arigato: btw, our readline doesn't work with non-ascii chars
<cfbolz>
either a utf-8 bug, or it never did?
<arigato>
uh
<cfbolz>
yes, utf-8 it seems
<cfbolz>
works pre-merge
<arigato>
well, "works"
<cfbolz>
try typing 'ä = 1' into a prompt
<arigato>
for me typing "éèà" in my non-utf8 terminal gives a different result than in cpython
<cfbolz>
ah, yes, I fear you probably need a unicode terminal anyway :P
<arigato>
indeed, in a utf8 terminal it's highly broken
<arigato>
but it works in pypy3 6.0.0
<cfbolz>
yes
<cfbolz>
so I fear somebody needs to dig :-(
<arigato>
(neither version works in my latin1 terminal)
<vstinner>
arigato, fijal : i sent 3 emails to capi-sig
<vstinner>
Update on my work to reorganize header files. fijal: I explained that I fixed an old issue which will allow to move more functions to internal APIs. it wasn't really possible previously
<vstinner>
I replied to Steve's PEP. in fact, i'm not convinced by his PEP, i'm not sure of which kind of problem he tries to solve?
<vstinner>
and i proposed the outline of 4 PEPs to solve the "C API" problem
<vstinner>
fijal: you should like my "PEP A: Ecosystem of C extensions in 2019" :)
<vstinner>
arigato: i proposed "PEP D: Completely new C API" where we can put your PyHandle idea
<arigato>
cool
marky1991 has joined #pypy
marky1991 has quit [Ping timeout: 245 seconds]
speeder39_ has joined #pypy
Arfrever has joined #pypy
senyai has quit [Ping timeout: 268 seconds]
<arigato>
vstinner: note that my PyHandle idea can also be expressed as: don't change the C API at all except kill Py_INCREF(), replacing it with something else
<arigato>
well that and stop using "==" to compare two "PyObject *"
<Alex_Gaynor>
maybe extensions should be C++ so we can use RIAA for reference count management
<arigato>
no
<arigato>
or at least, that's a completely unrelated topic :-)
<Alex_Gaynor>
I'm mostly joking :-)
danieljabailey has quit [Ping timeout: 246 seconds]
forgottenone has joined #pypy
<cfbolz>
arigato: that's a serious bug somehow, I think
<cfbolz>
I can get things to crash with a MemoryError by typing unicode chars in pyrepl
marky1991 has joined #pypy
<arigato>
:-/
<cfbolz>
found a way to debug stuff somewhat though
<nanonyme>
arigato, is it possible to disable CFFI preprocessing? Looks like there's something recently implemented that's apparently doign a subset of what my simplepreprocessor can do but fails at some input so is emitting warnings
<nanonyme>
Eg it doesn't seem to like string literals which is totally silly
<arigato>
ah
<arigato>
well, it's giving a warning for a good reason
Taggnostr has quit [Ping timeout: 250 seconds]
<arigato>
it's not doing anything with them, it's just giving a warning when it detects them
<arigato>
#389
<arigato>
...is the number of the issue, if you want to know
Taggnostr has joined #pypy
<nanonyme>
IOW basically at the point when CFFI gets my sources, all macros are expanded already
<nanonyme>
(also my preprocessor detects which literals are okay and which are not)
<nanonyme>
Oh, that's an interesting bug
<nanonyme>
Anyhow. I intentionally feed 1:1 headers to CFFI in ABI mode. They may sometimes have constants which may be strings
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
<nanonyme>
arigato, any chance you could create a warning class so it would be easier for me to suppress those?
marky1991 has joined #pypy
<nanonyme>
We're currently getting that a lot and it's confusing people
<nanonyme>
Their first impression is something is wrong :)
<nanonyme>
I guess I could just suppress all warnings during processing the CFFI object but but...
<arigato>
well, you may also get random nonsense depending on what is in the strings, which is what the warning is really warning you about
<nanonyme>
My strings are fine
<arigato>
then it should be easy to remove them with a regexp, e.g. that would replace a string constant with 0 for example
<nanonyme>
Errr, doesn't that result in invalid C?
<arigato>
then ``char *c = 0;`` is valid and even compiles (0 means NULL) but you don't need it to compile anyway
<nanonyme>
Oh. Thanks. I guess I'll add a flag for that conversion into simplecpreprocessor then
<arigato>
you can replace the string with NULL or any undefined symbol or whatever
<nanonyme>
I already tokenize stuff and handle strings as single tokens, I can always replace then with 0
<nanonyme>
(or NULL)
<cfbolz>
arigato: the length is 2 instead
<nanonyme>
Yeah, I guess that's the best way forward. I could always add /* */ around the stuff but that implies manual work on headers and I'm trying to avoid that at work
<nanonyme>
arigato, thanks
senyai has joined #pypy
<cfbolz>
arigato: right, if I run it untranslated I get an interplevel assert