<kenaan_>
arigo cffi/cffi f06010593f74 /setup.py: List _cffi_errors.h here (oops, thanks gabriel-m)
<arigo>
yes, that's it. thanks a lot!
<gabriel-m>
I'm glad I could help :)
<gabriel-m>
arigo, do you have any idea why I get this: ImportError: unable to load extension module '/work/pypy/site-packages/wx/_core.pypy-41.so': /work/pypy/site-packages/wx/_core.pypy-41.so: undefined symbol: PyExc_ValueError
<gabriel-m>
I installed wxpython-phoenix
<gabriel-m>
and when I try to import wx I get that
amaury has quit [Ping timeout: 255 seconds]
<cfbolz>
gabriel-m: what's your pypy version?
<gabriel-m>
5.8.0
amaury has joined #pypy
<cfbolz>
sure? because the "pypy-41.so" is a bit suspicious, no?
<cfbolz>
PyExc_ValueError exists in current pypy :P
rokujyouhitoma has quit [Ping timeout: 255 seconds]
arigato has quit [Ping timeout: 258 seconds]
<arigo>
gabriel-m: sorry, missed what you said
<arigo>
"PyExc_ValueError" is a symbol that shouldn't be needed
<arigo>
it should be called "PyPyExc_ValueError"
<arigo>
do you get a compilation warning mentioning PyExc_ValueError? e.g. if the name is used in a file where "#include <Python.h>" is missing, you'd get a warning but it would work on CPython
egregius313 has quit [Remote host closed the connection]
rokujyouhitoma has joined #pypy
vkirilichev has quit [Remote host closed the connection]
<antocuni>
cfbolz, gabriel-m: I think that you get the .pypy-41.so extension because 4.1 it's the ABI version
vkirilichev has joined #pypy
<antocuni>
i.e., it didn't change since pypy 4.1; not sure if it's correct or not, though
egregius313 has joined #pypy
<cfbolz>
I suppose not (we have added a ton new features since then, no?)
<antocuni>
yes, likely
<antocuni>
I don't really know what are the requisites for being compatible
rokujyouhitoma has quit [Ping timeout: 260 seconds]
vkirilichev has quit [Read error: Connection reset by peer]
vkirilichev has joined #pypy
arigato has joined #pypy
<antocuni>
arigato: about cffi's package_data: you might be interested also in trying "include_package_data=True"; which, despite the name, does something different: in particular, it automatically includes all files tracked by hg/git
<ionelmc>
antocuni: it includes everything in the MANIFEST.in
<ionelmc>
for tracked files if you use something like setuptools_scm i guess
<antocuni>
ah ok
<antocuni>
I indeed do use setuptools_scm, but its docs does not mention anything about package data
<antocuni>
so I assumed it was a setuptools feature
<ronny>
antocuni: include_package_data is unrelated to file listers ^^ manifest.in is used as fallback
<ronny>
file listers list files for a sdist, package_data and include_package_data are intended for wheels, eggs and tradtional installs
<antocuni>
but how can you include them in the wheels if they are not in the sdist?
<ronny>
not
<ronny>
manifest.in or file listers put stuff into the sdist
<arigato>
I suppose there's a deep and mysterious reason for why there are several ways to list files
<ronny>
package_data/include_pacakge_data put things into bdists
<antocuni>
ronny: ok, got it (I think)
<ronny>
arigato: there is simply a setuptools entrypoint to register file listers, and manifest.in is the fallback
<ronny>
setuptools_scm includes file listers for the supported scms
<antocuni>
ronny: oh, I see now in the setuptools README: "It also handles file finders for the supported scm's."
<antocuni>
I must say that this info is not particularly prominent: yesterday I looked at the README to check whether setuptools_scm handles only the version or more and I concluded it was "only the version"
<antocuni>
that sentence is a bit obscure if you don't know what is a file lister :)
<ronny>
antocuni: yes, i#ll log an issue about documenting it comprehensible without prior knowledge
<antocuni>
I can open the issue on github, if you like
<ronny>
man, intestine infects really take one down
<ronny>
i#ll be afk again
egregius313 has quit [Remote host closed the connection]
<dstufft>
there are two concerns, what files get added to the sdist, and what files get installed. Typically you need files to be listed both places because if it's not added to the sdist, it can't then be installed later, but sometimes you want to add a file without installing it (doc files for instance)
<dstufft>
MANIFEST.in and setuptools_scm etc control what files get added to the sdist
amaury has quit [Ping timeout: 240 seconds]
<antocuni>
dstufft: thanks, now it's clearer :)
<dstufft>
(package_data, py_modules, package, data_files, include_package_data are all about controlling what files get installed)
egregius313 has joined #pypy
jamescampbell has joined #pypy
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 260 seconds]
Rhy0lite has joined #pypy
egregius313 has quit [Remote host closed the connection]
rokujyouhitoma has quit [Ping timeout: 240 seconds]
adamholmberg has joined #pypy
Ryanar has joined #pypy
mattip has joined #pypy
realitix has quit [Quit: Leaving]
<antocuni>
gabriel-m: I don't see anything enlightening in the logs
<antocuni>
ah now wait
<antocuni>
maybe I found it
<antocuni>
I see this line: "Checking for program /usr/bin/pypy-config,python2.7-config,python-config-2.7,python2.7m-config : /usr/bin/python2.7-config"
<antocuni>
which means that the build tool is looking for something like "pypy-config" but it doesn't find it
<antocuni>
and then it finds python2.7-config and assumes it is fine to use it... which is wrong
<antocuni>
so I bet that what happens is that the "#include <Python.h>" find the *CPython* header, not the pypy one
<antocuni>
I don't think that pypy provides a pypy-config tool... maybe we should
<antocuni>
you can also try to sketch one by yourself: put a script called /usr/bin/pypy-config, make sure that the build uses it, and try to return reasonable values
girish946 has joined #pypy
* arigato
tries to use cffi for embedding for real on Windows
<arigato>
I'm getting RuntimeError: recursion depth exceeded after getting a number of errors, so I *think* something is wrong (e.g. the recursion counter is not balanced)
<antocuni>
although for gabriel-m uses cases it's probably easier to write something ad-hoc
<squeaky_pl>
Yeah definitely, python-config has python-config [--prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help|--abiflags|--configdir]
larstiq_ has joined #pypy
<arigato>
ah bah
<arigato>
ok: the problem is, I invoke some C function, and that function indirectly causes a setjmp() that interrupts everything including my Python frames in the middle
egregius313 has joined #pypy
adamholmberg has quit [Remote host closed the connection]
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 276 seconds]
Ryanar has quit [Quit: Ryanar]
Ryanar has joined #pypy
tilgovi has joined #pypy
egregius313 has quit [Remote host closed the connection]
egregius313 has joined #pypy
egregius313 has quit [Ping timeout: 240 seconds]
<mjacob>
arigato: i didn't really progress with the lto debug information issue because i wasn't able to produce a binary that contained debug symbols for some reason
<mjacob>
arigato: i.e. with -flto -g seems to be ignored
<mjacob>
arigato: did you have the same problem?
tilgovi has quit [Ping timeout: 276 seconds]
<arigato>
no
<arigato>
did you really specify -g to both the compiling and linking steps?
<mjacob>
at first i forgot it on the linking step, but then i added it, cleaned the object files and tried again
<mjacob>
however the resulting binary was only a little bit larger and didn't have all debug symbols
<mjacob>
gcc loaded it in a few seconds
<arigato>
no clue
<mjacob>
i will certainly try it again with a fresh mind in the next days
<mjacob>
when i need a break from learning for exams ;)
<mjacob>
mhh, that's a contradiction
jneen_ is now known as jneen
tilgovi has joined #pypy
rokujyouhitoma has joined #pypy
cstratak has quit [Quit: Leaving]
girish946 has quit [Quit: Leaving]
antocuni has quit [Ping timeout: 255 seconds]
rokujyouhitoma has quit [Ping timeout: 240 seconds]
jamescam_ has joined #pypy
jamescampbell has quit [Ping timeout: 240 seconds]
jamescampbell has joined #pypy
jamescampbell has quit [Read error: Connection reset by peer]
jamescam_ has quit [Ping timeout: 240 seconds]
vkirilichev has quit [Remote host closed the connection]
vkirilichev has joined #pypy
vkirilichev has quit [Ping timeout: 240 seconds]
yuyichao has joined #pypy
yuyichao has quit [Quit: Konversation terminated!]
tilgovi has quit [Ping timeout: 240 seconds]
yuyichao has joined #pypy
tilgovi has joined #pypy
arigo has quit [Quit: Leaving]
tilgovi has quit [Ping timeout: 246 seconds]
marr123 has joined #pypy
marr has quit [Ping timeout: 258 seconds]
marr123 is now known as marr
lazka has joined #pypy
pilne has joined #pypy
oberstet has quit [Read error: Connection reset by peer]
<lazka>
hey, I've recently tried building pypy under MSYS2 for fun, but have given up for now. If someone is interested here is what I have: https://github.com/lazka/msys2-pypy-test
<squeaky_pl>
dstufft, how is warehouse doing on PyPy, did the latency numbers go down?
girish946 has joined #pypy
girishj has joined #pypy
girishj has quit [Read error: Connection reset by peer]
rokujyouhitoma has quit [Ping timeout: 268 seconds]
<dstufft>
squeaky_pl: looks like it's elevated, but I think it's not very indicative ATM. The memory is growing past 1GB in size, which it's currently running on a Heroku 1GB Dyno
froztbyt1 is now known as froztbyte
<dstufft>
unfortunately going beyond a 1GB dyno gets real expensive, like 3x more
oberstet has joined #pypy
girish946 has quit [Quit: Leaving]
rmesta has joined #pypy
rmesta has left #pypy [#pypy]
squeaky_pl has quit [Read error: Connection reset by peer]
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 268 seconds]
<benbangert>
dstufft: we see memory leaks as well, still adding more debugging to try and determine why
<dstufft>
benbangert: dunno if mine is a leak or just PyPy using a bunch of ram
<Alex_Gaynor>
benbangert: was going to offer my assistance, then I remembered pjenvey_ is lareayd on your team :-)
<dstufft>
soon we'll hopefully move off of Heroku onto a place where we can get more ram easily
<benbangert>
Alex_Gaynor: yea, pjenvey_ added more detailed tracking, but if there's more we're missing that could shed insight, that'd be great. especially since we started seeing larger and faster leak after switching to using cryptography for our VAPID auth (the PR you saw awhile back)
<Alex_Gaynor>
benbangert: my immediate instinct is that your problem is memory allocated external to the gc (e.g. when we let openssl malloc crap) not contributing to GC pressure which means it's not running as frequently as it should.
<Alex_Gaynor>
Basically all of the mmeory that OpenSSL allocates via cryptography is freed with ffi.gc(), so only free'd when the GC runs.
<Alex_Gaynor>
Now what would be the best way to test that... stick a `gc.collect()` in an itimer callback and see if it helps I guess
jamescampbell has joined #pypy
<pjenvey_>
I still need to narrow down exactly where the memory allocations are, but some initial runs suggest it's just the GC
<pjenvey_>
if that's correct (even if it's not) one thing pypy could benefit from is extended debug statistics about the GCs underlying memory usage
<Alex_Gaynor>
pjenvey_: gc.dump_rpy_heap() (might not be the right name) can be used to do this FWIW.
<pjenvey_>
dump_rpy_heap reports on all known objects and their sizes and totals
<Alex_Gaynor>
Right, you can sort of manually compare this with RSS.
<pjenvey_>
however the GC allocates in larger chunks: arenas/pages, you basically want dump_arena_info
Ryanar has quit [Quit: Ryanar]
<pjenvey_>
Alex_Gaynor: situation I have currently is dump_rpy_heap size dramatically smaller than RSS
<Alex_Gaynor>
pjenvey_: That'd be expected if it's memory allocated external to the gc, e.g. by openssl
<Alex_Gaynor>
I'm not really sure what the right way to fix this would be. Maybe there's some hook we could replace OpenSSL's malloc with ffi.new() or something that the GC knows about, but I'm loath to do that, since it puts us in an awkward place if we're sharing the OpenSSL .so with someone else
<pjenvey_>
and what looks like (somewhat conjecture here) potentially large arenas lying around (from pmaps info) larger than dump_rpy_heap sizes
<pjenvey_>
so I'm somewhat doubting its openssl, but I'm not ruling it out yet
jamescampbell has quit [Remote host closed the connection]
rokujyouhitoma has joined #pypy
black_ant has joined #pypy
rokujyouhitoma has quit [Ping timeout: 260 seconds]
kipras`away is now known as kipras
vkirilichev has joined #pypy
<benbangert>
Alex_Gaynor: I was wondering what else could be built into PyPy to facilitate tracking and reporting. ie, how to make up the gap between information one can get about GC/object/memory-usage in pypy vs. the JVM
<Alex_Gaynor>
pjenvey_: so testing with this, the symtoms I see is that with `gc.collect()` the max resident memory is like 50% of without it, both seem to stablize, not leak indefeinitely. feels like this confirms my hypothesis https://www.irccloud.com/pastebin/XjoVLGhy/t.py
<Alex_Gaynor>
benbangert: Easier on the JVM, since there's basically no external allocations, which appear to be the issue here :-/ Not sure, will have to think.
jamescampbell has joined #pypy
jamesaxl has quit [Ping timeout: 255 seconds]
<pjenvey_>
the memory I'm seeing could certainly be from malloc vs GC's
<pjenvey_>
Alex_Gaynor: what version of pypy is that
<Alex_Gaynor>
pjenvey_: 5.8.0
<Alex_Gaynor>
2.7
<pjenvey_>
5.7 started adding memory pressure for each cffi alloc
egregius313 has joined #pypy
<Alex_Gaynor>
pjenvey_: these aren't cffi allocs though, they're OPENSSL_malloc allocs
jamesaxl has joined #pypy
<Alex_Gaynor>
we call, say, X509_new() and then the malloc itself happens inside OpenSSL
<pjenvey_>
Alex_Gaynor: aaah
<pjenvey_>
cryptography needs to add memory pressure then
<pjenvey_>
like everywhere
<Alex_Gaynor>
That does not feel like a tractable solution; though I can probably hack up a prototype easily that'd make it work :-/
Ryanar has joined #pypy
<Alex_Gaynor>
let me do the hack and see if it works
<pjenvey_>
it's a bit horrible but you're responsible for those allocations so..
<pjenvey_>
I dunno if cffi could really help here
pjenvey_ is now known as pjenvey
<Alex_Gaynor>
pjenvey: hack appears to work; installs custom malloc into OpenSSL which adds memory pressure. That gibberish has to be run very first thing on process start, before anyone else imports cryptography, hashlib, or ssl. https://www.irccloud.com/pastebin/FXmZ6Ucn/t.py
<pjenvey>
Alex_Gaynor: maybe you want ffi.new_allocator ?
<pjenvey>
Alex_Gaynor: oh boy, awesome
<Alex_Gaynor>
pjenvey: do I? what can I do with ffi.new_allocator
<Alex_Gaynor>
pjenvey: want to move to #cryptography-dev to involve other crypto devs?
jamescampbell has quit [Remote host closed the connection]
amaury has joined #pypy
antocuni has joined #pypy
adamholmberg has joined #pypy
adamholmberg has quit [Ping timeout: 255 seconds]
rokujyouhitoma has joined #pypy
jamescampbell has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
vkirilichev has quit [Remote host closed the connection]