antocuni changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "PyPy: the Gradual Reduction of Magic (tm)"
danieljabailey has quit [Quit: ZNC 1.6.5+deb2build2 - http://znc.in]
<menip>
I'm trying to import a noise library. Using pypy. Getting ImportError: cannot import name '_perlin'. Could I get some help with what it means, and how to overcome?
<menip>
I see a _perlin.cpython-35m-x86_64-linux-gnu.so in the package directory, but no _perlin. Works fine with python3
<njs>
that's a C extension module, which needs to be built separately for each python interpreter you want to use it with
<njs>
right now it's only been built for CPython 3.5. (CPython is the name of the regular python interpreter you'd used to.) If you want to use it on pypy, or even CPython 3.6, then it needs to be built for them.
<njs>
Usually this is taken care of when you install the package into whichever virtualenv you're using -- are you directly manipulating PYTHONPATH or something?
<menip>
Yes, I am. noise isn't supported by pypy it seems like
<menip>
Unless I'm trying to install package incorrectly
<njs>
menip: how are you installing the package?
<njs>
if you are ending up with a file named something.cpython-something inside your pypy site-packages, then something is going very wrong with how you're installing
<menip>
Oh. I meant that yes, I did change PYTHONPATH. I checked http://packages.pypy.org/ and didn't find the noise package listed
<njs>
that doesn't necessarily mean anything -- pypy works on lots of packages that haven't been explicitly checked
<njs>
instead of messing with PYTHONPATH, try 'pypy -m pip install noise'?
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tbodt has joined #pypy
tbodt has quit [Client Quit]
kirma has quit [Ping timeout: 250 seconds]
asmeurer__ has quit [Quit: asmeurer__]
asmeurer__ has joined #pypy
<menip>
njs: Do I need to just have python-dev installed, or is there something special need to be done to have pypy know where it while compiling stuff?
<arigato>
antocuni: using malloc() seems to create a lot of fragmentation
<arigato>
we end up with a lot of arenas grouped in very small numbers, with some space in-between that is smaller than one arena
<arigato>
calling mmap directly bypasses this problem almost entirely
<antocuni>
ok
<antocuni>
so the end result is that top reports we are using more memory than we actually need?
<fijal>
maybe :-)
<arigato>
yes, I think so
jacob22__ has quit [Ping timeout: 248 seconds]
<arigato>
or rather, we have an example (hard to reproduce however) where top reports really too much, and this could be one of the reasons
<arigato>
fijal: no news about that?
<fijal>
arigato: will pester pjenvey tonight
<fijal>
not last time I managed to get hold of him
<antocuni>
"hard to reproduce" means that it happens only once in a while?
<arigato>
no, "hard to reproduce" means pjenvey reports it happens all the time, but can't just let us go there and look
<antocuni>
ah :(
<arigato>
I think that after months and months, fijal could log in, but it's a machine where you don't have development environment
<antocuni>
"hard" for very bad reasons
<arigato>
yes
<fijal>
antocuni: I think it was not successfully reproduced on test data
<antocuni>
"top reports really too much": how much?
<fijal>
unbound
<fijal>
if you leave it for longer, it'll be more
<antocuni>
ouch
Nizumzen has quit [Ping timeout: 260 seconds]
zmt00 has joined #pypy
tbodt has joined #pypy
<antocuni>
arigato, fijal: how long do you expect a gc-collect-step to take on a process which consumes ~1GB of ram
<antocuni>
I'm seeing pauses up to 300ms, which seems like a lot to me
<fijal>
eah should be shorter
<antocuni>
although the fact that it's caused by gc-collect-step it's just a hypotesis
<fijal>
in fact there shouldn't be pauses but we never split some of the parts
<fijal>
ah
<fijal>
it usually is the JIT compilation time, not the GC
<fijal>
because incremental
<fijal>
you can quite easily look into times of the GC by looking into PYPYLOG=log
<antocuni>
no, I have already excluded the JIT, I don't see any compilation
<fijal>
well check the GC and come back
<antocuni>
not so easy because it's a production environment, it's hard to set PYPYLOG, it's eventlet based so it's hard to know which gc-collect-step corresponds to each greenlet, etc. etc. :)
<fijal>
sure sure
<fijal>
but no you can't randomly assume stuff anyway :)
<antocuni>
what I see is that there is a greenlet which usually takes a short period of time to handle back control, but every ~2-4 minutes I see a peak of ~100-300ms
<antocuni>
I tried to force a gc.collect every 4 minutes
<antocuni>
and the peaks disappear
<antocuni>
that's why I pointed the finger against gc-collect-step :)
<fijal>
it still occurs to me that setting PYPYLOG is still much much easier than speculating what happened
<antocuni>
yes sure
<antocuni>
I'll do
<fijal>
there are a few steps that are not incremental, but it will not be easy to find out which one is it I think
<fijal>
(if it's the GC)
<antocuni>
I just wondered what is a reasonable time for it to take
<fijal>
reasonable is below 1ms
<fijal>
I've seen odd ones that are higher than 1ms, like 15ms
<antocuni>
ok, perfect
<fijal>
and I think 100ms once
<fijal>
but I never investigated
<fijal>
I did some statistics for GC pauses for translation
<antocuni>
anyway yes, inspecting PYPYLOG seems to be the thing to to next
<arigato>
I'm not sure, but on a 1GB process I'd expect full collections to occur much much more often than every 4 minutes
<antocuni>
arigato: consider that this is a process which is data-driven, so the rate at which it allocates stuff depends on what data it receives from the network
<antocuni>
and I think that in general the CPU is sleeping most of the time
<antocuni>
so "4 minutes" doesn't mean much
<antocuni>
do we have any hook similar to pypyjit.set_compile_hook but for the GC?
jamesaxl has joined #pypy
raynold has joined #pypy
adamholmberg has quit []
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
adamholmberg has joined #pypy
tbodt has joined #pypy
tbodt has quit [Client Quit]
tbodt has joined #pypy
asmeurer_ has joined #pypy
asmeurer_ has quit [Quit: asmeurer_]
antocuni has quit [Ping timeout: 255 seconds]
<pjenvey>
antocuni: should a "symbol-file /path/to/libpypy-c.so.debug" in gdb just work?
<pjenvey>
aw, missed him by 20 seconds
marr has joined #pypy
slacky has joined #pypy
drolando has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]