cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
jamesaxl has quit [Quit: WeeChat 2.1]
mattip has quit [Ping timeout: 276 seconds]
<kenaan> alex_gaynor alex_gaynor/remove-an-unneeded-call-into-openssl-th-1526429141011 7a78467a6e72 /lib_pypy/_cffi_ssl/_stdssl/__init__.py: Remove an unneeded call into OpenSSL This call doe...
<Alex_Gaynor> mattip_: ^ should be an easy review whenever you're up :-)
TheAdversary has joined #pypy
Hasimir has joined #pypy
marr has quit [Ping timeout: 260 seconds]
Hasimir has quit [Ping timeout: 240 seconds]
TheAdversary has quit [Ping timeout: 265 seconds]
oberstet2 has quit [Ping timeout: 255 seconds]
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
jcea has quit [Remote host closed the connection]
tbodt has joined #pypy
stevenja_ has joined #pypy
stevenja_ has quit [Remote host closed the connection]
TheAdversary has joined #pypy
Hasimir has joined #pypy
dfee has quit [Ping timeout: 260 seconds]
inhahe__ has joined #pypy
inhahe__ has quit [Ping timeout: 240 seconds]
dfee has joined #pypy
Hasimir has quit [Ping timeout: 240 seconds]
TheAdversary has quit [Disconnected by services]
TheAdversary has joined #pypy
Hasimir has joined #pypy
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
dfee has quit [Ping timeout: 240 seconds]
R3d_Sky has joined #pypy
<R3d_Sky> hello
<R3d_Sky> so i've been trying to run a discord (chat service) utility bot on pypy instead of CPython
<R3d_Sky> and the total memory allocated/resident mem is much higher than the memory reported as consumed
<R3d_Sky> GC used: 221.6MB (peak: 755.5MB); GC allocated: 1070.7MB (peak: 1168.3MB)
<R3d_Sky> is there any way to optimize memory usage?
<simpson> R3d_Sky: Hi. Yes.
<simpson> PYPY_GC_MAX can forcefully limit the footprint. If your bot can't fit inside your desired footprint, we might be able to help you optimize further.
<simpson> But just setting some firm boundaries can be extremely helpful.
<R3d_Sky> i wish there was an easy way to test at this scale, heh; prod has about 8.6k guilds, 143k channels and 300k total users
<R3d_Sky> restarting takes quite a while
<R3d_Sky> my test instance is only on one guild, so I can't replicate any of the memory usage stats
<simpson> Oh. You should probably have a testbed of some sort. I can't help you with that, although I *am* a production SRE looking for work~
<simpson> Aha. Well, you can at least get an idea of how various GC tweaks will perform as a smoke-test.
<R3d_Sky> that, and I have to move the image processing out of the main bot and into a separate API because that's probably using a lot of the memory
<R3d_Sky> i'm also rewriting the whole thing (there's a WIP repo with like 10% of the features) to make it easier to go through
<R3d_Sky> simpson: also, this is a purely hobby project, I'm a high-school going teen :P
<simpson> I know these feelings.
<simpson> Yeah, it can be hard to find time for side projects.
<R3d_Sky> holidays right now
<R3d_Sky> so I can spend all the time I need
R3d_Sky has quit [Quit: Textual IRC Client: www.textualapp.com]
energizer has quit [Remote host closed the connection]
energizer has joined #pypy
dddddd has quit [Remote host closed the connection]
<kenaan> wlav cppyy-packaging ba933501a318 /pypy/module/_cppyy/: add helper to extract outer namespace from a C++ name
<kenaan> wlav cppyy-packaging 10c6393b2cd5 /pypy/module/_cppyy/: pythonization improvements
mcyprian has joined #pypy
mcyprian has quit [Client Quit]
dfee has joined #pypy
user24 has joined #pypy
user24 has quit [Remote host closed the connection]
dfee has quit [Ping timeout: 260 seconds]
dfee has joined #pypy
tayfun26 has joined #pypy
xorAxAx has quit [Remote host closed the connection]
xorAxAx has joined #pypy
xorAxAx has joined #pypy
xorAxAx has quit [Changing host]
marself has joined #pypy
antocuni_ has joined #pypy
tsutsumi has quit [Ping timeout: 268 seconds]
kipras is now known as kipras`away
antocuni_ has quit [Ping timeout: 256 seconds]
wleslie has joined #pypy
xorAxAx has quit [Remote host closed the connection]
xorAxAx has joined #pypy
wleslie has quit [Quit: ~~~ Crash in JIT!]
xorAxAx has quit [Remote host closed the connection]
xorAxAx has joined #pypy
__peke__ has quit [Excess Flood]
__peke__ has joined #pypy
marr has joined #pypy
TheAdversary has quit [Ping timeout: 256 seconds]
Hasimir has quit [Ping timeout: 240 seconds]
amz3` has joined #pypy
stevenja_ has joined #pypy
stevenja_ has quit [Ping timeout: 240 seconds]
inhahe__ has joined #pypy
oberstet2 has joined #pypy
mattip has joined #pypy
lritter has joined #pypy
mattip has quit [Quit: bye]
mcyprian has joined #pypy
dfee has quit [Ping timeout: 265 seconds]
antocuni_ has joined #pypy
mattip has joined #pypy
<bbot2> Started: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/6760 [mattip: force build, py3.6]
<mattip> Alex_Gaynor: the pull request is simple, but the branch does not produce a binary tar.gz, the compress step fails
<mattip> so I cannot test it
mattip has quit [Remote host closed the connection]
jamesaxl has joined #pypy
energizer has quit [Ping timeout: 240 seconds]
dddddd has joined #pypy
antocuni_ has quit [Ping timeout: 256 seconds]
stevenja_ has joined #pypy
stevenja_ has quit [Ping timeout: 248 seconds]
mcyprian has quit [Ping timeout: 256 seconds]
dfee has joined #pypy
citizen_stig has joined #pypy
dfee has quit [Ping timeout: 276 seconds]
mcyprian has joined #pypy
<bbot2> Failure: http://buildbot.pypy.org/builders/own-linux-x86-64/builds/6760 [mattip: force build, py3.6]
demonimin has joined #pypy
R3d_Sky has joined #pypy
amz3` has quit [Ping timeout: 255 seconds]
amz3` has joined #pypy
jcea has joined #pypy
<R3d_Sky> so, I tried to run it on pypy, and the code that worked just fine on CPython, 2 hours later, occupied 100% of system memory and got killed by the oom reapoer
<R3d_Sky> (it = utility bot for chat service)
<R3d_Sky> a bug on my end or CPython's?
<R3d_Sky> it worked for about 24 days on CPython, crashed and burned in 6 hours on PyPy
marky1991 has joined #pypy
<R3d_Sky> seems like a PyPy thing
Hasimir has joined #pypy
jcea has quit [Remote host closed the connection]
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-app-level-linux-x86-64/builds/3960 [Carl Friedrich Bolz-Tereick: force build, py3.6]
antocuni_ has joined #pypy
jcea has joined #pypy
TheAdversary has joined #pypy
<kenaan> cfbolz py3.5 c697fd791c27 /pypy/objspace/std/test/: fix more raises/skip imports
<kenaan> cfbolz py3.5 e149edfccd49 /pypy/objspace/std/test/: merge
<Alex_Gaynor> mattip_: is there a better branch to send it to?
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-app-level-linux-x86-64/builds/3960 [Carl Friedrich Bolz-Tereick: force build, py3.6]
marky1991 has quit [Remote host closed the connection]
lazka has joined #pypy
TheAdversary has quit [Ping timeout: 240 seconds]
Hasimir has quit [Ping timeout: 240 seconds]
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-app-level-linux-x86-64/builds/3961 [Carl Friedrich Bolz-Tereick: force build, py3.6]
<kenaan> cfbolz py3.5 a6beb8e7ff9c /lib-python/conftest.py: add missing file to unbreak lib-python tests
<kenaan> cfbolz py3.6 455b5c86ee7b /: merge py3.5
TheAdversary has joined #pypy
Hasimir has joined #pypy
marky1991 has joined #pypy
<R3d_Sky> i'm still stuck on the GC issue
<R3d_Sky> it doesn't free the RAM and it results in an OOM kill
<fijal> R3d_Sky: what's the memory consumption of cpy process?
<fijal> seems like a pypy problem, yes
<fijal> we did add some tools for improving memory reporting
<R3d_Sky> fijal: consumes about 670M RSS
<fijal> yeah so should not be a problem
<R3d_Sky> (stabilized about that amount over 24 days)
<R3d_Sky> pypy OOM'd in 6 hours
<fijal> gc.get_stats()
<R3d_Sky> I have the stats from when pypy used about 1.4GB RSS, ~2 hours before crash
<R3d_Sky> give me a min
<fijal> and it OOMed using how much RAM?
<R3d_Sky> 2.5GB RSS when it was killed
<fijal> give me a second to think
<fijal> maybe bit more than a second :)
<fijal> R3d_Sky: we're now at a trade show (pypy unrelated), would you mind sending an email to pypy-dev? my main worry here is that arenas + rawmalloced does not add up to allocated - this is BAD I think
<fijal> where did it go?
<R3d_Sky> i have no idea
<fijal> R3d_Sky: I'm sorry, I didn't want to sound like I wanted you to answer :-)
<R3d_Sky> haha
<R3d_Sky> is oki
<fijal> I would like to know an answer, and yes would be lovely if you can give me one, but I can live with the fact that you might not know ;-)
<R3d_Sky> also of note fijal: this problem only showed up on my production instance
<R3d_Sky> testing didn't break at all
<R3d_Sky> it's probably because it's caching a much larger number of users
mcyprian has left #pypy [#pypy]
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-app-level-linux-x86-64/builds/3961 [Carl Friedrich Bolz-Tereick: force build, py3.6]
__citizen_stig has joined #pypy
marky1991 has quit [Ping timeout: 264 seconds]
stevenja_ has joined #pypy
citizen_stig has quit [Ping timeout: 265 seconds]
__citizen_stig has quit [Ping timeout: 240 seconds]
<fijal> R3d_Sky: is there any chance to get more than one dump of that stats?
citizen_stig has joined #pypy
<R3d_Sky> fijal: besides breaking production once again to get a dump, not really
stevenja_ has quit [Ping timeout: 255 seconds]
citizen_stig has quit [Ping timeout: 256 seconds]
<bbot2> Started: http://buildbot.pypy.org/builders/pypy-c-app-level-linux-x86-64/builds/3962 [Carl Friedrich Bolz-Tereick: force build, py3.6-wordcode]
<bbot2> Failure: http://buildbot.pypy.org/builders/pypy-c-app-level-linux-x86-64/builds/3962 [Carl Friedrich Bolz-Tereick: force build, py3.6-wordcode]
R3d_Sky has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
R3d_Sky has joined #pypy
R3d_Sky has quit [Quit: Textual IRC Client: www.textualapp.com]
citizen_stig has joined #pypy
citizen_stig has quit [Remote host closed the connection]
citizen_stig has joined #pypy
citizen_stig has quit [Ping timeout: 248 seconds]
citizen_stig has joined #pypy
citizen_stig has quit [Client Quit]
tayfun26 has quit [Remote host closed the connection]
Rhy0lite has joined #pypy
marself has quit [Ping timeout: 255 seconds]
<kenaan> antocuni gc-more-logging 42a01bd21795 /: close merged branch
<kenaan> antocuni default 49ede3b5afe6 /rpython/memory/gc/incminimark.py: merge the gc-more-logging branch, which logs some extra gc-minor and gc-collect-step info in the PYPYLOG
raynold has joined #pypy
amz3` has quit [Quit: WeeChat 1.9.1]
antocuni_ has quit [Ping timeout: 240 seconds]
oberstet has joined #pypy
oberstet2 has quit [Ping timeout: 256 seconds]
dfee has joined #pypy
tbodt has joined #pypy
marself has joined #pypy
lritter has quit [Quit: Leaving]
Hasimir has quit [Ping timeout: 240 seconds]
TheAdversary has quit [Ping timeout: 265 seconds]
marky1991 has joined #pypy
energizer has joined #pypy
energizer has quit [Remote host closed the connection]
energizer has joined #pypy
lazka has quit [Quit: Leaving]
jaffachief has joined #pypy
stevenja_ has joined #pypy
Rhy0lite has quit [Quit: Leaving]
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
nunatak has joined #pypy
jaffachief has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
jaffachief has joined #pypy
jaffachief has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
stevenja_ has quit [Remote host closed the connection]
nunatak has quit [Quit: Leaving]
jaffachief has joined #pypy
jaffachief has quit [Client Quit]
marky1991 has quit [Ping timeout: 248 seconds]
jaffachief has joined #pypy
<dfee> i've written a package that relies on dict being ordered by default. so it's compatible with python3.6, and pypy3.5 how can i specify in my setup.py that CPyton3.5 is excluded?
<simpson> dfee: I'm not sure. Have you considered changing your package to explicitly use collections.OrderedDict instead? It'll get you correct behavior on earlier CPython too.
jamesaxl has quit [Ping timeout: 268 seconds]
jaffachief has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<dfee> simpson: yeah the only problem is that i'm relying on call argument order
<dfee> i.e. `def func(*args, **kwargs)` where kwargs is ordered
<simpson> dfee: I'm so sorry.
<dfee> simpson: ?
<simpson> Not in the "I broke your vase" sense, in the "I sympathize with you on the matter of your broken vase" sense.
<lesshaste> when you run scipy.test('full') where are some tests skipped?
<lesshaste> I meant.. why are
lesshaste has quit [Remote host closed the connection]
lesshaste has joined #pypy
tbodt has joined #pypy
jamesaxl has joined #pypy
jamesaxl has quit [Ping timeout: 260 seconds]
jamesaxl has joined #pypy
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tbodt has joined #pypy
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tbodt has joined #pypy
marself has quit [Ping timeout: 264 seconds]
wleslie has joined #pypy
energizer has quit [Quit: Leaving]
jamesaxl has quit [Quit: WeeChat 2.1]
tbodt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tbodt has joined #pypy
irclogs_io_bot has quit [Ping timeout: 250 seconds]
irclogs_io_bot has joined #pypy
wleslie has quit [Quit: ~~~ Crash in JIT!]