<kenaan>
mattip default b746e83d4c54 /lib-python/2.7/: add some pieces needed to start building opencv-python with scikit-build
whitequark has joined #pypy
<whitequark>
i'm getting a crash (assertion failure) in: File "pypy_objspace_std.c", line 47561, in _string_to_int_or_long
<whitequark>
i believe it's related to literals with underscores and i'd like to fix it
<whitequark>
where do i start?
<Alex_Gaynor>
whitequark: generally you start by writing a unit test that can be run untranslated (where you get better tracebacks, easier to debug, faster dev cycle)
<Alex_Gaynor>
I don't know anythign about that part of the code, so I can't really suggest how you fix that
<Alex_Gaynor>
but based on where the assertion was, pypy/objspace/std/test/test_something.py is where the testcase goes
<Alex_Gaynor>
well, unless it's a parser bug (might be able to guess if there's a full stack trace) that happens to be using some shared function, then it'd go somewhere else
<whitequark>
the unit test is: 0xffff_ffff_ff20_0000
<whitequark>
(the entire thing)
<Alex_Gaynor>
just having that in source code?
<whitequark>
yep, a file with that one literal crashes
<whitequark>
it's in prev_digit which doesn't handle underscores
<whitequark>
i don't really understand why prev_digit is even needed
<whitequark>
but i can fix it i think
<Alex_Gaynor>
ok, so pypy/interpreter/ is where the tests for tha tgo
<whitequark>
well a) myhdl uses a fundamentally inferior approach in my opinion and b) i have a huge design in migen that i spent a year working on, i'm not going to rewrite it
<whitequark>
(full disclosure: i wrote the successor to migen. i think migen is one freakin good idea)
<mattip>
heh. myhdl is what got me involved in pypy, so I am partial to it. I kind of drifted away from that world though
<whitequark>
glasgow (the project i'm working on right now) heavily uses things like FSMs parameterized by data
<whitequark>
let me show you some representative example
<whitequark>
stuff like this isn't really amenable to myhdl's approach
<whitequark>
it could be made amenable if myhdl gained an unquote operation
<whitequark>
but at that point you're stuck maintaining a python parser and that's just a world of pain
<whitequark>
(i maintain a python parser, i would know)
<mattip>
ok. Is there a fundamental reason getting migen / glasgow to run on PyPy would be hard? It seems to be pure python
<whitequark>
there isn't. that's what i'm trying to do right now
<mattip>
cool
<whitequark>
the bytecode parsing stuff in migen is mostly required for debuggability, there's only 2 places in glasgow where it's required to function
<whitequark>
(and i might be able to fix that in nmigen)
<whitequark>
hm
<whitequark>
so glasgow does run on pypy
<whitequark>
with a 2 line patch besides the bugfix we discussed earlier
<whitequark>
unfortunately performance is really bad
<whitequark>
the 2nd command is actually limited by USB line rate
<mattip>
most of the core devs are on european time, maybe we can pull them in soon
<simpson>
The standard profiling tools work, as long as you don't need to peek into the (nonexistent) libpython.so.
<whitequark>
i'm theoretically in europe too i just dont sleep like a normal person
<whitequark>
hmm
<simpson>
e.g. I would reach for profile and pstats, if I didn't have any good ideas and I didn't know the codebase well.
<mattip>
better to use vmprof on pypy, less intrusive than cProfile, and it can show jit traces
<whitequark>
i'm not sure if i've ever profiled this before, getting it to that level of performance mostly meant finding and eliminating USB bugs and n^2 algorithms :D
<whitequark>
let's see what vmprof says
<mattip>
the "upload to our server" part does not work, but there is a local viewer
<simpson>
Sure. TBH I'm expecting that you'll find an infelicity not in your code, but in PyPy. It's (un?)common for folks to discover that some library method is poorly implemented.
<whitequark>
infelicity, heheh
<simpson>
Well, not a correctness bug so much as a slowly-implemented library routine.
<whitequark>
oh i've just never heard that word before
<whitequark>
it's a good word
<simpson>
It is! It's on my list of terrible baby names.
<whitequark>
is pypy expected to have a huge startup time?
<mattip>
is there a benchmark that just tests io over lib-usb?
<whitequark>
mattip: that's pretty much what that benchmark is
<whitequark>
it tests libusb, my async wrapper for libusb, and some extremely thin code for buffering
<mattip>
code paths need to run ~1000 times before they are jitted
<whitequark>
hmm so what you're saying i should run that test in a loop
<whitequark>
it looks like an FSM but that's misleading, what it does is it parameterizes over datapath width
<whitequark>
so you write a parser "as if" you're processing symbols one by one and it allows you to stick a gearbox in front of it and process them by two or by four
<whitequark>
this is especially interesting when you need to decide what to do e.g. for symbol 3 based on something you have in symbol 2 when parsing them in pairs
<whitequark>
you might have to use a register or a combinatorial circuit for deciding in that case
<whitequark>
this is handled transparently
<mattip>
looks cool
* mattip
off
<mattip>
let us know if you find anything
<whitequark>
i can't figure out how to run vmprof
<whitequark>
i install vmprof and there are no files in site-packages/...egg
<mattip>
did you use pypy's pip?
<whitequark>
pypy3.6 -m pip, yes
<mattip>
I see it in /path/to/pypy/site-packages/vmprof
<mattip>
and there is a vmprofshow in /path/to/pypy/bin
<whitequark>
so what i'm doing is $ pypy3.6 -m pip install --prefix .pypy3.6 vmprof
<whitequark>
this seems to have worked with setup.py
<whitequark>
no vmprofshow though
<mattip>
`pip install --user` I think
<whitequark>
i have my stuff in a custom prefix so it doesn't change my existing binaries in ~/.local/bin
<mattip>
dunno, it all depends what version of pip and what phase of the moon. I just use a virtualenv and rm it every week or so
<arigato>
note that LIBUSB_ERROR_INTERRUPTED is likely a bug inside this libusb, which doesn't correctly catch and retry on EINTR. E.g. if you resize your terminal window at just the wrong time, it would get this error too
<arigato>
you know the problem, right? on POSIX many OS functions can return EINTR if a signal happens to be received by the application while they are blocked in the kernel. The problem is that signals are rare, and so this kind of problem is kind of common. But a typical sampling profiler works by delivering regular signals