<njs>
bonus: it's actually part of RLock's public API that it's guaranteed to be implemented in C, so even if all of eventlet's hacks worked perfectly, they would still be broken
inhahe has quit [Read error: Connection reset by peer]
<kenaan>
mattip buildbot 50073fc35d4e /bot2/pypybuildbot/arm_master.py: disable here too
<kenaan>
mattip buildbot 726d580f241a /bot2/pypybuildbot/arm_master.py: missed these too
bbot2 has joined #pypy
antocuni has quit [Ping timeout: 240 seconds]
<arigato>
this line fails an assertion, when run in pyinteractive.py
<arigato>
assert not isinstance(ctx, AbstractMatchContext)
<arigato>
this doesn't make sense, because 'ctx' is an AbstractMatchContext
<mattip>
yeah, it needs that for translation
<mattip>
but breaks untranslated
<arigato>
sorry, I even remember you adding this line, but I didn't connect the dots at the time
<mattip>
there are untranslated test failures
<arigato>
for translation it might as well be "assert False":
<arigato>
it "fixes" translation by making sure the rest is not translated
<mattip>
assert we_are_not_translated() ?
<arigato>
I mean, that's a wrong fix
<arigato>
it's similar to saying "there is a line here that breaks translation, let's add "assert False" just before
<mattip>
agreed
<mattip>
like I said, can of worms
<arigato>
no, it's a simple fix
<arigato>
I documented it and I'm now testing it
<kenaan>
arigo default 8d12729e7465 /rpython/rlib/rsre/rsre_core.py: fix rsre_core.py (hard to test for, as it was working in non-translated tests but accidentally killing the second h...
<kenaan>
arigo pread/pwrite d417efd34983 /: close branch, which I think was abandoned
<kenaan>
arigo default f535756918e0 /rpython/rlib/rutf8.py: These functions are elidable
<kenaan>
arigo default 1ab6eb502ac8 /rpython/rlib/: Optimized version of next_codepoint_pos() without the JIT
jcea has joined #pypy
<Ninpo>
Any tricks for making things like os.walk faster in pypy? cpython 3.6 kicking its butt in my tests
<mattip>
arigato: thanks
dddddd has joined #pypy
sknebel has quit [Remote host closed the connection]
sknebel has joined #pypy
<Ninpo>
mattip: unicode in with 3.6 yet?
<Ninpo>
holy _shit_. os.walk over a bunch of directories/files that creates a 24k entry set of full file paths, cpython3.6 and pypy3.5-7.0 both take between 45-50 seconds to complete a 100 iteration timeit run. pypy3.5-7.1.0 with unicode takes _33 seconds_
<Ninpo>
mattip: you've seriously gotta get this released! The improvements across the board are insane.
<mattip>
Ninpo: hopefully the nightly will build
<kenaan>
mattip py3.6 bf156a807410 /rpython/rlib/: merge default into py3.6
siddhesh has joined #pypy
siddhesh has quit [Remote host closed the connection]
lritter has joined #pypy
demonimin has quit [Ping timeout: 246 seconds]
demonimin has joined #pypy
demonimin has joined #pypy
<Alex_Gaynor>
arigato: It looks like there's an issue with cffi 1.12 using python3x.dll on Windows -- that file doesn't exist on python 3.4
<arigato>
OK, but libpython3.so somehow exists on Posix on python 3.4?
<Alex_Gaynor>
I guess. I didn't dig too far. /cc reaperhulk
beystef has quit [Quit: beystef]
demonimin has quit [Ping timeout: 246 seconds]
Rhy0lite has joined #pypy
antocuni has joined #pypy
WGH has quit [Read error: Connection reset by peer]
marvin has quit [Remote host closed the connection]
marvin has joined #pypy
marvin has quit [Remote host closed the connection]
marvin_ has joined #pypy
marvin___ has joined #pypy
marvin_ has quit [Remote host closed the connection]
marvin_ has joined #pypy
marky1991 has joined #pypy
marky1991 has quit [Ping timeout: 245 seconds]
jacob22__ has joined #pypy
fryguybob has joined #pypy
jcea has quit [Quit: jcea]
WGH has joined #pypy
jacob22__ has quit [Ping timeout: 240 seconds]
<cfbolz>
arigato: nice optimizations
marky1991 has joined #pypy
jacob22__ has joined #pypy
Zaab1t has joined #pypy
antocuni has quit [Ping timeout: 240 seconds]
illume has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
marvin_ has quit [Remote host closed the connection]
marvin_ has joined #pypy
marvin_ has quit [Remote host closed the connection]
92AABJD9Z has joined #pypy
92AABJD9Z has quit [Remote host closed the connection]
marvin_ has joined #pypy
marvin_ has quit [Remote host closed the connection]
marvin has joined #pypy
marvin has quit [Remote host closed the connection]
marvin has joined #pypy
marvin has quit [Remote host closed the connection]
marvin___ has quit [Remote host closed the connection]
marvin has joined #pypy
marvin___ has joined #pypy
marvin___ has quit [Remote host closed the connection]
marvin__ has joined #pypy
marvin__ has quit [Remote host closed the connection]
marvin__ has joined #pypy
zmt01 is now known as zmt00
marvin__ has quit [Remote host closed the connection]
marvin__ has joined #pypy
lritter has quit [Ping timeout: 272 seconds]
marvin__ has quit [Remote host closed the connection]
jacob22__ has quit [Remote host closed the connection]
jacob22__ has joined #pypy
marky1991 has joined #pypy
marky1991 has quit [Remote host closed the connection]
marky1991 has joined #pypy
Rhy0lite has quit [Quit: Leaving]
<Ninpo>
Does anyone have any pointers for interfacing with libc with CFFI?
<Ninpo>
All I can find are ctypes examples
<Ninpo>
I want to poke at inotify stuff
<Ninpo>
I can't figure out what I need to pass to FFI() and what not, docs almost look like I need to copy/paste the C from inotify in but that can't be right?
<cfbolz>
Ninpo: that's correct
<Ninpo>
oh
<Ninpo>
So I have to go rummage in the inotify libc source and copy/paste things from it? Why?
<cfbolz>
Ninpo: in the .h file
<Ninpo>
Oh I just need to drag in the relevant headers?
<Ninpo>
cfbolz: any good examples anywhere you know of? The docs are a tad over my head and quite dry
<fijal>
Ninpo: 0x871200001200
<Ninpo>
?
<fijal>
that's a pointer
<fijal>
or a more serious note - you need to copy paste C because there are different flavors
<fijal>
typically what you want to pass is something from a man page
<fijal>
int (*compar)(const struct dirent **, const struct dirent **));
marky1991 has quit [Ping timeout: 268 seconds]
<fijal>
that means you need to add "struct dirent { ...; };"
<fijal>
because it has fields, but you don't care about any single one
<fijal>
while the C header will contain the fields that are not public and not really relevant
<Ninpo>
Why does that mean I need to add struct dirent onwards...how do I know I don't care about scandir(const char etc?
<fijal>
you don't care about the details of dirent struct
<fijal>
(what fields it has, what size etc.)
<fijal>
you might care about scandir
<fijal>
how do you know - unfortunately by carefully reading the man page
<Ninpo>
but you said that means I need to add "struct dirent {
<fijal>
yes, struct dirent { ...; };
<fijal>
I mean literally ...
<Ninpo>
That _exactly_?
<Ninpo>
oh
<fijal>
which means "please figure out details on your own"
<fijal>
and unfortunately which details are relevant, which are not is not specified anywhere but in those man pages (which are not robot-parseable)
<Ninpo>
So I've been poking at what other people have done and one cffi interface into inotify (what I am currently interested in) has the stuff from the man page (just looked thanks to your hint) AS WELL AS stuff copy and pasted in from the header file
<Ninpo>
I'm looking at man inotify atm and man inotify_init1 and the stuff in the man page is as you describe it
<Ninpo>
Which looks a lot more straight forward :D
<fijal>
right so which part you are confused about?
<Ninpo>
Well going by what you said about I need to use what's in the man page...if I do that, I'd only include lines 13 to 19 there in that file
<Ninpo>
But that source has the contents of inotify.h in it too
<fijal>
you need all the functions and macros you gonna use
<fijal>
note the ... which means not having the exact numeric value
<fijal>
I *presume* someone uses those constants somewhere
<Ninpo>
Oh so I do need all the header file stuff as well?
<Ninpo>
That's not in the man page :|
jacob22__ has quit [Quit: Konversation terminated!]
<fijal>
if it's not in the man page how are you supposed to use it?
<fijal>
you need all the things you are going to *use* which usually means all the things in the manpage
agronholm_ has quit [Read error: Connection reset by peer]
<fijal>
how is this not in the manpage? it has a massive section on inotify events
<Ninpo>
It's telling me about them but the only copy-able bit is the struct inotify_event, I thought that's what you meant
<Ninpo>
So what I _actually_ want is to find header references to anything mentioned in the man page I want to make use of..?
<Ninpo>
and copy those...but cffi lets me use ... to go find and figure stuff out
<Ninpo>
Like that source doesn't specify all the hex/octal mask values
<fijal>
yeah, precisely
<Ninpo>
ok, starting to make sense now...last dumb question, I don't see anywhere in that file to tell it libc.so.6 or whatever, is that assumed?
<fijal>
will gcc know it's libc.so.6?
<Ninpo>
I don't know :|
<fijal>
it should!
<Ninpo>
I see ok.
<Ninpo>
Thank you
<fijal>
so the way it works is gcc is compiling (or cc whatever) a small thing with the default linker settings
<fijal>
if the default linker settings work, you should be fine
<fijal>
if they don't, you have the same problem as invoking a C compiler and you have to specify it somewhere
<Ninpo>
This feels much less daunting now, thanks for helping me get my head around it
<Ninpo>
trying to take steps to be more than just someone that hacks together bits of python using Everyone Else's code, get more understanding for how things actually work
<Ninpo>
Something that uses inotify with cffi (because yay pypy) and has some actual operations built on the events on top
<fijal>
pleasure
<Ninpo>
To the Pycharm! nananananananana
<fijal>
Ninpo: always happy to help
<Ninpo>
tbh people like you/the people here the general python community are why I stuck at python in the first place
<fijal>
thanks, means a lot
<Ninpo>
I've learned so much and keep learning in huge part to the people involved
agronholm has joined #pypy
user24 has quit [Remote host closed the connection]
Ai9zO5AP has quit [Quit: WeeChat 2.3]
<cfbolz>
fijal: I am pondering why utf-8 is better than CPython's flexible string representations
<fijal>
cfbolz: because of stuff like that 👍
<fijal>
?
<cfbolz>
yes, but in the flexible string representations you can *also* represent the string as utf8
<fijal>
we did some measurements for things like barrack obama wikipedia article
<fijal>
right, but you consume all the memory while at it, right?
<cfbolz>
well, you can also *just* have it in utf-8 and not one of the others
<fijal>
what can you do with it then?
<cfbolz>
so I think the real value comes from being forced to go through all operations and really trying to express them as operating on utf-8
<fijal>
so wait
<fijal>
if you call foo.decode("utf8") you get what?
<fijal>
a PyUnicodeObject with none of the fields filled? I somehow doubt that
<cfbolz>
the utf8 field filled, I think
<tos9>
wait so for unicode being fast we get to say... "thanks obama"?
<cfbolz>
fijal: I should poke around in GDB, I suppose
<fijal>
tos9: if this is what you want to take from this story, sure....
<tos9>
fijal: hey I always get the important bits.
<fijal>
cfbolz: from the C API it would seem as one of the byte representations has to always be there
<cfbolz>
yes, was my first impression from reading the PEP too
<fijal>
but is it not the case?
<cfbolz>
don't know for sure yet
<fijal>
ok
agronholm has quit [Read error: Connection reset by peer]
<cfbolz>
fijal: in a way I might not even care, if we can measure we are faster :-P
<fijal>
heh :D
<Alex_Gaynor>
fijal: I haven't looked at the branch how do we do `unicode.__getitem__` ?
<fijal>
Alex_Gaynor: we create an index (lazily) that indexes every 4th codepoint
<Alex_Gaynor>
fijal: ah ok, not too bad
<fijal>
(I think it's 4th)
<Alex_Gaynor>
fijal: if the string happens to all be single byte codepoints, is there a special case?
<fijal>
yes
<fijal>
every 64th it seems
<Alex_Gaynor>
there's also some cool vectorized bits, right?
<fijal>
ah, no it's more sophisticated
<fijal>
it's every 64 codepoints we have a full int and then we have byte offsets
<fijal>
Alex_Gaynor: no
<fijal>
there can be!
<fijal>
but not implemented yet
<fijal>
I *think* all of that ended up not really showing up in any benchamrks
<Alex_Gaynor>
too bad
agronholm has joined #pypy
<fijal>
turns out in most cases arithmetic is free
<fijal>
except parsing JSON ;-
kipras has joined #pypy
<cfbolz>
Alex_Gaynor, fijal: I am fairly convinced I can get a bit of speedups with vectorization, after my json experiments
<cfbolz>
but we should benchmark and profile first
Zaab1t has quit [Quit: bye bye friends]
Zaab1t has joined #pypy
jacob22__ has joined #pypy
<reaperhulk>
arigato, Alex_Gaynor: in Python 3.4.4 (the last non-source-only release for Windows) there is no python34.dll at all. python3.dll exists in the DLLs directory, but not in the top level python dir. In Python 3.5 and above python3.dll and python3x.dll both appear in the top level python dir.
<reaperhulk>
I have no idea if that is by design or an issue that was never corrected with the 3.4 windows releases
<reaperhulk>
but it means that right now if you build a package on windows with cffi 1.12.0 + python 3.4 it won't work
<reaperhulk>
oh lol, it installs itself in system32!
<reaperhulk>
but virtualenv isn't grabbing a copy from there
<reaperhulk>
I don't understand how dll resolution occurs in Windows so I'm not sure where the bug is. Is this a virtualenv bug for not copying python34.dll from system32 when in the presence of a python3.4 interpreter? Or is it a cffi issue?
<reaperhulk>
(If I copy python34.dll and python3.dll into a python 3.4 virtualenv it fixes the problem, suggesting this may be a virtualenv issue)
<reaperhulk>
Just to thicken the plot a bit more, python3.dll *is* copied into a venv using the venv module in python 3.4, but not python34.dll. On python 3.5+ both python3.dll and python3x.dll are copied in via venv.
<reaperhulk>
Okay, so this is a virtualenv bug.
<reaperhulk>
but cffi probably needs to avoid linking against python3.dll in python 3.4 now :(