<mjacob>
in function pypy_g_tuple_alloc, there's a line "RPY_REVDB_CALL(l_result_713 = PyPyTuple_New(l_itemcount_8);, PyObject *_e, l_result_713);"
<mjacob>
only in record mode, PyPyTuple_New (which is written in C) is actually called
<mjacob>
the call chain goes like this: PyPyTuple_New -> _PyPyObject_GC_NewVar -> _PyPyObject_NewVar -> _generic_alloc -> _PyPyPy_Malloc -> pypy_g__ll_malloc_varsize_no_length_zero__Signed_Signed
<mjacob>
the latter one has a line "RPY_REVDB_EMIT(OP_RAW_MALLOC(l_tot_size_0, 1, l_result_905);, void* _e, l_result_905);", pushing an extra word to the log
<mjacob>
since in replay mode the call to PyPyTuple_New is not actually made, it doesn't know about the extra word
<mjacob>
if i understand correctly, in record mode it should be noted in the log that the call back to rpython was made
<mjacob>
for some reason, this information is missing
<mjacob>
i'm translation with --no-shared, BTW
<mjacob>
both in the default and the py3.5-reverse-debugger branch, the issue seems to be present -- it can be triggered by importing cpyext
<mjacob>
it seems like cpyext is broken with revdb since a long time (maybe 6d880cc2b4 broke it; c_only doesn't add a stack bottom and therefore revdb doesn't know it's a callback)
<zaytsev>
mattip: so first re. pypy-2.7. you told me to remove vc90, although you maybe didn't mean to, and so i did, so of course it doesn't work anymore. i've just reinstalled it. i hope this takes care of this problem.
<zaytsev>
mattip: i have also upgraded visual studio installer to the latest version, and removed 14.1 leaving only 14.0, which i understood is what you want
<mjacob>
arigato: i'm going back to release_gil issue in the py3.5 branch. i observe that the addresses of some objects are off by 4096 bytes between record and replay. is that a problem?
<mattip>
zaytsev: thanks, sorry for any confusion
antocuni has quit [Ping timeout: 246 seconds]
jcea has quit [Quit: jcea]
kipras has joined #pypy
oberstet has quit [Remote host closed the connection]
lritter has quit [Ping timeout: 268 seconds]
dddddd has joined #pypy
<zaytsev>
mattip: alright, py2.7 is working again so it seems. let me try py3.5 with vc 14.0.
<kenaan>
mattip py3.5 49aade4a2c18 /rpython/translator/platform/windows.py: merge default into branch
<arigato>
mjacob: thanks for debugging revdb! you're right, that's probably exactly the problem
<arigato>
mjacob: yes, the addresses should be the same during record and replay
<mjacob>
on py3.5, boehm returns pointers which are slightly off between record and replay
<mjacob>
i don't really have an idea how to debug
<zaytsev>
mattip: at least it started translation now, so apparently it found the compiler after i deleted the variable. i'm curious if it will be able to compile the rest after the translation is over
<arigato>
mjacob: ah, boehm malloc()? right, no, that's not a problem
<arigato>
what must not change is the address of static objects, like functions or global variables
<mjacob>
i guess you implemented a hack to ensure stable hashes?
<arigato>
yes: the id() function(s) in RPython are implemented differently than directly returning the address of a GC object
<arigato>
they return the 64-bit unique object id, which increments from 1, 2, 3... and is never reused
<arigato>
we store this 64-bit number inside the object, both during recording and replaying
<arigato>
(and we make sure there is no cast between a GC address and a non-GC "void *" or integer
<arigato>
)
<mjacob>
i see
<mjacob>
i'm unsure what's the best fix the missing callback information on default
<mjacob>
marking it as a GC bottom would probably be the easiest way
<arigato>
yes
<arigato>
but it's a bit obscure
<arigato>
how does it work in a regular pypy?
<mjacob>
the code inside c_only doesn't trigger the GC
<mjacob>
it only calls raw malloc() and free()
<arigato>
ah I see
<arigato>
OK, which is an optimization we did at some point
<mjacob>
so it shouldn't really be a GC bottom; we just want revdb to know it's a callback
<arigato>
there's also a problem at the call to PyPyTuple_New
<arigato>
I think it must know it's a call that can possibly invoke a callback
<arigato>
well, wait a sec
<arigato>
...ok yes, another solution might be that the calls to things like state.C.PyTuple_New are done in such a way that they disable all recordings
<arigato>
so even if there is a op_raw_malloc inside, it's not recorded
<mjacob>
as far as i remember, that's not that easy
<zaytsev>
mattip: well, now that part of the log loaded, bad news :( it picked up vc90, which i understand is NOT what you wnat?
<zaytsev>
mattip: http://doc.pypy.org/en/latest/windows.html - "They will attempt to locate the same compiler version that was used to build the Python interpreter doing the translation." - bingo? pypy 5.7.1 was compiled with vc90, wasn't it?
<arigato>
mjacob: I think that these things like state.C.PyTuple_New are special, in the sense that they should not do any GC at all (as you said) or release the GIL or in general do anything, but they can still call back some rpython code that doesn't do any of these things
<arigato>
we need to check if we can add "don't mutate GC objects" to the list
<mjacob>
would that result in the operations inside the callback not being recorded?
<arigato>
yes
<arigato>
ah, "@c_only" is only used for _PyPy_Malloc and _PyPy_Free
<arigato>
then yes, it can be ignored
<arigato>
note that a raw malloc is usually recorded as "this returned the address 0x12345678" and replayed as "return 0x12345678 without actually allocating anything"
<arigato>
so if we don't record them and don't call the state.C.PyTuple_New at all during replaying, then everything is fine
<arigato>
(I'm only 90% sure of that sentence :-)
<mattip>
zaytsev: don't always believe the documentation :)
<arigato>
...one of these days I should look at "./rpython --gc=boehm", which crashes at the moment
<zaytsev>
mattip: so anyways, we have established that it picked 14.0 only because the variable was present. now that you've told me to delete it, it can't find it anymore and picks vc90. i assume this is NOT what you want. however, after i removed vc 14.1 as you told me and updated msvs / 14.0 to the latest version, the contents of the wariable is wrong
<zaytsev>
mattip: so what do we do now, set the variable but with some other content :) ?
<mattip>
the easiest would be to run `python pytest.py rpython/translator/platform/test/test_makefile.py -k test_simple`
<mattip>
stop in a debugger and see what is going wrong
<mattip>
but since that is not possible, I will try to add more printing/start build/stop build/ till something makes sense
<mattip>
zaytsev: could you install the build tools for 14.1?
<zaytsev>
mattip: your requests sound a bit like recursion to me, but alright, i'll try to and let you kno
<zaytsev>
i ran the test from the msvc prompt and it picks up vc90
<mattip>
ahh, hang on. I see the other build bot has
<zaytsev>
mattip: i've just ran the installer again and to my amazement the list of components looks entirely different from the last time, wtf
<zaytsev>
mattip: could you please tell me what do you want to have? i assume it did exactly the opposite of what i wanted, and specifically, removed 14.0 and isntalled 15.0 ?
<zaytsev>
mattip: you want 2015.3 and don't want 2017, is that right?
<zaytsev>
mattip: okay i can check 14.0 tolls no problem. but if i uncheck 14.1, then it says that's requiired by VC ++ build tools and tools for Cmake. do you want to proceed?
<zaytsev>
mattip: okay so what i have is already correct :(
<zaytsev>
let me try to set the variabvle
<mattip>
+1
<zaytsev>
mattip: by the way, i understand you're gonna come to ddorf. it's around 1 hours drive for me. i won't have time to come to the sprint but on thursday or tuesday i could come if you are going to have some sort of social event / wanna meet me
<zaytsev>
mattip: it seems that the variable magick did the trick it says picked up vsver 140 (?! which is actually not the case, because ti's 141 and we both don't even have 140 installed) but it runs. i will kick of a build now
<mattip>
zaytsev: I would like to meet up. I am not really sure of the schedule.
<zaytsev>
mattip: alright, my schedule is pretty much inflexible, so... just ping me per email / irc. we've met with fijal / arigato last week, so i guess they can vouch that i'm not dangerous now :)
themsay has quit [Ping timeout: 268 seconds]
<arigato>
:-)
<arigato>
if you want to come for any dinner, just say so. we're likely to have dinner with at least part of the group all evenings
k1nd0f has quit [Remote host closed the connection]
k1nd0f has joined #pypy
k1nd0f has quit [Quit: Leaving]
Zaab1t has quit [Quit: bye bye friends]
themsay has joined #pypy
<zaytsev>
mattip: damn it, it's still somehow broken :(
<zaytsev>
and it's only the problem for cffi imports, pypy exe gets created
<zaytsev>
i think it tried vcvarsall.bat instead of vcvars32.bat as it does at the beginning, and for some reason can't find it. maybe it's not installed by the latest version of the installer? i can have a look, but sadly now i have to go to sleep