<avsm>
mfp: do you mean due to Lwt.wakeup, or underlying pthreads?
hkBst has quit [Ping timeout: 276 seconds]
<avsm>
mfp: i've tended to ban Lwt.wakeup now, as the Lwt-level switching is just too dangerous (and Async guarantees the same behavior too)
<mfp>
avsm: if you're not careful, a finaliser can be invoked at any time by a GC run, which can happen in the middle of code where you would not expect a context switch, i.e. without any calls to lwt's machinery
<avsm>
hm, I wonder how Async deals with that
<mfp>
well, in safe-finalizer I just make sure to Lwt_unix.yield () before the actual finaliser
<avsm>
(* Ocaml permits finalisers to be run in any thread and at any time after the object
<avsm>
becomes unreachable -- they are essentially concurrent. This changes forces all
<avsm>
finaliser code to run sequentially and in a fixed thread. *)
<avsm>
so Async has a dedicated finaliser thread too
<avsm>
this really ought to be in Lwt upstream indeed...
mye has quit [Ping timeout: 256 seconds]
adotbrown has quit [Ping timeout: 248 seconds]
<mfp>
diml: ^^^^^^^^
f[x] has joined #ocaml
ttamttam has quit [Quit: ttamttam]
f[x] has quit [Read error: Connection reset by peer]
ttamttam has joined #ocaml
hkBst has joined #ocaml
f[x] has joined #ocaml
ttamttam has quit [Quit: ttamttam]
ttamttam has joined #ocaml
yacks has joined #ocaml
hkBst has quit [Ping timeout: 255 seconds]
|jbrown| has quit [Remote host closed the connection]
mye has joined #ocaml
hkBst has joined #ocaml
<diml>
mfp: it is already safe in lwt
<diml>
the finalizer just save the value into an option and send a notification
<diml>
also note that using Lwt_unix.yield in a finalizer may not be safe even if you are in the main thread: it does an allocation and so may run the gc
mye has quit [Ping timeout: 248 seconds]
mcclurmc has quit [Ping timeout: 240 seconds]
Yoric has quit [Ping timeout: 252 seconds]
jbrown has joined #ocaml
mye has joined #ocaml
<mfp>
diml: is this a recent change? with lwt 2.4.3, I observe both finalisers being executed outside the main thread and "surprise" context switches
<mfp>
you're very right about Lwt_unix.yield (), but can't yet see any way around it. The minor heap was just emptied, but if there are many finalisers executed in the same run or they allocate a lot we can't prevent another GC run, can we?
<mfp>
sorry, the above is with the new finaliser reg function, here's the one with Lwt_gc.finalise -> http://paste.ubuntu.com/1657397/
<mfp>
my bad, I was running that code in another box with 2.4.0; 2.4.3 seems fine, and I can see you've changed the code
Thorn has quit [Ping timeout: 264 seconds]
mye has quit [Ping timeout: 256 seconds]
q66 has quit [Remote host closed the connection]
q66 has joined #ocaml
q66 has quit [Remote host closed the connection]
q66 has joined #ocaml
q66 has quit [Remote host closed the connection]
q66 has joined #ocaml
Sergey_ has quit [Read error: Connection reset by peer]
mye has joined #ocaml
ahokaomaeha has joined #ocaml
tane has joined #ocaml
fasta_ is now known as fasta
mye has quit [Ping timeout: 248 seconds]
chambart has joined #ocaml
Yoric has joined #ocaml
tautologico has joined #ocaml
mye has joined #ocaml
dabd has joined #ocaml
jbrown has quit [Remote host closed the connection]
Yoric has quit [Ping timeout: 252 seconds]
q66 has quit [Remote host closed the connection]
q66 has joined #ocaml
mye has quit [Ping timeout: 256 seconds]
q66 has quit [Remote host closed the connection]
ontologiae has quit [Ping timeout: 252 seconds]
travisbrady has joined #ocaml
q66 has joined #ocaml
karswell has quit [Read error: Connection reset by peer]
travisbrady has quit [Client Quit]
q66 has quit [Remote host closed the connection]
smondet has joined #ocaml
travisbrady has joined #ocaml
q66 has joined #ocaml
jbrown has joined #ocaml
Cyanure has joined #ocaml
mye has joined #ocaml
mcclurmc has joined #ocaml
mye has quit [Ping timeout: 248 seconds]
darkf has quit [Quit: Leaving]
dabd has quit [Quit: Leaving]
ttamttam has left #ocaml []
S11001001 has joined #ocaml
S11001001 has quit [Changing host]
S11001001 has joined #ocaml
mye has joined #ocaml
cfy has quit [Quit: ERC Version 5.3 (IRC client for Emacs)]
Kakadu has quit [Quit: Page closed]
hkBst has quit [Quit: Konversation terminated!]
ahokaomaeha has quit [Ping timeout: 248 seconds]
mye has quit [Ping timeout: 256 seconds]
q66 has quit [Remote host closed the connection]
q66 has joined #ocaml
mcclurmc has quit [Ping timeout: 255 seconds]
Yoric has joined #ocaml
anderse has quit [Quit: anderse]
mye has joined #ocaml
osa1 has joined #ocaml
djcoin has quit [Quit: WeeChat 0.3.9.2]
travisbrady has quit [Quit: travisbrady]
tac has joined #ocaml
mye has quit [Ping timeout: 248 seconds]
travisbrady has joined #ocaml
cago has left #ocaml []
mika1 has quit [Quit: Leaving.]
myx has joined #ocaml
dwmw2 is now known as dwmw2_gone
mye has joined #ocaml
travisbrady has quit [Quit: travisbrady]
travisbrady has joined #ocaml
Guest34021 has quit [Ping timeout: 255 seconds]
Guest34021 has joined #ocaml
yacks has quit [Quit: Leaving]
ollehar has joined #ocaml
Guest34021 has quit [Ping timeout: 245 seconds]
<osa1>
does anyone here know a good introduction to HM(X) algorithm ?
<osa1>
other than the part in "advanced topics in types and programming languages" by pottier and odersky's phd thesis ?
Guest34021 has joined #ocaml
thomasga has quit [Quit: Leaving.]
m4b has joined #ocaml
<m4b>
i've got a program which uses hashtables of arrays and does a lot of updating; when i profile the program, caml_initialize occupies 40-50% of run-time. From what I understand, this is the garbage collector. This is unacceptable. Is there anyway to change this? Is it because I'm using mutable objects? Would switch the hashtable implementation to Maps, for example, change the matter?
<f[x]>
caml_initialize has nothing to do with gc
<m4b>
f[x]: ok, that's good, the internet lied to me, and then it means perhaps I can do something about the run time. What does it do?
<mfp>
m4b: it is called to initialize arrays
<mfp>
that is, initialize the elements
<_andre>
mfp: thanks for the lwt tip yesterday, i think i found my memory leak :)
<mfp>
m4b: do you often copy arrays or such?
<mfp>
_andre: good to hear, was it exactly the Lwt.choose thing?
<_andre>
not exactly like your example, but i was using choose when i should be using pick instead
<m4b>
mfp: it is a vm that is running a benchmark; here is the gprof flat output http://hpaste.org/82420
<_andre>
so i had a bunch of threads that were never dying
<f[x]>
gprof is useless, use pmp
<m4b>
mfp: the vm is using a hashtable of arrays; in the benchmark, it's making and setting a lot of the elements in the arrays. There is a haskell implementation that is running significantly faster, although our two implementations essentially differ in no significant details. This is somewhat perplexing.
<m4b>
f[x]: ok, I will put the hashtable version up; I'm currently switching out the hashtables for a map, don't think it'll make much of a difference though...
<mfp>
do you happen to have around 1400 elements per array on average? :)
ocp has quit [Ping timeout: 252 seconds]
<mfp>
m4b: it's the Array.copy/sub that's killing you
<m4b>
mfp: (1) I don't know the average amount of elements in the array. for the benchmark inary it runs, I know it copies an array with 11650 instructions. (2) ok, the only Array.cop operation is on the load op; I _need_ to return a fresh copy, otherwise it alias, which is no good. I don't Array.sub anywhere...
<m4b>
mfp: also, did you figuer out I am calling Array.sub from the gprof lines you pasted above? f[x]: sorry, not familiar with the profiler pmp; running pacman -Ss pmp returns no results either, so dunno...
<mfp>
m4b: a guess, other choices (= functions that call caml_array_gather) are caml_array_append and _concat
<m4b>
mfp: cool, good to know :)
<mfp>
this looks like a likely culprit let program = Array.make words (instruction_of_hex 0x0) in
<mfp>
since you use a boxed type for the instructions, it's going to make sure the values are added to the remembered set
leoncamel has quit [Ping timeout: 252 seconds]
<m4b>
mfp: why is it a culprit, and how else would I initialize a new array?
<mfp>
if you kept it in the original format, with 4 consecutive ints representing an instruction, the compiler might be able to avoid caml_initialize
<m4b>
mfp: are you suggesting I don't represent the instruction as a record type, or as any type at all?
<mfp>
caml_initialize is called to add each element of the array being initialized to the remembered set (for GC purposes)
<mfp>
indeed, it should be much faster if you avoid using a boxed type
<m4b>
mfp: ok I am sort of understanding; for the instruction type, you say that it is a "boxed type"; could you elaborate on this poit more?
<m4b>
mfp: also, thank you very much for the detailed and insightful feedback
<mfp>
the instruction record is represented as a block w/ 5 words (1 for header, 4 integer fields)
ttamttam has joined #ocaml
<mfp>
then your array of instructions becomes an array of pointers to such blocks
Anarchos has joined #ocaml
<m4b>
ok
<mfp>
which is costly because (1) initialization is more expensive (caml_initialize is called on each element of the array, to be more precise on each position of the array) and (2) you have an indirection each time you read an instruction
<m4b>
mfp: so how do you recommend refactoring the "instruction" type? simply passing in the 32-bit number?
<mfp>
which can easily mean a cache miss
<m4b>
*"instruction" type... and associated functions/code
<mfp>
if you're running a 64-bit ocaml, that'd be indeed the easiest solution
<m4b>
mfp: I am; if I wasn't, however, or I wanted it to be portable to 32-bit machines, what would you suggest?
<mfp>
if you're on a 32-bit platform, it depends: if 31-bit are enough (OCaml takes 1 for tagging), int arrays are fine, otherwise you have to use the bigarray lib
<m4b>
mfp: exact 32-bits are necessary; well i'll leave that for now, because I'm sort of cheating with the nand implementation, it masks out the most significant 32-bits to get the unsigned 32-bit number; not sure how to go about emulating unsigned 32-bits on a 32-bit machine in ocaml off the top of my head
<m4b>
mfp: but to be clear: you are suggesting that I refactor all of the functions for instructions; in other words, remove hex of instruction, etc., and simply deal with the raw 32-bit instruction directly.
<m4b>
_andre: thanks for the link; I looked around for something simple, but ended up doing the nand hack i mentioned above, since that was the only specific case where the ocaml rep wasn't working (i.e., ¬ (0x0 ∧ 0x0) = -1)
<mfp>
the other funny thing is your use of Hashtbl to hold blocks of mem
<m4b>
mfp: I'm a funny guy ;) I couldn't think of a better way to emulate the deallocate operation
<mfp>
you probably want to use a custom Hashtbl with a better hashing function, or another structure like patricia trees
<m4b>
mfp: if you have a better suggestion for a different data structure, I'm all ears. I don't like that my code is slower than the haskell version ;)
<m4b>
ok; I will try to implement the patricia trees asap; but forgive me ignorance, I'm not too great with module syntax, etc. For example, I have trouble with the map module, and getting it to use ints as keys, i get syntax errors and shit. but that' another issue.
<mfp>
you can instantiate the module with something like module M = Map.Make(struct type t = int let compare = (-) end);; <- note that this compare is NOT correct, though (overflow)
<m4b>
mfp: so I have a question; for the instruction of hex, would it be more efficient to just return a 4-tuple of the masked bits? or is it smarter to get the bits individually per request, so to speak
<mfp>
the latter
<m4b>
mfp: so 4-tuples will have similar issues?
<mfp>
creating a tuple is like creating a record, it means allocating a 5-word block
<m4b>
mfp: ah, ok, thank you for the clarification
<mfp>
gtg now, have fun :)
<m4b>
mfp: great thank you for the advice and suggestions, merci!
S11001001 has quit [Quit: ERC Version 5.3 (IRC client for Emacs)]
ollehar has quit [Ping timeout: 252 seconds]
<dsheets>
suggestions for parameter-space search parallelization libraries?
travisbrady has quit [Quit: travisbrady]
<dsheets>
I could retrofit an old process/pipe system but i have a sneaking suspicion someone has constructed a nice library to do just this. Suggestions?