<adrien>
beginner42: yes, lablwebkit 1.8.2 will just work with webkit-gtk 1.8.3 (or anything later than 1.8.0 if you can get it)
larhat has quit [Quit: Leaving.]
<beginner42>
adrien: is it possible to use webkit to crawl a page for a specific content?
<adrien>
beginner42: I don't know if there's a direct API that exists for that
<adrien>
how precise do you want to get?
fusillia has quit [Read error: Connection reset by peer]
<beginner42>
adrien: the ideal case would be to look for a specific xpath or dom, but i think you said that wont work?
<adrien>
well, the best path is typically to have a C API for that
<adrien>
for the DOM, I haven't made the bindings to it yet because the API conventions are different compared to the "usual" API of webkit-gtk
<adrien>
I can add it within a month but can't say when exactly
<beginner42>
thats great
<beginner42>
btw how would i tell the webkit to download a website?
ontologiae has quit [Ping timeout: 240 seconds]
<adrien>
load_uri method typically
<beginner42>
i thought about wrapping such a call inside lwt to do some of them in parallel, would that work?
<pippijn>
yes
<adrien>
no
<pippijn>
adrien: why not?
<adrien>
wrong way
<pippijn>
well, I don't know what webkit does
<adrien>
lwt has specific functions to not block on I/O
<pippijn>
how much it exposes
<beginner42>
adrien: could you elaborate that
<pippijn>
if it exposes its file descriptors, then lwt can poll them
<adrien>
here we're talking about C functions
<adrien>
on which we have no control
<pippijn>
ok
<pippijn>
then no
<adrien>
lwt can only do as much as what the functions you call will let it do
<adrien>
it cannot do magic
<adrien>
that's for the general case
<adrien>
now, for this in particular
<adrien>
just create several webview objects and use them in parallel
<adrien>
load_uri returns at once
<pippijn>
I can't imagine webkit has only blocking download functions
<adrien>
then you get a "webview-ready" signal
<adrien>
and then you can start processing
<adrien>
if you're single-threaded (webkit isn't single-threaded for everything), the processing will block
leoncamel has joined #ocaml
<adrien>
btw, since you'll be using callbacks and blocking in callbacks is a common thing to avoid, try to minimize that
<beginner42>
but i could start several processes and pass them the uri and then collecting the webview-ready signal and then start processing on each of them?
<adrien>
you don't need to start several processes to do the download part; the blocking part will be accessing the DOM (then you use threads to process the data you've extracted)
<adrien>
but start with only one process, no additional thread, one webview
<adrien>
you'll quickly see how things work out and where it blocks, where it could be made parallel
<beginner42>
adrien: i will try that, but one thing i really dont understand about such a big project is that why there is no proper tutorial material?
<adrien>
you mean, webkitgtk?
<beginner42>
yes
<adrien>
no idea \o/
<adrien>
quick-moving target
<adrien>
rather recent thing
<beginner42>
i got me now the tarball of the webkit. i just need ./configure and make && make install?
mye has joined #ocaml
<adrien>
no :P
<adrien>
beginner42: you really should use the packages in your distribution
fusillia has joined #ocaml
<beginner42>
adrien: the libwebkitgtk-1.0-0 or libwebkitgtk-3.0-0 or the corresponding dev files?
thomasga has joined #ocaml
<beginner42>
adrien: and thanks for the patience :)
<adrien>
webkitgtk-3 is webkitgtk but configured with --with-gtk+=3.0
<adrien>
not compatible with lablgtk
<beginner42>
those are the packages ubuntu lists, if i look for webkitgtk
<adrien>
the other one, I don't know; some distributions don't put actual versions at a location you can easily see them
<beginner42>
someone here who had the same problem with lablgtk2 and ubuntu?
larhat has joined #ocaml
<beginner42>
adrien: in case i can find it some day, what would be the first step in webkit to download a website?
<adrien>
the load_uri method of the webview object
<beginner42>
webview is an object not a module?
bobry has quit [Remote host closed the connection]
ontologiae has quit [Ping timeout: 255 seconds]
ontologiae has joined #ocaml
fusillia has quit [Ping timeout: 245 seconds]
lopex has joined #ocaml
Kakadu has joined #ocaml
gnuvince has quit [Ping timeout: 252 seconds]
gnuvince has joined #ocaml
fusillia has joined #ocaml
Kakadu has quit [Read error: Connection reset by peer]
Kakadu has joined #ocaml
Kakadu has quit [Read error: Connection reset by peer]
Kakadu has joined #ocaml
Kakadu has quit [Read error: Connection reset by peer]
Kakadu has joined #ocaml
Kakadu has quit [Read error: Connection reset by peer]
testcocoon has quit [Quit: Coyote finally caught me]
Kakadu has joined #ocaml
Kakadu has quit [Read error: Connection reset by peer]
Kakadu has joined #ocaml
astertronistic has quit [Read error: Operation timed out]
Kakadu has quit [Read error: Connection reset by peer]
Kakadu has joined #ocaml
Kakadu has quit [Read error: Connection reset by peer]
testcocoon has joined #ocaml
beckerb has joined #ocaml
_andre has joined #ocaml
justdit has quit [Ping timeout: 252 seconds]
Xizor has quit [Remote host closed the connection]
beginner42 has quit [Quit: irc2go]
ocp has joined #ocaml
mort___ has joined #ocaml
beckerb has quit [Ping timeout: 260 seconds]
ocp has quit [Ping timeout: 260 seconds]
fusillia has quit [Ping timeout: 245 seconds]
paolooo has quit [Ping timeout: 245 seconds]
chambart has quit [Ping timeout: 246 seconds]
jamii has joined #ocaml
justdit has joined #ocaml
mort___ has quit [Quit: Leaving.]
chambart has joined #ocaml
ontologiae has quit [Ping timeout: 240 seconds]
fusillia has joined #ocaml
hkBst has quit [Ping timeout: 268 seconds]
hkBst has joined #ocaml
jamii has quit [Ping timeout: 245 seconds]
bobry has joined #ocaml
fusillia has quit [Remote host closed the connection]
Xizor has joined #ocaml
mye has quit [Quit: mye]
ocp has joined #ocaml
Cyanure has quit [Ping timeout: 245 seconds]
Cyanure has joined #ocaml
nooy has quit [Remote host closed the connection]
ppseafield has joined #ocaml
Neros has quit [Ping timeout: 265 seconds]
ontologiae has joined #ocaml
chambart has quit [Ping timeout: 246 seconds]
Cyanure has quit [Remote host closed the connection]
mjonsson has joined #ocaml
ocp has quit [Ping timeout: 260 seconds]
X1z0r has joined #ocaml
hkBst has quit [Ping timeout: 245 seconds]
Xizor has quit [Ping timeout: 260 seconds]
hkBst has joined #ocaml
mathieui has quit [Ping timeout: 252 seconds]
hkBst has quit [Excess Flood]
hkBst has joined #ocaml
nimred has quit [Ping timeout: 260 seconds]
jpdeplaix has quit [Ping timeout: 255 seconds]
nimred has joined #ocaml
nimred has quit [Changing host]
nimred has joined #ocaml
notdan has quit [Ping timeout: 255 seconds]
notdan has joined #ocaml
hkBst has quit [Remote host closed the connection]
djcoin has joined #ocaml
zoie has joined #ocaml
<zoie>
that's weird: i have a function let rsort l = List.sort (fun a b -> compare b a) l, that i use to sort all kinds of lists; but my program doesn't compile when i declare just using only "let rsort = List.sort (fun a b -> compare b a)" (while it compiles when i explicitely put the "l" on both sides); it gives me some typing error, as though it assumes rsort only applies to certain kind of lists
Neros has joined #ocaml
hkBst has joined #ocaml
hkBst has quit [Changing host]
hkBst has joined #ocaml
<pippijn>
zoie: something like "can't be generalised", I guess
chambart has joined #ocaml
TDJACR has quit [Read error: Connection reset by peer]
mye has joined #ocaml
<zoie>
pippijn: no it says that its expecting another kind of list for rsort
<pippijn>
ah yeah
<pippijn>
makes sense
<pippijn>
because at first, the type variables are unknown
<pippijn>
but as soon as you use it, global type inference will fill in the types
<pippijn>
and then they are fixed
<pippijn>
let rsort () = List.sort (fun a b -> compare b a)
<pippijn>
rsort () [1; 2; 3]
<pippijn>
zoie: the code generated for explicit passing of l is better, anyway ;)
<pippijn>
better as in faster
<zoie>
ok thanks
<zoie>
so when we put let rsort l = ... it will just infer the type as 'a list -> 'a list that's it?
<pippijn>
yes, but let rsort = ... will also do that
<pippijn>
sorry, I can't explain further now, I got to go
hkBst has quit [Read error: Connection reset by peer]
<zoie>
it will do that, but as soon as it is used it will instanciate 'a
hkBst has joined #ocaml
hkBst has quit [Changing host]
hkBst has joined #ocaml
zoie has quit [Ping timeout: 245 seconds]
Neros has quit [Ping timeout: 260 seconds]
ocp has joined #ocaml
Neros has joined #ocaml
csag8264 has quit [Remote host closed the connection]
Neros has quit [Ping timeout: 260 seconds]
invariant has joined #ocaml
<invariant>
How can I obtain the directory size (number of files in a directory)?
<invariant>
The names of the files are of no interest and should not be processed.
<hcarty>
thelema: Looking back at the BatIO.Foo.pp functions, is there a reason to have a concrete implementation aside from BatIO.Enum.pp? Everything else, or almost everything else, could be implemented as something close to (Foo.enum |- Enum.pp ~first ~last ~sep ...)
<hcarty>
invariant: Aside from (Sys.readdir |- Array.length), you could look at the extunix library and/or fileutils.
<hcarty>
invariant: One of those may have something.
<invariant>
Where is |- defined?
<hcarty>
invariant: Batteries - that snippet is equivalent to (fun x -> Array.length (Sys.readdir x))
<hcarty>
let ( |- ) f g x = g (f x)
Neros has joined #ocaml
<pippijn>
hcarty: I don't use enum at all
Neros has quit [Remote host closed the connection]
<thelema>
hcarty: minor efficiency reasons, but that's all I can think of.
<flux>
(Enum.pp?)
<thelema>
pippijn: you wouldn't have to, it would be under-the-hood
<pippijn>
I used enum before
<invariant>
hcarty, it's plenty fast for my purposes, but a shell command was too slow.
<pippijn>
but the programs where I used it were ones that need high performance
<thelema>
pippijn: but had bad experience because the semantics...
<invariant>
hcarty, it's about 10 times faster than obvious shell code.
<pippijn>
I can use it in my game, though
<pippijn>
that doesn't need high performance
<pippijn>
but there is no use for it, yet
<thelema>
ah, yes - ocamlopt can't optimize enums down to the same level that C++ iterators get optimized to
<pippijn>
I had some simple loops like Enum.iter/Enum.fold f (1 -- n)
<pippijn>
I ended up making an ExtInt.iter/fold f 1 n
<invariant>
thelema, is there anything in the language which would stand in the way of that?
<pippijn>
because I do like enum's interface, but its genericity comes with a cost
<pippijn>
invariant: not that I can think of
<thelema>
pippijn: for loops are very nice.
<pippijn>
invariant: it#s just a lot of work
<pippijn>
thelema: for loops don't fold
<flux>
I'm not extremely excited with the destructive nature of Enums :(
<thelema>
pippijn: add an accumulator.
<pippijn>
a ref?
<thelema>
flux: it's not different from reading from an IO channel.
<pippijn>
thelema: I did, and it was slower in my tests
<thelema>
pippijn: really? wow. I'll keep that in mind. I guess the mutation has a cost
<pippijn>
yes
<pippijn>
caml_modify is super expensive
<pippijn>
I have one program where caml_modify takes 40% of the time
<pippijn>
actually now it's less, because I reduced mutation (and the time went from 19 to 16 seconds)
<flux>
thelema, so, it's always impure, for one. and it's way more difficult to accidentally consume things from a single file in multiple places than to consume things from an Enum in multiple places
cdidd has quit [Ping timeout: 268 seconds]
<flux>
given I'm yet to see any desire for implementing 'clone' for input_channel..
alexnixon has left #ocaml []
<thelema>
flux: the multiple consume problem is just a programming style problem. The python folks have very similar mutable iterators, and their brains don't break trying to use them.
<flux>
yes, it's a simple matter of programming imperatively :P
<flux>
or converting the enumeration to lazy lists what I've done at times
<flux>
wayyyy nicer to do certain things
<thelema>
true
X1z0r has quit [Quit: So yes it's mIRC under wine under debian double peche capital. ;) I'll soon see in kfreeBSD.]
<pippijn>
I don't find myself needing lazy lists or enums very often
<thelema>
when dealing with mutable values, some imperative style may be a good thing. Or kind of the opposite - making sure that values are consumed once.
<invariant>
This is kind of nice in a lazy language: (sort . head) xs
<invariant>
er (head .sort)
<thelema>
invariant: yes, depending on the sort algorithm.
<thelema>
although that's usually called `reduce min xs`
<invariant>
thelema, what if you want the top 10?
<thelema>
then a lazy sort is what you want.
<invariant>
thelema, are there already libraries for those kinds of use cases in OCaml?
<invariant>
Or more generally, the equivalent of 'yield'?
<invariant>
Or delimited continuations?
<thelema>
there is a delimited continuation library
<thelema>
there's no library that exactly implements yield
<thelema>
the times I've wanted yield, I've just used explicit continuations
<pippijn>
yep
<pippijn>
me too
smondet has joined #ocaml
<invariant>
Do you ever use state monads in OCaml?
<thelema>
invariant: just the builtin one. ';'
<invariant>
thelema, but you cannot overload the meaning of that one, can you?
<thelema>
no.
<flux>
I've not yet needed to combine different monads with each other, and for that I'm grateful :P
<flux>
but I guess I should learn that as well
<pippijn>
flux: lwt and option
<flux>
I haven't really used Option with the monadic binding operators
<pippijn>
I've used it twice, I think
<flux>
did you do Lwt operations inside the Option monad?
<pippijn>
I can't remember
<pippijn>
but it's simple, because the syntax for the two monads is different
<flux>
in any case, in simple cases it doesn't require anything special
<pippijn>
I use lwt a = b in
<flux>
in non-simple cases you need monad transformers
<pippijn>
and a <-- b; for option
<flux>
..and who knows how those work ;-)
ocp has quit [Ping timeout: 252 seconds]
<hcarty>
thelema: Given that the code is in the repository history now, it's probably worth basing everything on Enum.pp. I would hope that a minor speed decrease in a collection pretty-printer wouldn't cause trouble.
<fasta>
Some of the OUD talks were quite cool, btw. It does seem that nobody cares, though, since most videos only had 50 views or so.
<hcarty>
thelema: If I get time to do so I'll work on a pull request for that and maybe some implementations for other modules.
<fasta>
Who recorded it?
<fasta>
I am also not sure why it's not there in HD.
<hcarty>
invariant: Sys.readdir was fast enough? That's good.
<thelema>
hcarty: ok, I'll pull your changes
<invariant>
hcarty, 160000 files in 0.120 seconds of real time.
<invariant>
hcarty, shell code is over 1.3 seconds
<hcarty>
flux: BatIO.Enum.pp = Format-based printer for Enums (and lists and arrays...)
jpdeplaix has joined #ocaml
<flux>
pippijn, btw, did you achieve fast IO with ocaml?
<pippijn>
flux: Unix.read is fast
<flux>
hcarty, hmm, sounds interesting
<pippijn>
I don't need fast IO right now
<invariant>
hcarty, perhaps I would even go so far to declare OCaml as dominating over shell code for such tasks.
<pippijn>
when I do, I will probably use Unix.read
<pippijn>
input_line is conceptually difficult
<hcarty>
flux: It's handy!
<flux>
hcarty, can you give an oneliner example?-)
mjonsson has quit [Ping timeout: 240 seconds]
<thelema>
invariant: yes, compiled code generally beats shell scripts and executing programs to accomplish a system call
<pippijn>
flux: input_line can be fast for lines that fit into the buffer
<hcarty>
flux: All of the labeled arguments are optional
<thelema>
pippijn: really? I'd expect the scan for the \n to slow things down
<pippijn>
if the line was already buffered, it's easy
<pippijn>
thelema: sure, but that's just a scan
<fasta>
Is it because she is a women and knows how to spell F#?
<pippijn>
if the \n is not in the buffer, it needs to copy the buffer from "current position" to "end" and then fill the buffer again, looking for \n and if it didn't find it, append the whole buffer to the line buffer and try again
travisbrady has quit [Client Quit]
<hcarty>
flux: The biggest benefit in my experience is getting nicely formatted nested structures. I usually write a short pp_foo function to wrap any optional/output pieces.
<thelema>
pippijn: relative to the cost of filling the input buffer, the scan is cheap, yes.
<pippijn>
yes
<pippijn>
the scan is the easiest part of input_line
<flux>
hcarty, I wonder if something like the haskell Lenses library for ocaml would be suitable for the same use as well
<pippijn>
easiest and cheapest
<pippijn>
I was going to write an input_line, but I gave up because it was late
<adrien>
I actually still don't see the point but I'll probably understand later on at once
<thelema>
The idea is to replace this kind of code:
<thelema>
(My_string.of_string "string literals")
<thelema>
by:
<thelema>
m"string literals"
ftrvxmtrx has joined #ocaml
<thelema>
for example, unicode strings and ropes in batteries
<adrien>
but you have to define what "m" stands for
<adrien>
so why not (m "string literal") ?
<hcarty>
adrien: Some compile-time conversion could be done - not sure if it is. It also avoids polluting the namespace with single letter values.
<orbitz>
argh bastards
<orbitz>
ocp-build makes something in your home dir
<thelema>
another namespace for the conversion functions?
<thelema>
heh.
<hcarty>
thelema: :-)
<adrien>
well, I've writte "let i = ignore" for lablgtk code ;-)
chambart has joined #ocaml
<adrien>
i (foo bar baz);
<adrien>
after 30 invocations, even with the unexplicit name, it's fine too, and it doesn't require camlp4 :P
<thelema>
yes, realizing that estring has a home outside batteries makes me happy to remove it. And since having pa_comprehension isn't worth camlp4 to me, batteries will shortly be camlp4-free
othiym23 has quit [Ping timeout: 244 seconds]
othiym23 has joined #ocaml
<hcarty>
thelema: I don't think either of those rely on anything in Batteries that they couldn't access as outside modules.
<hcarty>
thelema: If you remove pa_estring then it's probably worth removing BatPrint too.
<hcarty>
It doesn't serve much purpose without the syntax extension.
<thelema>
hcarty: agreed
<thelema>
we don't need three printer combinators
<thelema>
adrien: ah, that's one thing that estring can do that a simple function can't do - parse format strings at compile-time
<hcarty>
Foo.print and Foo.pp (or IO.Foo.(print, pp)) are likely plenty. Syntax extensions can build on those if it's helpful.
<thelema>
yes, the batPrint infrastructure was added without too much thought on using existing infrastructure
ftrvxmtrx has quit [Read error: Connection reset by peer]
<thelema>
I think batteries 2.0 release will have to be called "le petit batteries", after the Exupery quote
<thelema>
A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.
<thelema>
there's still more to take away in batteries.
lolcathost has joined #ocaml
answer_42 has quit [Ping timeout: 276 seconds]
hongboz has joined #ocaml
smondet has quit [Read error: Operation timed out]
tane has quit [Quit: Verlassend]
hongboz has quit [Read error: Operation timed out]
eikke has joined #ocaml
Yoric has quit [Ping timeout: 252 seconds]
BiDOrD has joined #ocaml
BiDOrD_ has quit [Ping timeout: 246 seconds]
larhat has joined #ocaml
Snark has quit [Quit: Quitte]
ontologiae has joined #ocaml
Yoric has joined #ocaml
larhat has quit [Quit: Leaving.]
larhat has joined #ocaml
ppseafield has quit [Quit: Leaving.]
emmanuelux has joined #ocaml
chambart has quit [Ping timeout: 246 seconds]
larhat has quit [Quit: Leaving.]
larhat has joined #ocaml
Yoric has quit [Remote host closed the connection]
Yoric has joined #ocaml
Xizor has quit [Quit: So yes it's mIRC under wine under debian double peche capital. ;) I'll soon see in kfreeBSD.]
eikke has quit [Ping timeout: 246 seconds]
contempt has quit [Disconnected by services]
contempt has joined #ocaml
eikke has joined #ocaml
dwmw2_gone has quit [Ping timeout: 260 seconds]
lolcathost has quit [Ping timeout: 248 seconds]
Submarine has joined #ocaml
Submarine has quit [Changing host]
Submarine has joined #ocaml
mye has quit [Quit: mye]
cdidd has joined #ocaml
Cyanure has quit [Remote host closed the connection]
cdidd has quit [Ping timeout: 265 seconds]
pkrnj has quit [Quit: Computer has gone to sleep.]
ontologiae has quit [Ping timeout: 252 seconds]
justdit has joined #ocaml
dwmw2_gone has joined #ocaml
cdidd has joined #ocaml
justdit has quit [Quit: Lost terminal]
xavierm02 has quit [Quit: Leaving]
jamii has quit [Ping timeout: 246 seconds]
pkrnj has joined #ocaml
mjonsson has joined #ocaml
eikke has quit [Ping timeout: 260 seconds]
gnuvince has quit [Quit: Remember when men were men and regular expressions recognized regular languages?]
gnuvince has joined #ocaml
Anarchos has quit [Quit: Vision[0.9.7-H-090423]: i've been blurred!]
emmanuelux has quit [Remote host closed the connection]