nisanharamati has quit [Quit: Connection closed for inactivity]
<sknebel>
I'm thinking about trying wallaroo for an application that basically does web crawling. Is it reasonable to get rate limiting etc inside wallaroo, or would I need an external fetch component that has an input task queue from wallaroo and feeds results back through an Input?
<SeanTAllen>
Hi sknebel
<SeanTAllen>
Can you tell me a bit more about the web crawling you are looking to do?
<SeanTAllen>
How do you decide what you want to crawl?
<sknebel>
More or less a feed reader with some additions. Fetch a feed regularly or on external prompt, fetch the feed, parse it, fetch sites referenced in the feed, parse those for information. Wallaroo might be Overkill, but it seemed at least interesting to investigate.
<SeanTAllen>
You could use Wallaroo for parallelization like that. You'd be looking at each web scraping attempt as a stateless computation so, the more Wallaroo workers you add to the cluster, the more scraping you'd do in parallel. I think you could get a definite boost in that sense of not having to deal with that yourself.
<SeanTAllen>
Wallaroo is event-driven so streaming in the sites you want to scrape would make sense or even a single incoming message that is a list of everything you want to scrape. Then your first step in the pipeline would be splitting that list up and sending off to the stateless scraping steps.
<SeanTAllen>
There's nothing for doing rate limiting though in what I just described.
<SeanTAllen>
What were you thinking on the rate limiting front?
<SeanTAllen>
Let me rephrase that, there's nothing explicit doing rate limiting
<SeanTAllen>
If your cluster was say 3 Python processes then at most, you'd be doing 3 scraping jobs at once, so that is rate limiting.
<SeanTAllen>
If you wanted to do something like limit based on number of requests to a website then you could do something like the following:
<SeanTAllen>
key on each website you want to scrap and use that to feed into a stateful step
<SeanTAllen>
actually just a stateless one
<SeanTAllen>
then you are limited to one scraping per website at a time
<SeanTAllen>
i was going to say you could use a stateful step but i realized there's no way in wallaroo to say "don't run this computation right now, run it later" which would be a form of rate limiting. the ramifications of that are rather interesting though and potentially problematic.
<SeanTAllen>
if the "don't run computation at this time" sort of rate-limiting would be interesting to you sknebel, i'd love to hop on a call and chat about it for a bit as there are some serious complexities lurking beneath the surface on that.
<sknebel>
Yeah, growing queue sizes etc become much more of an issue then. Thanks for the info for now, I'll report back if I look at it closer
<sknebel>
(really in the early thinking stages for now)
<SeanTAllen>
You're welcome sknebel. Glad to help.
nisanharamati has joined #wallaroo
cajually has quit [Read error: Connection reset by peer]