<FromGitter>
<Blacksmoke16> that seems to be working better
<FromGitter>
<Blacksmoke16> bumped the backoff limit to 30, seems to be working quite well now, with it any lower it bleeds an error or 2 and still breaks
<FromGitter>
<Blacksmoke16> well not true, an rate limit sneaks in there somehow
<FromGitter>
<Blacksmoke16> yea the sleep causes it to go to the next fiber and do more requests
<FromGitter>
<Blacksmoke16> but i dont see how thats possible,
<FromGitter>
<Blacksmoke16> i think the requests currently in flight by other fibers are putting me over the limit
<FromGitter>
<Blacksmoke16> prob should use some semaphore or something to limit the active fibers maybe?
<FromGitter>
<Blacksmoke16> yea that seems to be helping
<FromGitter>
<Blacksmoke16> i should probably do something so i dont have to make requests i know will fail every time this runs tho
<FromGitter>
<Blacksmoke16> because this sucks
<FromGitter>
<Blacksmoke16> and because i bet 80% of them i wont have access to...
[gnubie] has joined #crystal-lang
[gnubie] has quit [Ping timeout: 246 seconds]
postmodern has quit [Remote host closed the connection]
<FromGitter>
<Blacksmoke16> now i also need to fetch more than 1 page...
[gnubie] has quit [Ping timeout: 264 seconds]
[gnubie] has joined #crystal-lang
postmodern has joined #crystal-lang
[gnubie] has quit [Ping timeout: 240 seconds]
<postmodern>
is there a way for a dependency to hook into `crystal build` to run after the build or somehow determine the current directory of the codebase being built?
<FromGitter>
<Blacksmoke16> sounds like a usecase for a makefile
<FromGitter>
<plambert> For the rate limiting, maybe use a two-way channel with a spawned fiber to receive the remaining number of errors, and if it is > 0, then write it to the channel. In addition, it should write to the channel every N seconds, so that when you hit zero, you eventually make another request. Each fiber that attempts to make a request, first reads from the channel to block until a request is allowed, then makes
<FromGitter>
... the request, then sends the number of errors remaining.
<FromGitter>
<plambert> There's a race condition: if fiber A makes a request, B makes a request, B receives 10 errors remaining, A receives 11 errors remaining, you'll have an invalid count.
<FromGitter>
<plambert> Also, change my "> 0" to "> the number of fibers making API requests." So if there are 10 fibers making API requests, you assume all 10 will get an error, so you stop allowing requests when only 10 remain.
<FromGitter>
<plambert> My question would be whether the API's 'remaining errors" value is a sliding window or a fixed interval. If it's a fixed interval, you can adjust your sleep to wait until the next interval starts. For example, at the start of the next minute. If it's a sliding window, and you *really* care, you could track the errors yourself with the same sliding window, so you can predict when they "drop off."
<FromGitter>
<Blacksmoke16> there's a header that gets returned with how long until the next window, atm i set things up to sleep that long + 2 just to be sure
<FromGitter>
<plambert> Oh! I just read some of the API docs. It always gives you the reset time, so you know exactly when the window ends. Send both through the "gatekeeper" channel, and the gatekeeper can use the last number it received in its sleep. So you just need to keep the remaining errors above the number of threads.
<FromGitter>
<Blacksmoke16> yea
<FromGitter>
<plambert> Did what I said make sense? It came out in brain order, not useful order... ;)
<FromGitter>
<Blacksmoke16> what i have pushed up atm seems to be working well, processing 20 fibers are once and having the backoff error limit set to 20 as well so if all fibers have an outstanding request before the sleep i'll end up with 0 errors left and not have a rate limited request
<FromGitter>
<plambert> Yeah.
<FromGitter>
<Blacksmoke16> bulk of the errors are going to be on the first run while data is being loaded into db, future runs should be much more robust
<FromGitter>
<plambert> If you really really really care, the gatekeeper can send the number of remaining errors and the reset time as the token, and the fibers hitting the API can have ID numbers 0..N and only continue if their ID is less than the remaining number of errors. If => then sleep before trying again. That would "drain" the pool until the reset time, so you'd let yourself get down to 0 errors before actually stopping
<FromGitter>
... the outgoing requests.
<FromGitter>
<plambert> This needs a whiteboard. ;)
<FromGitter>
<Blacksmoke16> hehe, atm this is more of a WIP/POC so no *super* worried atm
yxhuvud has quit [Read error: Connection reset by peer]
yxhuvud has joined #crystal-lang
postmodern has quit [Quit: Leaving]
alexherbo2 has joined #crystal-lang
deavmi has quit [Read error: Connection reset by peer]
deavmi has joined #crystal-lang
<FromGitter>
<guglielmocg_gitlab> Good day everyone, I would like to know where I can find use examples of the HTTP::Handler and HTTP::Websocket commands if any of you knows any. Specifically, I would like to be able to do what I would do with `Rack::Static.new()` and `Rack::Handler::WEBrick.run my_server, Port: XXXX` in Ruby
<FromGitter>
<guglielmocg_gitlab> Hello Julien, thank you for your quick answer. I always read the documentation API before asking here. I meant if there's any project on github that ca give me some additional context (no pun intended but here it is). Also the very concept of context in HTTP::Server stays a bit of a mystery, that's why I was looking for simple projects to study as reference.
alexherbo2 has quit [Ping timeout: 240 seconds]
alexherbo2 has joined #crystal-lang
<FromGitter>
<kingsleyh> afternoon - SushiChain has been re-branded as Axentro: https://axentro.io - it's still and always will be written in Crystal
_whitelogger has joined #crystal-lang
zorp_ has quit [Ping timeout: 258 seconds]
<FromGitter>
<j8r> @guglielmocg_gitlab ok :) I don't know Rack. Can you be more specific about what you want to do?
<FromGitter>
<j8r> As you can see in the examples in the API docs, the handlers can be chained in an Array passed to `HTTP::Server.new`
dostoyevsky has quit [Quit: leaving]
dostoyevsky has joined #crystal-lang
ua_ has quit [Ping timeout: 256 seconds]
ua_ has joined #crystal-lang
<FromGitter>
<plambert> @guglielmocg_gitlab Rather than passing around the request object and the response object separately, they're bundled in a context object. It's mostly just a container for those two things.
<FromGitter>
<Blacksmoke16> right
postmodern has joined #crystal-lang
<postmodern>
what does multi-threaded support mean for C bindings that are not multi-threaded aware? do we have to put into place guards or such?
<oprypin>
postmodern: sorry i don't understand
<postmodern>
oprypin, when crystal finally gets multi-threading support (native threads with fibers), will there be a way to guard non-threaded C bindings with like Mutexes or something?
<oprypin>
postmodern: i think mutexes is not the issue (what makes you think it is?)
<oprypin>
the issue is that if any call in that library is blocking, it can't be used as is
<postmodern>
oprypin, i was more thinking about C bindings that access shared memory, hence needing a mutex
<oprypin>
postmodern: Crystal stdlib also accesses shared memory with no sync guaranteed whatsoever
alexherbo24 has joined #crystal-lang
alexherbo2 has quit [Ping timeout: 256 seconds]
alexherbo24 is now known as alexherbo2
<oprypin>
postmodern: i don't think a C library would expose its own mutexes as part of its API. if it did, yes, they wouldn't be interoperable
<oprypin>
if the mutexes are exposed only as blocking calls then there's also a problem but it's the problem of blocking calls again
<oprypin>
[23:35:10] <e90d7fpostmodern> oprypin, when crystal finally gets multi-threading support (native threads with fibers), will there be a way to guard non-threaded C bindings with like Mutexes or something?
<oprypin>
i guess there's also a direct answer to this question. "yes". t do mutexes
<oprypin>
* just do mutexes in Crystal, doesn't matter that the callee is in C
<oprypin>
actually i think C bindings are the least likely to be problematic
<oprypin>
C libs
<oprypin>
C libs usually don't assume operations are synced while Crystal code does tend to assume