<havenwood>
i was thinking it might be nice to have `gem list` match more than /^#{string}/, so `gem list bootstrap` for example would match 'twitter-bootstrap-sass' etc.
dangerousdave has quit [Read error: Connection reset by peer]
dangerousdave has joined #rubygems
<iamjarvo>
swills: hasn't been updated in awhile. are you currently using it
x1337807x has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<swills>
iamjarvo: yes
le_gars has joined #rubygems
iamjarvo_ has joined #rubygems
iamjarvo has quit [Read error: Operation timed out]
iamjarvo_ is now known as iamjarvo
jfoy has quit [Quit: jfoy]
x1337807x has joined #rubygems
vertis has joined #rubygems
reset has joined #rubygems
iamjarvo has quit [Quit: iamjarvo]
<drbrain>
indirect: ping
workmad3 has joined #rubygems
<drbrain>
evan: when you were maintaining 1.8.x how did you backport commits?
<evan>
cherry pick
<drbrain>
I think we should do another 2.0.x release with bug fixes from master
<drbrain>
that will go into ruby 2.0.x
<evan>
k
<drbrain>
ok
<evan>
yeah, cherry pick the commit
<drbrain>
I'll go through my changes and cherry pick them over
reset has quit [Quit: Leaving...]
kgrz has quit [Remote host closed the connection]
reset has joined #rubygems
adkron has quit [Quit: leaving]
workmad3 has quit [Read error: Operation timed out]
iamjarvo has joined #rubygems
le_gars has quit [Remote host closed the connection]
jstr has joined #rubygems
kgrz has joined #rubygems
reset has quit [Quit: Leaving...]
kgrz has quit [Ping timeout: 256 seconds]
yerhot has quit [Remote host closed the connection]
le_gars has joined #rubygems
havenwood has joined #rubygems
le_gars has quit [Remote host closed the connection]
autumn has quit [Ping timeout: 264 seconds]
yerhot has joined #rubygems
autumn has joined #rubygems
hakunin has quit [Remote host closed the connection]
hakunin has joined #rubygems
workmad3 has joined #rubygems
dvu has quit [Remote host closed the connection]
dvu has joined #rubygems
hakunin has quit [Ping timeout: 268 seconds]
dvu has quit [Ping timeout: 268 seconds]
reset has joined #rubygems
jfoy has joined #rubygems
vertis has quit [Quit: Leaving.]
iamjarvo has quit [Ping timeout: 246 seconds]
<drbrain>
ok, all the bugs fixed in master have been backported to the 2.0 branch
<drbrain>
indirect: ping
yerhot has quit [Remote host closed the connection]
pipework has quit [Remote host closed the connection]
pipework has joined #rubygems
pipework has quit [Remote host closed the connection]
kgrz has joined #rubygems
icco has quit [Ping timeout: 252 seconds]
yerhot has joined #rubygems
icco has joined #rubygems
<indirect>
drbrain: yo
<indirect>
what's up?
hakunin has joined #rubygems
<indirect>
I am officially starting on the grant stuff next week, btw
<drbrain>
oh cool!
<indirect>
:D
<drbrain>
so I have 1 issue blocking a 2.1 release which is awaiting feedback after a commit
<drbrain>
and I have backported bug fixes to a 2.0 branch
<indirect>
oh nice
<indirect>
when I looked earlier this week, bundler is currently green on rg master
<drbrain>
I'd like to release 2.0.4 maybe next week depending on a bundler sign-off
<indirect>
what's the 2.0 branch named?
<drbrain>
2.1 uses the new resolver evan wrote which is bundler-esque
<drbrain>
"2.0"
<indirect>
I can do a check against 1.3.5 on both branches probably tonight or tomorrow
<indirect>
great
kgrz has quit [Ping timeout: 246 seconds]
<drbrain>
for 2.1 I wonder if we should add a new command like `gem bundle` or if we should wait for 2.2
<drbrain>
2.1 will need a prerelease since `gem install` has new code underneath it
<drbrain>
I think it wouldn't hurt if 2.1 had a `gem bundle` that only did installation from a Gemfile, but I can't guess at user expectations
<indirect>
drbrain: my experience with user expectations when they hear that "rubygems will support bundler" is that all bundler 1.x gemfiles will work
<indirect>
most bundler gemfiles have git gems in them
<indirect>
so I think that maybe the rubygems team will not want to do that, yet?
<indirect>
or at least, caveat gem bundle pretty hard
<drbrain>
we don't want to support git gems
<indirect>
right
<indirect>
exactly
<drbrain>
at least, not in rubygems
<drbrain>
but evan mentioned switching bundler to use the rubygems resolver
<indirect>
which is why I think user expectations are going to be pretty strongly at odds with the actual shipping command
<indirect>
if he can do that while maintaining compatibility, that would be awesome
<evan>
indirect: hiya!
<drbrain>
in that case it will be better to delay `gem bundle` to 2.2
<indirect>
I find his resolver easier to follow :P
<indirect>
hey
<drbrain>
indirect: and it's not recursive!
<indirect>
drbrain: exactly!
<evan>
so I figured we could do that a couple different ways
<evan>
it could be that bundler 2 requires rubygems 2
<indirect>
evan: "that" being gemfiles with git gems?
<evan>
thats the easiest.
<drbrain>
git gems aside, there's more features than installation behind `bundler`
<indirect>
oh, the resolver thing
<evan>
yeah
<evan>
the resolver
<indirect>
yeah
<evan>
ok
<indirect>
I would be fine with that
<evan>
thats easy :)
<drbrain>
err, `bundle` which is my motivation for a `gem bundle`
<evan>
I can help on that front then.
<indirect>
bundler 2 is going to be the big delete
x1337807x has quit [Ping timeout: 248 seconds]
<indirect>
:D
<indirect>
no 1.8, etc etc
<evan>
whats 'gem bundle' ?
<indirect>
I'll let drbrain handle that one
<drbrain>
evan: a proposed command to incorporate bundler features besides the Gemfile
<indirect>
since I don't really know
<indirect>
drbrain: so the somewhat tricky part of gem bundle would be the other things, like update, update foo, git gems, and the rest
workmad3 has quit [Ping timeout: 252 seconds]
<drbrain>
eah
<drbrain>
yeah
<indirect>
but I don't know what kind of scope you're interested in for that
<evan>
so we could make a big push
<evan>
if we require bundler 2 for rubygems 2
<drbrain>
and I don't know what's appropriate
<indirect>
yeah
<evan>
where we push most of bundler into rubygems
<evan>
and have bundler 2 just be tiny gem that adds the 'bundle' CLI
<indirect>
evan: what does "most of" mean, to you?
<drbrain>
it doesn't sound like `gem bundle` is a thing we should worry about for rubygems 2.1, so let's table it
<indirect>
that would be totally fine by me
<evan>
everything but the CLI
<indirect>
ok
<indirect>
drbrain: seems reasonable
<evan>
git gems, paths, etc, etc.
<indirect>
evan: I think that's probably a rubygems 2.2 thing at the earliest?
<indirect>
and Bundler 1.4 needs to happen because of current issues like the stack depth, new index format, and other things
<evan>
since 2.1 is coming out in a week at least, yeah
<indirect>
at a minimum before we push for bundler 2
<evan>
ok
<evan>
so what you could consider as a stop-gap
<drbrain>
I think 2.0.4 next weekish, with 2.1 at least the following week
<evan>
for, say, the stack depth issue
<evan>
is vendor rubygem's dependencyresolver
<indirect>
so there's already a stopgap written by charlie
<evan>
unless you want bundler 1.4 to depend on RG 2
<evan>
oh yeah
<indirect>
we can ship that stopgap in 1.4
<indirect>
he said it fixes jruby in his tests
<evan>
very good point.
<indirect>
no need for the big refactor just yet
<indirect>
so we can make it more thorough and test it better when we do it for real
<indirect>
I don't know if I'll have 1.4 done by two weeks from now, but I am aiming to have it done within 4 weeks
<indirect>
since I want to have prereleases that include the stack depth fix, and fingers crossed include the new index client code
<drbrain>
I don't mind shipping follow-ups to rubygems 2.1, whatever version is necessary
<evan>
is a new index format in that 4 week timeframe?
<evan>
we can certainly roll it out to rubygems.org whenever we want.
<indirect>
that would be awesome, but if the format slips, I will ship 1.4 and then 1.5
<indirect>
because the stack depth thing basically needs to start QA now-ish
<evan>
k
<indirect>
so I can ship it 4 weeks from now at the latest without panicing about compatibility
<indirect>
cool
<evan>
so, wanna discuss the format for a moment?
<indirect>
sure
<indirect>
tell me your thoughts
<evan>
ok, did you see my little pindex thought experiment?
<indirect>
I did
<evan>
k
<evan>
it's basically the basis for something that is nearly infinitely flexible.
<evan>
via 2 elements:
<evan>
1) remote includes
<evan>
this would fix the "large index" problem
<evan>
by allowing us to split up the list of available gems
<evan>
and we can split it however we want
<evan>
because the format doesn't have anything than indicates how it's split
<evan>
2) it includes key:val metadata to be passed along with each nametuple
<evan>
real fast
<indirect>
I have basically two concerns: as few network requests as is humanly possible, and a parsing speed that is at least equivalent to marshal
<evan>
a nametuple is the <name, version, platform> bit
<indirect>
yeah
<evan>
sure
<evan>
so we can reduce network requires by adding a separate kind of remote include
<evan>
a frozen remote include
<evan>
which tells the client "if you have this file, don't go get it again"
<indirect>
evan: why not just use etags?
<evan>
this allows us to freeze the list everyweek
<evan>
thats network traffic
<evan>
:D
<evan>
not asking about an etag is faster than asking.
<indirect>
but "have this file" is completely different from "have a non-corrupted copy of this file"
<evan>
and consider all gems from before 2010
<indirect>
yes
<evan>
thats easy as well
<evan>
the frozen include can be <url, hash>
<evan>
since it's frozen.
<indirect>
sure
<drbrain>
why not use HTTP caching semantics for this?
<evan>
because that means doing the HTTP
<evan>
to find out that you don't need to do anything
<indirect>
I'm totally not convinced that the extra client complexity is worth it over just using a single file, http caching, and range headers
<indirect>
but it seems like this could also work :)
<evan>
ug
<evan>
well
<evan>
i see huge issues with hoping HTTP does it all for us
<evan>
for one, it's a hope and a prayer.
<indirect>
I don't think "download this file and then request the bytes after the bytes you have" is hoping?
<indirect>
even if the caching and etags parts don't work at all
<evan>
but basically all info goes in one file forever?
<indirect>
it's still a single small request to keep the local index up to date
<drbrain>
evan: HTTP covers that through through Cache-control though
<evan>
we're already in that boat
<indirect>
yes
<evan>
we desperately need to way to split up the data
<indirect>
one append-only file
<indirect>
what do we need to split up?
<indirect>
and why?
<indirect>
well, maybe a second append-only file for yanks
<indirect>
but that's a separate discussion
<evan>
yanks are a good comment
<evan>
ok
<evan>
here is an issue
<indirect>
yes
<evan>
how does gzip interact with range requests?
<drbrain>
evan: terribly
<evan>
that seems like a pretty big deal.
<drbrain>
range request is on the content-encoded body, not the content-decoded body
<evan>
because we must assume that we have to compress the list.
<evan>
drbrain: so we'd depend on the webserver to be able to on-the-fly gzip'ing
<evan>
I think the flexibility of being able to have a list pull in other lists
<evan>
is too much not to add
<evan>
it's far, far to useful
<evan>
indirect: adding includes is superset of your idea really
<evan>
you could do append-only with range requests on a file that also contains remote include markers
<drbrain>
evan: it is possible with HTTP to send response headers that indicate "you don't need to request this ever again"
<indirect>
evan: I'm not averse to adding support for it, but I'd like to avoid using it for as long as we possibly can, because it adds both client and request complexity
<evan>
we have to add it to the client
<evan>
otherwise it's useless.
<evan>
i guess
<indirect>
speccing it for future use is different from implementing it and using it right now :)
<evan>
why does it have to be just one file?
<evan>
to flip the question
<indirect>
because latency is _horrible_
<indirect>
every single bundler user in japan and australia is miserable
<evan>
sure
<indirect>
one request is the goal
<indirect>
one file makes it easy to make one request
<evan>
thats why frozen includes are really important
<evan>
they allow us to breakup the list in a latency free way
<indirect>
how is it latency free?
<indirect>
includes mean more than one request
<evan>
frozen includes
<evan>
the frozen is important.
<evan>
let me explain.
<indirect>
it sounds like they mean a minimum of two requests if the master list points to weekly sublists
<indirect>
sure
<evan>
nope!
<evan>
ok
<evan>
so let's assume there are just 2 files for now.
<evan>
file 1 contains, say, all gems from before 2010
cowboyd has quit [Remote host closed the connection]
<evan>
the client is told to download file 2
<evan>
it seems the marker and says "oh, do I have that file?"
<evan>
if they do (the typical case), the make sure it's the same hash as is indicated
<evan>
if the hashes match, cool, they use the data
<evan>
never validating the file over the network
<evan>
fin
<indirect>
okay so the single-file version of that is
<indirect>
how big is the file I already have?
<indirect>
hey server, give me this file starting at X
<evan>
file 1? probably a couple megs.
<indirect>
done
<evan>
so the server is doing gzip on the fly then?
<indirect>
either the server does gzip on the fly or we use a streaming compression algorithm for the file that gets sent over the network so that we can decompress after concatting the bytes we don't have yet
<evan>
I guess I feel weird hanging 100% of the efficency on the HTTP range request header
<indirect>
efficiency?
<evan>
if the server doesn't support range request
<indirect>
you mean, on the server not actually sending bytes we told it to not send?
<evan>
you get the whole file
<evan>
everytime.
<indirect>
uhhhh
<indirect>
yes
<indirect>
let's not use servers that don't support the range header :)
<drbrain>
WEBrick supports the Range header
<evan>
the format will be used by servers we don't control
<indirect>
evan: you mean like the CDN?
<evan>
sure
<evan>
could be a CDN
<evan>
could be someone like gemfire
<indirect>
I control the servers supplied by the CDN I'm going to use?
<evan>
could be a lot of different things
<indirect>
sure
<evan>
could be a proxy server that a user is using
<indirect>
the range header is old
<evan>
imagine how bad it might be for that person
<evan>
actually let's move on
<evan>
this is not a super important part of it
<indirect>
I mean, I'm not married to the range header itself
<evan>
sure
<indirect>
I would just like to optimize for client/server simplicity, as well as speed and a minimum of requests
<evan>
I think there is an important thing to note here though
<evan>
we can't use Marshal
<evan>
or yaml.
<indirect>
of course not
<evan>
and so the format has to be append friendly
<indirect>
raggi already provided a spike that uses plaintext, is append-only, and benchmarks at the same speed as marshal
<indirect>
I tried it :)
<evan>
cool
<evan>
I'd like for the format to be able to embed metadata per spec
<evan>
so that the dep info can be embedded there.
<indirect>
hmm
<evan>
or the ruby_required_version
<indirect>
I worry about the speed implications of that, but it sounds nice
<evan>
here is the reason why I like that
<evan>
we can have that format be the universal format
<evan>
a dep API endpoint
<evan>
could simply append a bunch of files together that are already in this format
<evan>
and return them
<evan>
so we have 1 format parser then.
<indirect>
I would love to not need to download the gemspec, but I would prefer to have an index format that is much smaller and faster and still need to download the gemspec over having an index format that is big and slow
<indirect>
so maybe this can be settled by comparing some working options? :)
<evan>
sure
<indirect>
yes
<indirect>
it would be great to be able to do that
<evan>
I really do believe they can be the same format
x1337807x has joined #rubygems
<evan>
and there is benefit in doing so
<indirect>
I don't think a recursive dep api created by concatenation on the server makes sense
<evan>
it gives us flexibility to mix usages.
<indirect>
but I do believe that a single format could be great
<evan>
passing data in new ways based on future needs.
<indirect>
I worry about the speed that results, which is what I would like to test
<indirect>
but yes
<indirect>
I agree with all of those things, and think they are very good
<indirect>
:)
<evan>
i'm not concerned with the speed of String#split
<evan>
which is the primary parser here.
<indirect>
so in general
<indirect>
it would be awesome to have the thing you have just described as a complete replacement for fetching gemspecs
<drbrain>
(for caching, I prefer to use HTTP if possible since it aids reusability by non-rubygems users)
<indirect>
(yeah, I would also prefer that)
<evan>
(caching in this case being If-Not-Modified and such?)
<drbrain>
(we can send a Cache-control header that says "valid for 10 years" or longer so the client never needs to If-Modified-Since)
<indirect>
(a rubygems server with varnish in front of it is much better imo than a rubygems server running redis to cache strings it will concatenate to produce responses, for example)
<drbrain>
(HTTP allows clients to not fetch at all with such parametercs)
<indirect>
evan: fetching a small number of cacheable files to replace the bajillion .gemspec files would be a pretty big win
<evan>
indirect: right!
<evan>
so imagine dep-api hits being served via varnish
<indirect>
I think that is actually a separate concern from the small/fast primary index
<evan>
totally possible
<indirect>
the problem with the current dep api is that responses have effectively no overlap
<evan>
right
<indirect>
so varnish is more or less pointless
<evan>
yeah
<evan>
ok
<indirect>
yup
<evan>
so an important undiscussed thing is how to handle yanks in the presense of an append-only format
<indirect>
so these seem like similar things that we can maybe (hopefully?) merge into a single unified format
<indirect>
I've discussed that some already
<evan>
are they just marked inside the format?
<indirect>
my vote is pretty strongly for a parallel append-only yank file
<indirect>
oplogs require state inside the parser
<evan>
how does the client combine the 2 logs?
<indirect>
yanks are idempotent :P
dangerousdave has quit [Quit: Leaving...]
<evan>
pull in the yank file and consult it when considering a gem?
<evan>
"I see a 3.0, was that yanked?, no, ok."
<indirect>
use the yank file to reject… yes
<indirect>
in that case, the parser can stay very simple
<indirect>
since it doesn't have to worry about duplicate entries or special entries (of that type)
<evan>
if we use pindex, it can be the same format :D
<indirect>
and it gives us all the wins we just discussed in terms of caching and not re-fetching
<indirect>
yeah, totally
<evan>
I worry slightly about the size of that yanks file
<indirect>
you think it might get big?
<evan>
being parsed and consulted often
<indirect>
it only needs to be parsed once
<evan>
once per bundler run
<evan>
or rubygems run
<indirect>
and it shouldn't take longer than parsing the index+yanks file would have
<indirect>
how would merging them into one file make it faster, though? :P
<evan>
oh, it wouldn't.
<indirect>
oh, okay
<evan>
i'm not advocating that.
<evan>
just thinking it through.
<indirect>
haha alright
yerhot has quit [Remote host closed the connection]
<indirect>
I don't think the file will be big enough or frequently updated enough to warrant a huge concern
<evan>
ok
<evan>
and we can compensate in the client if so
<indirect>
and we want to make sure that yanked gems stay in the master index for the cases where yanked gems should be allowed
<evan>
caching as Marshal, for instance.
<indirect>
(eg they're already locked, or you explicitly asked for them, or… whatever)
<indirect>
yes
<evan>
ok
<evan>
so i'm going to go grab a coffee
<evan>
I'm down to implement whatever part of this
<evan>
and to get it into rubygems.org whenever we want.
<evan>
if we have something soon, we can shadow launch it on rubygems.org
<indirect>
sounds great
<evan>
so we can play with real world usage
<indirect>
I also just got ahold of bundler.io
<indirect>
(:D :D :D)
<evan>
:D
<indirect>
anyway… on the whole, I am +++ on trying out all this stuff
<indirect>
and getting some people to start testing it
<indirect>
so we can see what we need to tweak
<indirect>
and I will be working on that in the very near future
<indirect>
like, next week :)
<evan>
I think that we could get something done on the index quickly
<evan>
:D
<evan>
it will also end up being a reason for people to upgrade to RG 2
<evan>
and newer bundler
dvu has joined #rubygems
pipework has joined #rubygems
cowboyd has joined #rubygems
vertis has joined #rubygems
reset has quit [Quit: Leaving...]
reset has joined #rubygems
dvu has quit [Remote host closed the connection]
ckrailo has quit [Quit: Computer has gone to sleep.]
dvu has joined #rubygems
samkottler has quit [Read error: Connection reset by peer]
samkottler has joined #rubygems
dvu has quit [Ping timeout: 252 seconds]
havenwood has quit [Remote host closed the connection]
mootpointer has joined #rubygems
mootpointer has quit [Read error: Connection reset by peer]