MichaelSmith has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
rikkipitt has joined #rubygems
johnmwilliams___ has quit [Read error: Network is unreachable]
denym_OFF has quit [Ping timeout: 265 seconds]
akahn has quit [Ping timeout: 244 seconds]
havenwood has quit [Remote host closed the connection]
avdi has quit [Read error: Network is unreachable]
fidothe has quit [Read error: Connection reset by peer]
havenwood has joined #rubygems
workmad3 has joined #rubygems
johnmwilliams___ has joined #rubygems
avdi has joined #rubygems
fidothe has joined #rubygems
akahn has joined #rubygems
cpuguy83 has left #rubygems [#rubygems]
qrush has quit [Read error: Connection reset by peer]
denym_OFF has joined #rubygems
qrush has joined #rubygems
workmad3 has quit [Ping timeout: 244 seconds]
sj26 has quit [Ping timeout: 244 seconds]
electrical has quit [Ping timeout: 244 seconds]
electrical has joined #rubygems
whoojemaflip has quit [Ping timeout: 244 seconds]
whoojemaflip has joined #rubygems
ckrailo__ has quit [Ping timeout: 244 seconds]
sj26 has joined #rubygems
ckrailo__ has joined #rubygems
djbkd has quit [Remote host closed the connection]
jnh has joined #rubygems
<jnh>
Hey all.
<jnh>
to run a gem server do I actually need `gem serve` or just an HTTP server pointing at my mirror?
<jnh>
(ie nginx)
ngpestelos has joined #rubygems
<drbrain>
jnh: with `gem generate index` just an HTTP server
<drbrain>
generate_indfex
<drbrain>
(you get the idea)
<jnh>
cool.
<jnh>
thanks :)
<jnh>
so it doesn't need anything fancy to support the bundler api?
<drbrain>
it doesn't support the bundler API at all
<drbrain>
nobody has bulit that
<drbrain>
(typo city over here today
<jnh>
ahh
<jnh>
okay then :)
<drbrain>
it shouldn't much of a performance issue unless you have hundreds of versions per gem uploaded
<jnh>
by the way. gem generate_index takes days and ultimately dies on my machine, so I'm mirroring all the specs.hz and gemspec.rz files directly to seed it.
x1337807x has joined #rubygems
<drbrain>
how many gems do you have?
<jnh>
all of the,
<jnh>
all of them.
<jnh>
I'm setting up a Rubygems mirror for Rails Camp NZ
<drbrain>
100+ GB?
<jnh>
yup
<drbrain>
yeah… you might want to look at rubygems.org or geminabox
<drbrain>
generate_index isn't meant to scale that big
elia has joined #rubygems
<drbrain>
jnh: I asked evan to pop in and help you out here
Evan_ has joined #rubygems
<jnh>
so how do you generate the indexes for rubygems.org in the first place? generate_index can update, once they're there but if it ever has to start again from first principals we're rooted.
<drbrain>
evan_: jnh is setting up a mirror for RailsCamp NZ
<drbrain>
jnh: generate_index might work up to around 100,000 gems, maybe 200,000 on modern hardware in a reasonable time
<jnh>
drbrain: I left it running on my laptop for days (3GHz i7 16GB RAM, SSD)
<jnh>
it needs to be modified to work in batches.
<Evan_>
we don't use generate_index
<drbrain>
rubygems.org is designed to work against the all the gems and uses different techniques
<Evan_>
way back when
<Evan_>
years ago now.
<Evan_>
we did
<Evan_>
and it took a long ass time, as you've seen.
<Evan_>
so I redid it special in rubygems.org to read the data out of the database and build the index directly
<Evan_>
it's nearly instant now.
<Evan_>
you could go that route
elia has quit [Client Quit]
<jnh>
ahh. that's heaps better.
<jnh>
I'm just about done now.
<drbrain>
maybe we can get jnh a partial dump?
<Evan_>
of what?
<drbrain>
(no users)
<drbrain>
of our DB
<jnh>
I have almost all the gemspec.rz files now and the spec.gz files too
<jnh>
once I have those I'll be able to run generate_index --update
<Evan_>
i'll be sure to send you the $150+ bandwidth bill
<Evan_>
:D :D
<jnh>
Yup. Feel free to send it to RubyNZ
<jnh>
I have room for it in the camp budget
<Evan_>
only kidding
<jnh>
I'm not
<Evan_>
I generally try to not have people make mirrors
<jnh>
if it's an issue we'll pay
<Evan_>
it's a long process
<Evan_>
it's expensive
<Evan_>
and it's a pain to keep updated.
<drbrain>
hopefully everything was fetched through fastly?
<jnh>
yes
<jnh>
through fastly.
<Evan_>
drbrain: though all those 0.0.1 gems from 10 years ago went through fastly too
<Evan_>
evicting good stuff
<Evan_>
for stuff that they'll never use :)
<jnh>
I have a Vagrant box ready to go now, and the mirror sitting on an external USB3 SSD
<jnh>
so we'll be set from now on.
rikkipitt has quit []
<Evan_>
ok
<drbrain>
evan_: more likely it evicted old pages from all our sites
<drbrain>
at least it should have cut down on some of the bandwidth
<Evan_>
yeah, some.
<jnh>
there's a cache in Sydney, right?
<Evan_>
jnh: well, check out the rubygems.org Indexer code if you want to see how I did it
<Evan_>
jnh: there was, run by ninefold
<Evan_>
but it went offline mysteriously
<Evan_>
and we've not seen it return.
<jnh>
:(
volte has joined #rubygems
<jnh>
AWS has a Sydney availability region
<Evan_>
yeah
<Evan_>
we're going to be redoing the mirrors soon.
<dwradcliffe>
jnh: when do you need the mirror?
<Evan_>
though fastly is a better option all around than our hand-rolled mirrors.
<jnh>
yeah. I expect that they are
<drbrain>
there's Fastly POPs in Sydney, Aukland, Wellington
<jnh>
dwradcliffe: camp is 30th of Jan.
<jnh>
ahh, then we don't need a mirror in NZ or AU then.
x1337807x has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]