Geminabox


#1

Hi All!

Sort of off-topic, but related to chef in that we need ruby gems to run chef-client.

Has anybody implemented “geminabox” or “geminaboxplus”? We’ve been using the rubygems-mirror gem to mirror rubygems.orghttp://rubygems.org/ behind our firewall and like everyone else, have run into the SystemStackError when “gem generate_index” is run against the full gem set. Any known scaling problems with geminabox or geminaboxplus that you know of? What have you done to mirror rubygems.org behind your firewall?

Thanks,
Dang H. Nguyen


#2

Dang,

We’ve implemented geminabox behind our firewall, we use a couple cookbooks
to stand it up, one of them is custom application deployment the rest is
standard ruby, nginx, passenger install cookbooks.

We only use geminabox for custom internal gems, we then have
a separate location for the mirror of rubygems itself - we use the nginx
cookbook to deploy this and a custom cookbook to deploy a cron job and the
nginx configuration for the mirror. This is the cron I use to mirror and
index for rubygems.org:

45 1-11,13-23 * * * gem mirror; gem generate_index -d /data/rubygems
–update;
0 12 * * * gem generate_index -d /data/rubygems --no-legacy;

The first is incremental updates, the second is recreating the entire
index. Make sure you update rubygems itself to the newest version (gem
update --system) via your cookbook and make sure you install the builder
3.0 gem too (rubygems-mirror doesn’t specify it as a dependency, nor does
rubygems itself)

Our setup is multiple nginx servers running those crons and all sharing the
same NFS mount (rubygems mirror takes about 60GB), then in our gemrc file
we put this:

:sources:


Have had no scaling issues with using just nginx to serve up gems from
rubygems.org, geminabox doesn’t scale well when looking at a very large set
of gems in its index, that’s why we kept those separate and only for
private internal gems.

And we do point our bootstrapper script/chef to use the private and public
gem hosts behind our firewall.

Thanks,
Nick Willever

On Tue, Sep 25, 2012 at 2:32 PM, Nguyen, Dang Dang.Nguyen@disney.comwrote:

Hi All!

Sort of off-topic, but related to chef in that we need ruby gems to run
chef-client.

Has anybody implemented “geminabox” or “geminaboxplus”? We’ve been using
the rubygems-mirror gem to mirror rubygems.org behind our firewall and
like everyone else, have run into the SystemStackError when “gem
generate_index” is run against the full gem set. Any known scaling problems
with geminabox or geminaboxplus that you know of? What have you done to
mirror rubygems.org behind your firewall?

Thanks,
Dang H. Nguyen


#3

Yo!

Heavy Water to the rescue: https://github.com/chrisroberts/cookbook-geminabox

Cheers,

–AJ

On 26 September 2012 18:11, Nick Willever nickwillever@gmail.com wrote:

Dang,

We’ve implemented geminabox behind our firewall, we use a couple cookbooks
to stand it up, one of them is custom application deployment the rest is
standard ruby, nginx, passenger install cookbooks.

We only use geminabox for custom internal gems, we then have a separate
location for the mirror of rubygems itself - we use the nginx cookbook to
deploy this and a custom cookbook to deploy a cron job and the nginx
configuration for the mirror. This is the cron I use to mirror and index
for rubygems.org:

45 1-11,13-23 * * * gem mirror; gem generate_index -d /data/rubygems
–update;
0 12 * * * gem generate_index -d /data/rubygems --no-legacy;

The first is incremental updates, the second is recreating the entire index.
Make sure you update rubygems itself to the newest version (gem update
–system) via your cookbook and make sure you install the builder 3.0 gem
too (rubygems-mirror doesn’t specify it as a dependency, nor does rubygems
itself)

Our setup is multiple nginx servers running those crons and all sharing the
same NFS mount (rubygems mirror takes about 60GB), then in our gemrc file we
put this:

:sources:


Have had no scaling issues with using just nginx to serve up gems from
rubygems.org, geminabox doesn’t scale well when looking at a very large set
of gems in its index, that’s why we kept those separate and only for private
internal gems.

And we do point our bootstrapper script/chef to use the private and public
gem hosts behind our firewall.

Thanks,
Nick Willever

On Tue, Sep 25, 2012 at 2:32 PM, Nguyen, Dang Dang.Nguyen@disney.com
wrote:

Hi All!

Sort of off-topic, but related to chef in that we need ruby gems to run
chef-client.

Has anybody implemented “geminabox” or “geminaboxplus”? We’ve been using
the rubygems-mirror gem to mirror rubygems.org behind our firewall and like
everyone else, have run into the SystemStackError when "gem generate_index"
is run against the full gem set. Any known scaling problems with geminabox
or geminaboxplus that you know of? What have you done to mirror rubygems.org
behind your firewall?

Thanks,
Dang H. Nguyen


#4

We don’t mirror all of Rubygems internally, but we do use geminabox for any
internal only gems we write. It’s served us well.

Josiah

On Tue, Sep 25, 2012 at 11:14 PM, AJ Christensen aj@junglist.gen.nz wrote:

Yo!

Heavy Water to the rescue:
https://github.com/chrisroberts/cookbook-geminabox

Cheers,

–AJ

On 26 September 2012 18:11, Nick Willever nickwillever@gmail.com wrote:

Dang,

We’ve implemented geminabox behind our firewall, we use a couple
cookbooks
to stand it up, one of them is custom application deployment the rest is
standard ruby, nginx, passenger install cookbooks.

We only use geminabox for custom internal gems, we then have a separate
location for the mirror of rubygems itself - we use the nginx cookbook to
deploy this and a custom cookbook to deploy a cron job and the nginx
configuration for the mirror. This is the cron I use to mirror and index
for rubygems.org:

45 1-11,13-23 * * * gem mirror; gem generate_index -d /data/rubygems
–update;
0 12 * * * gem generate_index -d /data/rubygems --no-legacy;

The first is incremental updates, the second is recreating the entire
index.
Make sure you update rubygems itself to the newest version (gem update
–system) via your cookbook and make sure you install the builder 3.0 gem
too (rubygems-mirror doesn’t specify it as a dependency, nor does
rubygems
itself)

Our setup is multiple nginx servers running those crons and all sharing
the
same NFS mount (rubygems mirror takes about 60GB), then in our gemrc
file we
put this:

:sources:


Have had no scaling issues with using just nginx to serve up gems from
rubygems.org, geminabox doesn’t scale well when looking at a very large
set
of gems in its index, that’s why we kept those separate and only for
private
internal gems.

And we do point our bootstrapper script/chef to use the private and
public
gem hosts behind our firewall.

Thanks,
Nick Willever

On Tue, Sep 25, 2012 at 2:32 PM, Nguyen, Dang Dang.Nguyen@disney.com
wrote:

Hi All!

Sort of off-topic, but related to chef in that we need ruby gems to run
chef-client.

Has anybody implemented “geminabox” or “geminaboxplus”? We’ve been using
the rubygems-mirror gem to mirror rubygems.org behind our firewall and
like

everyone else, have run into the SystemStackError when “gem
generate_index”

is run against the full gem set. Any known scaling problems with
geminabox

or geminaboxplus that you know of? What have you done to mirror
rubygems.org

behind your firewall?

Thanks,
Dang H. Nguyen