Omnibus thoughts

Chef Infra (archive)
1

(Not sure the best place to post this but doing it via twitter was getting
a bit long in the tooth. I figured this was as good a place as any)

So I’ve been using omnibus pretty heavily over the past week and I’ve
started to feel some pain points that I’d like to get input on.

Omnibus is obviously very focused on the idea of building fat packages for
Opscode’s various products. Not a big deal and totally understandable. It
feels like at this point though, omnibus has grown into a product in its
own right and has a community entirely outside of Chef users (folks are
packaging puppet this way, for instance). I feel like there’s a bit of
conflict in those two things.

Right now, the omnibus-software repo is the master source/fallback for all
things. I haven’t looked to see if this is hardcoded into omnibus itself or
if I’m missing something simple to override that. The software definitions
in omnibus-software are always going to be designed to be for
building/packaging all the bits of a chef-server/chef-client package.
They’re carefully curated to ensure they all play nice with each other.

This is great for reuse/community as long as the omnibus-software repo
versions of packages are what you need. But if they aren’t, you have to
start copying out individual software files into your own project. If you
have multiple projects internally that you build with omnibus, you have to
duplicate that effort across all the projects.

So I have a proposal that I think would at least help part of that (and
leads to a bigger question):

  • Extended the project DSL to support defining an alternate source at the
    project level as well as at the individual dependency level. Obviously the
    sane default here (and sane default for fallback) is the opscode
    omnibus-software repo. Something like this (in config/projects/foo.rb)

name 'foo’
master_source ', :branch => ‘master’

dependency ‘erlang’, :source => '‘
dependency ‘nginx’ # checks local ‘software’ dir first -> checks
’master_source’ -> checks opscode upstream

This would allow me to reuse common components internally when other folks
want to build custom packages using OUR known good
configuration/combination.

The questions are if opscode would merge something like this functionality
being that omnibus is so sensitive to the needs of building chef packages
first and foremost? I read something like this:

“The first version of this project used Opscode’s tool, but they don’t seem
to take pull requests, so I enhanced bernd’s superb fpm-cookery to create
Omnibus packages, and switched this project to use it.” (from here:
https://github.com/andytinycat/puppet-omnibus/blob/master/README.md)

And it gives me pause. As I said, I understand that omnibus exists to
support opscode and that is the primary driver. I’m just wondering if this
is something that others would value?

2

When i started working with omnibus, i were confused by the same reasons.

For our purposes we thinking about creating our own fork of omnibus ruby-gem and modifying it as we want.

Just omnibus is open-source project of OpsCode, and they don't provide any support for our problems. I think, that OpsCode doesn't have enough time for implementation of all of our requests.

--
With best regards, Anton Baranov.

среда, 4 сентября 2013 г. в 21:41, John E. Vincent (lusis) написал:

(Not sure the best place to post this but doing it via twitter was getting a bit long in the tooth. I figured this was as good a place as any)

So I've been using omnibus pretty heavily over the past week and I've started to feel some pain points that I'd like to get input on.

Omnibus is obviously very focused on the idea of building fat packages for Opscode's various products. Not a big deal and totally understandable. It feels like at this point though, omnibus has grown into a product in its own right and has a community entirely outside of Chef users (folks are packaging puppet this way, for instance). I feel like there's a bit of conflict in those two things.

Right now, the omnibus-software repo is the master source/fallback for all things. I haven't looked to see if this is hardcoded into omnibus itself or if I'm missing something simple to override that. The software definitions in omnibus-software are always going to be designed to be for building/packaging all the bits of a chef-server/chef-client package. They're carefully curated to ensure they all play nice with each other.

This is great for reuse/community as long as the omnibus-software repo versions of packages are what you need. But if they aren't, you have to start copying out individual software files into your own project. If you have multiple projects internally that you build with omnibus, you have to duplicate that effort across all the projects.

So I have a proposal that I think would at least help part of that (and leads to a bigger question):

  • Extended the project DSL to support defining an alternate source at the project level as well as at the individual dependency level. Obviously the sane default here (and sane default for fallback) is the opscode omnibus-software repo. Something like this (in config/projects/foo.rb)

name 'foo'
master_source ', :branch => 'master'

dependency 'erlang', :source => ''
dependency 'nginx' # checks local 'software' dir first -> checks 'master_source' -> checks opscode upstream

This would allow me to reuse common components internally when other folks want to build custom packages using OUR known good configuration/combination.

The questions are if opscode would merge something like this functionality being that omnibus is so sensitive to the needs of building chef packages first and foremost? I read something like this:

"The first version of this project used Opscode's tool, but they don't seem to take pull requests, so I enhanced bernd's superb fpm-cookery to create Omnibus packages, and switched this project to use it." (from here: puppet-omnibus/README.md at master · andytinycat/puppet-omnibus · GitHub)

And it gives me pause. As I said, I understand that omnibus exists to support opscode and that is the primary driver. I'm just wondering if this is something that others would value?

3

Lusis,

A few comments below...

lusis.org+chef-list@gmail.com writes:

Omnibus is obviously very focused on the idea of building fat packages for
Opscode's various products. Not a big deal and totally understandable. It
feels like at this point though, omnibus has grown into a product in its
own right and has a community entirely outside of Chef users (folks are
packaging puppet this way, for instance). I feel like there's a bit of
conflict in those two things.

I think we'd all benefit from Omnibus being generally useful. The
conflict I'm aware of here at Opscode is one of available time. And it
is true that, at present, building mind share around omnibus is not a
top priority.

Right now, the omnibus-software repo is the master source/fallback for all
things. I haven't looked to see if this is hardcoded into omnibus itself or
if I'm missing something simple to override that. The software definitions
in omnibus-software are always going to be designed to be for
building/packaging all the bits of a chef-server/chef-client package.
They're carefully curated to ensure they all play nice with each other.

This is great for reuse/community as long as the omnibus-software repo
versions of packages are what you need. But if they aren't, you have to
start copying out individual software files into your own project. If you
have multiple projects internally that you build with omnibus, you have to
duplicate that effort across all the projects.

So I have a proposal that I think would at least help part of that (and
leads to a bigger question):

  • Extended the project DSL to support defining an alternate source at the
    project level as well as at the individual dependency level. Obviously the
    sane default here (and sane default for fallback) is the opscode
    omnibus-software repo. Something like this (in config/projects/foo.rb)

Extending things to allow specifying a list of omnibus software repos
such that package search does PATH style lookup in the repo list sounds
like a nice enhancement.

name 'foo'
master_source ', :branch => 'master'

dependency 'erlang', :source => ''
dependency 'nginx' # checks local 'software' dir first -> checks
'master_source' -> checks opscode upstream

If you can specify the location on a per dependency basis, what happens
when two different things depend on 'nginx' from two different sources?
Is the per dependency thing something you are sure you need? Seems
easier to reason about a list of software repos defining a software
package search path.

Thinking out loud here... I could see a desire to be able to install
different versions of the same thing in a given package. It's unclear to
me if that would be best managed within omnibus or by the user
(e.g. create packages nginx-1.2, nginx-2.0).

The questions are if opscode would merge something like this functionality
being that omnibus is so sensitive to the needs of building chef packages
first and foremost?

The short answer is: yes, this sounds like a good change and we'd love
to see omnibus be more generally useful.

If our omnibus projects contined to work with the patch, that would make
merging it much easier.

"The first version of this project used Opscode's tool, but they don't seem
to take pull requests, so I enhanced bernd's superb fpm-cookery to create
Omnibus packages, and switched this project to use it." (from here:
puppet-omnibus/README.md at master · andytinycat/puppet-omnibus · GitHub)

The context of that was very much: we open sourced a project as soon as
we could, but were super busy with six other projects and just couldn't
prioritize looking at PRs at the time. That sucks, I know, but it was
absolutely not a "we don't take pull requests" and instead a "we opened
this project while we were very busy and didn't plan ahead on responding
to PRs for it" thing.

Cheers,

  • seth

--
Seth Falcon | Development Lead | Opscode | @sfalcon

4

On 9/4/13 2:53 PM, Seth Falcon wrote:

If you can specify the location on a per dependency basis, what happens
when two different things depend on 'nginx' from two different sources?
Is the per dependency thing something you are sure you need? Seems
easier to reason about a list of software repos defining a software
package search path.

Thinking out loud here... I could see a desire to be able to install
different versions of the same thing in a given package. It's unclear to
me if that would be best managed within omnibus or by the user
(e.g. create packages nginx-1.2, nginx-2.0).

I just checked the source in omnibus-ruby and it looks like
omnibus-software is very hardcoded in omnibus.rb:

def self.omnibus_software_root
@omnibus_software_root ||= begin
if spec =
Gem::Specification.find_all_by_name('omnibus-software').first
Pathname.new(spec.gem_dir)
else
nil
end
end
end

Talked with Seth a little and one small step towards this would be to
remove the hard-coding and allow the project file to specify a list of
gems to search for software definitions (in-order). Sources of the gems
could then just be specified in the Gemfile.

And I had actually assumed this is kind of how it worked already...
We're definitely making architectural choices in omnibus-software that
will not take into consideration any downstream consumers other than
opscode projects...

5

Comments inline

On Wed, Sep 4, 2013 at 5:53 PM, Seth Falcon seth@opscode.com wrote:

Lusis,

A few comments below...

lusis.org+chef-list@gmail.com writes:

Omnibus is obviously very focused on the idea of building fat packages
for
Opscode's various products. Not a big deal and totally understandable. It
feels like at this point though, omnibus has grown into a product in its
own right and has a community entirely outside of Chef users (folks are
packaging puppet this way, for instance). I feel like there's a bit of
conflict in those two things.

I think we'd all benefit from Omnibus being generally useful. The
conflict I'm aware of here at Opscode is one of available time. And it
is true that, at present, building mind share around omnibus is not a
top priority.

Yeah I think conflict might have been to harsh of a word. Sorry about that.

Right now, the omnibus-software repo is the master source/fallback for
all
things. I haven't looked to see if this is hardcoded into omnibus itself
or
if I'm missing something simple to override that. The software
definitions
in omnibus-software are always going to be designed to be for
building/packaging all the bits of a chef-server/chef-client package.
They're carefully curated to ensure they all play nice with each other.

This is great for reuse/community as long as the omnibus-software repo
versions of packages are what you need. But if they aren't, you have to
start copying out individual software files into your own project. If you
have multiple projects internally that you build with omnibus, you have
to
duplicate that effort across all the projects.

So I have a proposal that I think would at least help part of that (and
leads to a bigger question):

  • Extended the project DSL to support defining an alternate source at the
    project level as well as at the individual dependency level. Obviously
    the
    sane default here (and sane default for fallback) is the opscode
    omnibus-software repo. Something like this (in config/projects/foo.rb)

Extending things to allow specifying a list of omnibus software repos
such that package search does PATH style lookup in the repo list sounds
like a nice enhancement.

This would be the simplest MVP kind of thing to implement I think while the
specific semantics and needs get fleshed out.

name 'foo'
master_source ', :branch =>
'master'

dependency 'erlang', :source => ''
dependency 'nginx' # checks local 'software' dir first -> checks
'master_source' -> checks opscode upstream

If you can specify the location on a per dependency basis, what happens
when two different things depend on 'nginx' from two different sources?
Is the per dependency thing something you are sure you need? Seems
easier to reason about a list of software repos defining a software
package search path.

I think I ended up mixing two use cases here and didn't define it clearly.
It would be nice to be able to define a dependency as a gist raw url source
for instance. Ostensibly it's no different than me curling that raw gist
url but the overall idea was that you could, by marking a dependency as
coming from a versioned source somehow, "freeze" things a bit. Right now
I'm always pulling from the opscode "master".

Then again this smells like a road that every damn language, project,
framework and tool on the planet has been down. Maybe simply opting in to a
search path at the project level does make more sense. Definitely much
easier to implement. I'm not too keen on implementing a proper dependency
graph here :wink:

Thinking out loud here... I could see a desire to be able to install

different versions of the same thing in a given package. It's unclear to
me if that would be best managed within omnibus or by the user
(e.g. create packages nginx-1.2, nginx-2.0).

Yeah this is a use case I've thought about but haven't really fleshed out.
I like what you're thinking though.

The questions are if opscode would merge something like this functionality

being that omnibus is so sensitive to the needs of building chef packages
first and foremost?

The short answer is: yes, this sounds like a good change and we'd love
to see omnibus be more generally useful.

If our omnibus projects contined to work with the patch, that would make
merging it much easier.

"The first version of this project used Opscode's tool, but they don't
seem
to take pull requests, so I enhanced bernd's superb fpm-cookery to create
Omnibus packages, and switched this project to use it." (from here:
https://github.com/andytinycat/puppet-omnibus/blob/master/README.md)

The context of that was very much: we open sourced a project as soon as
we could, but were super busy with six other projects and just couldn't
prioritize looking at PRs at the time. That sucks, I know, but it was
absolutely not a "we don't take pull requests" and instead a "we opened
this project while we were very busy and didn't plan ahead on responding
to PRs for it" thing.

Apologies because I realize now that section sounded totally harsh on you
all. You didn't have to open a lick of code and I think everyone would say
they're grateful for it being opened up.

Cheers,

  • seth

--
Seth Falcon | Development Lead | Opscode | @sfalcon

6

Lamont, I was actually about to respond to your previous email when I read
this one. I honestly don't mind the idea of using the Gemfile for this sort
of thing but as you pointed out (and I suspected but hadn't confirmed) the
opscode stuff is hardcoded.

I'm going to try and slot some time to learn the codebase a bit more and
get some POC type things done. This might be an easy first stab along with
Seth's idea about a search path.

On Wed, Sep 4, 2013 at 7:33 PM, Lamont Granquist lamont@opscode.com wrote:

On 9/4/13 2:53 PM, Seth Falcon wrote:

If you can specify the location on a per dependency basis, what happens
when two different things depend on 'nginx' from two different sources?
Is the per dependency thing something you are sure you need? Seems
easier to reason about a list of software repos defining a software
package search path.

Thinking out loud here... I could see a desire to be able to install
different versions of the same thing in a given package. It's unclear to
me if that would be best managed within omnibus or by the user
(e.g. create packages nginx-1.2, nginx-2.0).

I just checked the source in omnibus-ruby and it looks like
omnibus-software is very hardcoded in omnibus.rb:

def self.omnibus_software_root
@omnibus_software_root ||= begin
if spec = Gem::Specification.find_all_**by_name('omnibus-software').
**first
Pathname.new(spec.gem_dir)
else
nil
end
end
end

Talked with Seth a little and one small step towards this would be to
remove the hard-coding and allow the project file to specify a list of gems
to search for software definitions (in-order). Sources of the gems could
then just be specified in the Gemfile.

And I had actually assumed this is kind of how it worked already... We're
definitely making architectural choices in omnibus-software that will not
take into consideration any downstream consumers other than opscode
projects...