CHEF-50 - Portage use flags & keywords

Dear chefs

I’ve updated CHEF-50 with some information, pasted in this email as well. I
would like to solicit some feedback on how we can get this issue resolved.

Best

Currently Chef::Resource::Package doesn’t give us the ability to pass
arbitrary attributes that can be used by Chef::Provider::package::Portage to
manipulate the USE & KEYWORDS environment variables.

I can think of three possible routes to follow here, outlined below:

The first is to expand Chef::Resource::Package to accept a hash of
environment variables that can be passed to the underlying package
providers. This is generic enough, and could be used by other providers as
well:

package "net-dns/pdns" do
  action :install
  environment "USE" => "mysql"
end

The above can then be interpreted by the Portage provider and executed as
"USE=mysql emerge net-dns/pdns"

The second could be to implement two new providers specifically for Gentoo,
one for manipulating use flags and the other for manipulating keywords (both
through the corresponding /etc/portage/package.{use|keywords} files

use "net-dns/pdns" do
  flags "mysql"
end

keywords "dev-db/couchdb" do
  accept "~x86"
end

The last, and least desirable option would be to expand
Chef::Resource::Package to accept ‘use’ and ‘keywords’ as attributes, but
that blows it open to expansion from every child provider.


Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za
@kennethkalmer

+1 on #2 (managing /etc/portage/package.(use|keywords)

setting the USE variables when emerging a package is considered "temporary".

If for any reason the package(s) installed via this method get
re-emerged outside of chef (like by a sysadmin troubleshooting a
problem) the use flags will be lost (unless the sysadmin knows the
flags to tell emerge, which is doubtful).

On Mon, Feb 8, 2010 at 4:24 AM, Kenneth Kalmer kenneth.kalmer@gmail.com wrote:

Dear chefs
I've updated CHEF-50 with some information, pasted in this email as well. I
would like to solicit some feedback on how we can get this issue resolved.
Best

Currently Chef::Resource::Package doesn't give us the ability to pass
arbitrary attributes that can be used by Chef::Provider::package::Portage to
manipulate the USE & KEYWORDS environment variables.
I can think of three possible routes to follow here, outlined below:
The first is to expand Chef::Resource::Package to accept a hash of
environment variables that can be passed to the underlying package
providers. This is generic enough, and could be used by other providers as
well:
package "net-dns/pdns" do
action :install
environment "USE" => "mysql"
end
The above can then be interpreted by the Portage provider and executed as
"USE=mysql emerge net-dns/pdns"
The second could be to implement two new providers specifically for Gentoo,
one for manipulating use flags and the other for manipulating keywords (both
through the corresponding /etc/portage/package.{use|keywords} files
use "net-dns/pdns" do
flags "mysql"
end
keywords "dev-db/couchdb" do
accept "~x86"
end
The last, and least desirable option would be to expand
Chef::Resource::Package to accept 'use' and 'keywords' as attributes, but
that blows it open to expansion from every child provider.

Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za
@kennethkalmer

--
Edward Muller
Sr. Systems Engineer
Engine Yard Inc. : Support, Scalability, Reliability
+1.866.518.9273 x209 - Mobile: +1.417.844.2435
IRC: edwardam - XMPP/GTalk: emuller@engineyard.com
Pacific/US

On Mon, Feb 8, 2010 at 4:24 AM, Kenneth Kalmer kenneth.kalmer@gmail.com wrote:

Currently Chef::Resource::Package doesn't give us the ability to pass
arbitrary attributes that can be used by Chef::Provider::package::Portage to
manipulate the USE & KEYWORDS environment variables.
I can think of three possible routes to follow here, outlined below:
The first is to expand Chef::Resource::Package to accept a hash of
environment variables that can be passed to the underlying package
providers. This is generic enough, and could be used by other providers as
well:
package "net-dns/pdns" do
action :install
environment "USE" => "mysql"
end
The above can then be interpreted by the Portage provider and executed as
"USE=mysql emerge net-dns/pdns"
The second could be to implement two new providers specifically for Gentoo,
one for manipulating use flags and the other for manipulating keywords (both
through the corresponding /etc/portage/package.{use|keywords} files
use "net-dns/pdns" do
flags "mysql"
end
keywords "dev-db/couchdb" do
accept "~x86"
end
The last, and least desirable option would be to expand
Chef::Resource::Package to accept 'use' and 'keywords' as attributes, but
that blows it open to expansion from every child provider.

I would prefer not to have the generic terms "use" and "keywords" be
tied to this. It seems like this is indeed package-level options,
that just happen to set some things in the filesystem for you prior to
packaging.

My gut says the thing to do is to have a re-usable generic for hints
of this kind to the underlying package providers. One of the prime
reasons people choose one system over another are exactly this kind of
implementation detail - you pick Gentoo partly because you love the
use and keywords systems. Something like:

package "dev-db/couchdb" do
hints :keywords => { :accept => "~x86" }
end

Might work, and the underlying package provider can choose to do with
it as it will.

Another option would be to make these attributes available only
through the specific resource for the provider:

portage_package "dev-db/couchdb" do
keywords :accept => "~x86"
use :flags => "mysql"
end

You could build the nicer version (with the keywords and use
attributes) as sugar on top of the generic hints attribute.

Thoughts?

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

Hi!
One reference I can think of is that "execute" resource has an
"environment" attribute. However thats not a feature of the package
resource.

I don't understand as much about this as I would like. But it seems
you cant use "options" here because these are package options which
are set elsewhere and not appended to the end of the command. Maybe
the portage tools could help you out a bit more by exposing some
helper commands to manipulate those configuration files. If yes then
it should be relatively straightforward.

Otherwise, for adding / subtracting some arbitrary lines to a config
file, I'm not sure that's so well catered for in chef. Mainly we rely
on erb templates instead. Which may not be ideal.

In spirit I agree with Adam that its better to make and use a
platform-generic interface wherever possible to set these options. Its
easier to understand and to document those features in a more uniform
way.

For example:
If I were familiar with the Macports packaging system, I would know
that each package can be compiled with certain flavors, known as
"variants". However the "variants" feature is unique to Macports and
its either not used or called by that name on any other platforms.

But it could still seem reasonable and concise to write:

package "git-core" do
case node[:platform]
when "mac_os_x"
options "+bash_completion +svn"
end
end

It seems a lot harder to make such an elegant solution here. There
might be other ways. If you had made a seperate resource for these
hints things, then you could point it to your package resource with
notifies. Or you can write some overloaded resource methods into a
cookbook, which cater specifically for these tougher stuff (what I
would call the more "niche" behaviour). Hmm. Dunno.

On Mon, Feb 8, 2010 at 8:56 PM, Adam Jacob adam@opscode.com wrote:

On Mon, Feb 8, 2010 at 4:24 AM, Kenneth Kalmer kenneth.kalmer@gmail.com wrote:

Currently Chef::Resource::Package doesn't give us the ability to pass
arbitrary attributes that can be used by Chef::Provider::package::Portage to
manipulate the USE & KEYWORDS environment variables.
I can think of three possible routes to follow here, outlined below:
The first is to expand Chef::Resource::Package to accept a hash of
environment variables that can be passed to the underlying package
providers. This is generic enough, and could be used by other providers as
well:
package "net-dns/pdns" do
action :install
environment "USE" => "mysql"
end
The above can then be interpreted by the Portage provider and executed as
"USE=mysql emerge net-dns/pdns"
The second could be to implement two new providers specifically for Gentoo,
one for manipulating use flags and the other for manipulating keywords (both
through the corresponding /etc/portage/package.{use|keywords} files
use "net-dns/pdns" do
flags "mysql"
end
keywords "dev-db/couchdb" do
accept "~x86"
end
The last, and least desirable option would be to expand
Chef::Resource::Package to accept 'use' and 'keywords' as attributes, but
that blows it open to expansion from every child provider.

I would prefer not to have the generic terms "use" and "keywords" be
tied to this. It seems like this is indeed package-level options,
that just happen to set some things in the filesystem for you prior to
packaging.

My gut says the thing to do is to have a re-usable generic for hints
of this kind to the underlying package providers. One of the prime
reasons people choose one system over another are exactly this kind of
implementation detail - you pick Gentoo partly because you love the
use and keywords systems. Something like:

package "dev-db/couchdb" do
hints :keywords => { :accept => "~x86" }
end

Might work, and the underlying package provider can choose to do with
it as it will.

Another option would be to make these attributes available only
through the specific resource for the provider:

portage_package "dev-db/couchdb" do
keywords :accept => "~x86"
use :flags => "mysql"
end

You could build the nicer version (with the keywords and use
attributes) as sugar on top of the generic hints attribute.

Thoughts?

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

On Mon, Feb 8, 2010 at 10:56 PM, Adam Jacob adam@opscode.com wrote:

I would prefer not to have the generic terms "use" and "keywords" be
tied to this. It seems like this is indeed package-level options,
that just happen to set some things in the filesystem for you prior to
packaging.

I agreed, I mentioned it in the original message so we have something
'bad' to improve upon.

My gut says the thing to do is to have a re-usable generic for hints
of this kind to the underlying package providers. One of the prime
reasons people choose one system over another are exactly this kind of
implementation detail - you pick Gentoo partly because you love the
use and keywords systems. Something like:

package "dev-db/couchdb" do
hints :keywords => { :accept => "~x86" }
end

Might work, and the underlying package provider can choose to do with
it as it will.

This is a great suggestion and makes a lot of sense.

Another option would be to make these attributes available only
through the specific resource for the provider:

portage_package "dev-db/couchdb" do
keywords :accept => "~x86"
use :flags => "mysql"
end

You could build the nicer version (with the keywords and use
attributes) as sugar on top of the generic hints attribute.

This can also work quite nicely.

The guys in Chef Infra (archive) were also alive with suggestions, and I've updated
the ticket with links to a gist and the beginnings of a
gentoo-cookbooks repo [1]. Gábor Vészi had some tremendous insights
into how to configure portage properly on a package by package basis,
which allows a simple definition around template resource. At first it
sounded too much to have two/three blocks for emerging packages
(use/keyworks/package), but we quickly agreed that in case of manual
intervention by a system administrator the generated configs would
work great to keep things sane.

For the gentoo specifics like use flags and keywords, the
gentoo-cookbooks' portage recipe seems to have it well covered (I'm
still testing everything). But I can imagine edge cases where a
'hints' implementation would be really useful.

Thanks for the lively discussion around this and other gentoo issues.
I really need to create a wiki page and start consolidating this
information in there.

Best

[1] GitHub - veszig/gentoo-cookbooks: Chef cookbooks for gentoo.

--
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za
@kennethkalmer

+1 on using /etc/portage/package.(use|keywords)

I find the thought of using env variables to determine which 'USE' is
turned on... to be distasteful because then if you use portage outside
of chef to upgrade the package; you suddenly lost all your USE flags and
you just upgraded the package to fail possibly.

The quandry then becomes how do you have chef manage your
/etc/portage/package. files? Do you want it to be a remote_file?
template? or augeas(whatever alternatives are go here)?

Yes it's elegant to mention the use flags in your package block, it's
more elegant if that gets stored properly in /etc/portage/package.use also.

If you can hit both notes, a pint is required.

Scott

On 2/8/10 8:11 AM, Edward Muller wrote:

+1 on #2 (managing /etc/portage/package.(use|keywords)

setting the USE variables when emerging a package is considered "temporary"..

If for any reason the package(s) installed via this method get
re-emerged outside of chef (like by a sysadmin troubleshooting a
problem) the use flags will be lost (unless the sysadmin knows the
flags to tell emerge, which is doubtful).

On Mon, Feb 8, 2010 at 4:24 AM, Kenneth Kalmer kenneth.kalmer@gmail.com wrote:

Dear chefs
I've updated CHEF-50 with some information, pasted in this email as well. I
would like to solicit some feedback on how we can get this issue resolved..
Best

Currently Chef::Resource::Package doesn't give us the ability to pass
arbitrary attributes that can be used by Chef::Provider::package::Portage to
manipulate the USE & KEYWORDS environment variables.
I can think of three possible routes to follow here, outlined below:
The first is to expand Chef::Resource::Package to accept a hash of
environment variables that can be passed to the underlying package
providers. This is generic enough, and could be used by other providers as
well:
package "net-dns/pdns" do
action :install
environment "USE" => "mysql"
end
The above can then be interpreted by the Portage provider and executed as
"USE=mysql emerge net-dns/pdns"
The second could be to implement two new providers specifically for Gentoo,
one for manipulating use flags and the other for manipulating keywords (both
through the corresponding /etc/portage/package.{use|keywords} files
use "net-dns/pdns" do
flags "mysql"
end
keywords "dev-db/couchdb" do
accept "~x86"
end
The last, and least desirable option would be to expand
Chef::Resource::Package to accept 'use' and 'keywords' as attributes, but
that blows it open to expansion from every child provider.

Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za
@kennethkalmer

The quandry then becomes how do you have chef manage your
/etc/portage/package. files? Do you want it to be a remote_file?
template? or augeas(whatever alternatives are go here)?

It's much easier if you make /etc/portage/package.{keywords,use} into a directories and have chef manage files within them. This is what I've done in the past, and it works rather well.

Caleb