Installing from OS Package vs Download

Hi all,
I’m just starting out with Chef and I have a question about cookbook design.

I’ve noticed that some cookbooks use the OS packaging system to get what’s
required while other cookbooks download a binary file.
For example, the tomcat6 package will get the file from the apache website
and install from there while the mysql cookbook will use the package
resource.

What guidelines do people use to decide which one to do?

I personally prefer to use the packaging system and feel uncomfortable
installing from a download unless there’s something in that version I really
need.
This means I can be confident that what I’ve installed will work with my
system - because the package maintainer and the community have tested it
pretty thoroughly and the package will install it according to the OS layout
standards.

I’m using Ubuntu but I think my comments apply generally.

Thoughts?

In my case, I’m considering modifying the public tomcat6 cookbook to get
what I need via packages.

Thanks,
Edward

On 13.04.2011, at 17:58, Edward Sargisson wrote:

Hi all,
I'm just starting out with Chef and I have a question about cookbook design.

I've noticed that some cookbooks use the OS packaging system to get what's required while other cookbooks download a binary file.
For example, the tomcat6 package

this happens if there is no "right" choice in the packaging system. at the time I've written the tomcat6 cookbook there was no up-to-date tomcat6 package available for centos.

will get the file from the apache website and install from there while the mysql cookbook will use the package resource.

What guidelines do people use to decide which one to do?

if the package which comes through the packaging system is what you want to have use this one

I personally prefer to use the packaging system and feel uncomfortable installing from a download unless there's something in that version I really need.

me too.

This means I can be confident that what I've installed will work with my system - because the package maintainer and the community have tested it pretty thoroughly and the package will install it according to the OS layout standards.

I'm using Ubuntu but I think my comments apply generally.

Thoughts?

In my case, I'm considering modifying the public tomcat6 cookbook to get what I need via packages.

there is another tomcat cookbook available. it is called tomcat and uses the tomcat package of ubuntu

Thanks,
Edward

--
DI Edmund Haselwanter, edmund@haselwanter.com, http://edmund.haselwanter.com/
http://www.iteh.at | Facebook | http://at.linkedin.com/in/haselwanteredmund

The general trend seems to be to have a foo::package and a foo::source recipe in these situations, with the default recipe dispatching to one of them. Would this address your concerns in this case?

–Noah

Haselwanter Edmund edmund@haselwanter.com wrote:

On 13.04.2011, at 17:58, Edward Sargisson wrote:

Hi all,
I’m just starting out with Chef and I have a question about cookbook design.

I’ve noticed that some cookbooks use the OS packaging system to get what’s required while other cookbooks download a binary file.
For example, the tomcat6 package

this happens if there is no “right” choice in the packaging system. at the time I’ve written the tomcat6 cookbook there was no up-to-date tomcat6 package available for centos.

will get the file from the apache website and install from there while the mysql cookbook will use the package resource.

What guidelines do people use to decide which one to do?

if the package which comes through the packaging system is what you want to have use this one

I personally prefer to use the packaging system and feel uncomfortable installing from a download unless there’s something in that version I really need.

me too.

This means I can be confident that what I’ve installed will work with my system - because the package maintainer and the community have tested it pretty thoroughly and the package will install it according to the OS layout standards.

I’m using Ubuntu but I think my comments apply generally.

Thoughts?

In my case, I’m considering modifying the public tomcat6 cookbook to get what I need via packages.

there is another tomcat cookbook available. it is called tomcat and uses the tomcat package of ubuntu

Thanks,
Edward

DI Edmund Haselwanter, edmund@haselwanter.com, http://edmund.haselwanter.com/

http://www.iteh.at | http://facebook.com/iTeh.solutions | http://at.linkedin.com/in/haselwanteredmund

Hi,
Yes - I think that answers the question and I can see what I need to do.

BTW, I meant no criticism of Edmund for the download method in the tomcat6
package - I just wanted to be clear on the reasoning.

Thanks,
Edward

On Wed, Apr 13, 2011 at 9:29 AM, Noah Kantrowitz noah@coderanger.netwrote:

The general trend seems to be to have a foo::package and a foo::source
recipe in these situations, with the default recipe dispatching to one of
them. Would this address your concerns in this case?

--Noah

Haselwanter Edmund edmund@haselwanter.com wrote:

On 13.04.2011, at 17:58, Edward Sargisson wrote:

Hi all,
I'm just starting out with Chef and I have a question about cookbook
design.

I've noticed that some cookbooks use the OS packaging system to get what's
required while other cookbooks download a binary file.
For example, the tomcat6 package

this happens if there is no "right" choice in the packaging system. at the
time I've written the tomcat6 cookbook there was no up-to-date tomcat6
package available for centos.

will get the file from the apache website and install from there while the
mysql cookbook will use the package resource.

What guidelines do people use to decide which one to do?

if the package which comes through the packaging system is what you want
to have use this one

I personally prefer to use the packaging system and feel uncomfortable
installing from a download unless there's something in that version I really
need.

me too.

This means I can be confident that what I've installed will work with my
system - because the package maintainer and the community have tested it
pretty thoroughly and the package will install it according to the OS layout
standards.

I'm using Ubuntu but I think my comments apply generally.

Thoughts?

In my case, I'm considering modifying the public tomcat6 cookbook to get
what I need via packages.

there is another tomcat cookbook available. it is called tomcat and uses
the tomcat package of ubuntu

Thanks,
Edward

--
DI Edmund Haselwanter, edmund@haselwanter.com,
http://edmund.haselwanter.com/
http://www.iteh.at | Facebook |
http://at.linkedin.com/in/haselwanteredmund

The 'tomcat' cookbook also supports installation of tomcat 6 via packages on
centos/rhel systems using the jpackage yum repository. And as Noah
mentioned, a separate source and package recipe is a good approach. The
'default' recipe then chooses the appropriate recipe based on platform...or
you can do a 'hard' override by selecting the appropriate recipe directly.

Seth

--
Opscode, Inc.
Seth Chisamore, Technical Evangelist
IRC, Skype, Twitter, Github: schisamo

On Wed, Apr 13, 2011 at 12:20 PM, Edward Sargisson esarge@pobox.com wrote:

Hi,
Yes - I think that answers the question and I can see what I need to do.

BTW, I meant no criticism of Edmund for the download method in the tomcat6
package - I just wanted to be clear on the reasoning.

Thanks,
Edward

On Wed, Apr 13, 2011 at 9:29 AM, Noah Kantrowitz noah@coderanger.netwrote:

The general trend seems to be to have a foo::package and a foo::source
recipe in these situations, with the default recipe dispatching to one of
them. Would this address your concerns in this case?

--Noah

Haselwanter Edmund edmund@haselwanter.com wrote:

On 13.04.2011, at 17:58, Edward Sargisson wrote:

Hi all,
I'm just starting out with Chef and I have a question about cookbook
design.

I've noticed that some cookbooks use the OS packaging system to get
what's required while other cookbooks download a binary file.
For example, the tomcat6 package

this happens if there is no "right" choice in the packaging system. at
the time I've written the tomcat6 cookbook there was no up-to-date tomcat6
package available for centos.

will get the file from the apache website and install from there while
the mysql cookbook will use the package resource.

What guidelines do people use to decide which one to do?

if the package which comes through the packaging system is what you want
to have use this one

I personally prefer to use the packaging system and feel uncomfortable
installing from a download unless there's something in that version I really
need.

me too.

This means I can be confident that what I've installed will work with my
system - because the package maintainer and the community have tested it
pretty thoroughly and the package will install it according to the OS layout
standards.

I'm using Ubuntu but I think my comments apply generally.

Thoughts?

In my case, I'm considering modifying the public tomcat6 cookbook to get
what I need via packages.

there is another tomcat cookbook available. it is called tomcat and uses
the tomcat package of ubuntu

Thanks,
Edward

--
DI Edmund Haselwanter, edmund@haselwanter.com,
http://edmund.haselwanter.com/
http://www.iteh.at | Facebook |
http://at.linkedin.com/in/haselwanteredmund

On Wed, Apr 13, 2011 at 10:37 AM, Seth Chisamore schisamo@opscode.com wrote:

The 'tomcat' cookbook also supports installation of tomcat 6 via packages on
centos/rhel systems using the jpackage yum repository. And as Noah
mentioned, a separate source and package recipe is a good approach. The
'default' recipe then chooses the appropriate recipe based on platform...or
you can do a 'hard' override by selecting the appropriate recipe directly.

While this is a reasonable way to install from source if necessary,
I'd discourage anyone from doing this in practice on boxes you care
about (i.e. production systems or dev/qa systems in an enterprise
environment). If there isn't a package for your platform, grab the
source and build the package yourself. You won't get the benefits of
community testing of the build, but you'll still get the benefits of
package management to install, remove and manage dependencies for the
package. Bonus points for contributing your package build back to the
community (shout out to the folks on this list that have published
their chef and chef dependency packages that weren't publicly packaged
before!).

KC

I was going to ask about that.

Let's say I really need the latest version of something that already exists
as a package. (e.g. PostGis for ubuntu is currently 1.5.1 and I might want
1.5.2).

Would I be better to:
a) Run a source build, put the binaries on a file server and write recipes
to download and install them.
b) Grab the packaging scripts, attempt to build my own upgrade, place it
into a local package repository and contribute the patches back.

What do people normally do?

Cheers,
Edward

On Wed, Apr 13, 2011 at 10:52 AM, KC Braunschweig
kcbraunschweig@gmail.comwrote:

On Wed, Apr 13, 2011 at 10:37 AM, Seth Chisamore schisamo@opscode.com
wrote:

The 'tomcat' cookbook also supports installation of tomcat 6 via packages
on
centos/rhel systems using the jpackage yum repository. And as Noah
mentioned, a separate source and package recipe is a good approach. The
'default' recipe then chooses the appropriate recipe based on
platform...or
you can do a 'hard' override by selecting the appropriate recipe
directly.

While this is a reasonable way to install from source if necessary,
I'd discourage anyone from doing this in practice on boxes you care
about (i.e. production systems or dev/qa systems in an enterprise
environment). If there isn't a package for your platform, grab the
source and build the package yourself. You won't get the benefits of
community testing of the build, but you'll still get the benefits of
package management to install, remove and manage dependencies for the
package. Bonus points for contributing your package build back to the
community (shout out to the folks on this list that have published
their chef and chef dependency packages that weren't publicly packaged
before!).

KC

build it and run your own repo. Just gotta watch how you build if you want to eventually get back to what upstream is providing. Most places with a significant amount of machines will be mirroring the repos of whatever distro they use anyhow.

On Apr 13, 2011, at 11:13 AM, Edward Sargisson wrote:

I was going to ask about that.

Let's say I really need the latest version of something that already exists as a package. (e.g. PostGis for ubuntu is currently 1.5.1 and I might want 1.5.2).

Would I be better to:
a) Run a source build, put the binaries on a file server and write recipes to download and install them.
b) Grab the packaging scripts, attempt to build my own upgrade, place it into a local package repository and contribute the patches back.

What do people normally do?

Cheers,
Edward

On Wed, Apr 13, 2011 at 10:52 AM, KC Braunschweig kcbraunschweig@gmail.com wrote:
On Wed, Apr 13, 2011 at 10:37 AM, Seth Chisamore schisamo@opscode.com wrote:

The 'tomcat' cookbook also supports installation of tomcat 6 via packages on
centos/rhel systems using the jpackage yum repository. And as Noah
mentioned, a separate source and package recipe is a good approach. The
'default' recipe then chooses the appropriate recipe based on platform...or
you can do a 'hard' override by selecting the appropriate recipe directly.

While this is a reasonable way to install from source if necessary,
I'd discourage anyone from doing this in practice on boxes you care
about (i.e. production systems or dev/qa systems in an enterprise
environment). If there isn't a package for your platform, grab the
source and build the package yourself. You won't get the benefits of
community testing of the build, but you'll still get the benefits of
package management to install, remove and manage dependencies for the
package. Bonus points for contributing your package build back to the
community (shout out to the folks on this list that have published
their chef and chef dependency packages that weren't publicly packaged
before!).

KC

On Wed, Apr 13, 2011 at 11:13 AM, Edward Sargisson esarge@pobox.com wrote:

What do people normally do?

If there is an upstream package, I'll backport it myself and put it in
a local apt repository, or put the deb in the cookbook if there isn't
one (boo). A file server is a reasonable alternative. Be sure you
document where these packages came from. For debian packages I tend to
change the version to whatever_it_was_1~bpo1 to signify it is a
backport, or whatever_it_was_1opscode1 to signal that there are local
changes.

Contributing back is always preferred, except that with it comes
varying degrees of difficult. If you're back porting a newer version
of a package to an older distribution, there isn't anything to push
upstream. Upstream distributions usually only backport blocking
bugfixes and security patches. Sometimes upstream moves to slow for
your patches. Sometimes they don't have the resources to deal with all
of your packages and don't see it worth the time to put them in. It
depends.

On Wed, Apr 13, 2011 at 10:52 AM, KC Braunschweig
kcbraunschweig@gmail.com wrote:

While this is a reasonable way to install from source if necessary,
I'd discourage anyone from doing this in practice on boxes you care
about (i.e. production systems or dev/qa systems in an enterprise
environment). If there isn't a package for your platform, grab the
source and build the package yourself. You won't get the benefits of
community testing of the build, but you'll still get the benefits of
package management to install, remove and manage dependencies for the
package. Bonus points for contributing your package build back to the
community (shout out to the folks on this list that have published
their chef and chef dependency packages that weren't publicly packaged
before!).

When you log into a server you didn't build because someone told you
the web service is down and find that there are multiple copies of
apache installed; one as a system package in /usr/sbin, and two built
from source in /usr/local, you curse. Which one is the right one? One
benefit of packaging is that you can cleanly remove software when
you're done with it or want a different version. So packaging helps
this, but so does documentation and good practice. Packaging as makes
it easier to know where files are installed, and that they are
installed where they should be, depending on the quality of the
package.

Some of the benefits of packaging are also served by best practices
these days. When your servers are fully automated, and you want to
know what binary is the web server, it should be clear from your chef
recipes. If it isn't, you might want to consider that they're living
documentation and clean them up a bit.

I'm disappointed by the trend of using source packages whenever there
aren't binary packages, rather than working with upstream
distributions to create them. Sooner or later I believe you end up
with a very carefully managed set of custom dependencies and you start
to lose the benefits of a shared base operating system. Still, there
are certainly times when scripting an installation from source is the
easiest way to go, and perfectly sane. If it doesn't work out, rebuilt
the system from scratch. That should be automated anyway.

Bryan

btm++

There are also some practical considerations:

Installing binary is much faster than compiling from source, which can
make all the difference when doing automated server builds, especially
when testing things on ec2. (even gentoo has .tbz2).

Source builds require you to map out all your build prerequisites, run
prerequisites, compile flags, prefixes, etc and include them in the
cookbook you're building. If you've done all that, you may as well
just go ahead and make the package, since that's all a debian or rpm
spec is anyway.

IMHO procedural logic should be used as sparingly as possible in favor
of declarative resources.

The "install package, write config, notify service" (all using Chef
resources) design pattern is much more readable and maintainable than
a mess of "unless ::File.exists?" blocks littered throughout a recipe

The idempotency check to see if a packages is installed cleaner and safer.

All that being said, there are situations when you simply need to Get
It Done Now and might not have time or resources to build a proper
package, and might opt to build from source instead.

I just did this myself yesterday

/2 cents

-s

On Wed, Apr 13, 2011 at 5:01 PM, Bryan McLellan btm@loftninjas.org wrote:

On Wed, Apr 13, 2011 at 11:13 AM, Edward Sargisson esarge@pobox.com wrote:

What do people normally do?

If there is an upstream package, I'll backport it myself and put it in
a local apt repository, or put the deb in the cookbook if there isn't
one (boo). A file server is a reasonable alternative. Be sure you
document where these packages came from. For debian packages I tend to
change the version to whatever_it_was_1~bpo1 to signify it is a
backport, or whatever_it_was_1opscode1 to signal that there are local
changes.

Contributing back is always preferred, except that with it comes
varying degrees of difficult. If you're back porting a newer version
of a package to an older distribution, there isn't anything to push
upstream. Upstream distributions usually only backport blocking
bugfixes and security patches. Sometimes upstream moves to slow for
your patches. Sometimes they don't have the resources to deal with all
of your packages and don't see it worth the time to put them in. It
depends.

On Wed, Apr 13, 2011 at 10:52 AM, KC Braunschweig
kcbraunschweig@gmail.com wrote:

While this is a reasonable way to install from source if necessary,
I'd discourage anyone from doing this in practice on boxes you care
about (i.e. production systems or dev/qa systems in an enterprise
environment). If there isn't a package for your platform, grab the
source and build the package yourself. You won't get the benefits of
community testing of the build, but you'll still get the benefits of
package management to install, remove and manage dependencies for the
package. Bonus points for contributing your package build back to the
community (shout out to the folks on this list that have published
their chef and chef dependency packages that weren't publicly packaged
before!).

When you log into a server you didn't build because someone told you
the web service is down and find that there are multiple copies of
apache installed; one as a system package in /usr/sbin, and two built
from source in /usr/local, you curse. Which one is the right one? One
benefit of packaging is that you can cleanly remove software when
you're done with it or want a different version. So packaging helps
this, but so does documentation and good practice. Packaging as makes
it easier to know where files are installed, and that they are
installed where they should be, depending on the quality of the
package.

Some of the benefits of packaging are also served by best practices
these days. When your servers are fully automated, and you want to
know what binary is the web server, it should be clear from your chef
recipes. If it isn't, you might want to consider that they're living
documentation and clean them up a bit.

I'm disappointed by the trend of using source packages whenever there
aren't binary packages, rather than working with upstream
distributions to create them. Sooner or later I believe you end up
with a very carefully managed set of custom dependencies and you start
to lose the benefits of a shared base operating system. Still, there
are certainly times when scripting an installation from source is the
easiest way to go, and perfectly sane. If it doesn't work out, rebuilt
the system from scratch. That should be automated anyway.

Bryan

On 13.04.2011, at 19:20, Edward Sargisson wrote:

Hi,
Yes - I think that answers the question and I can see what I need to do.

BTW, I meant no criticism of Edmund for the download method in the tomcat6 package - I just wanted to be clear on the reasoning.

criticism as in there-are-no-wrong-questions is always welcome. My reasoning was that I was not satisfied with the tomcat6 package on centos.
I always want to leverage what the os gives me. but if there isn't anything already packaged I tend to go the "this is how it should be installed" route
of the package. I really don't like source compiling install chef recipes either.

In this special case with tomcat, which IS already compiled I does not feel that dirty to install it from the archive. As it is true with installing e.g.
Wordpress from the archive (first which came to mind ..).

After this discussion I again gave the os package install a shot. But again they just give a starting point. I used the tomcat (not tomcat6) cookbook to install
tomcat 6 on a ubuntu box. And I get a tomcat home directory owned by "root:root". So the the next step would be to contact the package
maintainer for this change (which might be just desired by me to have it owned tomcat6:tomcat6) OR use chef for that.

The same is true with installing the mysql jdbc jar files which do not get picked up from tomcat automagically.
Same with the tomcat native stuff.

So in the end I feel better of (in such cases) to maintain my best practice aka cookbook.

cu edi

Thanks,
Edward

On Wed, Apr 13, 2011 at 9:29 AM, Noah Kantrowitz noah@coderanger.net wrote:
The general trend seems to be to have a foo::package and a foo::source recipe in these situations, with the default recipe dispatching to one of them. Would this address your concerns in this case?

--Noah

Haselwanter Edmund edmund@haselwanter.com wrote:

On 13.04.2011, at 17:58, Edward Sargisson wrote:

Hi all,
I'm just starting out with Chef and I have a question about cookbook design.

I've noticed that some cookbooks use the OS packaging system to get what's required while other cookbooks download a binary file.
For example, the tomcat6 package

this happens if there is no "right" choice in the packaging system. at the time I've written the tomcat6 cookbook there was no up-to-date tomcat6 package available for centos.

will get the file from the apache website and install from there while the mysql cookbook will use the package resource.

What guidelines do people use to decide which one to do?

if the package which comes through the packaging system is what you want to have use this one

I personally prefer to use the packaging system and feel uncomfortable installing from a download unless there's something in that version I really need.

me too.

This means I can be confident that what I've installed will work with my system - because the package maintainer and the community have tested it pretty thoroughly and the package will install it according to the OS layout standards.

I'm using Ubuntu but I think my comments apply generally.

Thoughts?

In my case, I'm considering modifying the public tomcat6 cookbook to get what I need via packages.

there is another tomcat cookbook available. it is called tomcat and uses the tomcat package of ubuntu

Thanks,
Edward

--
DI Edmund Haselwanter, edmund@haselwanter.com, http://edmund.haselwanter.com/
http://www.iteh.at | Facebook | http://at.linkedin.com/in/haselwanteredmund

--
DI Edmund Haselwanter, edmund@haselwanter.com, http://edmund.haselwanter.com/
http://www.iteh.at | Facebook | http://at.linkedin.com/in/haselwanteredmund