Re: RE: Re: berkshelf gem listed as a requirement in cookbook gemfile

Chef Infra (archive)
1

Why the hell would you do a bundle install on a node ?

Berkshelf is of use on your workstation to create a pack of cookbooks with correct versions to distribute as tarball or to upload to chef-server.

If your day to day workstation is a lightweight Vm you recreate often, include chef-dk or do bundle install in the source template box...

I don't see any use case where you would do a bundle install from a cookbook dir within a chef run, you should install needed gems for a recipe with chef-gem resource if you need it inside your recipes.

Please enlight me if I missed your point...
Le 2 janv. 2015 22:21, Nico Kadel-Garcia nkadel@skyhookwireless.com a écrit :

-----Original Message-----
From: AJ Christensen [mailto:aj@junglistheavy.industries]
Sent: Wednesday, December 31, 2014 3:14 AM
To: chef@lists.opscode.com
Subject: [chef] Re: berkshelf gem listed as a requirement in cookbook gemfile

Yo,

Sorry for the follow up.

It's worth mentioning the ChefDk includes gecode libs, pre-built, for all
supported target platforms, which greatly alleviates the installation heat-cost of
Berkshelf.

On machines where this (ChefDK) is not present and the prerequisite
environment variable and system libraries have not been configured, a full
installation of gecode from source will be automatically handled by the gem (via
bundle) installation.

It's problematic to compile. I mentioned this week: it requires a C++ compiler, takes hours on a modest VM or lightweight host, especially if the memory allocated is light, and it can cause other issues if the system runs out of swap and RAM.

Nico Kadel-Garcia
Lead DevOps Engineer
nkadel@skyhookwireless.com

cheers,

--aj

On Wed, Dec 31, 2014 at 9:07 PM, AJ Christensen aj@junglistheavy.industries
wrote:

chefdk includes bundler and berkshelf and the Gemfile will resolve.
Please try to resolve and report back.

I think it's OK. In some projects, it's not necessary to use a Gemfile any more.

cheers,

--aj

On Wed, Dec 31, 2014 at 9:03 PM, William Jimenez
william.jimenez@itsoninc.com wrote:

Hi Folks,
I'm noticing that the berkshelf gem is listed as a requirement in
Gemfiles of a lot of cookbooks I've been working with lately. I
suspect that this is bad behavior now that ChefDK is the recommended
deployment method for berkshelf. Just wanted to confirm with you all that
my reasoning is correct.

Thanks,
William

2

-----Original Message-----
From: Tensibai Zhaoying [mailto:tensibai@iabis.net]
Sent: Friday, January 02, 2015 5:21 PM
Cc: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: berkshelf gem listed as a requirement in cookbook
gemfile

Why the hell would you do a bundle install on a node ?

Berkshelf is of use on your workstation to create a pack of cookbooks with
correct versions to distribute as tarball or to upload to chef-server.

Or using it under chef-solo to pull and apply on the local host.

   cd /var/chef-solo
   git pull [ designated git repo, tag, branch, etc., etc. ]
       [ Modify locally in testable environments ]
       [ Or checkout out git tag for production ]
   berks install
   berks vendor
       [ Consistent Berksfile.lock allows use of hard-coded cookbook versions, including rollback to previous version without touching upstream ]
   chef-solo [ with configuration that looks in /var/chef-solo/berks-cookbooks ]
      [ test ]
      [ Record git changes ]
      [ Push git changes back upstream ]

'Deploy from tarball' involves maintaining tarballs and an available server to deploy them from, and potentially pushing tarballs from a development server to the chef server, which raises serious management and security concerns. This also gets really hairy when older cookbooks are incompatible with newer cookbooks or different chef releases, and some servers need one or the other. Recent architectural changes in the yum and mysql cookbooks are good examples where the chef server, itself, might need to have multiple cookbooks for multiple categories of servers. And Berksfile and a chef server do not trivially support deploying multiple cookbook versions, especially legacy ones that have not historically been deployed on that chef server. And they break horribly when different versions of the same numbered cookbook are in use on two development hosts with the same chef server or the same "tarball building setup" as you mentioned, which happens when two different people are working on different parts of the same cookbook at the same time in the same chef server environment.

If your day to day workstation is a lightweight Vm you recreate often, include
chef-dk or do bundle install in the source template box...

"Recreating vm's" is a surprisingly resource expensive operation in many environments. It requires privileges to create VM's, and the sys-admin time and resources to manage those privileges to prevent mistakes which may corrupt critical resources. Being able to spin up a VM or test environment with only local git privileges and no chef server privileges is much safer in many environments.

I don't see any use case where you would do a bundle install from a cookbook
dir within a chef run, you should install needed gems for a recipe with chef-gem
resource if you need it inside your recipes.

Roles and node configurations and environments benefit from having consistent, locked cookbooks especially between QA, staging, and production environments. A "template box" is not good enough for production if you've not recorded your chef changes, especially including cookbook versions, and the 'Berksfile.lock' for chef-solo on individual test environments provides that.

There are trade-offs: you can't use "search" functions as you might on a chef server. But for many test environments, testing on a single chef-solo instance and then propagating it to the relevant staging or production environment is quite useful.

Please enlight me if I missed your point...

I'm happy to do so: see above.

Nico Kadel-Garcia
Lead DevOps Engineer
nkadel@skyhookwireless.com

3

Few inline comments below

On Mon, Jan 5, 2015 at 4:57 PM, Nico Kadel-Garcia
nkadel@skyhookwireless.com wrote:

-----Original Message-----
From: Tensibai Zhaoying [mailto:tensibai@iabis.net]
Sent: Friday, January 02, 2015 5:21 PM
Cc: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: berkshelf gem listed as a requirement in cookbook
gemfile

Why the hell would you do a bundle install on a node ?

Berkshelf is of use on your workstation to create a pack of cookbooks with
correct versions to distribute as tarball or to upload to chef-server.

Or using it under chef-solo to pull and apply on the local host.

   cd /var/chef-solo
   git pull [ designated git repo, tag, branch, etc., etc. ]
       [ Modify locally in testable environments ]
       [ Or checkout out git tag for production ]
   berks install
   berks vendor
       [ Consistent Berksfile.lock allows use of hard-coded cookbook versions, including rollback to previous version without touching upstream ]
   chef-solo [ with configuration that looks in /var/chef-solo/berks-cookbooks ]
      [ test ]
      [ Record git changes ]
      [ Push git changes back upstream ]

'Deploy from tarball' involves maintaining tarballs and an available server to deploy them from, and potentially pushing tarballs from a development server to the chef server, which raises serious management and security concerns. This also gets really hairy when older cookbooks are incompatible with newer cookbooks or different chef releases, and some servers need one or the other. Recent architectural changes in the yum and mysql cookbooks are good examples where the chef server, itself, might need to have multiple cookbooks for multiple categories of servers. And Berksfile and a chef server do not trivially support deploying multiple cookbook versions, especially legacy ones that have not historically been deployed on that chef server. And they break horribly when different versions of the same numbered cookbook are in use on two development hosts with the same chef server or the same "tarball building setup" as you mentioned, which happens when two different people are working on different parts of the same cookbook at the same time in the same chef server environment.

That's my workflow as well, except that I'm not bundle installing it
on the node, but rather from the outside. If that is a desired
scenario for you then Vagrant is a perfect fit.

Example here: https://github.com/tknerr/sample-toplevel-cookbook/blob/master/Gemfile

If your day to day workstation is a lightweight Vm you recreate often, include
chef-dk or do bundle install in the source template box...

"Recreating vm's" is a surprisingly resource expensive operation in many environments. It requires privileges to create VM's, and the sys-admin time and resources to manage those privileges to prevent mistakes which may corrupt critical resources. Being able to spin up a VM or test environment with only local git privileges and no chef server privileges is much safer in many environments.

I don't see any use case where you would do a bundle install from a cookbook
dir within a chef run, you should install needed gems for a recipe with chef-gem
resource if you need it inside your recipes.

Roles and node configurations and environments benefit from having consistent, locked cookbooks especially between QA, staging, and production environments. A "template box" is not good enough for production if you've not recorded your chef changes, especially including cookbook versions, and the 'Berksfile.lock' for chef-solo on individual test environments provides that.

There are trade-offs: you can't use "search" functions as you might on a chef server. But for many test environments, testing on a single chef-solo instance and then propagating it to the relevant staging or production environment is quite useful.

You can use chef-zero though which is almost a drop-in replacement for
chef-solo, especially when using Vagrant (simply replace the
:chef_solo with the :chef_zero provisioner). This will give you more
chef-client like behaviour and especially lets you use search as with
chef server.

Cheers,
Torben

Please enlight me if I missed your point...

I'm happy to do so: see above.

Nico Kadel-Garcia
Lead DevOps Engineer
nkadel@skyhookwireless.com

4

you are just recreating a chef server by all this. you can certain use git
or any other SCM server itself, but this only mean you are using source
code management server as an artifact server, which is fine, but then you
are bound to face with other limitations (some of which you have already
encountered). I am failing to understand the access control bits, they are
same, its just now you have coupled it with source control (unless your git
repos are public).

Regarding your versioning concerns, git sha != cookbook version. in general
source code revision version != artifact version. Source code versions
solves different problems than artifact versions (which can be compared,
hence build tools can use them).

the approach you are taking is way more insecure and dubious, by your own
arguments.

if you do take this route, you should just check in all dependency
cookbooks inside your own repo (because by all means you are treating it as
a artifact server), may be do some tooling with git sub-modules etc. This
will solve your current problems, you wont need a build/assembly tool. In
fact this is what most of us did with puppet or early days of chef.

cheers
ranjib

On Mon, Jan 5, 2015 at 7:57 AM, Nico Kadel-Garcia <
nkadel@skyhookwireless.com> wrote:

-----Original Message-----
From: Tensibai Zhaoying [mailto:tensibai@iabis.net]
Sent: Friday, January 02, 2015 5:21 PM
Cc: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: berkshelf gem listed as a requirement in
cookbook
gemfile

Why the hell would you do a bundle install on a node ?

Berkshelf is of use on your workstation to create a pack of cookbooks
with
correct versions to distribute as tarball or to upload to chef-server.

Or using it under chef-solo to pull and apply on the local host.

   cd /var/chef-solo
   git pull [ designated git repo, tag, branch, etc., etc. ]
       [ Modify locally in testable environments ]
       [ Or checkout out git tag for production ]
   berks install
   berks vendor
       [ Consistent Berksfile.lock allows use of hard-coded cookbook

versions, including rollback to previous version without touching upstream ]
chef-solo [ with configuration that looks in
/var/chef-solo/berks-cookbooks ]
[ test ]
[ Record git changes ]
[ Push git changes back upstream ]

'Deploy from tarball' involves maintaining tarballs and an available
server to deploy them from, and potentially pushing tarballs from a
development server to the chef server, which raises serious management and
security concerns. This also gets really hairy when older cookbooks are
incompatible with newer cookbooks or different chef releases, and some
servers need one or the other. Recent architectural changes in the yum and
mysql cookbooks are good examples where the chef server, itself, might need
to have multiple cookbooks for multiple categories of servers. And
Berksfile and a chef server do not trivially support deploying multiple
cookbook versions, especially legacy ones that have not historically been
deployed on that chef server. And they break horribly when different
versions of the same numbered cookbook are in use on two development hosts
with the same chef server or the same "tarball building setup" as you
mentioned, which happens when two different people are working on different
parts of the same cookbook at the same time in the same chef server
environment.

If your day to day workstation is a lightweight Vm you recreate often,
include
chef-dk or do bundle install in the source template box...

"Recreating vm's" is a surprisingly resource expensive operation in many
environments. It requires privileges to create VM's, and the sys-admin time
and resources to manage those privileges to prevent mistakes which may
corrupt critical resources. Being able to spin up a VM or test environment
with only local git privileges and no chef server privileges is much
safer in many environments.

I don't see any use case where you would do a bundle install from a
cookbook
dir within a chef run, you should install needed gems for a recipe with
chef-gem
resource if you need it inside your recipes.

Roles and node configurations and environments benefit from having
consistent, locked cookbooks especially between QA, staging, and production
environments. A "template box" is not good enough for production if you've
not recorded your chef changes, especially including cookbook versions, and
the 'Berksfile.lock' for chef-solo on individual test environments provides
that.

There are trade-offs: you can't use "search" functions as you might on a
chef server. But for many test environments, testing on a single chef-solo
instance and then propagating it to the relevant staging or production
environment is quite useful.

Please enlight me if I missed your point...

I'm happy to do so: see above.

Nico Kadel-Garcia
Lead DevOps Engineer
nkadel@skyhookwireless.com

5

On Mon, Jan 5, 2015 at 7:20 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

you are just recreating a chef server by all this. you can certain use git
or any other SCM server itself, but this only mean you are using source code
management server as an artifact server, which is fine, but then you are
bound to face with other limitations (some of which you have already
encountered). I am failing to understand the access control bits, they are
same, its just now you have coupled it with source control (unless your git
repos are public).

Regarding your versioning concerns, git sha != cookbook version. in general
source code revision version != artifact version. Source code versions
solves different problems than artifact versions (which can be compared,
hence build tools can use them).

...but you can use tags for sure, which you should represent the same artifact
that would be uplaoded to an artifact repository.

the approach you are taking is way more insecure and dubious, by your own
arguments.

if you do take this route, you should just check in all dependency cookbooks
inside your own repo (because by all means you are treating it as a artifact
server), may be do some tooling with git sub-modules etc. This will solve
your current problems, you wont need a build/assembly tool. In fact this is
what most of us did with puppet or early days of chef.

...or instead of using submodules or checking in vendored cookbooks you can
resort to only using tag references in the Berksfile.

Using SCM instead of an artifact repository is especially suitable if you have
a chef zero / solo based workflow where you don't need a chef server otherwise
in my opinion.

Cheers,
Torben

cheers
ranjib

On Mon, Jan 5, 2015 at 7:57 AM, Nico Kadel-Garcia
nkadel@skyhookwireless.com wrote:

-----Original Message-----
From: Tensibai Zhaoying [mailto:tensibai@iabis.net]
Sent: Friday, January 02, 2015 5:21 PM
Cc: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: berkshelf gem listed as a requirement in
cookbook
gemfile

Why the hell would you do a bundle install on a node ?

Berkshelf is of use on your workstation to create a pack of cookbooks
with
correct versions to distribute as tarball or to upload to chef-server.

Or using it under chef-solo to pull and apply on the local host.

   cd /var/chef-solo
   git pull [ designated git repo, tag, branch, etc., etc. ]
       [ Modify locally in testable environments ]
       [ Or checkout out git tag for production ]
   berks install
   berks vendor
       [ Consistent Berksfile.lock allows use of hard-coded cookbook

versions, including rollback to previous version without touching upstream ]
chef-solo [ with configuration that looks in
/var/chef-solo/berks-cookbooks ]
[ test ]
[ Record git changes ]
[ Push git changes back upstream ]

'Deploy from tarball' involves maintaining tarballs and an available
server to deploy them from, and potentially pushing tarballs from a
development server to the chef server, which raises serious management and
security concerns. This also gets really hairy when older cookbooks are
incompatible with newer cookbooks or different chef releases, and some
servers need one or the other. Recent architectural changes in the yum and
mysql cookbooks are good examples where the chef server, itself, might need
to have multiple cookbooks for multiple categories of servers. And Berksfile
and a chef server do not trivially support deploying multiple cookbook
versions, especially legacy ones that have not historically been deployed on
that chef server. And they break horribly when different versions of the
same numbered cookbook are in use on two development hosts with the same
chef server or the same "tarball building setup" as you mentioned, which
happens when two different people are working on different parts of the same
cookbook at the same time in the same chef server environment.

If your day to day workstation is a lightweight Vm you recreate often,
include
chef-dk or do bundle install in the source template box...

"Recreating vm's" is a surprisingly resource expensive operation in many
environments. It requires privileges to create VM's, and the sys-admin time
and resources to manage those privileges to prevent mistakes which may
corrupt critical resources. Being able to spin up a VM or test environment
with only local git privileges and no chef server privileges is much safer
in many environments.

I don't see any use case where you would do a bundle install from a
cookbook
dir within a chef run, you should install needed gems for a recipe with
chef-gem
resource if you need it inside your recipes.

Roles and node configurations and environments benefit from having
consistent, locked cookbooks especially between QA, staging, and production
environments. A "template box" is not good enough for production if you've
not recorded your chef changes, especially including cookbook versions, and
the 'Berksfile.lock' for chef-solo on individual test environments provides
that.

There are trade-offs: you can't use "search" functions as you might on a
chef server. But for many test environments, testing on a single chef-solo
instance and then propagating it to the relevant staging or production
environment is quite useful.

Please enlight me if I missed your point...

I'm happy to do so: see above.

Nico Kadel-Garcia
Lead DevOps Engineer
nkadel@skyhookwireless.com