Listing files in cookbook/files/default/

Chef Infra (archive)
1

Is there a better way to dynamically list the contents of the
/files/default/ directory from a recipe at runtime than
inspecting that path in /var/chef/cache/cookbooks ?

My concern is that the Chef sometimes removes files from the cache
that are in the cookbook but it doesn’t consider currently needed.
Thank you in advance.

-J

2

Hey,

On 09/25/2011 03:06 AM, Jason J. W. Williams wrote:

Is there a better way to dynamically list the contents of the
/files/default/ directory from a recipe at runtime than
inspecting that path in /var/chef/cache/cookbooks ?

My concern is that the Chef sometimes removes files from the cache
that are in the cookbook but it doesn't consider currently needed.
Thank you in advance.

I can't say anything about a better way than listing. But I can't agree
that it doesn't cconsider if file is needed. Unless you remove a file
resource that uses /files/default/* the file will persist in the node cache.
Maybe you could think of defining a file resource with action :nothing
that obviously can help to solve the problem.

Denis

-J

3

Hello,

On Sep 24, 2011, at 17:06, “Jason J. W. Williams” jasonjwwilliams@gmail.com
wrote:

Is there a better way to dynamically list the contents of the
/files/default/ directory from a recipe at runtime than
inspecting that path in /var/chef/cache/cookbooks ?

My concern is that the Chef sometimes removes files from the cache
that are in the cookbook but it doesn’t consider currently needed.

Chef only downloads cookbook_file and template assets if it needs to render
the target file. Otherwise they are cleaned up at the end of the chef run.

4

Is there a way to iterate the files list through the API? Do templates follow the same rule?

-J

Sent via iPhone

Is your email Premiere?

On Sep 25, 2011, at 13:44, Joshua Timberman joshua@opscode.com wrote:

Hello,

On Sep 24, 2011, at 17:06, "Jason J. W. Williams" jasonjwwilliams@gmail.com wrote:

Is there a better way to dynamically list the contents of the
/files/default/ directory from a recipe at runtime than
inspecting that path in /var/chef/cache/cookbooks ?

My concern is that the Chef sometimes removes files from the cache
that are in the cookbook but it doesn't consider currently needed.

Chef only downloads cookbook_file and template assets if it needs to render the target file. Otherwise they are cleaned up at the end of the chef run.

5

When do templates get cached? When the cookbook is loaded or when the
template is first called?

-J

On Sun, Sep 25, 2011 at 1:44 PM, Joshua Timberman joshua@opscode.com wrote:

Hello,

On Sep 24, 2011, at 17:06, "Jason J. W. Williams"
jasonjwwilliams@gmail.com wrote:

Is there a better way to dynamically list the contents of the
/files/default/ directory from a recipe at runtime than
inspecting that path in /var/chef/cache/cookbooks ?

My concern is that the Chef sometimes removes files from the cache
that are in the cookbook but it doesn't consider currently needed.

Chef only downloads cookbook_file and template assets if it needs to render
the target file. Otherwise they are cleaned up at the end of the chef run.

6

When the template is called.

Adam

On Tue, Sep 27, 2011 at 1:59 PM, Jason J. W. Williams
jasonjwwilliams@gmail.com wrote:

When do templates get cached? When the cookbook is loaded or when the
template is first called?

-J

On Sun, Sep 25, 2011 at 1:44 PM, Joshua Timberman joshua@opscode.com wrote:

Hello,

On Sep 24, 2011, at 17:06, "Jason J. W. Williams"
jasonjwwilliams@gmail.com wrote:

Is there a better way to dynamically list the contents of the
/files/default/ directory from a recipe at runtime than
inspecting that path in /var/chef/cache/cookbooks ?

My concern is that the Chef sometimes removes files from the cache
that are in the cookbook but it doesn't consider currently needed.

Chef only downloads cookbook_file and template assets if it needs to render
the target file. Otherwise they are cleaned up at the end of the chef run.

--
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com

7

Thanks Adam. I'm trying to avoid hardcoding a list of templates into
my recipe, and instead introspect a sub-directory off templates/ to
auto-discover new files. Tried this with cookbook files instead of
templates, but those get cleaned out of the cache. Is there any
reliable way to introspect the authoritative contents of a templates/
subdirectory?

-J

On Tue, Sep 27, 2011 at 4:08 PM, Adam Jacob adam@opscode.com wrote:

When the template is called.

Adam

On Tue, Sep 27, 2011 at 1:59 PM, Jason J. W. Williams
jasonjwwilliams@gmail.com wrote:

When do templates get cached? When the cookbook is loaded or when the
template is first called?

-J

On Sun, Sep 25, 2011 at 1:44 PM, Joshua Timberman joshua@opscode.com wrote:

Hello,

On Sep 24, 2011, at 17:06, "Jason J. W. Williams"
jasonjwwilliams@gmail.com wrote:

Is there a better way to dynamically list the contents of the
/files/default/ directory from a recipe at runtime than
inspecting that path in /var/chef/cache/cookbooks ?

My concern is that the Chef sometimes removes files from the cache
that are in the cookbook but it doesn't consider currently needed.

Chef only downloads cookbook_file and template assets if it needs to render
the target file. Otherwise they are cleaned up at the end of the chef run.

--
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com

8

Yep. You can do this:

run_context.cookbook_collection["COOKBOOKNAME"].template_filenames

Which will result in an array of filenames. If you had
'default/foo.rb' and 'default/bar.rb', you'll get:

[ 'foo.rb', 'bar.rb' ]

Should give you what you want, with a bit of munging.

Best,
Adam

On Tue, Sep 27, 2011 at 3:15 PM, Jason J. W. Williams
jasonjwwilliams@gmail.com wrote:

Thanks Adam. I'm trying to avoid hardcoding a list of templates into
my recipe, and instead introspect a sub-directory off templates/ to
auto-discover new files. Tried this with cookbook files instead of
templates, but those get cleaned out of the cache. Is there any
reliable way to introspect the authoritative contents of a templates/
subdirectory?

-J

On Tue, Sep 27, 2011 at 4:08 PM, Adam Jacob adam@opscode.com wrote:

When the template is called.

Adam

On Tue, Sep 27, 2011 at 1:59 PM, Jason J. W. Williams
jasonjwwilliams@gmail.com wrote:

When do templates get cached? When the cookbook is loaded or when the
template is first called?

-J

On Sun, Sep 25, 2011 at 1:44 PM, Joshua Timberman joshua@opscode.com wrote:

Hello,

On Sep 24, 2011, at 17:06, "Jason J. W. Williams"
jasonjwwilliams@gmail.com wrote:

Is there a better way to dynamically list the contents of the
/files/default/ directory from a recipe at runtime than
inspecting that path in /var/chef/cache/cookbooks ?

My concern is that the Chef sometimes removes files from the cache
that are in the cookbook but it doesn't consider currently needed.

Chef only downloads cookbook_file and template assets if it needs to render
the target file. Otherwise they are cleaned up at the end of the chef run.

--
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com

--
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com

9

Hi,

Is there a chef way to store the output of a bash command as a ruby variable?

Or should I use plain old ruby?

E.g. I want to do this:

$ hostname -d

  • and store the result in a variable, which I can pass via variables to a template.

Thanks

Geoff

10

v = hostname -d

but you should look at ohai attributes before.

2011/9/29 Geoff Meakin Acid geoffmeakin@aciddevelopments.co.uk

Hi,

Is there a chef way to store the output of a bash command as a ruby
variable?

Or should I use plain old ruby?

E.g. I want to do this:

$ hostname -d

  • and store the result in a variable, which I can pass via variables to a
    template.

Thanks

Geoff

11

On Thu, Sep 29, 2011 at 5:44 AM, Akzhan Abdulin
akzhan.abdulin@gmail.com wrote:

v = hostname -d
but you should look at ohai attributes before.

Right. Akzhan's example is using backticks, which is the simple Ruby
way to execute a command and grab its output. Ohai and Chef use more
complicated methods for running commands to deal with numerous edge
cases, like where STDOUT and STDERR go, and what happens if the
command blocks on something and hangs.

You can get all the hostname information from ohai. Hostname, domain,
and fqdn are all stored in top level node attributes. So in a recipe
you could have:

h = node[:hostname]
d = node[:domain]
f = node[:fqdn]

Chef::Log.info "My hostname is #{h}, my domain is #{d}, and my FQDN is #{f}"

These are automatic attributes [1] populated on each run by Ohai data.

If you want to use them in a template you can do:

template "/tmp/foo" do
source "foo.erb"
variables({
:h => node[:hostname],
:d => node[:domain]
})
end

Then you will have an h and d variable available to you in the
template that will automatically be populated with the data from Ohai.
This is important because if you ever decided that you needed to get
this variable a different way, you don't have to change your template
but just change the above template resource so that 'd' or 'h' is
populated from somewhere else.

Bryan

[1] http://wiki.opscode.com/display/chef/Automatic+Attributes

12

Hi all,

I came across an annoyance in that, I now have set up several chef infrastructures, production/staging/test/dev etc., and opted to have each one have its own chef server as they span different physical locations.

However the standard knife client reads its config from /etc/chef/.pem , or ~/.chef/.pem by default.

I notice you can override this to locations of your choice via a knife.rb file, which can be located in $(pwd)/.chef/knife.rb or ~/.chef/knife.rb or /etc/chef/knife.rb

Which is great, as now I can have multiple knife files in different locations, for each of the chef servers (which I’ve done).

Only problem is “knife configure -i” which is the documented way to register yourself as a client of each chef server… this attempts to override whatever knife.rb file it finds… not very helpful as I’ve already configured my various knife.rb files to look at different servers with different ‘environment’ attributes. I dont want to have all these manual steps of post-configuring knife.rb files, I want my knife.rb files committed to git so I can just run&go (™)

For my solution, I created a bunch of different directories - chef-production / chef-staging / chef-dev etc., and in each of those had a .chef/knife.rb file configuring knife to the correct place.

Furthermore, in each of the directories, I had a subdir “setup” with a script which copies the relevant validation/webui.pems into ~/.chef, runs knife against them with no setup/.chef/knife.rb to be found (it doesnt exist), and copies the resultant client.pem out into the correct directories, so each knife client is totally self contained with all its certs. It drives knife configure -i via expect to do this.

A bit of jumping through hoops, but it works a treat.

However, it strikes me as inelegant and I would be interested to hear your views on how you solved this (for those that did).

–

Geoff

13

Thanks guys, thats perfect and a real help

--

Geoff

On 29 Sep 2011, at 13:19, Bryan McLellan wrote:

On Thu, Sep 29, 2011 at 5:44 AM, Akzhan Abdulin
akzhan.abdulin@gmail.com wrote:

v = hostname -d
but you should look at ohai attributes before.

Right. Akzhan's example is using backticks, which is the simple Ruby
way to execute a command and grab its output. Ohai and Chef use more
complicated methods for running commands to deal with numerous edge
cases, like where STDOUT and STDERR go, and what happens if the
command blocks on something and hangs.

You can get all the hostname information from ohai. Hostname, domain,
and fqdn are all stored in top level node attributes. So in a recipe
you could have:

h = node[:hostname]
d = node[:domain]
f = node[:fqdn]

Chef::Log.info "My hostname is #{h}, my domain is #{d}, and my FQDN is #{f}"

These are automatic attributes [1] populated on each run by Ohai data.

If you want to use them in a template you can do:

template "/tmp/foo" do
source "foo.erb"
variables({
:h => node[:hostname],
:d => node[:domain]
})
end

Then you will have an h and d variable available to you in the
template that will automatically be populated with the data from Ohai.
This is important because if you ever decided that you needed to get
this variable a different way, you don't have to change your template
but just change the above template resource so that 'd' or 'h' is
populated from somewhere else.

Bryan

[1] http://wiki.opscode.com/display/chef/Automatic+Attributes

14

In my Knife file I'm just using a simple if block like so:

environment = "Prod"
#environment = "Stage"
#environment = "Dev"

if environment ==" Prod"

elseif environment == "Stage"

elseif environment == "Dev"

end

All I do is change the comment to the environment/application that I want.
It's worked well and simple.

On Thu, Sep 29, 2011 at 7:34 AM, Geoff Meakin Acid <
geoffmeakin@aciddevelopments.co.uk> wrote:

Hi all,

I came across an annoyance in that, I now have set up several chef
infrastructures, production/staging/test/dev etc., and opted to have each
one have its own chef server as they span different physical locations.

However the standard knife client reads its config from /etc/chef/.pem ,
or ~/.chef/.pem by default.

I notice you can override this to locations of your choice via a knife.rb
file, which can be located in $(pwd)/.chef/knife.rb or ~/.chef/knife.rb or
/etc/chef/knife.rb

Which is great, as now I can have multiple knife files in different
locations, for each of the chef servers (which I've done).

Only problem is "knife configure -i" which is the documented way to
register yourself as a client of each chef server… this attempts to override
whatever knife.rb file it finds.. not very helpful as I've already
configured my various knife.rb files to look at different servers with
different 'environment' attributes. I dont want to have all these manual
steps of post-configuring knife.rb files, I want my knife.rb files committed
to git so I can just run&go (™)

For my solution, I created a bunch of different directories -
chef-production / chef-staging / chef-dev etc., and in each of those had a
.chef/knife.rb file configuring knife to the correct place.

Furthermore, in each of the directories, I had a subdir "setup" with a
script which copies the relevant validation/webui.pems into ~/.chef, runs
knife against them with no setup/.chef/knife.rb to be found (it doesnt
exist), and copies the resultant client.pem out into the correct
directories, so each knife client is totally self contained with all its
certs. It drives knife configure -i via expect to do this.

A bit of jumping through hoops, but it works a treat.

However, it strikes me as inelegant and I would be interested to hear your
views on how you solved this (for those that did).

--

Geoff

15

Thanks for the inspiration! In my situation i basically just have 2
chef servers, preprod and production. I was just symlinking
~/.chef/knife.rb to the knife config of the server i wanted at the
moment (I didn't really want the files in my repo). Now I've instead
symlinked ~/bin/pknife to /usr/bin/knife and created 1 knife.rb with
this:

if File.basename($0) == 'pknife' then

else

end

This way by running knife I get preproduction. When I wanna touch
prod, I just use pknife instead. No need to edit any files/symlinks
and its always clear what I'm gonna touch if I forget what state I
left things in. Ruby config files are cool!

KC

On Thu, Sep 29, 2011 at 6:18 AM, Bryan Brandau agent462@gmail.com wrote:

In my Knife file I'm just using a simple if block like so:
environment = "Prod"
#environment = "Stage"
#environment = "Dev"
if environment ==" Prod"

elseif environment == "Stage"

elseif environment == "Dev"

end

All I do is change the comment to the environment/application that I want.
It's worked well and simple.

On Thu, Sep 29, 2011 at 7:34 AM, Geoff Meakin Acid
geoffmeakin@aciddevelopments.co.uk wrote:

Hi all,
I came across an annoyance in that, I now have set up several chef
infrastructures, production/staging/test/dev etc., and opted to have each
one have its own chef server as they span different physical locations.
However the standard knife client reads its config from /etc/chef/.pem ,
or ~/.chef/.pem by default.
I notice you can override this to locations of your choice via a knife.rb
file, which can be located in $(pwd)/.chef/knife.rb or ~/.chef/knife.rb or
/etc/chef/knife.rb
Which is great, as now I can have multiple knife files in different
locations, for each of the chef servers (which I've done).
Only problem is "knife configure -i" which is the documented way to
register yourself as a client of each chef server… this attempts to override
whatever knife.rb file it finds.. not very helpful as I've already
configured my various knife.rb files to look at different servers with
different 'environment' attributes. I dont want to have all these manual
steps of post-configuring knife.rb files, I want my knife.rb files committed
to git so I can just run&go (™)

For my solution, I created a bunch of different directories -
chef-production / chef-staging / chef-dev etc., and in each of those had a
.chef/knife.rb file configuring knife to the correct place.
Furthermore, in each of the directories, I had a subdir "setup" with a
script which copies the relevant validation/webui.pems into ~/.chef, runs
knife against them with no setup/.chef/knife.rb to be found (it doesnt
exist), and copies the resultant client.pem out into the correct
directories, so each knife client is totally self contained with all its
certs. It drives knife configure -i via expect to do this.
A bit of jumping through hoops, but it works a treat.
However, it strikes me as inelegant and I would be interested to hear your
views on how you solved this (for those that did).

--
Geoff

16

I used a similar approach to select the chef server based on git branch name.

begin
require 'grit'
branch = Grit::Repo.new("#{current_dir}/..").head.name
rescue LoadError
puts "Unable to load grit. Are you using the correct RVM gemset?"
branch = "master"
end

case branch
when "production"
chef_server_url "https://chef.production.example.com"
else
chef_server_url "http://chef.development.example.com:4000"
end

On Fri, Sep 30, 2011 at 4:59 PM, KC Braunschweig
kcbraunschweig@gmail.com wrote:

Thanks for the inspiration! In my situation i basically just have 2
chef servers, preprod and production. I was just symlinking
~/.chef/knife.rb to the knife config of the server i wanted at the
moment (I didn't really want the files in my repo). Now I've instead
symlinked ~/bin/pknife to /usr/bin/knife and created 1 knife.rb with
this:

if File.basename($0) == 'pknife' then

else

end

This way by running knife I get preproduction. When I wanna touch
prod, I just use pknife instead. No need to edit any files/symlinks
and its always clear what I'm gonna touch if I forget what state I
left things in. Ruby config files are cool!

KC

On Thu, Sep 29, 2011 at 6:18 AM, Bryan Brandau agent462@gmail.com wrote:

In my Knife file I'm just using a simple if block like so:
environment = "Prod"
#environment = "Stage"
#environment = "Dev"
if environment ==" Prod"

elseif environment == "Stage"

elseif environment == "Dev"

end

All I do is change the comment to the environment/application that I want.
It's worked well and simple.

On Thu, Sep 29, 2011 at 7:34 AM, Geoff Meakin Acid
geoffmeakin@aciddevelopments.co.uk wrote:

Hi all,
I came across an annoyance in that, I now have set up several chef
infrastructures, production/staging/test/dev etc., and opted to have each
one have its own chef server as they span different physical locations.
However the standard knife client reads its config from /etc/chef/.pem ,
or ~/.chef/.pem by default.
I notice you can override this to locations of your choice via a knife.rb
file, which can be located in $(pwd)/.chef/knife.rb or ~/.chef/knife.rb or
/etc/chef/knife.rb
Which is great, as now I can have multiple knife files in different
locations, for each of the chef servers (which I've done).
Only problem is "knife configure -i" which is the documented way to
register yourself as a client of each chef server… this attempts to override
whatever knife.rb file it finds.. not very helpful as I've already
configured my various knife.rb files to look at different servers with
different 'environment' attributes. I dont want to have all these manual
steps of post-configuring knife.rb files, I want my knife.rb files committed
to git so I can just run&go (™)

For my solution, I created a bunch of different directories -
chef-production / chef-staging / chef-dev etc., and in each of those had a
.chef/knife.rb file configuring knife to the correct place.
Furthermore, in each of the directories, I had a subdir "setup" with a
script which copies the relevant validation/webui.pems into ~/.chef, runs
knife against them with no setup/.chef/knife.rb to be found (it doesnt
exist), and copies the resultant client.pem out into the correct
directories, so each knife client is totally self contained with all its
certs. It drives knife configure -i via expect to do this.
A bit of jumping through hoops, but it works a treat.
However, it strikes me as inelegant and I would be interested to hear your
views on how you solved this (for those that did).

--
Geoff

--
Opscode, Inc
Joshua Timberman
IRC, Skype, Twitter, Github: jtimberman

17

On Thu, Sep 29, 2011 at 8:34 AM, Geoff Meakin Acid
geoffmeakin@aciddevelopments.co.uk wrote:

Only problem is "knife configure -i" which is the documented way to register
yourself as a client of each chef server… this attempts to override whatever
knife.rb file it finds.. not very helpful as I've already configured my
various knife.rb files to look at different servers with different
'environment' attributes. I dont want to have all these manual steps of
post-configuring knife.rb files, I want my knife.rb files committed to git
so I can just run&go (™)

"knife configure -i" is really for a standard configuration and for
new users. It just uses the webui client, which is an admin, to create
another client. You can do this easily with:

sudo knife client create an_admin -u chef-webui -k /etc/chef/webui.pem
--admin -f .chef/an_admin.pem

You'll need to fix up the file permissions since you ran as sudo,
which you need to do so you can read the webui.pem file.

For my solution, I created a bunch of different directories -
chef-production / chef-staging / chef-dev etc., and in each of those had a
.chef/knife.rb file configuring knife to the correct place.

I have a directory in which I keep all of my chef repositories. Each
of these has a separate .chef directory. For Opscode Hosted Chef
organizations, the directory is the same name as the organization and
my .chef/knife.rb looks like:

config_dir = File.dirname(FILE)
cookbook_dir = File.join(config_dir, ".." ,"cookbooks")
organization = File.basename(File.expand_path(File.join(File.dirname(FILE),
"..")))

log_level :info
log_location STDOUT
node_name ENV['USER']
client_key "#{config_dir}/#{ENV['USER']}.pem"
chef_server_url "Sign In - Chef Manage"
cache_type 'BasicFile'
cache_options( :path => "#{config_dir}/checksums" )
cookbook_path [ cookbook_dir ]

validation_client_name "#{organization}-validator"
validation_key "#{config_dir}/#{organization}-validator.pem"

I also have a couple directories for my local and testing chef servers
which contain pretty standard knife.rb configurations.

Bryan