Knife bootstrap Command Not Applying the Run List

Chef Infra (archive)
1

I’m sending this to the mailing list because I have posted a bug on 3/6/2012 and haven’t gotten a response. Any help is much appreciated. Here’s the ticket:

http://tickets.opscode.com/browse/CHEF-2962

When running the knife bootstrap command and passing in the -r option to specify the run list, the node installs the chef-client but does not perform a chef-client run and completely ignores the run list. There is also no first-boot.json file present in /etc/chef. We get these results when using the -d option and a distro file to install chef-client from apt packages and we also get these results when not using -d option the chef-client installs through gem. Adding the -VV option gives us no informative error. knife-bootstrap just stops after the chef-client installation. We get this behavior from knife on two different Linux workstations. One in Ubuntu 10.04 Server LTS and the other is Linux Mint 12. Here are the commands that we have used that produce these results:

knife bootstrap uscheflab015 -x root -P ROOTPASSWORD -d ubuntu-wt-apt -r ‘role[base]’ -N uscheflab015.ourdomain.com -VV --bootstrap-proxy http://proxy.ourdomain.com:8080http://proxy.ourdomain.com:8080/

knife bootstrap uscheflab015 -x root -P ROOTPASSWORD -r ‘role[base]’ -N uscheflab015.ourdomain.com -VV --bootstrap-proxy http://proxy.ourdomain.com:8080http://proxy.ourdomain.com:8080/

Here is the ubuntu-wt-apt distro template we are using. We had to modify it not to include the proxy information in the client.rb file. The knife bootstrap command needs the proxy to get out of our internal network only to download the Opscode repository key via wget, but the node does not need the proxy in order to contact the chef-server on our internal network.

Here is our ubuntu-wt-apt template. It’s identical to ubuntu10.04-apt except it doesn’t place the proxy info into the client.rb.

http://pastie.org/3677629

I can confirm that this is also happening with the default ubuntu10.04-apt bootstrap template. Here is the command I’m running that still stops short of doing a chef run:

knife bootstrap 10.116.27.100 -x root -N uscheflabext001.field.jw.org --bootstrap-proxy http://proxy.ourproxy.com:8080http://proxy.ourproxy.com:8080/ -r ‘role[base]’ -s https://usmgmtext016.field.jw.orghttps://usmgmtext016.field.jw.org/ -d ubuntu10.04-apt -VV

Also, doing the following (as mentioned in the Knife Bootstrap wiki page) does not resolve the issue as there is no such file /etc/chef/first-boot.json

root@USCHEFLABEXT001:~# chef-client -j /etc/chef/first-boot.json
[Mon, 26 Mar 2012 16:53:50 -0400] FATAL: I cannot find /etc/chef/first-boot.json

Kind regards,

Ian D. Rossi
CD - Server Infrastructure
Phone 23834

2

Hi Ian,

On Mar 27, 2012, at 5:11 AM, IDROSSI@jw.org IDROSSI@jw.org wrote:

I'm sending this to the mailing list because I have posted a bug on 3/6/2012 and haven't gotten a response. Any help is much appreciated. Here's the ticket:
http://tickets.opscode.com/browse/CHEF-2962

When running the knife bootstrap command and passing in the -r option to specify the run list, the node installs the chef-client but does not perform a chef-client run and completely ignores the run list. There is also no first-boot.json file present in /etc/chef. We get these results when using the -d option and a distro file to install chef-client from apt packages and we also get these results when not using -d option the chef-client installs through gem. Adding the -VV option gives us no informative error. knife-bootstrap just stops after the chef-client installation. We get this behavior from knife on two different Linux workstations. One in Ubuntu 10.04 Server LTS and the other is Linux Mint 12. Here are the commands that we have used that produce these results:

So the bootstrap completes successfully? Have you verified the client.rb is written correctly? Namely, if you're using a proxy I would image that knife[:bootstrap_proxy] should be properly set.

Also, doing the following (as mentioned in the Knife Bootstrap wiki page) does not resolve the issue as there is no such file /etc/chef/first-boot.json

root@USCHEFLABEXT001:~# chef-client -j /etc/chef/first-boot.json
[Mon, 26 Mar 2012 16:53:50 -0400] FATAL: I cannot find /etc/chef/first-boot.json

as far as this is concerned, verify that the contents of first-boot.jason is something like:

{"run_list":["role[base]"]}

You can create this file yourself if it doesn't exist.

3

Hi Sean,

Yes, the bootstrap finishes with no errors, it just doesn’t add the run list or perform a chef run. The client.rb file is written properly, except it didn’t set the proxy options. Also, sometimes bootstrap fails to start the chef-client daemon. I did this twice for confirmation–the second I reverted to a clean snapshot of the VM. This is the command that produced the aforementioned behavior and the following client.rb:

knife bootstrap 10.116.27.100 -x root -N uscheflabext001.field.jw.org --bootstrap-proxy http://proxy.ourdomain.net:8080 -r ‘role[base]’ -s https://usmgmtext016.ourdomain.net -d ubuntu10.04-apt -VV

log_level :info
log_location STDOUT
ssl_verify_mode :verify_none
chef_server_url "https://usmgmtext016.field.jw.org"
file_cache_path "/var/cache/chef"
file_backup_path "/var/lib/chef/backup"
pid_file "/var/run/chef/client.pid"
cache_options({ :path => “/var/cache/chef/checksums”, :skip_expires => true})
signing_ca_user "chef"
Mixlib::Log::Formatter.show_time = true
node_name “uscheflabext001.field.jw.org”

Ian D. Rossi
CD - Server Infrastructure
Phone 23834

From: sean escriva [sean.escriva@gmail.com]
Sent: Tuesday, March 27, 2012 2:41 PM
To: chef@lists.opscode.com
Subject: [chef] Re: knife bootstrap Command Not Applying the Run List

Hi Ian,

On Mar 27, 2012, at 5:11 AM, <IDROSSI@jw.orgmailto:IDROSSI@jw.org> <IDROSSI@jw.orgmailto:IDROSSI@jw.org> wrote:

I’m sending this to the mailing list because I have posted a bug on 3/6/2012 and haven’t gotten a response. Any help is much appreciated. Here’s the ticket:
http://tickets.opscode.com/browse/CHEF-2962

When running the knife bootstrap command and passing in the -r option to specify the run list, the node installs the chef-client but does not perform a chef-client run and completely ignores the run list. There is also no first-boot.json file present in /etc/chef. We get these results when using the -d option and a distro file to install chef-client from apt packages and we also get these results when not using -d option the chef-client installs through gem. Adding the -VV option gives us no informative error. knife-bootstrap just stops after the chef-client installation. We get this behavior from knife on two different Linux workstations. One in Ubuntu 10.04 Server LTS and the other is Linux Mint 12. Here are the commands that we have used that produce these results:

So the bootstrap completes successfully? Have you verified the client.rb is written correctly? Namely, if you’re using a proxy I would image that knife[:bootstrap_proxy] should be properly set.

Also, doing the following (as mentioned in the Knife Bootstrap wiki page) does not resolve the issue as there is no such file /etc/chef/first-boot.json

root@USCHEFLABEXT001:~# chef-client -j /etc/chef/first-boot.json
[Mon, 26 Mar 2012 16:53:50 -0400] FATAL: I cannot find /etc/chef/first-boot.json

as far as this is concerned, verify that the contents of first-boot.jason is something like:

{“run_list”:[“role[base]”]}

You can create this file yourself if it doesn’t exist.

4

Ian,

You can see the steps the bootstrap would follow by looking at the
’ubuntu10.04-apt.erb’ file in the gem (gem contents chef | grep
ubuntu10.04-apt).

As Sean recommended, please check to see that
/etc/chef/first-boot.json has been created. This will give a better
idea of where you’re having a problem.

Bryan

5

Hi Bryan,

Thank you. I have examined the ubuntu10.04.apt.erb file as well as the bootstrap.erb file itself and I cannot seem to find any clues. The -VV verbose output of the bootstrap command also does not produce any error messages or actionable clues. Also, the bootstrap session did not create a first-boot.json file.

Here is the verbose output of the bootstrap command in two parts. Also here is the bootstrap command:
http://pastie.org/3685719
http://pastie.org/3685721

knife bootstrap 10.116.27.100 -x root -N uscheflabext001.ourdomain.net --bootstrap-proxy http://proxy.ourdomain.net:8080 -r ‘role[base]’ -s https://usmgmtext016.ourdomain.net -d ubuntu10.04-apt -VV

Ian D. Rossi
CD - Server Infrastructure
Phone 23834

From: Bryan McLellan [btm@loftninjas.org]
Sent: Wednesday, March 28, 2012 10:21 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: knife bootstrap Command Not Applying the Run List

Ian,

You can see the steps the bootstrap would follow by looking at the
’ubuntu10.04-apt.erb’ file in the gem (gem contents chef | grep
ubuntu10.04-apt).

As Sean recommended, please check to see that
/etc/chef/first-boot.json has been created. This will give a better
idea of where you’re having a problem.

Bryan

6

Ian,

You didn't mention /etc/chef/first-boot.json. Did you look for this file?

On Wed, Mar 28, 2012 at 10:21 AM, Bryan McLellan btm@loftninjas.org wrote:

As Sean recommended, please check to see that
/etc/chef/first-boot.json has been created. This will give a better
idea of where you're having a problem.

7

Hi Bryan,

I believe I did mention it. That file does not exist /etc/chef when the problem occurs.

Ian D. Rossi
CD - Server Infrastructure
Phone 23834

From: Bryan McLellan [btm@loftninjas.org]
Sent: Wednesday, March 28, 2012 11:04 AM
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Re: knife bootstrap Command Not Applying the Run List

Ian,

You didn't mention /etc/chef/first-boot.json. Did you look for this file?

On Wed, Mar 28, 2012 at 10:21 AM, Bryan McLellan btm@loftninjas.org wrote:

As Sean recommended, please check to see that
/etc/chef/first-boot.json has been created. This will give a better
idea of where you're having a problem.

8

On Wed, Mar 28, 2012 at 11:06 AM, IDROSSI@jw.org wrote:

I believe I did mention it. That file does not exist /etc/chef when the problem occurs.

Ah! Apparently you mentioned it twice and I'm a terrible reader.

Is /etc/chef/validation.pem populated?

What happens if you delete all the commented lines out of your custom
bootstrap? (Lines 38-44 here: http://pastie.org/3677629)

Bryan

9

No worries :slight_smile:

Yes, the validation.pem is correct. I have also tried with the stock ubuntu10.04-apt template and I get the same behavior while using the --bootstrap-proxy option.

Ian D. Rossi
CD - Server Infrastructure
Phone 23834

From: Bryan McLellan [btm@loftninjas.org]
Sent: Wednesday, March 28, 2012 11:15 AM
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: RE: Re: Re: knife bootstrap Command Not Applying the Run List

On Wed, Mar 28, 2012 at 11:06 AM, IDROSSI@jw.org wrote:

I believe I did mention it. That file does not exist /etc/chef when the problem occurs.

Ah! Apparently you mentioned it twice and I'm a terrible reader.

Is /etc/chef/validation.pem populated?

What happens if you delete all the commented lines out of your custom
bootstrap? (Lines 38-44 here: http://pastie.org/3677629)

Bryan

10

On Wed, Mar 28, 2012 at 11:22 AM, IDROSSI@jw.org wrote:

No worries :slight_smile:

Yes, the validation.pem is correct. I have also tried with the stock ubuntu10.04-apt template and I get the same behavior while using the --bootstrap-proxy option.

I would probably:

  1. remove the commented code from the bootstrap template
  2. Add 'echo begin > /tmp/bootstrap.log' to the top of the template
  3. Add 'echo n >> /tmp/bootstrap.log' in between each step, but start
    n at 1 and incrementally increase it.

Then run the bootstrap again. This will tell you how far it got.

Bryan

11

We have figured out another way to set the proxy on the host, so we don't need to the use the --bootstrap-proxy option. It works fine without that option. However, I believe this is a bug that still remains. I imagine that this is a roadblock for that must use --bootstrap-proxy. The Chef team may want to investigate this.

Ian D. Rossi
CD - Server Infrastructure
Phone 23834

From: Bryan McLellan [btm@loftninjas.org]
Sent: Wednesday, March 28, 2012 11:58 AM
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: RE: Re: RE: Re: Re: knife bootstrap Command Not Applying the Run List

On Wed, Mar 28, 2012 at 11:22 AM, IDROSSI@jw.org wrote:

No worries :slight_smile:

Yes, the validation.pem is correct. I have also tried with the stock ubuntu10.04-apt template and I get the same behavior while using the --bootstrap-proxy option.

I would probably:

  1. remove the commented code from the bootstrap template
  2. Add 'echo begin > /tmp/bootstrap.log' to the top of the template
  3. Add 'echo n >> /tmp/bootstrap.log' in between each step, but start
    n at 1 and incrementally increase it.

Then run the bootstrap again. This will tell you how far it got.

Bryan