Failing to feed run_list to first-boot.json from userdata


#1

hai. i’m failing to feed a run_list present in ec2 userdata to a
first-boot.json file. what am i doing wrong? maybe run_list shouldn’t
be within attributes? (see userdata below).

i’m using a client.rb based on jtimberman’s gist. my modification
is here: https://gist.github.com/1648988

my knife ec2 launch command:

knife ec2 server create -N “testme” -S sshkey -f m1.large -I
ami-xxx -G sg-xxx,sg-xxx --user-data /tmp/userdata --distro distroX

distroX

bash -c ‘
S3CFG="–config=/root/.s3cfg-w-https"
S3CMDOPT="–force $S3CFG"
X_ENV=dev
X_AV_REGION=us-east-1
S3_BUCKET="x-ops-$X_ENV-$X_AV_REGION"
s3cmd get $S3CMDOPT s3://$S3_BUCKET/chef/client.frak.rb /etc/chef/client.rb
/usr/bin/chef-client -j /etc/chef/first-boot.json -l debug -L /var/log/chef/client.log

and the output from the launch:

Waiting for sshd…done
Bootstrapping Chef on ec2-xx-xx-xx-xx.compute-1.amazonaws.com
ec2-xx-xx-xx-xx.compute-1.amazonaws.com s3://x-ops-dev-east-1/chef/client.frak.rb -> /etc/chef/client.rb [1 of 1]
1502 of 1502 100% in 0s 10.31 kB/s done
ec2-xx-xx-xx-xx.compute-1.amazonaws.com FATAL: I cannot find /etc/chef/first-boot.json

root@ip-10-190-xx-xx:/etc/chef# curl -s http://169.254.169.254/latest/user-data
{
“validation_key”: “-----BEGIN RSA PRIVATE KEY-----\nMIxx_snip\n”,
“attributes”: {
“role”: “chefsrv”,
“node_name_override”: “testme”,
“env”: “dev”,
“cluster”: “dev”,
“vol_list”: [
“vol-f1xxxx”,
“vol-p3xxxx”
],
“run_list”: [
“role[base]”,
“role[postfix-client]”,
“role[syslog-ng-client]”
]
},
“chef_server”: “https://10.xx.xx.113”,
“validation_client_name”: “chef-validator”
}

i tried another test of this by placing run_list at the same level as
attributes in the userdata.

Waiting for sshd…done
Bootstrapping Chef on ec2-50-yy-yy-xx.compute-1.amazonaws.com
ec2-50-yy-yy-xx.compute-1.amazonaws.com s3://x-ops-dev-us-east-1/chef/client.frak.rb -> /etc/chef/client.rb [1 of 1]
1507 of 1507 100% in 0s 10.35 kB/s done
ec2-50-yy-yy-xx.compute-1.amazonaws.com /usr/lib/ruby/1.8/chef/client.rb:313:in converge': undefined methodrun_list’ for nil:NilClass (NoMethodError)
ec2-50-yy-yy-xx.compute-1.amazonaws.com from /usr/lib/ruby/1.8/chef/client.rb:87:in run' ec2-50-yy-yy-xx.compute-1.amazonaws.com from /usr/lib/ruby/1.8/chef/application/client.rb:215:inrun_application’
ec2-50-yy-yy-xx.compute-1.amazonaws.com from /usr/lib/ruby/1.8/chef/application/client.rb:207:in loop' ec2-50-yy-yy-xx.compute-1.amazonaws.com from /usr/lib/ruby/1.8/chef/application/client.rb:207:inrun_application’
ec2-50-yy-yy-xx.compute-1.amazonaws.com from /usr/lib/ruby/1.8/chef/application.rb:62:in `run’
ec2-50-yy-yy-xx.compute-1.amazonaws.com from /usr/bin/chef-client:25

root@domU-12-31-39-0E-E4-xx:/etc/chef# cat first-boot.json
[
“role[base]”,
“role[postfix-client]”,
“role[syslog-ng-client]”
]root@domU-12-31-39-0E-E4-xx:/etc/chef#

also, within this block (from client.rb), what does “json_attribs” do?

if chef_config.has_key?(“run_list”)
File.open("/etc/chef/first-boot.json", “w”) do |f|
f.print(JSON.pretty_generate(chef_config[“run_list”]))
end
json_attribs "/etc/chef/first-boot.json"
end

thanks!
kallen


#2

got a little farther. now i’m getting the run_list contents into a
json file. but the first chef run fails. chef client is natty, 0.10.8.
chef server is natty, 0.10.4. i’ll show the end of the debug output
at the bottom.

any help?

knife ec2 launch command is the same.

client.firstboot.rb gist updated: https://gist.github.com/1648988

the first chef run is run like so:

/usr/bin/chef-client -c /etc/chef/client.firstboot.rb -j /etc/chef/first-boot.json -l debug -L /var/log/chef/client.log

[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Sending HTTP Request via GET to chef.dev.spergatronic.com:443/nodes/cheftain03.dev.spergatronic.com
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Synchronizing cookbooks
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Signing the request as cheftain03.dev.spergatronic.com
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: String to sign: ‘Method:GET
Hashed Path:ElB0/CXW2+Oojbo2+os6SdIsdWs=
X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
X-Ops-Timestamp:2012-01-20T23:19:41Z
X-Ops-UserId:cheftain03.dev.spergatronic.com’
Header hash: {“X-Ops-Content-Hash”=>“2jmj7l5rSw0yVb/vlWAYkK/YBwk=”, “X-Ops-Authorization-1”=>“V2fmlwDSnSH5vEiQtPs86OhEsyV567RJ8W6z
zPEdxEFVT0ZKs7VDGptwzo1T”, “X-Ops-Userid”=>“cheftain03.dev.spergatronic.com”, “X-Ops-Sign”=>“version=1.0”, “X-Ops-Authorization-2”=>“O
Nfrm3C9YxBbqHgeCJcVUemHQb2WLPIBlkUPIcJtqpTsH3gZVnxbK9uGfv4s”, “X-Ops-Authorization-3”=>“womX5+3X8tntwvnFaQ2KvdzbLIySbHoixWL9ObFGBp
ZXd5sJ0h2zWyI8t0pW”, “X-Ops-Authorization-4”=>“aOsXzUKrM1CLsU0gOBSamwZgJxZoBn+WmFwehXPn3EalnZPzP9DIjob/8b+I”, “X-Ops-Authorization
-5”=>“L7ZUj7rSRWfFSLxKNKSwAME/1y3lkJIlnifOKU1zyGQMPwYWXqa97Mb8LD8z”, “X-Ops-Authorization-6”=>“QtJyIcM6XuNy3CeZk3g6M7rsTAFY95oMcO4
8YHADSA==”, “X-Ops-Timestamp”=>“2012-01-20T23:19:41Z”}
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Sending HTTP Request via GET to 10.xx.xx.xx:443/nodes/cheftain03.dev.spergatronic.com/cookbo
oks
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Cookbooks to load: {}
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Compiling recipes for node cheftain03.dev.spergatronic.com
/usr/lib/ruby/1.8/chef/client.rb:313:in converge': undefined methodrun_list’ for nil:NilClass (NoMethodError)
from /usr/lib/ruby/1.8/chef/client.rb:87:in run' from /usr/lib/ruby/1.8/chef/application/client.rb:215:inrun_application’
from /usr/lib/ruby/1.8/chef/application/client.rb:207:in loop' from /usr/lib/ruby/1.8/chef/application/client.rb:207:inrun_application’
from /usr/lib/ruby/1.8/chef/application.rb:62:in `run’
from /usr/bin/chef-client:25

root@ip-10-46-1xx-xx:/etc/chef# curl -s http://169.254.169.254/latest/user-data
{
“validation_key”: “-----BEGIN RSA PRIVATE KEY-----\nMIxx_snip\n”,
“attributes”: {
“role”: “chefsrv”,
“node_name”: “cheftain03.dev.spergatronic.com”,
“environment”: “dev”,
“cluster”: “dev”,
“vol_list”: [
“vol-f1xxxxxx”,
“vol-q2xxxxxx”
]
},
“chef_server”: “https://10.xx.xx.xx”,
“validation_client_name”: “chef-validator”,
“run_list”: [
“role[base]”,
“role[postfix-client]”,
“role[syslog-ng-client]”
]
}

On Fri, 20 Jan 2012, kallen@groknaut.net wrote:

hai. i’m failing to feed a run_list present in ec2 userdata to a
first-boot.json file. what am i doing wrong? maybe run_list shouldn’t
be within attributes? (see userdata below).

i’m using a client.rb based on jtimberman’s gist. my modification
is here: https://gist.github.com/1648988

my knife ec2 launch command:

knife ec2 server create -N “testme” -S sshkey -f m1.large -I
ami-xxx -G sg-xxx,sg-xxx --user-data /tmp/userdata --distro distroX

distroX

bash -c ‘
S3CFG="–config=/root/.s3cfg-w-https"
S3CMDOPT="–force $S3CFG"
X_ENV=dev
X_AV_REGION=us-east-1
S3_BUCKET="x-ops-$X_ENV-$X_AV_REGION"
s3cmd get $S3CMDOPT s3://$S3_BUCKET/chef/client.frak.rb /etc/chef/client.rb
/usr/bin/chef-client -j /etc/chef/first-boot.json -l debug -L /var/log/chef/client.log


#3

It looks like you are telling chef to use /etc/chef/first-boot.json
and then showing us the output of a curl command.

Show us the contents of /etc/chef/first-boot.json – it looks like
your crazy client.rb is generating it, based on the user data?

Seems awfully complex, maybe even unnecessarily so. What problem are
you solving for, here?

–AJ

On 21 January 2012 12:35, kallen@groknaut.net wrote:

got a little farther. now i’m getting the run_list contents into a
json file. but the first chef run fails. chef client is natty, 0.10.8.
chef server is natty, 0.10.4. i’ll show the end of the debug output
at the bottom.

any help?

knife ec2 launch command is the same.

client.firstboot.rb gist updated: https://gist.github.com/1648988

the first chef run is run like so:

/usr/bin/chef-client -c /etc/chef/client.firstboot.rb -j /etc/chef/first-boot.json -l debug -L /var/log/chef/client.log

[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Sending HTTP Request via GET to chef.dev.spergatronic.com:443/nodes/cheftain03.dev.spergatronic.com
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Synchronizing cookbooks
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Signing the request as cheftain03.dev.spergatronic.com
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: String to sign: ‘Method:GET
Hashed Path:ElB0/CXW2+Oojbo2+os6SdIsdWs=
X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
X-Ops-Timestamp:2012-01-20T23:19:41Z
X-Ops-UserId:cheftain03.dev.spergatronic.com’
Header hash: {“X-Ops-Content-Hash”=>“2jmj7l5rSw0yVb/vlWAYkK/YBwk=”, “X-Ops-Authorization-1”=>“V2fmlwDSnSH5vEiQtPs86OhEsyV567RJ8W6z
zPEdxEFVT0ZKs7VDGptwzo1T”, “X-Ops-Userid”=>“cheftain03.dev.spergatronic.com”, “X-Ops-Sign”=>“version=1.0”, “X-Ops-Authorization-2”=>“O
Nfrm3C9YxBbqHgeCJcVUemHQb2WLPIBlkUPIcJtqpTsH3gZVnxbK9uGfv4s”, “X-Ops-Authorization-3”=>“womX5+3X8tntwvnFaQ2KvdzbLIySbHoixWL9ObFGBp
ZXd5sJ0h2zWyI8t0pW”, “X-Ops-Authorization-4”=>“aOsXzUKrM1CLsU0gOBSamwZgJxZoBn+WmFwehXPn3EalnZPzP9DIjob/8b+I”, “X-Ops-Authorization
-5”=>“L7ZUj7rSRWfFSLxKNKSwAME/1y3lkJIlnifOKU1zyGQMPwYWXqa97Mb8LD8z”, “X-Ops-Authorization-6”=>“QtJyIcM6XuNy3CeZk3g6M7rsTAFY95oMcO4
8YHADSA==”, “X-Ops-Timestamp”=>“2012-01-20T23:19:41Z”}
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Sending HTTP Request via GET to 10.xx.xx.xx:443/nodes/cheftain03.dev.spergatronic.com/cookbo
oks
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Cookbooks to load: {}
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Compiling recipes for node cheftain03.dev.spergatronic.com
/usr/lib/ruby/1.8/chef/client.rb:313:in converge': undefined methodrun_list’ for nil:NilClass (NoMethodError)
from /usr/lib/ruby/1.8/chef/client.rb:87:in run' from /usr/lib/ruby/1.8/chef/application/client.rb:215:inrun_application’
from /usr/lib/ruby/1.8/chef/application/client.rb:207:in loop' from /usr/lib/ruby/1.8/chef/application/client.rb:207:inrun_application’
from /usr/lib/ruby/1.8/chef/application.rb:62:in `run’
from /usr/bin/chef-client:25

root@ip-10-46-1xx-xx:/etc/chef# curl -s http://169.254.169.254/latest/user-data
{
“validation_key”: “-----BEGIN RSA PRIVATE KEY-----\nMIxx_snip\n”,
“attributes”: {
“role”: “chefsrv”,
“node_name”: “cheftain03.dev.spergatronic.com”,
“environment”: “dev”,
“cluster”: “dev”,
“vol_list”: [
“vol-f1xxxxxx”,
“vol-q2xxxxxx”
]
},
“chef_server”: “https://10.xx.xx.xx”,
“validation_client_name”: “chef-validator”,
“run_list”: [
“role[base]”,
“role[postfix-client]”,
“role[syslog-ng-client]”
]
}

On Fri, 20 Jan 2012, kallen@groknaut.net wrote:

hai. i’m failing to feed a run_list present in ec2 userdata to a
first-boot.json file. what am i doing wrong? maybe run_list shouldn’t
be within attributes? (see userdata below).

i’m using a client.rb based on jtimberman’s gist. my modification
is here: https://gist.github.com/1648988

my knife ec2 launch command:

knife ec2 server create -N “testme” -S sshkey -f m1.large -I
ami-xxx -G sg-xxx,sg-xxx --user-data /tmp/userdata --distro distroX

distroX

bash -c ‘
S3CFG="–config=/root/.s3cfg-w-https"
S3CMDOPT="–force $S3CFG"
X_ENV=dev
X_AV_REGION=us-east-1
S3_BUCKET="x-ops-$X_ENV-$X_AV_REGION"
s3cmd get $S3CMDOPT s3://$S3_BUCKET/chef/client.frak.rb /etc/chef/client.rb
/usr/bin/chef-client -j /etc/chef/first-boot.json -l debug -L /var/log/chef/client.log


#4

On Sat, 21 Jan 2012, AJ Christensen wrote:

It looks like you are telling chef to use /etc/chef/first-boot.json
and then showing us the output of a curl command.

Show us the contents of /etc/chef/first-boot.json – it looks like
your crazy client.rb is generating it, based on the user data?

sorry, i forgot to provide /etc/chef/first-boot.json. and since my
last reply, i see a problem, and i’m trying to fix the problem, but
slow at doing it because i’m not experienced at ruby.

the bad /etc/chef/first-boot.json that is gen’d from my example:

[
“role[base]”,
“role[postfix-client]”,
“role[syslog-ng-client]”
]

so yeah, that should be:

{
“run_list”: [
“role[base]”,
“role[postfix-client]”,
“role[syslog-ng-client]”
]
}

any ruby tips on writing out /etc/chef/first-boot.json correctly from
userdata?

Seems awfully complex, maybe even unnecessarily so. What problem are
you solving for, here?

i’m trying to pass an initial run_list (and other attributes) to the
first chef run of an ec2 instance via userdata supplied on knife ec2
commandline. i’m considering a case where i want to launch 10s or 100s
of instances at a time, so i’d need this to be automated (i.e. i don’t
want to concoct a knife ec2 commandline each time with specific run_list
passed in on cli).

is there any wiki doc or blog post that put’s jtimberman’s gist into
more context? https://gist.github.com/319106. any docs on doing what
i say above?

thanks,
kallen

On 21 January 2012 12:35, kallen@groknaut.net wrote:

got a little farther. now i’m getting the run_list contents into a
json file. but the first chef run fails. chef client is natty, 0.10.8.
chef server is natty, 0.10.4. i’ll show the end of the debug output
at the bottom.

any help?

knife ec2 launch command is the same.

client.firstboot.rb gist updated: https://gist.github.com/1648988

the first chef run is run like so:

/usr/bin/chef-client -c /etc/chef/client.firstboot.rb -j /etc/chef/first-boot.json -l debug -L /var/log/chef/client.log

[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Sending HTTP Request via GET to chef.dev.spergatronic.com:443/nodes/cheftain03.dev.spergatronic.com
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Synchronizing cookbooks
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Signing the request as cheftain03.dev.spergatronic.com
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: String to sign: ‘Method:GET
Hashed Path:ElB0/CXW2+Oojbo2+os6SdIsdWs=
X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
X-Ops-Timestamp:2012-01-20T23:19:41Z
X-Ops-UserId:cheftain03.dev.spergatronic.com’
Header hash: {“X-Ops-Content-Hash”=>“2jmj7l5rSw0yVb/vlWAYkK/YBwk=”, “X-Ops-Authorization-1”=>“V2fmlwDSnSH5vEiQtPs86OhEsyV567RJ8W6z
zPEdxEFVT0ZKs7VDGptwzo1T”, “X-Ops-Userid”=>“cheftain03.dev.spergatronic.com”, “X-Ops-Sign”=>“version=1.0”, “X-Ops-Authorization-2”=>“O
Nfrm3C9YxBbqHgeCJcVUemHQb2WLPIBlkUPIcJtqpTsH3gZVnxbK9uGfv4s”, “X-Ops-Authorization-3”=>“womX5+3X8tntwvnFaQ2KvdzbLIySbHoixWL9ObFGBp
ZXd5sJ0h2zWyI8t0pW”, “X-Ops-Authorization-4”=>“aOsXzUKrM1CLsU0gOBSamwZgJxZoBn+WmFwehXPn3EalnZPzP9DIjob/8b+I”, “X-Ops-Authorization
-5”=>“L7ZUj7rSRWfFSLxKNKSwAME/1y3lkJIlnifOKU1zyGQMPwYWXqa97Mb8LD8z”, “X-Ops-Authorization-6”=>“QtJyIcM6XuNy3CeZk3g6M7rsTAFY95oMcO4
8YHADSA==”, “X-Ops-Timestamp”=>“2012-01-20T23:19:41Z”}
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Sending HTTP Request via GET to 10.xx.xx.xx:443/nodes/cheftain03.dev.spergatronic.com/cookbo
oks
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Cookbooks to load: {}
[Fri, 20 Jan 2012 23:19:41 +0000] DEBUG: Compiling recipes for node cheftain03.dev.spergatronic.com
/usr/lib/ruby/1.8/chef/client.rb:313:in converge': undefined methodrun_list’ for nil:NilClass (NoMethodError)
from /usr/lib/ruby/1.8/chef/client.rb:87:in run' from /usr/lib/ruby/1.8/chef/application/client.rb:215:inrun_application’
from /usr/lib/ruby/1.8/chef/application/client.rb:207:in loop' from /usr/lib/ruby/1.8/chef/application/client.rb:207:inrun_application’
from /usr/lib/ruby/1.8/chef/application.rb:62:in `run’
from /usr/bin/chef-client:25

root@ip-10-46-1xx-xx:/etc/chef# curl -s http://169.254.169.254/latest/user-data
{
“validation_key”: “-----BEGIN RSA PRIVATE KEY-----\nMIxx_snip\n”,
“attributes”: {
“role”: “chefsrv”,
“node_name”: “cheftain03.dev.spergatronic.com”,
“environment”: “dev”,
“cluster”: “dev”,
“vol_list”: [
“vol-f1xxxxxx”,
“vol-q2xxxxxx”
]
},
“chef_server”: “https://10.xx.xx.xx”,
“validation_client_name”: “chef-validator”,
“run_list”: [
“role[base]”,
“role[postfix-client]”,
“role[syslog-ng-client]”
]
}

On Fri, 20 Jan 2012, kallen@groknaut.net wrote:

hai. i’m failing to feed a run_list present in ec2 userdata to a
first-boot.json file. what am i doing wrong? maybe run_list shouldn’t
be within attributes? (see userdata below).

i’m using a client.rb based on jtimberman’s gist. my modification
is here: https://gist.github.com/1648988

my knife ec2 launch command:

knife ec2 server create -N “testme” -S sshkey -f m1.large -I
ami-xxx -G sg-xxx,sg-xxx --user-data /tmp/userdata --distro distroX

distroX

bash -c ‘
S3CFG="–config=/root/.s3cfg-w-https"
S3CMDOPT="–force $S3CFG"
X_ENV=dev
X_AV_REGION=us-east-1
S3_BUCKET="x-ops-$X_ENV-$X_AV_REGION"
s3cmd get $S3CMDOPT s3://$S3_BUCKET/chef/client.frak.rb /etc/chef/client.rb
/usr/bin/chef-client -j /etc/chef/first-boot.json -l debug -L /var/log/chef/client.log