Intermittent problems with default attributes

something i’m doing with attributes is not quite right, and here’s the
proof:

[root@headnode opt]# while [[ 1 ]]; do chef-client -E chefenv1 2>/dev/null
1>/dev/null; if [[ “$?” != “0” ]]; then echo -n ‘E’; elif grep 42559
/opt/moab/etc/moab.cfg >/dev/null && grep 42559
/opt/mws/etc/mws-config.groovy >/dev/null; then echo -n ‘.’; else echo -n
’!’; fi; done
!!!.!!!..!!!..!..!!!.!!!..!!!^CE^C

so, i’m running chef over and over without changing cookbooks, run lists,
override attributes, or anything else in between. if the run fails it
should print ‘E’, if the expected attributes are in the config files it’s
writing out it should print ‘.’ , and if the attributes are not in the file
it should print ‘!’. as you can see, the results are not the same in
subsequent runs. this is not because the chef runs are failing-- the value
simply ends up being nil on some runs all else seemingly being equal. i’ve
confirmed this manually many times, it can succeed the client run but
produce unexpected output.

here’s what i’m doing, and it seems pretty straightforward. there is a
default attributes file that sets the ‘sched_port’ as a default attribute,
and it is consumed by a template in that cookbook where it’s set (mwm) and
in a template from another cookbook which is also run on the same node
(mws):

===== mwm/attributes/default.rb =====
::Chef::Node.send(:include, Opscode::OpenSSL::Password)

set_unless[‘mcs’][‘mwm’][‘secret_key’] = secure_password
node.save
default[‘mcs’][‘mwm’][‘sched_port’] = ‘42559’

====== mwm/recipes/default.rb =======

write out the templated moab config

template “/opt/moab/etc/moab.cfg” do
source "moab-cfg.erb"
mode 0600
owner "root"
group "root"
action :create
end

===== mwm/templates/default/moab-cfg.erb =====

SCHEDCFG[Moab] SERVER=<%= node[‘hostname’] %>:<%=
node[‘mcs’][‘mwm’][‘sched_port’] %>

===== mws/recipes/default.rb =======

write mws config file

template “/opt/mws/etc/mws-config.groovy” do
source “mws-config-groovy.erb”

    mode 0600
    owner "tomcat"
    group "tomcat"

    variables(
            :moab_secretkey => node['mcs']['mwm']['secret_key'],
            :moab_server => node['hostname'],
            :moab_port => node['mcs']['mwm']['sched_port'],
            :mws_username => node['mcs']['mws']['username'],
            :mws_password => node['mcs']['mws']['password'],
            :mongo_host => mongo_node['hostname'],
            :mongo_port => mongo_node['mongodb']['port'],
            :mongo_database => node['mcs']['mws']['mongo_dbname']
    )
    action :create

end

====== mws/templates/default/mws-config-groovy.erb ====
moab.secretKey = "<%= @moab_secretkey %>"
moab.server = "<%= @moab_server %>"
moab.port = <%= @moab_port %>

also, it disappears from either file. every time i’ve checked it
disappears from both files at once:
[root@headnode opt]# while [[ 1 ]]; do chef-client -E chefenv1 2>/dev/null
1>/dev/null; if [[ “?" != "0" ]]; then echo -n 'E'; elif grep 42559 /opt/moab/etc/moab.cfg >/dev/null; then echo -n '.'; else echo -n '!'; fi; done .!!!!!!..!!!.!^CE^C [root@headnode opt]# while [[ 1 ]]; do chef-client -E chefenv1 2>/dev/null 1>/dev/null; if [[ "?” != “0” ]]; then echo -n ‘E’; elif grep 42559
/opt/mws/etc/mws-config.groovy >/dev/null; then echo -n ‘.’; else echo -n
’!’; fi; done
!!.!!!.!!..!!!..!..!!!

i’ve wrestled through several other attribute-related issues (one i
detailed here
http://www.devopstonoops.com/2012/07/04/what-ive-learned-from-using-chef-part-1/),
but i’m baffled about this one. any help would be appreciated.

  • jonathan

It's probably the node.save in your attribute file (at a glance), only
a hunch though!

Please post the relevant (full) debug logs. Alternately, reduce this
to the simplest example -- a template with an attribute; repeat your
tests. If you use an override attribute for the port in a role, does
it still happen? How about a 'set' attribute on the node?

--AJ

On 4 July 2012 22:52, Jonathan Dye chef@lulz.bz wrote:

something i'm doing with attributes is not quite right, and here's the
proof:

[root@headnode opt]# while [[ 1 ]]; do chef-client -E chefenv1 2>/dev/null
1>/dev/null; if [[ "$?" != "0" ]]; then echo -n 'E'; elif grep 42559
/opt/moab/etc/moab.cfg >/dev/null && grep 42559
/opt/mws/etc/mws-config.groovy >/dev/null; then echo -n '.'; else echo -n
'!'; fi; done
!!!!!!!.!!!...!!!!!..!.......!!!!!!.!!!!!..!!!!^CE^C

so, i'm running chef over and over without changing cookbooks, run lists,
override attributes, or anything else in between. if the run fails it
should print 'E', if the expected attributes are in the config files it's
writing out it should print '.' , and if the attributes are not in the file
it should print '!'. as you can see, the results are not the same in
subsequent runs. this is not because the chef runs are failing-- the value
simply ends up being nil on some runs all else seemingly being equal. i've
confirmed this manually many times, it can succeed the client run but
produce unexpected output.

here's what i'm doing, and it seems pretty straightforward. there is a
default attributes file that sets the 'sched_port' as a default attribute,
and it is consumed by a template in that cookbook where it's set (mwm) and
in a template from another cookbook which is also run on the same node
(mws):

===== mwm/attributes/default.rb =====
::Chef::Node.send(:include, Opscode::OpenSSL::Password)

set_unless['mcs']['mwm']['secret_key'] = secure_password
node.save
default['mcs']['mwm']['sched_port'] = '42559'

====== mwm/recipes/default.rb =======
...

write out the templated moab config

template "/opt/moab/etc/moab.cfg" do
source "moab-cfg.erb"
mode 0600
owner "root"
group "root"
action :create
end
...

===== mwm/templates/default/moab-cfg.erb =====
...
SCHEDCFG[Moab] SERVER=<%= node['hostname'] %>:<%=
node['mcs']['mwm']['sched_port'] %>
...

===== mws/recipes/default.rb =======
...

write mws config file

template "/opt/mws/etc/mws-config.groovy" do
source "mws-config-groovy.erb"

    mode 0600
    owner "tomcat"
    group "tomcat"

    variables(
            :moab_secretkey => node['mcs']['mwm']['secret_key'],
            :moab_server => node['hostname'],
            :moab_port => node['mcs']['mwm']['sched_port'],
            :mws_username => node['mcs']['mws']['username'],
            :mws_password => node['mcs']['mws']['password'],
            :mongo_host => mongo_node['hostname'],
            :mongo_port => mongo_node['mongodb']['port'],
            :mongo_database => node['mcs']['mws']['mongo_dbname']
    )
    action :create

end

====== mws/templates/default/mws-config-groovy.erb ====
moab.secretKey = "<%= @moab_secretkey %>"
moab.server = "<%= @moab_server %>"
moab.port = <%= @moab_port %>
...

also, it disappears from either file. every time i've checked it disappears
from both files at once:
[root@headnode opt]# while [[ 1 ]]; do chef-client -E chefenv1 2>/dev/null
1>/dev/null; if [[ "$?" != "0" ]]; then echo -n 'E'; elif grep 42559
/opt/moab/etc/moab.cfg >/dev/null; then echo -n '.'; else echo -n '!'; fi;
done
.!!!!!!..!!!.!^CE^C
[root@headnode opt]# while [[ 1 ]]; do chef-client -E chefenv1 2>/dev/null
1>/dev/null; if [[ "$?" != "0" ]]; then echo -n 'E'; elif grep 42559
/opt/mws/etc/mws-config.groovy >/dev/null; then echo -n '.'; else echo -n
'!'; fi; done
!!.!!!!!!!!.!!...!!!..!.......!!!!

i've wrestled through several other attribute-related issues (one i detailed
here
http://www.devopstonoops.com/2012/07/04/what-ive-learned-from-using-chef-part-1/),
but i'm baffled about this one. any help would be appreciated.

  • jonathan