Supervisor missing templates


#1

Hi –

This is probably due to my own misunderstanding, but a little help would
be appreciated.

tldr;
Okay, I specified “supervisor”, “= 0.4.2” and it worked!
So, why doesn’t 0.4.10 work, and why doesn’t the template show up in the
files downloaded to /var/cache/cookbooks/supervisor?

I’ve got it working now, but I’d like to understand this!


I am including the supervisor cookbook in a recipe.
berks list shows

  • supervisor (0.4.10)
    I have run berks upload.
    The management interface show three versions of supervisor
    0.4.10
    0.4.2
    0.4.0

When I converge, I get the error below.

The command itself is fine. If I execute it by hand on the node
"/home/ubuntu/virtualenv/bin/python
/home/ubuntu/virtualenv/current/tools/clientutilities/jobs/job_activity_search.py"
it runs.

Apparently the job is not getting created. The template is missing:

ls /var/chef/cache/cookbooks/supervisor/templates/*
/var/chef/cache/cookbooks/supervisor/templates/default:
supervisord.conf.erb

/var/chef/cache/cookbooks/supervisor/templates/ubuntu:
supervisor.default.erb supervisor.init.erb
Yet when I look at the content of the 0.4.10 supervisor cookbook (at
manage.opscode.com) I see the templates directory contains:
supervisor.default.erb
supervisor.init.erb
fcgi.conf.erb
group.conf.erb
program.conf.erb
supervisord.conf.erb
manifest.xml.erb
supervisor.default.erb
supervisor.init.erb

  • supervisor_service[job_activity_search] action restart
    ================================================================================
    Error executing action restart on resource
    ’supervisor_service[job_activity_search]’
    ================================================================================

RuntimeError

Supervisor service job_activity_search cannot be restarted because it
does not exist

Cookbook Trace:

/var/chef/cache/cookbooks/supervisor/providers/service.rb:72:in `block
in class_from_file’

Resource Declaration:

In /var/chef/cache/cookbooks/bti_main/recipes/default.rb

108: supervisor_service “#{job}” do
109: command "#{virtual_env_dir}/bin/python
#{virtual_env_dir}/#{jobs_path}/#{job}.py"
110: process_name "#{job}-%(process_num)s"
111: action [:enable, :restart]
112: autostart true
113: autorestart true
114: user "root"
115: numprocs 1
116: end
117: end

Compiled Resource:

Declared in

/var/chef/cache/cookbooks/bti_main/recipes/default.rb:108:in `block in
from_file’

supervisor_service(“job_activity_search”) do
action [:enable, :restart]
updated true
retries 0
retry_delay 2
cookbook_name "bti_main"
recipe_name "default"
command "/home/ubuntu/virtualenv/bin/python
/home/ubuntu/virtualenv/current/tools/clientutilities/jobs/job_activity_search.py"
process_name "job_activity_search-%(process_num)s"
autostart true
autorestart true
user "root"
numprocs 1
service_name "job_activity_search"
end


Liam Kirsher
PGP: http://liam.numenet.com/pgp/


#2

I have also seen this behavior, but haven’t gotten to the bottom of it. If I knife download my cookbook, the template is there, but chef-client doesn’t get it. Isn’t in /var/chef/cache at all.

On Jul 21, 2014, at 7:15 PM, Liam Kirsher liamk@numenet.com wrote:

Hi –

This is probably due to my own misunderstanding, but a little help would be appreciated.

tldr;
Okay, I specified “supervisor”, “= 0.4.2” and it worked!
So, why doesn’t 0.4.10 work, and why doesn’t the template show up in the files downloaded to /var/cache/cookbooks/supervisor?

I’ve got it working now, but I’d like to understand this!


I am including the supervisor cookbook in a recipe.
berks list shows

  • supervisor (0.4.10)
    I have run berks upload.
    The management interface show three versions of supervisor
    0.4.10
    0.4.2
    0.4.0

When I converge, I get the error below.

The command itself is fine. If I execute it by hand on the node
"/home/ubuntu/virtualenv/bin/python /home/ubuntu/virtualenv/current/tools/clientutilities/jobs/job_activity_search.py"
it runs.

Apparently the job is not getting created. The template is missing:

ls /var/chef/cache/cookbooks/supervisor/templates/*
/var/chef/cache/cookbooks/supervisor/templates/default:
supervisord.conf.erb

/var/chef/cache/cookbooks/supervisor/templates/ubuntu:
supervisor.default.erb supervisor.init.erb
Yet when I look at the content of the 0.4.10 supervisor cookbook (at manage.opscode.com) I see the templates directory contains:
supervisor.default.erb
supervisor.init.erb
fcgi.conf.erb
group.conf.erb
program.conf.erb
supervisord.conf.erb
manifest.xml.erb
supervisor.default.erb
supervisor.init.erb

  • supervisor_service[job_activity_search] action restart
    ================================================================================
    Error executing action restart on resource ‘supervisor_service[job_activity_search]’
    ================================================================================

RuntimeError

Supervisor service job_activity_search cannot be restarted because it does not exist

Cookbook Trace:

/var/chef/cache/cookbooks/supervisor/providers/service.rb:72:in `block in class_from_file’

Resource Declaration:

In /var/chef/cache/cookbooks/bti_main/recipes/default.rb

108: supervisor_service “#{job}” do
109: command "#{virtual_env_dir}/bin/python #{virtual_env_dir}/#{jobs_path}/#{job}.py"
110: process_name "#{job}-%(process_num)s"
111: action [:enable, :restart]
112: autostart true
113: autorestart true
114: user "root"
115: numprocs 1
116: end
117: end

Compiled Resource:

Declared in /var/chef/cache/cookbooks/bti_main/recipes/default.rb:108:in `block in from_file’

supervisor_service(“job_activity_search”) do
action [:enable, :restart]
updated true
retries 0
retry_delay 2
cookbook_name "bti_main"
recipe_name "default"
command "/home/ubuntu/virtualenv/bin/python /home/ubuntu/virtualenv/current/tools/clientutilities/jobs/job_activity_search.py"
process_name "job_activity_search-%(process_num)s"
autostart true
autorestart true
user "root"
numprocs 1
service_name "job_activity_search"
end


Liam Kirsher
PGP: http://liam.numenet.com/pgp/


#3

On Monday, July 21, 2014 at 6:59 PM, Ben Bytheway wrote:

I have also seen this behavior, but haven’t gotten to the bottom of it. If I knife download my cookbook, the template is there, but chef-client doesn’t get it. Isn’t in /var/chef/cache at all.

Templates are downloaded on demand, as they are used by resources. If you never use them, then chef doesn’t download them. If I remember correctly, there might also be a small bug where templates that are used can be removed at the end of a chef-client run (but they’re downloaded the next time so it’s just wasting a bit of IO). Have you seen a cookbook fail to execute for this reason?

Also, if you just don’t like this behavior, you can configure no_lazy_load true to make chef-client fetch all templates (and cookbook files, which have the same behavior) eagerly.

FWIW, eager load will be the default in Chef 12 since the URLs to fetch the files are time-limited and on long chef-client runs (an hour or more), the URLs expire and cause errors.


Daniel DeLeo


#4

Yes, the chef run fails with an error that the template cannot be found, even when specifying the cookbook where the template is located in the resource call.

On Jul 21, 2014, at 8:09 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, July 21, 2014 at 6:59 PM, Ben Bytheway wrote:
I have also seen this behavior, but haven’t gotten to the bottom of it. If I knife download my cookbook, the template is there, but chef-client doesn’t get it. Isn’t in /var/chef/cache at all.

Templates are downloaded on demand, as they are used by resources. If you never use them, then chef doesn’t download them. If I remember correctly, there might also be a small bug where templates that are used can be removed at the end of a chef-client run (but they’re downloaded the next time so it’s just wasting a bit of IO). Have you seen a cookbook fail to execute for this reason?

Also, if you just don’t like this behavior, you can configure no_lazy_load true to make chef-client fetch all templates (and cookbook files, which have the same behavior) eagerly.

FWIW, eager load will be the default in Chef 12 since the URLs to fetch the files are time-limited and on long chef-client runs (an hour or more), the URLs expire and cause errors.


Daniel DeLeo


#5

The stack-trace in OP is not a template resource.

“Supervisor service job_activity_search cannot be restarted because it
does not exist”

Perhaps :create needs to be called before :enable/:restart. I’m not
familiar with the supervisor cookbook.

111: action [:enable, :restart]

One might anticipate :enable to call :create, but would have to dig
into the supervisor (L|H)WRP itself, to find out!

Cheers,

AJ

On Tue, Jul 22, 2014 at 2:30 PM, Ben Bytheway bbytheway@gmail.com wrote:

Yes, the chef run fails with an error that the template cannot be found, even when specifying the cookbook where the template is located in the resource call.

On Jul 21, 2014, at 8:09 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, July 21, 2014 at 6:59 PM, Ben Bytheway wrote:
I have also seen this behavior, but haven’t gotten to the bottom of it. If I knife download my cookbook, the template is there, but chef-client doesn’t get it. Isn’t in /var/chef/cache at all.

Templates are downloaded on demand, as they are used by resources. If you never use them, then chef doesn’t download them. If I remember correctly, there might also be a small bug where templates that are used can be removed at the end of a chef-client run (but they’re downloaded the next time so it’s just wasting a bit of IO). Have you seen a cookbook fail to execute for this reason?

Also, if you just don’t like this behavior, you can configure no_lazy_load true to make chef-client fetch all templates (and cookbook files, which have the same behavior) eagerly.

FWIW, eager load will be the default in Chef 12 since the URLs to fetch the files are time-limited and on long chef-client runs (an hour or more), the URLs expire and cause errors.


Daniel DeLeo


#6

I guess I’m getting different behavior then. I was able to reproduce my
missing template problem.

It happens when you have a render in a template that is part of an lwrp
called from another cookbook.

Here’s the stack trace (all custom cookbooks, this isn’t the community
apache cookbook). Notice the template and the render specify the cookbook
explicitly. this is using chef-client 11.12.8 and open source chef server
11.1.3 on RHEL 6.

================================================================================
Error executing action create_if_missing on resource
’template[/usr/local/common/apache2/confs/ben_ssl/httpd.conf]’

Chef::Mixin::Template::TemplateError

Cookbook ‘use_template’ (0.4.0) does not contain a file at any of these
locations:
templates/redhat-6.5/httpd_ssl.conf.erb
templates/redhat/httpd_ssl.conf.erb
templates/default/httpd_ssl.conf.erb

Resource Declaration:

In /var/chef/cache/cookbooks/apache/providers/instance.rb

177: template “/usr/local/common/apache2/confs/#{flavor}/httpd.conf” do
178: source 'httpd.conf.erb’
179: variables(
180: server_name: node[:fqdn],
181: server_port: (ssl?(flavor) ? 443 : 80),
182: vhost_name: vhost_name,
183: vhost_aliases: vhost_aliases,
184: ssl_enabled: ssl?(flavor),
185: )
186: owner 'root’
187: group group_name
188: mode '0644’
189: cookbook 'apache’
190: action :create_if_missing
191: end
192:

Compiled Resource:

Declared in /var/chef/cache/cookbooks/apache/providers/instance.rb:177:in

`config_files’

template("/usr/local/common/apache2/confs/ben_ssl/httpd.conf") do
provider Chef::Provider::Template
action [:create_if_missing]
retries 0
retry_delay 2
guard_interpreter :default
path "/usr/local/common/apache2/confs/ben_ssl/httpd.conf"
backup 5
atomic_update true
source "httpd.conf.erb"
cookbook "apache"
variables {:server_name=>“hostname”, :server_port=>443,
:vhost_name=>“hostname”, :vhost_aliases=>[“hostname”], :ssl_enabled=>true}
cookbook_name "use_template"
owner "root"
group "webadmin"
mode "0644"
end

Template Context:

on line #195
193:
194: <% if @ssl_enabled %>
195: <%= render “httpd_ssl.conf.erb”, variables: { vhost_name:
@vhost_name }, coobook: ‘apache’ %>
196: <% end %>
197:

On Mon, Jul 21, 2014 at 8:32 PM, AJ Christensen aj@junglist.io wrote:

The stack-trace in OP is not a template resource.

“Supervisor service job_activity_search cannot be restarted because it
does not exist”

Perhaps :create needs to be called before :enable/:restart. I’m not
familiar with the supervisor cookbook.

111: action [:enable, :restart]

One might anticipate :enable to call :create, but would have to dig
into the supervisor (L|H)WRP itself, to find out!

Cheers,

AJ

On Tue, Jul 22, 2014 at 2:30 PM, Ben Bytheway bbytheway@gmail.com wrote:

Yes, the chef run fails with an error that the template cannot be found,
even when specifying the cookbook where the template is located in the
resource call.

On Jul 21, 2014, at 8:09 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, July 21, 2014 at 6:59 PM, Ben Bytheway wrote:
I have also seen this behavior, but haven’t gotten to the bottom of
it. If I knife download my cookbook, the template is there, but chef-client
doesn’t get it. Isn’t in /var/chef/cache at all.

Templates are downloaded on demand, as they are used by resources. If
you never use them, then chef doesn’t download them. If I remember
correctly, there might also be a small bug where templates that are used
can be removed at the end of a chef-client run (but they’re downloaded the
next time so it’s just wasting a bit of IO). Have you seen a cookbook fail
to execute for this reason?

Also, if you just don’t like this behavior, you can configure
no_lazy_load true to make chef-client fetch all templates (and cookbook
files, which have the same behavior) eagerly.

FWIW, eager load will be the default in Chef 12 since the URLs to fetch
the files are time-limited and on long chef-client runs (an hour or more),
the URLs expire and cause errors.


Daniel DeLeo


#7

Yes, that’s true. I’m pretty sure the reason it won’t start is because
the file does not get created from the template. Could be wrong, but…
Looking at the code, the providers/service.rb line 72 is where the
problem is in the traceback, so the current_resource.state value is
’UNAVAILABLE’.
The :enable action has been called, and it calls enable_service, which
is where the template lwrp is. It references program.conf.erb. Since
that doesn’t seem to exist anywhere (I was expecting it in
/var/chef/cache/cookbooks/supervisor/templates/) I don’t see how
supervisor can start. This would seem to explain it – but leaves the
question, why is the template not available?

Don’t know if this is relevant, but… this is my code:
attributes

default[:main][:jobs] = %w[
job_activity_search
job_user_call_log
]

In recipe

app_name = 'main’
app_node = node[app_name]
jobs = app_node[:jobs]

jobs.each do |job|
supervisor_service “#{job}” do
command "#{virtual_env_dir}/bin/python #{virtual_env_dir}/#{jobs_path}/#{job}.py"
process_name "#{job}-%(process_num)s"
action [:enable, :restart]
autostart true
autorestart true
user "root"
numprocs 1
end

On 07/21/2014 07:32 PM, AJ Christensen wrote:

The stack-trace in OP is not a template resource.

“Supervisor service job_activity_search cannot be restarted because it
does not exist”

Perhaps :create needs to be called before :enable/:restart. I’m not
familiar with the supervisor cookbook.

111: action [:enable, :restart]

One might anticipate :enable to call :create, but would have to dig
into the supervisor (L|H)WRP itself, to find out!

Cheers,

AJ

On Tue, Jul 22, 2014 at 2:30 PM, Ben Bytheway bbytheway@gmail.com wrote:

Yes, the chef run fails with an error that the template cannot be found, even when specifying the cookbook where the template is located in the resource call.

On Jul 21, 2014, at 8:09 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, July 21, 2014 at 6:59 PM, Ben Bytheway wrote:
I have also seen this behavior, but haven’t gotten to the bottom of it. If I knife download my cookbook, the template is there, but chef-client doesn’t get it. Isn’t in /var/chef/cache at all.

Templates are downloaded on demand, as they are used by resources. If you never use them, then chef doesn’t download them. If I remember correctly, there might also be a small bug where templates that are used can be removed at the end of a chef-client run (but they’re downloaded the next time so it’s just wasting a bit of IO). Have you seen a cookbook fail to execute for this reason?

Also, if you just don’t like this behavior, you can configure no_lazy_load true to make chef-client fetch all templates (and cookbook files, which have the same behavior) eagerly.

FWIW, eager load will be the default in Chef 12 since the URLs to fetch the files are time-limited and on long chef-client runs (an hour or more), the URLs expire and cause errors.


Daniel DeLeo


Liam Kirsher
PGP: http://liam.numenet.com/pgp/