Is there a timeout attr for the service resource?


#1

hai.

In a recipe, the service resource to start mysql threw an exception because it
took too long for it to start mysql. The node that runs this recipe should be
expected to possibly take several minutes to start mysql, since innodb is doing
a recovery.

[2013-09-13T21:37:46+00:00] DEBUG: service[mysql] supports status, running
[2013-09-13T21:47:56+00:00] DEBUG: Re-raising exception: Mixlib::ShellOut::CommandTimeout - service[mysql] (dbci::drone line 61) had an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of /sbin/service mysql start ----
STDOUT: Starting MySQL (Percona Server)…

After this exception was thrown, mysql actually did finish successfully starting
up.

I looked for a way to set a timeout for the service resource, but didn’t see
a way to do it. Can anyone advise?

thanks,
kallen


#2

While it’s not a timeout change, what you can do instead is set a retry.

All resources support a retries attribute, which takes an int as number of retries to attempt before marking the resource failed. There’s also a retry_delay attribute, to set a timer between retries.

More info here: http://docs.opscode.com/chef/resources.html#attributes

On September 13, 2013 at 2:58:58 PM, kallen@groknaut.net (kallen@groknaut.net) wrote:

hai.

In a recipe, the service resource to start mysql threw an exception because it
took too long for it to start mysql. The node that runs this recipe should be
expected to possibly take several minutes to start mysql, since innodb is doing
a recovery.

[2013-09-13T21:37:46+00:00] DEBUG: service[mysql] supports status, running
[2013-09-13T21:47:56+00:00] DEBUG: Re-raising exception: Mixlib::ShellOut::CommandTimeout - service[mysql] (dbci::drone line 61) had an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of /sbin/service mysql start ----
STDOUT: Starting MySQL (Percona Server)…

After this exception was thrown, mysql actually did finish successfully starting
up.

I looked for a way to set a timeout for the service resource, but didn’t see
a way to do it. Can anyone advise?

thanks,
kallen


#3

On Fri, 13 Sep 2013, Charles Johnson wrote:

While it’s not a timeout change, what you can do instead is set a retry.

All resources support a retries attribute, which takes an int as number of retries to attempt before marking the resource failed. There’s also a retry_delay attribute, to set a timer between retries.

More info here: http://docs.opscode.com/chef/resources.html#attributes

I thought about that, but thought that might not work because the “start” is
already still in flight, waiting for innodb recovery to finish. I assumed
a retry of “start” wouldn’t work, might exit non-zero. But I haven’t tried it
yet. I’ll give it a try. Thanks.

Still, I would think a timeout on service resource would be useful.

And I guess I could also try starting mysql without the service resource, but
within a ruby_block that creates the timeout…

On September 13, 2013 at 2:58:58 PM, kallen@groknaut.net (kallen@groknaut.net) wrote:

hai.

In a recipe, the service resource to start mysql threw an exception because it
took too long for it to start mysql. The node that runs this recipe should be
expected to possibly take several minutes to start mysql, since innodb is doing
a recovery.

[2013-09-13T21:37:46+00:00] DEBUG: service[mysql] supports status, running
[2013-09-13T21:47:56+00:00] DEBUG: Re-raising exception: Mixlib::ShellOut::CommandTimeout - service[mysql] (dbci::drone line 61) had an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of /sbin/service mysql start ----
STDOUT: Starting MySQL (Percona Server)…

After this exception was thrown, mysql actually did finish successfully starting
up.

I looked for a way to set a timeout for the service resource, but didn’t see
a way to do it. Can anyone advise?

thanks,
kallen


#4

On Fri, 13 Sep 2013, kallen@groknaut.net wrote:

On Fri, 13 Sep 2013, Charles Johnson wrote:

While it’s not a timeout change, what you can do instead is set a retry.

All resources support a retries attribute, which takes an int as number of retries to attempt before marking the resource failed. There’s also a retry_delay attribute, to set a timer between retries.

More info here: http://docs.opscode.com/chef/resources.html#attributes

I thought about that, but thought that might not work because the “start” is
already still in flight, waiting for innodb recovery to finish. I assumed
a retry of “start” wouldn’t work, might exit non-zero. But I haven’t tried it
yet. I’ll give it a try. Thanks.

When I issue another “start”, the init script (from percona pkg) isn’t smart
enough to see that mysql is already trying to start, and the result of the second
start is the following errors spewed over and over again into the log:

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.

Next I guess I’ll try a timeout inside a ruby_block without using the service
resource, unless someone has other suggestions.

thanks,
kallen

Still, I would think a timeout on service resource would be useful.

And I guess I could also try starting mysql without the service resource, but
within a ruby_block that creates the timeout…

On September 13, 2013 at 2:58:58 PM, kallen@groknaut.net (kallen@groknaut.net) wrote:

hai.

In a recipe, the service resource to start mysql threw an exception because it
took too long for it to start mysql. The node that runs this recipe should be
expected to possibly take several minutes to start mysql, since innodb is doing
a recovery.

[2013-09-13T21:37:46+00:00] DEBUG: service[mysql] supports status, running
[2013-09-13T21:47:56+00:00] DEBUG: Re-raising exception: Mixlib::ShellOut::CommandTimeout - service[mysql] (dbci::drone line 61) had an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of /sbin/service mysql start ----
STDOUT: Starting MySQL (Percona Server)…

After this exception was thrown, mysql actually did finish successfully starting
up.

I looked for a way to set a timeout for the service resource, but didn’t see
a way to do it. Can anyone advise?

thanks,
kallen


#5

You could try subclassing the service resource. (I haven’t looked at it
but if it uses Chef::Mixin::ShellOut you just have to find a way to add
a :timeout value to the options hash passed to shell_out.)

On 13-09-13 06:21 PM, kallen@groknaut.net wrote:

On Fri, 13 Sep 2013, kallen@groknaut.net wrote:

On Fri, 13 Sep 2013, Charles Johnson wrote:

While it’s not a timeout change, what you can do instead is set a retry.

All resources support a retries attribute, which takes an int as number of retries to attempt before marking the resource failed. There’s also a retry_delay attribute, to set a timer between retries.

More info here: http://docs.opscode.com/chef/resources.html#attributes
I thought about that, but thought that might not work because the “start” is
already still in flight, waiting for innodb recovery to finish. I assumed
a retry of “start” wouldn’t work, might exit non-zero. But I haven’t tried it
yet. I’ll give it a try. Thanks.
When I issue another “start”, the init script (from percona pkg) isn’t smart
enough to see that mysql is already trying to start, and the result of the second
start is the following errors spewed over and over again into the log:

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.

Next I guess I’ll try a timeout inside a ruby_block without using the service
resource, unless someone has other suggestions.

thanks,
kallen

Still, I would think a timeout on service resource would be useful.

And I guess I could also try starting mysql without the service resource, but
within a ruby_block that creates the timeout…

On September 13, 2013 at 2:58:58 PM, kallen@groknaut.net (kallen@groknaut.net) wrote:

hai.

In a recipe, the service resource to start mysql threw an exception because it
took too long for it to start mysql. The node that runs this recipe should be
expected to possibly take several minutes to start mysql, since innodb is doing
a recovery.

[2013-09-13T21:37:46+00:00] DEBUG: service[mysql] supports status, running
[2013-09-13T21:47:56+00:00] DEBUG: Re-raising exception: Mixlib::ShellOut::CommandTimeout - service[mysql] (dbci::drone line 61) had an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of /sbin/service mysql start ----
STDOUT: Starting MySQL (Percona Server)…

After this exception was thrown, mysql actually did finish successfully starting
up.

I looked for a way to set a timeout for the service resource, but didn’t see
a way to do it. Can anyone advise?

thanks,
kallen