What kind of limit am I hitting to?

Chef Infra (archive)
1

Hi everybody,

I got following trace while trying to execute a bash block:
Mixlib::ShellOut::ShellCommandFailed: bash[copy-to-pickup]
(xyzcookbook::default line 56) had an error:
Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0],
but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20130129-19058-1ltcctd-0” ----
STDOUT:
STDERR:
/usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:93:in
uid=': Resource temporarily unavailable (Errno::EAGAIN) from /usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:93:inset_user’
from
/usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:226:in
fork_subprocess' from /usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:222:infork’
from
/usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:222:in
fork_subprocess' from /usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:35:inrun_command’
from
/usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout.rb:222:in
run_command' from /usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/mixin/shell_out.rb:30:inshell_out’
from
/usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/…/lib/chef/mixin/shell_out.rb:35:in
shell_out!' from /usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/provider/execute.rb:62:inaction_run’
from
/usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/…/lib/chef/mixin/why_run.rb:52:in
call' from /usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/mixin/why_run.rb:52:inadd_action’
from
/usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/…/lib/chef/provider.rb:152:in
converge_by' from /usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/provider/execute.rb:61:inaction_run’

Contents of that bash block is:
bash “copy-to-pickup” do
code "cp -f #{xyz_filename} #{pickup_filename}"
user node[:abcattr][:user]
group node[:abcattr][:group]
action :run
end

I’ve increased nproc and nofile limits in /etc/security/limits.conf file
before starting and is doesn’t resolve my problem:

grep userid /etc/security/limits.conf

userid - nproc 4096
userid - nofile 4096

ps -eLF | grep userid | wc -l

1756

I have a plenty of disk space available on that host.

What else to check?

Andrey.

2

Have you tried running the block outside of chef? The error EAGAIN is a
blocking error. Can you try to capture your process count in your actual
recipe ?

This Stackoverflow article hints that it could be something in
Mixlib::ShellOut. Probably worth looking at it.

3

On Wed, Jan 30, 2013 at 12:52 PM, Jesse Nelson spheromak@gmail.com wrote:

Have you tried running the block outside of chef? The error EAGAIN is a
blocking error. Can you try to capture your process count in your actual
recipe ?

This Stackoverflow article hints that it could be something in
Mixlib::ShellOut. Probably worth looking at it.

exception - Ruby's `Process.fork` and `Errno::EAGAIN` due to batch forking - Stack Overflow

Jesse,

I'm getting error on following block in Mixlib::ShellOut:

  def set_user
    if user
      Process.euid = uid
      Process.uid = uid
    end
  end

It's not a fork, it's actually after fork.

Same command outside chef recipe works fine.

I actually found a workaround yesterday:

bash "copy-to-pickup" do
code "su -c 'cp -f #{xyz_filename} #{pickup_filename}' #{
node[:abcattr][:user] }"

user node[:abcattr][:user]

group node[:abcattr][:group]

action :run

end

Works like a charm!

Andrey.

4

this seems like a bug to me. Its prob worth making a ticket so someone can
investigate.

On Wed, Jan 30, 2013 at 1:07 PM, Andrey Brindeyev <
abrindeyev@griddynamics.com> wrote:

On Wed, Jan 30, 2013 at 12:52 PM, Jesse Nelson spheromak@gmail.comwrote:

Have you tried running the block outside of chef? The error EAGAIN is a
blocking error. Can you try to capture your process count in your actual
recipe ?

This Stackoverflow article hints that it could be something in
Mixlib::ShellOut. Probably worth looking at it.

exception - Ruby's `Process.fork` and `Errno::EAGAIN` due to batch forking - Stack Overflow

Jesse,

I'm getting error on following block in Mixlib::ShellOut:

  def set_user
    if user
      Process.euid = uid
      Process.uid = uid
    end
  end

It's not a fork, it's actually after fork.

Same command outside chef recipe works fine.

I actually found a workaround yesterday:

bash "copy-to-pickup" do
code "su -c 'cp -f #{xyz_filename} #{pickup_filename}' #{
node[:abcattr][:user] }"

user node[:abcattr][:user]

group node[:abcattr][:group]

action :run

end

Works like a charm!

Andrey.

5

Hey.

Everything points of the limits.conf changes not being effective for the
chef-client process. But, you've taken care to do this before starting the
client.
If you can absolutely rule out node[:abcattr][:user] being a sane user with
a sane unreserved uid, and that the new limits are effective, the only
outside punt I can think of is, from the setuid(2) manpage:

"Linux has the concept of the file system user ID, normally equal to the
effective user ID. The setuid() call also sets the file system user ID of
the calling process. See setfsuid(2)."

There's nothing weird about either your working directory at that point, or
any kind of mounted volume that you might encounter?
Do you have an strace of the client following the child process?

S.

On Tue, Jan 29, 2013 at 11:16 PM, Andrey Brindeyev <
abrindeyev@griddynamics.com> wrote:

Hi everybody,

I got following trace while trying to execute a bash block:
Mixlib::ShellOut::ShellCommandFailed: bash[copy-to-pickup]
(xyzcookbook::default line 56) had an error:
Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0],
but received '1'
---- Begin output of "bash" "/tmp/chef-script20130129-19058-1ltcctd-0"

STDOUT:
STDERR:
/usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:93:in
uid=': Resource temporarily unavailable (Errno::EAGAIN) from /usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:93:in set_user'
from
/usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:226:in
fork_subprocess' from /usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:222:in fork'
from
/usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:222:in
fork_subprocess' from /usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout/unix.rb:35:in run_command'
from
/usr/lib/ruby/gems/1.8/gems/mixlib-shellout-1.1.0/lib/mixlib/shellout.rb:222:in
run_command' from /usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/mixin/shell_out.rb:30:in shell_out'
from
/usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/mixin/shell_out.rb:35:in
shell_out!' from /usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/provider/execute.rb:62:in action_run'
from
/usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/mixin/why_run.rb:52:in
call' from /usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/mixin/why_run.rb:52:in add_action'
from
/usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/provider.rb:152:in
converge_by' from /usr/lib/ruby/gems/1.8/gems/chef-10.18.0/bin/../lib/chef/provider/execute.rb:61:in action_run'

Contents of that bash block is:
bash "copy-to-pickup" do
code "cp -f #{xyz_filename} #{pickup_filename}"
user node[:abcattr][:user]
group node[:abcattr][:group]
action :run
end

I've increased nproc and nofile limits in /etc/security/limits.conf file
before starting and is doesn't resolve my problem:

grep userid /etc/security/limits.conf

userid - nproc 4096
userid - nofile 4096

ps -eLF | grep userid | wc -l

1756

I have a plenty of disk space available on that host.

What else to check?

Andrey.