Chef-solo: Is there a way to stream stderr/stdout output of executed processes without buffering?

Chef Infra (archive)
1

chef-solo seems to buffer all stderr/stdout output of any process
created by the execute resource and then dumps it at the end of the
run. This makes debugging issues more difficult because you don’t see
errors when they actually occur.

In one particular case, I had a long running execute’d script that was
causing chef-solo to increase it’s memory consumption rapidly. Soon
chef-solo was consuming 3.5GB heap at which point the machine was
swapping heavily. After waiting hours for the executed process to
finally finish, chef-solo dumped thousands of lines that looked like this:

Re-format filesystem in /var/hadoop/hdfs/data ? (Y or N) Re-format
filesystem in /var/hadoop/hdfs/data ? (Y or N) Re-format filesystem in
/var/hadoop/hdfs/data ? (Y or N) Re-format filesystem in
/var/hadoop/hdfs/data ? (Y or N) …

If chef-solo had simply relayed that instead of buffering it without
bound …

Thanks,

Josh

2

Looks like mixlib-shellout is configured to stream the output under
certain conditions:

https://github.com/opscode/chef/blob/master/chef/lib/chef/mixin/shell_out.rb#L27-29

HTH

--AJ

On 6 June 2012 03:41, Joshua Blatt jblatt@verticloud.com wrote:

chef-solo seems to buffer all stderr/stdout output of any process created by
the execute resource and then dumps it at the end of the run. This makes
debugging issues more difficult because you don't see errors when they
actually occur.

In one particular case, I had a long running execute'd script that was
causing chef-solo to increase it's memory consumption rapidly. Soon
chef-solo was consuming 3.5GB heap at which point the machine was swapping
heavily. After waiting hours for the executed process to finally finish,
chef-solo dumped thousands of lines that looked like this:

Re-format filesystem in /var/hadoop/hdfs/data ? (Y or N) Re-format
filesystem in /var/hadoop/hdfs/data ? (Y or N) Re-format filesystem in
/var/hadoop/hdfs/data ? (Y or N) Re-format filesystem in
/var/hadoop/hdfs/data ? (Y or N) ...

If chef-solo had simply relayed that instead of buffering it without bound
...

Thanks,

Josh