# Chef-solo: Is there a way to stream stderr/stdout output of executed processes without buffering?

**URL:** <https://discourse.chef.io/t/chef-solo-is-there-a-way-to-stream-stderr-stdout-output-of-executed-processes-without-buffering/2898>\
**Category:** Chef Infra (archive)\
**Created:** [June 5, 2012, 3:41pm UTC](https://discourse.chef.io/t/chef-solo-is-there-a-way-to-stream-stderr-stdout-output-of-executed-processes-without-buffering/2898 "2012-06-05T15:41:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joshua\_Blatt](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_blatt/32/683_2.png) [@Joshua\_Blatt](https://discourse.chef.io/u/Joshua_Blatt)\
**Post date:** [June 5, 2012, 3:41pm UTC](https://discourse.chef.io/t/chef-solo-is-there-a-way-to-stream-stderr-stdout-output-of-executed-processes-without-buffering/2898/1 "2012-06-05T15:41:52Z")

</div>

chef-solo seems to buffer all stderr/stdout output of any process  
created by the execute resource and then dumps it at the end of the  
run. This makes debugging issues more difficult because you don’t see  
errors when they actually occur.

In one particular case, I had a long running execute’d script that was  
causing chef-solo to increase it’s memory consumption rapidly. Soon  
chef-solo was consuming 3.5GB heap at which point the machine was  
swapping heavily. After waiting hours for the executed process to  
finally finish, chef-solo dumped thousands of lines that looked like this:

Re-format filesystem in /var/hadoop/hdfs/data ? (Y or N) Re-format  
filesystem in /var/hadoop/hdfs/data ? (Y or N) Re-format filesystem in  
/var/hadoop/hdfs/data ? (Y or N) Re-format filesystem in  
/var/hadoop/hdfs/data ? (Y or N) …

If chef-solo had simply relayed that instead of buffering it without  
bound …

Thanks,

Josh

---

<div class="post-metadata">

**Author:** ![AJ\_Christensen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/aj_christensen/32/525_2.png) [@AJ\_Christensen](https://discourse.chef.io/u/AJ_Christensen)\
**Post date:** [June 5, 2012, 11:05pm UTC](https://discourse.chef.io/t/chef-solo-is-there-a-way-to-stream-stderr-stdout-output-of-executed-processes-without-buffering/2898/2 "2012-06-05T23:05:35Z")

</div>

Looks like mixlib-shellout is configured to stream the output under  
certain conditions:

[https://github.com/opscode/chef/blob/master/chef/lib/chef/mixin/shell\_out.rb#L27-29](https://github.com/opscode/chef/blob/master/chef/lib/chef/mixin/shell_out.rb#L27-29)

HTH

--AJ

On 6 June 2012 03:41, Joshua Blatt [jblatt@verticloud.com](mailto:jblatt@verticloud.com) wrote:

> chef-solo seems to buffer all stderr/stdout output of any process created by  
> the execute resource and then dumps it at the end of the run. This makes  
> debugging issues more difficult because you don't see errors when they  
> actually occur.
> 
> In one particular case, I had a long running execute'd script that was  
> causing chef-solo to increase it's memory consumption rapidly. Soon  
> chef-solo was consuming 3.5GB heap at which point the machine was swapping  
> heavily. After waiting hours for the executed process to finally finish,  
> chef-solo dumped thousands of lines that looked like this:
> 
> Re-format filesystem in /var/hadoop/hdfs/data ? (Y or N) Re-format  
> filesystem in /var/hadoop/hdfs/data ? (Y or N) Re-format filesystem in  
> /var/hadoop/hdfs/data ? (Y or N) Re-format filesystem in  
> /var/hadoop/hdfs/data ? (Y or N) ...
> 
> If chef-solo had simply relayed that instead of buffering it without bound  
> ...
> 
> Thanks,
> 
> Josh