Logging less from resources, aka using the logger for logging and not STDOUT

Peter_Norton · March 19, 2012, 5:32pm

I’m doing some work to make launching multiple systems more friendly to
human observation. My specific goal right now is to force more things to
go through the logger, and to only log warning or error (and not debug and
info) to the console. Info and above goes to a file for later perusal if
needed.

However, I’m confused by this bit of code in some of the resources:

   opts[:log_tag] = @new_resource.to_s
    if STDOUT.tty? && !Chef::Config[:daemon] && Chef::Log.info?  # <<<

this specfically
opts[:live_stream] = STDOUT
end

if I have Log.info defined, why is chef spamming stdout instead of using
the logger? Wouldn’t it make more sense to have one of the following two
options:

Be able to disable live_stream from the resource
Have live_stream asynchronously call the logger to print log lines

The pain point here is that most of the commands that I run from chef using
the execute resource are not verbose (tar, cp, etc.). However, the apt LWRP
uses execute for apt-get update, etc. which is incredibly verbose and which
therefore drowns out all useful info in a parallel launch. Any suggestions
on how to reduce this is appreciated. The option I think I’m going to have
to go with at the moment is to start monkey-patching from within client.rb,
but I’d like it if there was a clearer way to deal with this.

Thanks,

-Peter

Vincent_Jorgensen · March 19, 2012, 5:55pm

Hi all!

I’m attempting to create a clone of the running chef server. Since it’s running in AWS, I created an EBS/AMI and re-launched it. Everything almosts, but when I attempt to do “knife cookbook upload mycookbook -V” I get the following error:

Uploading mycookbook [1.0.0]
INFO: Saving mycookbook
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Uploading files
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 3/5 in 9s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 4/5 in 18s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 5/5 in 52s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: internal server error
Response: Connection failed - user: chef

I suspect that “chef-client” is going to fail as well. As near as my rudimentary google search could ascertain, this appears related to rabbitmq’s DB. What’s the safe way to change chef hosts without regenerating all of my validation.pem and client.pem files? I have the original instance still running, so I can copy any files from the original if need be.

Thanks!
Vincent

kallistec · March 19, 2012, 6:03pm

On Monday, March 19, 2012 at 10:55 AM, Vincent Jorgensen wrote:

Hi all!

I'm attempting to create a clone of the running chef server. Since it's running in AWS, I created an EBS/AMI and re-launched it. Everything almosts, but when I attempt to do "knife cookbook upload mycookbook -V" I get the following error:

Uploading mycookbook [1.0.0]
INFO: Saving mycookbook
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Uploading files
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 3/5 in 9s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 4/5 in 18s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 5/5 in 52s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: internal server error
Response: Connection failed - user: chef

I suspect that "chef-client" is going to fail as well. As near as my rudimentary google search could ascertain, this appears related to rabbitmq's DB. What's the safe way to change chef hosts without regenerating all of my validation.pem and client.pem files? I have the original instance still running, so I can copy any files from the original if need be.

Thanks!
Vincent

Logs from chef-server would be helpful.

Assuming the problem is RabbitMQ (seems likely), it looks like rabbitmq isn't running at all. Try running rabbitmq-server in the foreground and see if you get any hints. You can try blowing away the RabbitMQ database (usually in /var/lib/rabbitmq/mnesia/ ) and reconfiguring rabbit (see this: https://github.com/opscode/cookbooks/blob/master/chef-server/recipes/rabbitmq.rb ).

HTH,

Dan DeLeo

KC_Braunschweig · March 19, 2012, 6:44pm

Dan is probably right about Rabbit. IIRC its configuration is tied to
its hostname so if you bring it up again with a new hostname, it will
ignore the config. Re-applying the configuration should sort it out.

KC

On Mon, Mar 19, 2012 at 11:03 AM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, March 19, 2012 at 10:55 AM, Vincent Jorgensen wrote:

Hi all!

I'm attempting to create a clone of the running chef server. Since it's running in AWS, I created an EBS/AMI and re-launched it. Everything almosts, but when I attempt to do "knife cookbook upload mycookbook -V" I get the following error:

Uploading mycookbook [1.0.0]
INFO: Saving mycookbook
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Uploading files
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 3/5 in 9s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 4/5 in 18s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 5/5 in 52s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: internal server error
Response: Connection failed - user: chef

I suspect that "chef-client" is going to fail as well. As near as my rudimentary google search could ascertain, this appears related to rabbitmq's DB. What's the safe way to change chef hosts without regenerating all of my validation.pem and client.pem files? I have the original instance still running, so I can copy any files from the original if need be.

Thanks!
Vincent

Logs from chef-server would be helpful.

Assuming the problem is RabbitMQ (seems likely), it looks like rabbitmq isn't running at all. Try running rabbitmq-server in the foreground and see if you get any hints. You can try blowing away the RabbitMQ database (usually in /var/lib/rabbitmq/mnesia/ ) and reconfiguring rabbit (see this: https://github.com/opscode/cookbooks/blob/master/chef-server/recipes/rabbitmq.rb ).

HTH,

Dan DeLeo

Vincent_Jorgensen · March 19, 2012, 6:52pm

Hi all!

Here is the output of rabbitmq-server:

rabbitmq-server

Activating RabbitMQ plugins ...
0 plugins activated:

node with name "rabbit" already running on "laertes"
diagnostics:

nodes and their ports on laertes: [{rabbit,34726},
{rabbitmqprelaunch2047,42450}]
current node: rabbitmqprelaunch2047@localhost
current node home dir: /var/lib/rabbitmq
current node cookie hash: tCzpZ8Rs9Ql4DuU5bRpWyA==

Also, it's nearly identical for both servers, so it's definitely running.

How do I re-apply the configuration?

Best,

Vincent

On Mar 19, 2012, at 11:44 AM, KC Braunschweig wrote:

Dan is probably right about Rabbit. IIRC its configuration is tied to
its hostname so if you bring it up again with a new hostname, it will
ignore the config. Re-applying the configuration should sort it out.

KC

On Mon, Mar 19, 2012 at 11:03 AM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, March 19, 2012 at 10:55 AM, Vincent Jorgensen wrote:

Hi all!

I'm attempting to create a clone of the running chef server. Since it's running in AWS, I created an EBS/AMI and re-launched it. Everything almosts, but when I attempt to do "knife cookbook upload mycookbook -V" I get the following error:

Uploading mycookbook [1.0.0]
INFO: Saving mycookbook
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Uploading files
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 3/5 in 9s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 4/5 in 18s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 5/5 in 52s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: internal server error
Response: Connection failed - user: chef

I suspect that "chef-client" is going to fail as well. As near as my rudimentary google search could ascertain, this appears related to rabbitmq's DB. What's the safe way to change chef hosts without regenerating all of my validation.pem and client.pem files? I have the original instance still running, so I can copy any files from the original if need be.

Thanks!
Vincent

Logs from chef-server would be helpful.

Assuming the problem is RabbitMQ (seems likely), it looks like rabbitmq isn't running at all. Try running rabbitmq-server in the foreground and see if you get any hints. You can try blowing away the RabbitMQ database (usually in /var/lib/rabbitmq/mnesia/ ) and reconfiguring rabbit (see this: https://github.com/opscode/cookbooks/blob/master/chef-server/recipes/rabbitmq.rb ).

HTH,

Dan DeLeo

kallistec · March 19, 2012, 6:55pm

On Monday, March 19, 2012 at 10:32 AM, Peter Norton wrote:

I'm doing some work to make launching multiple systems more friendly to human observation. My specific goal right now is to force more things to go through the logger, and to only log warning or error (and not debug and info) to the console. Info and above goes to a file for later perusal if needed.

However, I'm confused by this bit of code in some of the resources:

opts[:log_tag] = @new_resource.to_s
if STDOUT.tty? && !Chef::Config[:daemon] && Chef::Log.info (http://Log.info)? # <<< this specfically
opts[:live_stream] = STDOUT
end

if I have Log.info (http://Log.info) defined, why is chef spamming stdout instead of using the logger? Wouldn't it make more sense to have one of the following two options:

Be able to disable live_stream from the resource

Have live_stream asynchronously call the logger to print log lines

The pain point here is that most of the commands that I run from chef using the execute resource are not verbose (tar, cp, etc.). However, the apt LWRP uses execute for apt-get update, etc. which is incredibly verbose and which therefore drowns out all useful info in a parallel launch. Any suggestions on how to reduce this is appreciated. The option I think I'm going to have to go with at the moment is to start monkey-patching from within client.rb, but I'd like it if there was a clearer way to deal with this.

Thanks,

-Peter
The idea is that when you're watching the output (stdout is a tty and Chef isn't daemonized) and the log level is info or debug[0], the live stream is helpful to see what's going on with long running commands. You can disable it for everything by piping output through cat.

Index of Classes & Methods in logger: Ruby Standard Library Documentation (Ruby 1.9.3)
--
Dan DeLeo

Vincent_Jorgensen · March 19, 2012, 7:05pm

Got it! I don’t know if this was the right solution, but here’s what I did.

turn off chef rabbitmq
blow away /var/lib/rabbitmq/mnesia
scp mnesia contents from original server
rename mnesia directories to match current server
restart rabbitmq and chef

Best,

Vincent

On Mar 19, 2012, at 11:52 AM, Vincent Jorgensen wrote:

Hi all!

Here is the output of rabbitmq-server:

rabbitmq-server

Activating RabbitMQ plugins …
0 plugins activated:

node with name “rabbit” already running on "laertes"
diagnostics:

nodes and their ports on laertes: [{rabbit,34726},
{rabbitmqprelaunch2047,42450}]
current node: rabbitmqprelaunch2047@localhost
current node home dir: /var/lib/rabbitmq
current node cookie hash: tCzpZ8Rs9Ql4DuU5bRpWyA==

Also, it’s nearly identical for both servers, so it’s definitely running.

How do I re-apply the configuration?

Best,

Vincent

On Mar 19, 2012, at 11:44 AM, KC Braunschweig wrote:

Dan is probably right about Rabbit. IIRC its configuration is tied to
its hostname so if you bring it up again with a new hostname, it will
ignore the config. Re-applying the configuration should sort it out.

KC

On Mon, Mar 19, 2012 at 11:03 AM, Daniel DeLeo <dan@kallistec.com mailto:dan@kallistec.com> wrote:

On Monday, March 19, 2012 at 10:55 AM, Vincent Jorgensen wrote:

Hi all!

I’m attempting to create a clone of the running chef server. Since it’s running in AWS, I created an EBS/AMI and re-launched it. Everything almosts, but when I attempt to do “knife cookbook upload mycookbook -V” I get the following error:

Uploading mycookbook [1.0.0]
INFO: Saving mycookbook
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Uploading files
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 3/5 in 9s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 4/5 in 18s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: Server returned error for http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 5/5 in 52s
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ERROR: internal server error
Response: Connection failed - user: chef

I suspect that “chef-client” is going to fail as well. As near as my rudimentary google search could ascertain, this appears related to rabbitmq’s DB. What’s the safe way to change chef hosts without regenerating all of my validation.pem and client.pem files? I have the original instance still running, so I can copy any files from the original if need be.

Thanks!
Vincent

Logs from chef-server would be helpful.

Assuming the problem is RabbitMQ (seems likely), it looks like rabbitmq isn’t running at all. Try running rabbitmq-server in the foreground and see if you get any hints. You can try blowing away the RabbitMQ database (usually in /var/lib/rabbitmq/mnesia/ ) and reconfiguring rabbit (see this: https://github.com/opscode/cookbooks/blob/master/chef-server/recipes/rabbitmq.rb ).

HTH,

Dan DeLeo

Peter_Norton · March 19, 2012, 7:49pm

On Mon, Mar 19, 2012 at 2:55 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, March 19, 2012 at 10:32 AM, Peter Norton wrote:

I'm doing some work to make launching multiple systems more friendly to
human observation. My specific goal right now is to force more things to go
through the logger, and to only log warning or error (and not debug and
info) to the console. Info and above goes to a file for later perusal if
needed.

However, I'm confused by this bit of code in some of the resources:

opts[:log_tag] = @new_resource.to_s
if STDOUT.tty? && !Chef::Config[:daemon] && Chef::Log.info (
http://Log.info)? # <<< this specfically
opts[:live_stream] = STDOUT
end

if I have Log.info (http://Log.info) defined, why is chef spamming
stdout instead of using the logger? Wouldn't it make more sense to have one
of the following two options:

Be able to disable live_stream from the resource

Have live_stream asynchronously call the logger to print log lines

The pain point here is that most of the commands that I run from chef
using the execute resource are not verbose (tar, cp, etc.). However, the
apt LWRP uses execute for apt-get update, etc. which is incredibly verbose
and which therefore drowns out all useful info in a parallel launch. Any
suggestions on how to reduce this is appreciated. The option I think I'm
going to have to go with at the moment is to start monkey-patching from
within client.rb, but I'd like it if there was a clearer way to deal with
this.

Thanks,

-Peter
The idea is that when you're watching the output (stdout is a tty and Chef
isn't daemonized) and the log level is info or debug[0], the live stream is
helpful to see what's going on with long running commands. You can disable
it for everything by piping output through cat.

Index of Classes & Methods in logger: Ruby Standard Library Documentation (Ruby 1.9.3)
--
Dan DeLeo

But I don't want to ignore everything. My assumption is that if you have a
logger, and logging levels, etc. you want to use them. What I want to
happen is that I can log info to a file, and warning to the console if I'm
working interactively (e.g. a bootstrap). What happens in this case is
that instead of using the logger, chef spits to stdout, defeating the
purpose of configuring logging.

BTW, to actually get the logging that I want I need to do this:

Need to monkeypatch the logging initialization in Chef::Application

class Chef::Application
def configure_logging
l1 = Chef::Log.init("/var/log/chef/chef-info.log")
l1.level=(Mixlib::Log::LEVELS[:info])
l2 = Chef::Log.init(STDOUT)
l2.level=(Mixlib::Log::LEVELS[:warn])
STDOUT.sync = true
l2.formatter = Chef::Log.logger.formatter
Chef::Log::use_log_devices([l1, l2])
end
end
Chef::Log.warn("Logging INFO and higher to /var/log/chef/chef-info.log")

I'm not particularly happy about this, but I haven't found a better way to
both have my output logged and minimize interactive spam.

-Peter

AJ_Christensen · March 19, 2012, 7:52pm

What you consider spam other users may consider useful relevant
information accurately pertaining to a failure mode.

In some cases you will be asked to provide logs, often-times these are
grabbed from chef running on an interactive terminal after a
failure/crash. I don't really see the benefit here -- what are you
trying to do?

--AJ

On 20 March 2012 08:49, Peter Norton pn+chef-list@knewton.com wrote:

On Mon, Mar 19, 2012 at 2:55 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, March 19, 2012 at 10:32 AM, Peter Norton wrote:

I'm doing some work to make launching multiple systems more friendly to
human observation. My specific goal right now is to force more things to go
through the logger, and to only log warning or error (and not debug and
info) to the console. Info and above goes to a file for later perusal if
needed.

However, I'm confused by this bit of code in some of the resources:

opts[:log_tag] = @new_resource.to_s
if STDOUT.tty? && !Chef::Config[:daemon] && Chef::Log.info
(http://Log.info)? # <<< this specfically
opts[:live_stream] = STDOUT
end

if I have Log.info (http://Log.info) defined, why is chef spamming
stdout instead of using the logger? Wouldn't it make more sense to have one
of the following two options:

Be able to disable live_stream from the resource

Have live_stream asynchronously call the logger to print log lines

The pain point here is that most of the commands that I run from chef
using the execute resource are not verbose (tar, cp, etc.). However, the apt
LWRP uses execute for apt-get update, etc. which is incredibly verbose and
which therefore drowns out all useful info in a parallel launch. Any
suggestions on how to reduce this is appreciated. The option I think I'm
going to have to go with at the moment is to start monkey-patching from
within client.rb, but I'd like it if there was a clearer way to deal with
this.

Thanks,

-Peter
The idea is that when you're watching the output (stdout is a tty and Chef
isn't daemonized) and the log level is info or debug[0], the live stream is
helpful to see what's going on with long running commands. You can disable
it for everything by piping output through cat.

Index of Classes & Methods in logger: Ruby Standard Library Documentation (Ruby 1.9.3)
--
Dan DeLeo

But I don't want to ignore everything. My assumption is that if you have a
logger, and logging levels, etc. you want to use them. What I want to
happen is that I can log info to a file, and warning to the console if I'm
working interactively (e.g. a bootstrap). What happens in this case is that
instead of using the logger, chef spits to stdout, defeating the purpose of
configuring logging.

BTW, to actually get the logging that I want I need to do this:

Need to monkeypatch the logging initialization in Chef::Application

class Chef::Application
def configure_logging
l1 = Chef::Log.init("/var/log/chef/chef-info.log")
l1.level=(Mixlib::Log::LEVELS[:info])
l2 = Chef::Log.init(STDOUT)
l2.level=(Mixlib::Log::LEVELS[:warn])
STDOUT.sync = true
l2.formatter = Chef::Log.logger.formatter
Chef::Log::use_log_devices([l1, l2])
end
end
Chef::Log.warn("Logging INFO and higher to /var/log/chef/chef-info.log")

I'm not particularly happy about this, but I haven't found a better way to
both have my output logged and minimize interactive spam.

-Peter

Peter_Norton · March 19, 2012, 8:36pm

My use case is launching multiple systems (approx 8 right now). There are,
further, 2 sub-use-cases. The first is for automated deployment. This is
covered by notifications so I'm not concerned about failures here. The
second is developers bootstrapping, where they launch and are presented
with a matrix-like cascade of text. I want that text to be available to
investigate a failure. I also want the developer to know interactively
whether they should ^C the launch and kill the nodes because one has
failed. Some apps take 2.5 minutes to launch, and some take 15 minutes,
and I want to let them fail and investigate as fast as they can. The
solution I've come up with is to have messages at the info level sent to a
file, and to send warning to the console.

However, my use case may be a bit of a distraction from the core issue that
I'm trying to bring up:

    if STDOUT.tty? && !Chef::Config[:daemon] && Chef::Log.info?
      opts[:live_stream] = STDOUT

Code like this (from chef's lib/chef/provider/execute.rb, with similar code
in mixin/shell_out.rb, etc.) says that even If I configure chef to log to
debug to a file, and configure client.rb to log to a file (not via my
patching mechanism) my settings will be ignored and output will hit stdout.
I'm calling this spam because there's no way to configure it to actually
log to anywhere in particular - it's depending on 3 undocumented factors
(the state of your tty, the running mode of chef at the moment, and whether
logging is set up to lot at info) including the state of logging, and then
it's bypassing logging. Wat?
(https://www.destroyallsoftware.com/talks/watfor your amusement).

-Peter

On Mon, Mar 19, 2012 at 3:52 PM, AJ Christensen aj@junglist.gen.nz wrote:

What you consider spam other users may consider useful relevant
information accurately pertaining to a failure mode.

In some cases you will be asked to provide logs, often-times these are
grabbed from chef running on an interactive terminal after a
failure/crash. I don't really see the benefit here -- what are you
trying to do?

--AJ

On 20 March 2012 08:49, Peter Norton pn+chef-list@knewton.com wrote:

On Mon, Mar 19, 2012 at 2:55 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, March 19, 2012 at 10:32 AM, Peter Norton wrote:

I'm doing some work to make launching multiple systems more friendly
to
human observation. My specific goal right now is to force more things
to go
through the logger, and to only log warning or error (and not debug
and
info) to the console. Info and above goes to a file for later perusal
if
needed.

However, I'm confused by this bit of code in some of the resources:

opts[:log_tag] = @new_resource.to_s
if STDOUT.tty? && !Chef::Config[:daemon] && Chef::Log.info
(http://Log.info)? # <<< this specfically
opts[:live_stream] = STDOUT
end

if I have Log.info (http://Log.info) defined, why is chef spamming
stdout instead of using the logger? Wouldn't it make more sense to
have one
of the following two options:

Be able to disable live_stream from the resource

Have live_stream asynchronously call the logger to print log lines

The pain point here is that most of the commands that I run from chef
using the execute resource are not verbose (tar, cp, etc.). However,
the apt
LWRP uses execute for apt-get update, etc. which is incredibly
verbose and
which therefore drowns out all useful info in a parallel launch. Any
suggestions on how to reduce this is appreciated. The option I think
I'm
going to have to go with at the moment is to start monkey-patching
from
within client.rb, but I'd like it if there was a clearer way to deal
with
this.

Thanks,

-Peter
The idea is that when you're watching the output (stdout is a tty and
Chef
isn't daemonized) and the log level is info or debug[0], the live
stream is
helpful to see what's going on with long running commands. You can
disable
it for everything by piping output through cat.

Index of Classes & Methods in logger: Ruby Standard Library Documentation (Ruby 1.9.3)
--
Dan DeLeo

But I don't want to ignore everything. My assumption is that if you
have a
logger, and logging levels, etc. you want to use them. What I want to
happen is that I can log info to a file, and warning to the console if
I'm
working interactively (e.g. a bootstrap). What happens in this case is
that
instead of using the logger, chef spits to stdout, defeating the purpose
of
configuring logging.

BTW, to actually get the logging that I want I need to do this:

Need to monkeypatch the logging initialization in Chef::Application

class Chef::Application
def configure_logging
l1 = Chef::Log.init("/var/log/chef/chef-info.log")
l1.level=(Mixlib::Log::LEVELS[:info])
l2 = Chef::Log.init(STDOUT)
l2.level=(Mixlib::Log::LEVELS[:warn])
STDOUT.sync = true
l2.formatter = Chef::Log.logger.formatter
Chef::Log::use_log_devices([l1, l2])
end
end
Chef::Log.warn("Logging INFO and higher to /var/log/chef/chef-info.log")

I'm not particularly happy about this, but I haven't found a better way
to
both have my output logged and minimize interactive spam.

-Peter

KC_Braunschweig · March 19, 2012, 9:26pm

Nice to know that works. Reapplying the config manually looks like:

sudo rabbitmqctl add_vhost /chef
sudo rabbitmqctl add_user chef testing
sudo rabbitmqctl set_permissions -p /chef chef "." "." ".*"

If you configured chef server with chef-solo, rerunning that would also do it.

KC

On Mon, Mar 19, 2012 at 12:05 PM, Vincent Jorgensen
vincent.jorgensen@incentivenetworks.com wrote:

Got it! I don't know if this was the right solution, but here's what I did.

turn off chef rabbitmq

blow away /var/lib/rabbitmq/mnesia

scp mnesia contents from original server

rename mnesia directories to match current server

restart rabbitmq and chef

Best,

Vincent

On Mar 19, 2012, at 11:52 AM, Vincent Jorgensen wrote:

Hi all!

Here is the output of rabbitmq-server:

rabbitmq-server

Activating RabbitMQ plugins ...
0 plugins activated:

node with name "rabbit" already running on "laertes"
diagnostics:

nodes and their ports on laertes: [{rabbit,34726},
{rabbitmqprelaunch2047,42450}]

current node: rabbitmqprelaunch2047@localhost

current node home dir: /var/lib/rabbitmq

current node cookie hash: tCzpZ8Rs9Ql4DuU5bRpWyA==

Also, it's nearly identical for both servers, so it's definitely running.

How do I re-apply the configuration?

Best,

Vincent

On Mar 19, 2012, at 11:44 AM, KC Braunschweig wrote:

Dan is probably right about Rabbit. IIRC its configuration is tied to

its hostname so if you bring it up again with a new hostname, it will

ignore the config. Re-applying the configuration should sort it out.

KC

On Mon, Mar 19, 2012 at 11:03 AM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, March 19, 2012 at 10:55 AM, Vincent Jorgensen wrote:

Hi all!

I'm attempting to create a clone of the running chef server. Since it's
running in AWS, I created an EBS/AMI and re-launched it. Everything almosts,
but when I attempt to do "knife cookbook upload mycookbook -V" I get the
following error:

Uploading mycookbook [1.0.0]

INFO: Saving mycookbook

INFO: Validating ruby files

INFO: Validating templates

INFO: Syntax OK

INFO: Uploading files

INFO: HTTP Request Returned 500 Internal Server Error: Connection failed -
user: chef

ERROR: Server returned error for
http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 1/5 in 4s

INFO: HTTP Request Returned 500 Internal Server Error: Connection failed -
user: chef

ERROR: Server returned error for
http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 2/5 in 7s

INFO: HTTP Request Returned 500 Internal Server Error: Connection failed -
user: chef

ERROR: Server returned error for
http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 3/5 in 9s

INFO: HTTP Request Returned 500 Internal Server Error: Connection failed -
user: chef

ERROR: Server returned error for
http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 4/5 in 18s

INFO: HTTP Request Returned 500 Internal Server Error: Connection failed -
user: chef

ERROR: Server returned error for
http://107.21.8.91:4000/cookbooks/mycookbook/1.0.0, retrying 5/5 in 52s

INFO: HTTP Request Returned 500 Internal Server Error: Connection failed -
user: chef

ERROR: internal server error

Response: Connection failed - user: chef

I suspect that "chef-client" is going to fail as well. As near as my
rudimentary google search could ascertain, this appears related to
rabbitmq's DB. What's the safe way to change chef hosts without regenerating
all of my validation.pem and client.pem files? I have the original instance
still running, so I can copy any files from the original if need be.

Thanks!

Vincent

Logs from chef-server would be helpful.

Assuming the problem is RabbitMQ (seems likely), it looks like rabbitmq
isn't running at all. Try running rabbitmq-server in the foreground and
see if you get any hints. You can try blowing away the RabbitMQ database
(usually in /var/lib/rabbitmq/mnesia/ ) and reconfiguring rabbit (see this:
https://github.com/opscode/cookbooks/blob/master/chef-server/recipes/rabbitmq.rb
).

HTH,

--

Dan DeLeo

Topic		Replies	Views
Using the 'log' resource vs Chef::Log.<loglevel> Chef Infra (archive)	5	620	July 8, 2013
Stdout/stderr and exception/report framework Chef Infra (archive)	2	561	April 9, 2012
Replicating chef-client log and report settings when run as a service Chef Infra (archive)	4	678	February 2, 2016
How to use resource bash in chef handlers? Chef Infra (archive)	3	579	February 1, 2018
Logging Question Chef Infra (archive)	0	271	May 8, 2019

Logging less from resources, aka using the logger for logging and not STDOUT

HTH,

HTH,

rabbitmq-server

HTH,

rabbitmq-server

HTH,

Need to monkeypatch the logging initialization in Chef::Application

Need to monkeypatch the logging initialization in Chef::Application

Need to monkeypatch the logging initialization in Chef::Application

rabbitmq-server

Related Topics