Authentication error when indexing nodes


#1

Hey Chefs,

Please help to find the root cause of the problem – sometimes when
registering/updating nodes the indexing phase fails due to strange
authentication error. This happens not often but the influence is very
unpleasant for our system. We’re using Chef 0.9.8.

Thanks!

The error from server.log:

merb : chef-server (api) : worker (port 4000) ~ Started request handling:
Sun Feb 27 19:58:36 +0000 2011
merb : chef-server (api) : worker (port 4000) ~ Routed to:
{“action”=>“index”, “controller”=>“main”}
merb : chef-server (api) : worker (port 4000) ~ Params: {“action”=>“index”,
“controller”=>“main”}
merb : chef-server (api) : worker (port 4000) ~
Mixlib::Authentication::MissingAuthenticationHeader: missing required a
uthentication header(s) ‘X_OPS_SIGN’, ‘X_OPS_USERID’, ‘X_OPS_TIMESTAMP’,
‘X_OPS_CONTENT_HASH’ - (Merb::ControllerExcept
ions::Unauthorized)
/usr/lib/ruby/gems/1.8/gems/chef-server-api-0.9.8/app/controllers/application.rb:47:in
authenticate_every' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:346:insend’
/usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:346:in
_call_filters' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:338:ineach’
/usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:338:in
_call_filters' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:283:in_dispatch’
/usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:281:in
catch' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:281:in_dispatch’
/usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/merb_controller.rb:252:in
_dispatch' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/dispatch/dispatcher.rb:102:indispatch_action’
/usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/dispatch/dispatcher.rb:74:in
handle' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/dispatch/dispatcher.rb:36:inhandle’
/usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/rack/application.rb:17:in
call' /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:incall’
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:76:in
pre_process' /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:incatch’
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:74:in
pre_process' /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:57:inprocess’
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/connection.rb:42:in
receive_data' /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:inrun_machine’
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
run' /usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/backends/base.rb:57:instart’
/usr/lib/ruby/gems/1.8/gems/thin-1.2.5/lib/thin/server.rb:156:in start' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/rack/adapter/thin.rb:30:instart_server’
/usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/rack/adapter/abstract.rb:296:in
start_at_port' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/rack/adapter/abstract.rb:128:instart’
/usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:174:in
bootup' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:159:indaemonize’
/usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:143:in
fork' /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:143:indaemonize’


Andrew Klochkov


#2

On Sunday, February 27, 2011 at 12:03 PM, Andrey Klochkov wrote:
Hey Chefs,

Please help to find the root cause of the problem – sometimes when registering/updating nodes the indexing phase fails due to strange authentication error. This happens not often but the influence is very unpleasant for our system. We’re using Chef 0.9.8.

Thanks!

The error from server.log:

merb : chef-server (api) : worker (port 4000) ~ Started request handling: Sun Feb 27 19:58:36 +0000 2011
merb : chef-server (api) : worker (port 4000) ~ Routed to: {“action”=>“index”, “controller”=>“main”}
merb : chef-server (api) : worker (port 4000) ~ Params: {“action”=>“index”, “controller”=>“main”}
merb : chef-server (api) : worker (port 4000) ~ Mixlib::Authentication::MissingAuthenticationHeader: missing required a
uthentication header(s) ‘X_OPS_SIGN’, ‘X_OPS_USERID’, ‘X_OPS_TIMESTAMP’, ‘X_OPS_CONTENT_HASH’ - (Merb::ControllerExcept
ions::Unauthorized)

This request is completely unsigned. Also, it’s hitting the root URL (i.e., http://HOSTNAME/ ) which is strange since there’s nothing there. Can you show us the client log and the code that is making this request (assuming it’s something in your recipe)?

Is it possible that this request is from a monitoring system and the server log for your error is something else?


Dan DeLeo


#3

Daniel,

Thanks, found it - yes, it’s part of our monitoring system which just
performs GET request to Chef Server root url to check it’s alive (401 is Ok
in this case). These errors have nothing to do with our registration
problems, you’re right.

On Sun, Feb 27, 2011 at 11:10 PM, Daniel DeLeo dan@kallistec.com wrote:

On Sunday, February 27, 2011 at 12:03 PM, Andrey Klochkov wrote:

Hey Chefs,

Please help to find the root cause of the problem – sometimes when
registering/updating nodes the indexing phase fails due to strange
authentication error. This happens not often but the influence is very
unpleasant for our system. We’re using Chef 0.9.8.

Thanks!

The error from server.log:

merb : chef-server (api) : worker (port 4000) ~ Started request handling:
Sun Feb 27 19:58:36 +0000 2011
merb : chef-server (api) : worker (port 4000) ~ Routed to:
{“action”=>“index”, “controller”=>“main”}
merb : chef-server (api) : worker (port 4000) ~ Params: {“action”=>“index”,
“controller”=>“main”}
merb : chef-server (api) : worker (port 4000) ~
Mixlib::Authentication::MissingAuthenticationHeader: missing required a
uthentication header(s) ‘X_OPS_SIGN’, ‘X_OPS_USERID’, ‘X_OPS_TIMESTAMP’,
‘X_OPS_CONTENT_HASH’ - (Merb::ControllerExcept
ions::Unauthorized)

This request is completely unsigned. Also, it’s hitting the root URL (i.e.,
http://HOSTNAME/ ) which is strange since there’s nothing there. Can you
show us the client log and the code that is making this request (assuming
it’s something in your recipe)?

Is it possible that this request is from a monitoring system and the server
log for your error is something else?


Dan DeLeo


Andrew Klochkov
Grid Dynamics
+7 495 640 4944 ext 257 office
Skype: aklochkov_gd


aklochkov@griddynamics.com