Chef-validator authentication fails after Chef server upgrade. How to fix this?

Chef Infra (archive)
1

Hello,

This morning, I upgraded (via gems) the Chef installation on my Chef server
and all the managed nodes. This was a completely functioning setup where I
was using vagrant to test my cookbooks before taking them into production.

After the upgrade, all clients connected without any problems except for
one thing: any access via chef-validator fails with “401 Unauthorized:
Failed to authenticate”! I can create clients in the web UI, but the
auto-registration via chef-validator doesn’t work.

After a few such failures, I regenerated the key pair of chef-validator and
saved the private key part in validation.pem. Doesn’t resolve the problem.

Then I noticed the chef-validator client was not an admin user, so I edited
the user and made it admin. Still no luck.

Anyone an idea how to fix this?

Ringo

2

On Thu, Aug 30, 2012 at 8:25 AM, Ringo De Smet ringo.desmet@gmail.com wrote:

After a few such failures, I regenerated the key pair of chef-validator and
saved the private key part in validation.pem. Doesn't resolve the problem.

Can you set log_level to debug on the API server and look for a
relevant message there?

Then I noticed the chef-validator client was not an admin user, so I edited
the user and made it admin. Still no luck.

This doesn't matter, and generally the validation client should not be
an admin because then if someone got a hold of the validation key
which is commonly left around on nodes, they could cause a lot of
havoc.

The validation client gets special permissions that are hardcoded. See here:

https://github.com/opscode/chef/blob/master/chef-server-api/app/controllers/application.rb#L72

Bryan

3

Bryan,

On 30 August 2012 16:04, Bryan McLellan btm@loftninjas.org wrote:

On Thu, Aug 30, 2012 at 8:25 AM, Ringo De Smet ringo.desmet@gmail.com
wrote:

After a few such failures, I regenerated the key pair of chef-validator
and
saved the private key part in validation.pem. Doesn't resolve the
problem.

Can you set log_level to debug on the API server and look for a
relevant message there?

Here is the relevant output from the server.log at debug level:

Ringo

4

On Thu, Aug 30, 2012 at 10:42 AM, Ringo De Smet ringo.desmet@gmail.com wrote:

Here is the relevant output from the server.log at debug level:
chef-validator: 401 Unauthorized problem · GitHub

What's builder? Is that the node_name of the client you're trying to
create? Have you tried a different node_name?

This is what shows up in the logs for a new client I just build on a
fresh apt install.

merb : chef-server (api) : worker (port 4000) ~ Params:
{"admin"=>false, "name"=>"domU-12-31-39-0E-AD-32.compute-1.internal",
"controller"=>"clients", "action"=>"create"}

Bryan

5

On Thursday, 30 August 2012, Bryan McLellan wrote:

On Thu, Aug 30, 2012 at 10:42 AM, Ringo De Smet <ringo.desmet@gmail.com<javascript:;>>
wrote:

Here is the relevant output from the server.log at debug level:
chef-validator: 401 Unauthorized problem · GitHub

What's builder? Is that the node_name of the client you're trying to
create? Have you tried a different node_name?

Yes, builder is the node name. I used that name multiple times before
without any problems In my Vagrant setup before the Chef Server upgrade. I
can also create that client in the web UI manually.

Any other suggestions?

Ringo

6

On Thu, Aug 30, 2012 at 4:21 PM, Ringo De Smet ringo.desmet@gmail.com wrote:

Yes, builder is the node name. I used that name multiple times before
without any problems In my Vagrant setup before the Chef Server upgrade. I
can also create that client in the web UI manually.

I suggested trying a different node_name to see if perhaps there was
some kind of issue with broken objects.

Any other suggestions?

Do you see chef-validator in knife client list?

knife client delete chef-validator # is the client gone now?
/etc/init.d/chef-server restart
knife client list # Does chef-validator come back?

modify your knife.rb to use a node_name of 'chef-validator' and point
it at that key and see if you can list clients using it?

Check your server.rb validation_key setting and the timestamp on
/etc/chef/validation.pem to make sure the key is being saved where you
expect it to be?

Bryan

7

Bryan,

On 31 August 2012 00:16, Bryan McLellan btm@loftninjas.org wrote:

Do you see chef-validator in knife client list?

knife client delete chef-validator # is the client gone now?
/etc/init.d/chef-server restart
knife client list # Does chef-validator come back?

The process of recreating the chef-validator client seems to have resolved
the problem!

Thank you very much!

Ringo