Client registration fails with failed to authenticate - part way through series of installs

Chef Infra (archive)
1

Hi all,

I have been deploying chef-client (0.10.8) from gems to some instances
using a script which delivers a validation.pem key for the
chef-validator user. The script was working well with many instances
deployed.

However at some point all the new instances start to fail with the
following error; (The server log is here http://pastebin.com/XkYXZtqe)
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: HTTP Request Returned 401
Unauthorized: Failed to authenticate. Ensure that your client key is valid.

According to the debug log, the NTP time differences are OK, and the
Hash content is ok, but the request has failed a
"OpenSSL::PKey::RSAError: padding check failed"

See here…

  1. DEBUG: Failed to verify request signature: OpenSSL::PKey::RSAError:
    padding check failed
  2. DEBUG: Request time difference: 0.617022, within 900 seconds? : true
  3. DEBUG: Expected content hash is: ‘VD3sVc7y8Od5rhMPZqxkdaNP5Q8=’
  4. DEBUG: Request Content Hash is: ‘VD3sVc7y8Od5rhMPZqxkdaNP5Q8=’
  5. DEBUG: Hashes match?: true

Any suggestion on what might be the problem?

This seems to intermittently happen, and if I regenerate the
chef-validator private key I might get another week or 2 before that key
gets screwed up.

Thanks,
Tom

Linux opencirrus-g0801.hpl.hp.com 2.6.18-194.26.1.el5 #1 SMP Tue Nov 9
12:54:20 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@opencirrus-g0801 ~]# ruby -v
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

[root@opencirrus-g0801 ~]# gem search chef
*** LOCAL GEMS ***
chef (0.10.8)
chef-expander (0.10.8)
chef-server (0.10.8)
chef-server-api (0.10.8)
chef-server-webui (0.10.8)
chef-solr (0.10.8)

http://pastebin.com/XkYXZtqe

client log;
[root@i-00007740 ~]# chef-client -j /etc/chef/first-boot.json
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: *** Chef 0.10.8 ***
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: Client key /etc/chef/client.pem
is not present - registering
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: HTTP Request Returned 401
Unauthorized: Failed to authenticate. Ensure that your client key is valid.
[Tue, 10 Jan 2012 03:13:08 -0800] FATAL: Stacktrace dumped to
/var/cache/chef/chef-stacktrace.out
[Tue, 10 Jan 2012 03:13:08 -0800] FATAL: Net::HTTPServerException: 401
"Unauthorized"
[root@i-00007740 ~]#

The server log is here http://pastebin.com/XkYXZtqe

2

On Tuesday, January 10, 2012 at 3:58 AM, Tom wrote:

Hi all,

I have been deploying chef-client (0.10.8) from gems to some instances using a script which delivers a validation.pem key for the chef-validator user. The script was working well with many instances deployed.

However at some point all the new instances start to fail with the following error; (The server log is here (http://pastebin.com/XkYXZtqe))
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: HTTP Request Returned 401 Unauthorized: Failed to authenticate. Ensure that your client key is valid.

According to the debug log, the NTP time differences are OK, and the Hash content is ok, but the request has failed a "OpenSSL::PKey::RSAError: padding check failed"

See here...
DEBUG: Failed to verify request signature: OpenSSL::PKey::RSAError: padding check failed
DEBUG: Request time difference: 0.617022, within 900 seconds? : true

DEBUG: Expected content hash is: 'VD3sVc7y8Od5rhMPZqxkdaNP5Q8='

DEBUG: Request Content Hash is: 'VD3sVc7y8Od5rhMPZqxkdaNP5Q8='

DEBUG: Hashes match?: true
"padding check failed" is one of the errors that can occur when the public key used to decrypt the encrypted content does not correspond to the private key used to encrypt it.

I'd see if some external process is rekeying the validator. For example, if you remove the validator from its default location on the Chef Server and restart the server, it will rekey the validator and drop off the new one.

Do the times when this stops working correspond to something like restarts caused by log rotation?

--
Dan DeLeo

Any suggestion on what might be the problem?

This seems to intermittently happen, and if I regenerate the chef-validator private key I might get another week or 2 before that key gets screwed up.

Thanks,
Tom

Linux opencirrus-g0801.hpl.hp.com (http://opencirrus-g0801.hpl.hp.com) 2.6.18-194.26.1.el5 #1 SMP Tue Nov 9 12:54:20 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@opencirrus-g0801 ~]# ruby -v
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

[root@opencirrus-g0801 ~]# gem search chef
*** LOCAL GEMS ***
chef (0.10.8)
chef-expander (0.10.8)
chef-server (0.10.8)
chef-server-api (0.10.8)
chef-server-webui (0.10.8)
chef-solr (0.10.8)

client log;
[root@i-00007740 ~]# chef-client -j /etc/chef/first-boot.json
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: *** Chef 0.10.8 ***
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: Client key /etc/chef/client.pem is not present - registering
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: HTTP Request Returned 401 Unauthorized: Failed to authenticate. Ensure that your client key is valid.
[Tue, 10 Jan 2012 03:13:08 -0800] FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
[Tue, 10 Jan 2012 03:13:08 -0800] FATAL: Net::HTTPServerException: 401 "Unauthorized"
[root@i-00007740 ~]#

The server log is here (http://pastebin.com/XkYXZtqe)