Chef server (v12) : error when changing SSL ciphers

Chef Infra Server
1

Hello,

At my company, (for several reasons...), we're stuck with Chef-server version 12.
Unfortunatly, an audit pointed out that this version is using weak SSL ciphers.
I've tried to update the config to use more secure SSL ciphers :

  • chef-client is running fine on all my servers
  • But I "sometimes" have some trouble to push new files :

Apparently, If i'm the one who made the last upload (of a cookbook for example), it runs fine, I can see on the nginx logs:

1.2.3.4 - - [04/Oct/2023:15:16:49 +0000]  "PUT /bookshelf/organization-00000000000000000000000000000000/checksum-7b019b129e41c4f3e4643e5ad47dd639?AWSAccessKeyId=54ca32b30c0b21a9f8ea0a4b245bef1909ec7c82&Expires=1696433509&Signature=pemmaoinHVHALPncmlOs8MU1xck%3D HTTP/1.1" 204 "0.022" 0 "-" "Chef Knife/12.3.0 (ruby-2.1.4-p265; ohai-8.3.0; x86_64-linux; +http://opscode.com)" "127.0.0.1:4321" "204" "0.012" "12.3.0" "algorithm=sha1;version=1.0;" "username" "2023-10-04T15:17:22Z" "tq4D1Nofy61oH9Sv0r3tDu7ItdE=" 7284

But if I'm pushing a cookbook lastly pushed by someone else (or, that's what I feel, not 100% sure about that...) :

1.2.3.4 - - [04/Oct/2023:15:16:53 +0000]  "PUT /sandboxes/000000000000b7a1ff8b4dca2a381355 HTTP/1.1" 500 "0.039" 36 "-" "Chef Knife/12.3.0 (ruby-2.1.4-p265; ohai-8.3.0; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "500" "0.028" "12.3.0" "algorithm=sha1;version=1.0;" "username" "2023-10-04T15:17:22Z" "oMRtV6loUDnbKJuGcW6nqBbF8ww=" 1097

Is there a way (except upgrade to a latest chef-server version...) to use stronger ciphers than the original ones without breaking the system ?

2

Hi Nico,

I think we need a little more information. Are you seeing any kind off ssl error on upload or download of the cookbook?
It would also be helpful if you could include the portion of the erchef request and crash log that shows more details about what's causing the 500 error you're seeing. You can find those logs in /var/log/opscode/oc_erchef.

3

Hello

When I try to upload a cookbook that my colleague had uploaded earlier :

$ knife cookbook upload web -o cookbooks/
* KNIFE SITE: VDR
Uploading web          [0.0.2]
ERROR: Server returned error 500 for https://chef-server.com/sandboxes/00000000000041068702c03b2bd644f4, retrying 1/5 in 3s
ERROR: Server returned error 500 for https://chef-server.com/sandboxes/00000000000041068702c03b2bd644f4, retrying 2/5 in 5s
ERROR: Server returned error 500 for https://chef-server.com/sandboxes/00000000000041068702c03b2bd644f4, retrying 3/5 in 9s
ERROR: Server returned error 500 for https://chef-server.com/sandboxes/00000000000041068702c03b2bd644f4, retrying 4/5 in 26s
ERROR: Server returned error 500 for https://chef-server.com/sandboxes/00000000000041068702c03b2bd644f4, retrying 5/5 in 54s
ERROR: internal server error
Response: internal service error

This is what I can find in the erchef/requests.log.1 file

2023-10-20T12:15:56Z erchef@127.0.0.1 method=PUT; path=/sandboxes/0000000000000f942bda7567a7faa52a; status=500; user=username; req_id=/RTq4TQsXgiQ3tuoc/wfdA==; req_time=3372; rdbms_time=132; rdbms_count=3; s3_time=3233; s3_count=1;
2023-10-20T12:16:02Z erchef@127.0.0.1 method=PUT; path=/sandboxes/0000000000000f942bda7567a7faa52a; status=500; user=username; req_id=ofSVO9+Rn/vujOxCU/uGXA==; req_time=3457; rdbms_time=210; rdbms_count=3; s3_time=3240; s3_count=1;
2023-10-20T12:16:13Z erchef@127.0.0.1 method=PUT; path=/sandboxes/0000000000000f942bda7567a7faa52a; status=500; user=username; req_id=mt797hQu9wnAONg9WOfULg==; req_time=3120; rdbms_time=125; rdbms_count=3; s3_time=2991; s3_count=1;
2023-10-20T12:16:32Z erchef@127.0.0.1 method=PUT; path=/sandboxes/0000000000000f942bda7567a7faa52a; status=500; user=username; req_id=SLcY5zawRpWZsw+zKUf5iQ==; req_time=4489; rdbms_time=210; rdbms_count=3; s3_time=4272; s3_count=1;
2023-10-20T12:17:01Z erchef@127.0.0.1 method=PUT; path=/sandboxes/0000000000000f942bda7567a7faa52a; status=500; user=username; req_id=VK+SauiRZYKjR+u0F52ELw==; req_time=3466; rdbms_time=115; rdbms_count=3; s3_time=3346; s3_count=1;

In crash.log, I can see those kind of error :

2023-10-20 12:23:12 =ERROR REPORT====
SSL: hello: ssl_connection.erl:1724:Fatal error: handshake failure
2023-10-20 12:23:12 =ERROR REPORT====
Checking presence of file (checksum: <<"7c8c2fa5887396d790bcd44e53efc80d">>) for org <<"00000000000000000000000000000000">> from bucket "bookshelf" (key: "organization-00000000000000000000000000000000/checksum-7c8c2fa5887396d790bcd44e53efc80d") raised exception error:{aws_error,{socket_error,{conn_failed,{error,esslconnect}}}}
2023-10-20 12:23:12 =ERROR REPORT====
Checking presence of file (checksum: <<"17a7f0a8e086524ae51f3541d7093b9c">>) for org <<"00000000000000000000000000000000">> from bucket "bookshelf" (key: "organization-00000000000000000000000000000000/checksum-17a7f0a8e086524ae51f3541d7093b9c") raised exception error:{aws_error,{socket_error,{conn_failed,{error,esslconnect}}}}
2023-10-20 12:23:12 =ERROR REPORT====
SSL: hello: ssl_connection.erl:1724:Fatal error: handshake failure
2023-10-20 12:23:12 =ERROR REPORT====
SSL: hello: ssl_connection.erl:1724:Fatal error: handshake failure
2023-10-20 12:23:12 =ERROR REPORT====
SSL: hello: ssl_connection.erl:1724:Fatal error: handshake failure
2023-10-20 12:23:12 =ERROR REPORT====
SSL: hello: ssl_connection.erl:1724:Fatal error: handshake failure
2023-10-20 12:23:12 =ERROR REPORT====
SSL: hello: ssl_connection.erl:1724:Fatal error: handshake failure
2023-10-20 12:23:12 =ERROR REPORT====
Checking presence of file (checksum: <<"2a16ac57a474626e712fea3c48cc1bf7">>) for org <<"00000000000000000000000000000000">> from bucket "bookshelf" (key: "organization-00000000000000000000000000000000/checksum-2a16ac57a474626e712fea3c48cc1bf7") raised exception error:{aws_error,{socket_error,{conn_failed,{error,esslconnect}}}}
2023-10-20 12:23:12 =ERROR REPORT====
Checking presence of file (checksum: <<"6a2e278013a3fd2a96f015e5f94d66ce">>) for org <<"00000000000000000000000000000000">> from bucket "bookshelf" (key: "organization-00000000000000000000000000000000/checksum-6a2e278013a3fd2a96f015e5f94d66ce") raised exception error:{aws_error,{socket_error,{conn_failed,{error,esslconnect}}}}
2023-10-20 12:23:12 =ERROR REPORT====
Checking presence of file (checksum: <<"45b251f98e32959553652dbb39944b96">>) for org <<"00000000000000000000000000000000">> from bucket "bookshelf" (key: "organization-00000000000000000000000000000000/checksum-45b251f98e32959553652dbb39944b96") raised exception error:{aws_error,{socket_error,{conn_failed,{error,esslconnect}}}}
2023-10-20 12:23:12 =ERROR REPORT====
Checking presence of file (checksum: <<"f370a68cb796f94cbff35d89936c4f1f">>) for org <<"00000000000000000000000000000000">> from bucket "bookshelf" (key: "organization-00000000000000000000000000000000/checksum-f370a68cb796f94cbff35d89936c4f1f") raised exception error:{aws_error,{socket_error,{conn_failed,{error,esslconnect}}}}
2023-10-20 12:23:12 =ERROR REPORT====
Checking presence of file (checksum: <<"27ed6abe166d0e5bb63c049679c130a4">>) for org <<"00000000000000000000000000000000">> from bucket "bookshelf" (key: "organization-00000000000000000000000000000000/checksum-27ed6abe166d0e5bb63c049679c130a4") raised exception error:{aws_error,{socket_error,{conn_failed,{error,esslconnect}}}}

Other than that, I can see that the VMs are downloading the files correctly from the Chef-server. We only see error when trying to upload cookbooks (I've just successfully uploaded a "node" file which had been earlier uploaded by my colleague.)

I suspect it might be an issue with my knife client, but don't know how to verifiy that...

4

Digging a bit deeper, I did a tcpdump.
Apparently,

  • Client hello
  • Server hello
  • Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
  • New Session Ticket, Change Cipher Spec, Encrypted Handshake Message
  • Content Type: Change Cipher Spec (20)
  • 3 "Application data" packets
  • FIN/ACK (from server), ACK
  • Encrypted Alert
  • FIN/ACK (from client), ACK

Unfortunatly, I'm not able to see what is this "Encrypted alert"