Apt_repository started failing with SSL verification errors

Chef Infra
#1

From last 2 days we started seeing chef execution failures are happening related to apt_repository resource. Seen similar failures with chef-client execution too

Chef-solo version: 12.19.36
OS: Ubuntu v18 & Ubuntuv14

See below error from chef-solo execution.

==> core: [2021-10-04T14:36:46+00:00] ERROR: SSL Validation failure connecting to host: www.postgresql.org - SSL_connect returned=1 errno=0 state=error: certificate verify failed
==> core:
==> core:
==> core: ================================================================================
==> core: Error executing action create on resource 'remote_file[/var/chef/cache/https___www_postgresql_org_media_keys_ACCC4CF8_asc]'
==> core: ================================================================================
==> core:
==> core: OpenSSL::SSL::SSLError
==> core: ----------------------
==> core: SSL Error connecting to https://www.postgresql.org/media/keys/ACCC4CF8.asc - SSL_connect returned=1 errno=0 state=error: certificate verify failed
==> core:
==> core:
==> core: Resource Declaration:
==> core: ---------------------
==> core: # In /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/provider/apt_repository.rb
==> core:
==> core: 166: declare_resource(type, cached_keyfile) do
==> core: 167: source new_resource.key
==> core: 168: mode "0644"
==> core: 169: sensitive new_resource.sensitive
==> core: 170: action :create
==> core: 171: end
==> core: 172:
==> core:
==> core: Compiled Resource:
==> core:
==> core: ------------------
==> core: # Declared in /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/provider/apt_repository.rb:166:in `install_key_from_uri'
==> core:

#2

Encountering similar issues across multiple resources and primarily with remote_file resource

provider Chef::Provider::RemoteFile
==> core: action [:create]
==> core: retries 0
==> core: retry_delay 2
==> core: default_guard_interpreter :default
==> core: source ["https://getcomposer.org/download/1.0.0-alpha11/composer.phar"]
==> core: use_etag true
==> core: use_last_modified true
==> core: declared_type :remote_file

#3

Both https://www.postgresql.org/ and https://getcomposer.org/ are using Letsencrypt certificates so you are likely being bitten by the expiry of the DST X3 root certificate.

There is an alternate validation path to the ISRG X1 root which is still valid but older versions of openssl won't find it if the DST X3 certificate is still in the root certificate store. Unfortunately even chef 16 is still using an old openssl 1.0.2 release that is affected by this.

The latest chef 16 and 17 releases remove DST X3 from the root certificate store which should fix it, or you can remove it yourself - the file I had to edit was /opt/chef/embedded/ssl/certs/cacert.pem which fixed the problem for us before the chef 16 update came out.

As to chef-solo 12.19.36 that may be so old that you'd need to add ISRG X1 to the root store as well!

1 Like
#4

Thanks @tomhughes for the response. It helped fix the issue we are encountering. We use chef-solo provider(in Vagrant) and chef-client for bootstrapping instances. Is there a way to pass custom certs while bootstrapping?