We are having weird NET::HTTPServerException errors on a 2 out of many nodes when a recipe tries to render a template to a file. The file is created correctly but we get this 403 Forbidden error at that point of the chef-client run which corresponds to a request in the nginx access.log from one of these 2 nodes requesting via GET the /organizations/my_org_name URL. No other nodes which all run this same recipe and create the same file make this GET request to the server. Any idea why chef is requesting this URL and getting a 403 Forbidden response? I assume the forbidden is because this URL is for a folder not a file and browsing folders is disabled. The chef-stacktrace.out contains this information:
Generated at 2020-03-18 21:26:01 +0000 Net::HTTPServerException: template[D:/Hotfixes/InstallHotfix.ps1] (foo-core-provision::create-hotfix-framework line 59) had an error: Net::HTTPServerException: 403 "Forbidden" C:/opscode/chef/embedded/lib/ruby/2.5.0/net/http/response.rb:122:in error!'
C:/opscode/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29-universal-mingw32/lib/chef/http.rb:237:instreaming_request' C:/opscode/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29-universal-mingw32/lib/chef/cookbook/remote_file_vendor.rb:66:in get_filename'
C:/opscode/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29-universal-mingw32/lib/chef/cookbook_version.rb:276:inpreferred_filename_on_disk_location' C:/opscode/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29-universal-mingw32/lib/chef/provider/template_finder.rb:40:in find'
C:/opscode/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29-universal-mingw32/lib/chef/provider/template/content.rb:32:intemplate_location' C:/opscode/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29-universal-mingw32/lib/chef/provider/template.rb:43:in block in define_resource_requirements'
C:/opscode/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29-universal-mingw32/lib/chef/mixin/why_run.rb:309:inassert' C:/opscode/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29-universal-mingw32/lib/chef/provider/template.rb:41:in define_resource_requirements'
C:/opscode/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29-universal-mingw32/lib/chef/provider.rb:169:inrun_action'
I thought this sounded like the weird 403 errors due to lazy loading and expiration of links but none of the requests with expiration values have other than 200 HTTP status replies and the expiration value is hours into the future after our IT department set the chef server configuration to allow 36000 seconds.
Any ideas?