Chef Provisioning AWS - Resource ID not found errors and Eventual Consistency

Chef Infra (archive)
1

Hello All,

So I have been facing an intermittent error while using Chef to Provision AWS Infrastructure which is when Chef creates a resource and then tries to query AWS for its existence and create the ‘Name’ tag to the resource and even when the resource is created, Chef fails after it’s retry attempts with Resource with ID not found error. I can see the resource created on AWS console when Chef is unable to find it and attempting to look for it.

AWS Documentation says that it may happen at times and can be overcome by putting some delays. (Ref: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#eventual-consistency)

Is there any way to put a delay in between the retry attempts on top of what Chef Provisioning already has?

I also filed an issue on the Github repo for Chef Provisioning AWS.

Thanks,
Rahul

1 Like
2

Hello Rahul,

I usually add this to my knife.rb or client.rb

chef_provisioning({:aws_retry_limit => 10, :image_max_wait_time => 600,
:machine_max_wait_time => 600})

3

Hello,

Thanks for the response. But is it not just for machine_batch resource? Docs on Github mention machine_batch when describing these options. Because when I try to put this (:aws_retry_limit => 25) it still just runs for 10 times and then fails.
Again, this is intermittent and saw it one time in like 5 or 6 deployments.
See these logs:


[2016-11-14T19:29:45+00:00] INFO: [AWS EC2 200 0.093748 0 retries] create_network_acl(:vpc_id=>"vpc-41936926")  

[2016-11-14T19:29:45+00:00] INFO: [AWS EC2 400 0.046846 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:29:46+00:00] INFO: [AWS EC2 400 0.078124 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:29:48+00:00] INFO: [AWS EC2 400 0.078126 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:29:52+00:00] INFO: [AWS EC2 400 0.078126 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:30:00+00:00] INFO: [AWS EC2 400 0.0781 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:30:16+00:00] INFO: [AWS EC2 400 0.078102 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:30:32+00:00] INFO: [AWS EC2 400 0.078126 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:30:49+00:00] INFO: [AWS EC2 400 0.0781 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:31:05+00:00] INFO: [AWS EC2 400 0.07813 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:31:21+00:00] INFO: [AWS EC2 400 0.078125 0 retries] create_tags(:resources=>["acl-1704a570"],:tags=>[{:key=>"Name",:value=>"EG5974_DMZ_21.25_ACL_01"}]) AWS::EC2::Errors::InvalidNetworkAclID::NotFound The networkAcl ID 'acl-1704a570' does not exist

[2016-11-14T19:31:21+00:00] INFO: HTTP Request Returned 404 Object Not Found: Cannot load data bag item EG5974_DMZ_21.25_ACL_01 for data bag aws_network_acl
[2016-11-14T19:31:21+00:00] INFO: HTTP Request Returned 404 Object Not Found: Cannot load data bag item EG5974_DMZ_21.25_ACL_01 for data bag aws_network_acl
[2016-11-14T19:31:21+00:00] INFO: HTTP Request Returned 404 Object Not Found: Cannot load data bag item EG5974_DMZ_21.25_ACL_01 for data bag aws_network_acl
[2016-11-14T19:31:21+00:00] INFO: Running queued delayed notifications before re-raising exception
[2016-11-14T19:31:21+00:00] ERROR: Running exception handlers
[2016-11-14T19:31:21+00:00] ERROR: Exception handlers complete
[2016-11-14T19:31:21+00:00] FATAL: Stacktrace dumped to c:/chef/cache/chef-stacktrace.out
[2016-11-14T19:31:21+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2016-11-14T19:31:21+00:00] FATAL: AWS::EC2::Errors::InvalidNetworkAclID::NotFound: aws_network_acl[EG5974_DMZ_21.25_ACL_01] (no AWS object id) (network_stack::network_stack line 114) had an error: AWS::EC2::Errors::InvalidNetworkAclID::NotFound: The networkAcl ID 'acl-1704a570' does not exist