Problem when downloading Cookbooks from Chef-10 server


#1

Hi,

as part of migration from Chef-10 to Chef-11 i am trying to download
Chef-10 configuration and the proccess keeps failing

running knife10 download /
the log and the errors are attached.

any help would be appreciated.

Thanks
Michael


#2

On Wednesday, October 9, 2013 at 4:43 AM, Michael Vaknin wrote:

Hi,

as part of migration from Chef-10 to Chef-11 i am trying to download Chef-10 configuration and the proccess keeps failing

running knife10 download /
the log and the errors are attached.

any help would be appreciated.

Thanks
Michael

Attachments:

  • download_chef10.log

How is your hostname lookup configured? This error:

ERROR: Network Error: Error connecting to https://chef10.anyclipsrv.info:444/cookbooks/AnyClip/0.2.25/files/df73fda5125ced9f3542126fd3ceaa6d - getaddrinfo: Name or service not known
Check your knife configuration and network settings

typically indicates a failure to lookup the hostname. Does this error reproduce persistently?


Daniel DeLeo


#3

Hello Daniel,
The problem persist and i was able to download all the cookbooks after i
run the commannd sevral times.
it happens every time on a different cookbook.
does this mean that i have a network problem? i can add in the hosts file
the chef server name and ip to see if this will solve the problem.
Thanks
Michael

On Wed, Oct 9, 2013 at 7:16 PM, Daniel DeLeo dan@kallistec.com wrote:

On Wednesday, October 9, 2013 at 4:43 AM, Michael Vaknin wrote:

Hi,

as part of migration from Chef-10 to Chef-11 i am trying to download
Chef-10 configuration and the proccess keeps failing

running knife10 download /
the log and the errors are attached.

any help would be appreciated.

Thanks
Michael

Attachments:

  • download_chef10.log

How is your hostname lookup configured? This error:

ERROR: Network Error: Error connecting to
https://chef10.anyclipsrv.info:444/cookbooks/AnyClip/0.2.25/files/df73fda5125ced9f3542126fd3ceaa6d- getaddrinfo: Name or service not known
Check your knife configuration and network settings

typically indicates a failure to lookup the hostname. Does this error
reproduce persistently?


Daniel DeLeo


#4

Hi,

It looks like i had a network problem and it was fixed and i was able to
download all the data from the old server.

now i have a problem when i try to upload the data.
all data is uploaded but not the cookbooks.
attached is the log file with the errrors

I am able to run commands such knife client list and knife role list but it
fails in the cookbook upload so i guess that my knife.rb configuration is
correct.

a minor issue on the download is that environment was downloaded to
/var/chef/environments and not to transfer-repo folder where all other
stuff was downloaded.

Thanks
Michael

On Thu, Oct 10, 2013 at 8:24 AM, Michael Vaknin mvaknine@gmail.com wrote:

Hello Daniel,
The problem persist and i was able to download all the cookbooks after i
run the commannd sevral times.
it happens every time on a different cookbook.
does this mean that i have a network problem? i can add in the hosts file
the chef server name and ip to see if this will solve the problem.
Thanks
Michael

On Wed, Oct 9, 2013 at 7:16 PM, Daniel DeLeo dan@kallistec.com wrote:

On Wednesday, October 9, 2013 at 4:43 AM, Michael Vaknin wrote:

Hi,

as part of migration from Chef-10 to Chef-11 i am trying to download
Chef-10 configuration and the proccess keeps failing

running knife10 download /
the log and the errors are attached.

any help would be appreciated.

Thanks
Michael

Attachments:

  • download_chef10.log

How is your hostname lookup configured? This error:

ERROR: Network Error: Error connecting to
https://chef10.anyclipsrv.info:444/cookbooks/AnyClip/0.2.25/files/df73fda5125ced9f3542126fd3ceaa6d- getaddrinfo: Name or service not known
Check your knife configuration and network settings

typically indicates a failure to lookup the hostname. Does this error
reproduce persistently?


Daniel DeLeo


#5

On Thursday, October 10, 2013 at 3:55 AM, Michael Vaknin wrote:

Hi,

It looks like i had a network problem and it was fixed and i was able to download all the data from the old server.

now i have a problem when i try to upload the data.
all data is uploaded but not the cookbooks.
attached is the log file with the errrors

Looks like more networking issues?

The relevant error is:

DEBUG: Sending HTTP Request via GET to chef11.anyclipsrv.info:443/environments/qa1
ERROR: Network Error: Connection refused - connect(2)
Check your knife configuration and network settings

This means that either nothing is listening on port 443 of that host, or you have a firewall rule to refuse the connection and send a RST packet.

But earlier in your log, you have:

DEBUG: Sending HTTP Request via GET to chef11.anyclipsrv.info:443/data/anyclip/sqs

…which indicates a successful request was made to that host.

Does this error reproduce on every cookbook upload?

I am able to run commands such knife client list and knife role list but it fails in the cookbook upload so i guess that my knife.rb configuration is correct.
This seems to rule out the possibility that the server stopped running. How is hostname resolution for chef11.anyclipsrv.info configured? DNS? /etc/hosts ?

a minor issue on the download is that environment was downloaded to /var/chef/environments and not to transfer-repo folder where all other stuff was downloaded.

Thanks
Michael


Daniel DeLeo


#6

Hello Daniel,
i am not able to load any cookbook i get the same error.
if you look at the logs you can see something strange it is trying to load
to https://localhost:443 and i do not understand why.

Thanks
Michael

On Thu, Oct 10, 2013 at 5:26 PM, Daniel DeLeo dan@kallistec.com wrote:

On Thursday, October 10, 2013 at 3:55 AM, Michael Vaknin wrote:

Hi,

It looks like i had a network problem and it was fixed and i was able to
download all the data from the old server.

now i have a problem when i try to upload the data.
all data is uploaded but not the cookbooks.
attached is the log file with the errrors

Looks like more networking issues?

The relevant error is:

DEBUG: Sending HTTP Request via GET to
chef11.anyclipsrv.info:443/environments/qa1
ERROR: Network Error: Connection refused - connect(2)
Check your knife configuration and network settings

This means that either nothing is listening on port 443 of that host, or
you have a firewall rule to refuse the connection and send a RST packet.

But earlier in your log, you have:

DEBUG: Sending HTTP Request via GET to
chef11.anyclipsrv.info:443/data/anyclip/sqs

…which indicates a successful request was made to that host.

Does this error reproduce on every cookbook upload?

I am able to run commands such knife client list and knife role list but
it fails in the cookbook upload so i guess that my knife.rb configuration
is correct.

This seems to rule out the possibility that the server stopped running.
How is hostname resolution for chef11.anyclipsrv.info configured? DNS?
/etc/hosts ?

a minor issue on the download is that environment was downloaded to
/var/chef/environments and not to transfer-repo folder where all other
stuff was downloaded.

Thanks
Michael


Daniel DeLeo


#7

On Sunday, October 13, 2013 at 3:54 AM, Michael Vaknin wrote:

Hello Daniel,
i am not able to load any cookbook i get the same error.
if you look at the logs you can see something strange it is trying to load to https://localhost:443 and i do not understand why.

Thanks
Michael

On Thu, Oct 10, 2013 at 5:26 PM, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

On Thursday, October 10, 2013 at 3:55 AM, Michael Vaknin wrote:

Hi,

It looks like i had a network problem and it was fixed and i was able to download all the data from the old server.

now i have a problem when i try to upload the data.
all data is uploaded but not the cookbooks.
attached is the log file with the errrors

Looks like more networking issues?

The relevant error is:

DEBUG: Sending HTTP Request via GET to chef11.anyclipsrv.info:443/environments/qa1 (http://chef11.anyclipsrv.info:443/environments/qa1)
ERROR: Network Error: Connection refused - connect(2)
Check your knife configuration and network settings

This means that either nothing is listening on port 443 of that host, or you have a firewall rule to refuse the connection and send a RST packet.

But earlier in your log, you have:

DEBUG: Sending HTTP Request via GET to chef11.anyclipsrv.info:443/data/anyclip/sqs (http://chef11.anyclipsrv.info:443/data/anyclip/sqs)

…which indicates a successful request was made to that host.

Does this error reproduce on every cookbook upload?

I am able to run commands such knife client list and knife role list but it fails in the cookbook upload so i guess that my knife.rb configuration is correct.
This seems to rule out the possibility that the server stopped running. How is hostname resolution for chef11.anyclipsrv.info (http://chef11.anyclipsrv.info) configured? DNS? /etc/hosts ?

a minor issue on the download is that environment was downloaded to /var/chef/environments and not to transfer-repo folder where all other stuff was downloaded.

Thanks
Michael


Daniel DeLeo


#8

Sorry for the reply fail in the previous message :frowning:

On Monday, October 14, 2013 at 1:21 PM, Daniel DeLeo wrote:

On Sunday, October 13, 2013 at 3:54 AM, Michael Vaknin wrote:

Hello Daniel,
i am not able to load any cookbook i get the same error.
if you look at the logs you can see something strange it is trying to load to https://localhost:443 and i do not understand why.

Responses from the server contain URLs to other resources, exactly like a normal web page does. Currently, the hostname of the server is determined when you run chef-server-ctl reconfigure, and this hostname is used when creating URLs in responses. Most API operations ignore these URLs, but for cookbook operations, the workstation uses them to upload/download individual files from cookbooks. You can customize the URL that is handed out with the lb['api_fqdn'] = "chef.example.com" setting in /etc/chef-server/chef-server.rb More info on chef server configurables is here: http://docs.opscode.com/config_rb_chef_server.html

Thanks
Michael


Daniel DeLeo


#9

Ok,
so i have a config file chef-server.rb

bookshelf[‘url’] = "chef11.anyclipsrv.info"
bookshelf[‘listen’] = "chef11.anyclipsrv.info"
bookshelf[‘vip’] = "chef11.anyclipsrv.info"
bookshelf[‘port’] = 443
lb[‘api_fqdn’] = “chef11.anyclipsrv.info

i run chef-server-ctl reconfigure

now when i try to upload i get this errors

INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving AnyClip
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 4s
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving AnyClip-wordpress
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving ant
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 3s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 4s
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving anyclip_newrelic
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving apache2
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 3s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 2/5 in 8s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 2/5 in 8s

Thanks
Michael

On Mon, Oct 14, 2013 at 11:30 PM, Daniel DeLeo dan@kallistec.com wrote:

Sorry for the reply fail in the previous message :frowning:

On Monday, October 14, 2013 at 1:21 PM, Daniel DeLeo wrote:

On Sunday, October 13, 2013 at 3:54 AM, Michael Vaknin wrote:

Hello Daniel,
i am not able to load any cookbook i get the same error.
if you look at the logs you can see something strange it is trying to load
to https://localhost:443 and i do not understand why.

Responses from the server contain URLs to other resources, exactly like a
normal web page does. Currently, the hostname of the server is determined
when you run chef-server-ctl reconfigure, and this hostname is used when
creating URLs in responses. Most API operations ignore these URLs, but for
cookbook operations, the workstation uses them to upload/download
individual files from cookbooks. You can customize the URL that is handed
out with the lb['api_fqdn'] = "chef.example.com" setting in
/etc/chef-server/chef-server.rb More info on chef server configurables is
here: http://docs.opscode.com/config_rb_chef_server.html

Thanks
Michael


Daniel DeLeo


#10

On Monday, October 14, 2013 at 10:43 PM, Michael Vaknin wrote:

Ok,
so i have a config file chef-server.rb

bookshelf[‘url’] = "chef11.anyclipsrv.info (http://chef11.anyclipsrv.info)"
bookshelf[‘listen’] = "chef11.anyclipsrv.info (http://chef11.anyclipsrv.info)"
bookshelf[‘vip’] = "chef11.anyclipsrv.info (http://chef11.anyclipsrv.info)"
bookshelf[‘port’] = 443
lb[‘api_fqdn’] = “chef11.anyclipsrv.info (http://chef11.anyclipsrv.info)”

i run chef-server-ctl reconfigure

now when i try to upload i get this errors

INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving AnyClip
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes, retrying 1/5 in 4s
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving AnyClip-wordpress
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving ant
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes, retrying 1/5 in 3s
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes, retrying 1/5 in 4s
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving anyclip_newrelic
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving apache2
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes, retrying 1/5 in 3s
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes, retrying 2/5 in 8s
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes, retrying 2/5 in 8s

Thanks
Michael

These all look like retries that eventually succeed. Is there eventually a hard failure?

Also, for Chef 11, you can use chef-server-ctl tail to see the logs for all components. Any error messages in there?


Daniel DeLeo


#11

Hello Daniel,

I had some netowrk issues which are now fixed.
All data except the cookbooks are uploaded to the chef11 server correctly.
the cookbooks still fails with this errors

INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for
https://chef11.anyclipsrv.info/sandboxes/000000000000844a143dd4870d9d9444,
retrying 1/5 in 3s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for
https://chef11.anyclipsrv.info/sandboxes/000000000000844a143dd4870d9d9444,
retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for
https://chef11.anyclipsrv.info/sandboxes/000000000000844a143dd4870d9d9444,
retrying 3/5 in 9s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for
https://chef11.anyclipsrv.info/sandboxes/000000000000b9febbc2c5918119deed,
retrying 1/5 in 4s
ERROR: Failed to upload : 500 Internal Server Error
internal service error
ERROR: RestClient::InternalServerError: 500 Internal Server Error

I run chef-server-ctl tail on the server at the same time i tried the
upload and here are the logs from the server.

On Thu, Oct 17, 2013 at 12:57 AM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, October 14, 2013 at 10:43 PM, Michael Vaknin wrote:

Ok,
so i have a config file chef-server.rb

bookshelf[‘url’] = "chef11.anyclipsrv.info"
bookshelf[‘listen’] = "chef11.anyclipsrv.info"
bookshelf[‘vip’] = "chef11.anyclipsrv.info"
bookshelf[‘port’] = 443
lb[‘api_fqdn’] = “chef11.anyclipsrv.info

i run chef-server-ctl reconfigure

now when i try to upload i get this errors

INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving AnyClip
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 4s
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving AnyClip-wordpress
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving ant
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 3s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 4s
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving anyclip_newrelic
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving apache2
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 3s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 2/5 in 8s
INFO: HTTP Request Returned 500 Internal Server Error: internal service
error
ERROR: Server returned error for https://chef11.anyclipsrv.info/sandboxes,
retrying 2/5 in 8s

Thanks
Michael

These all look like retries that eventually succeed. Is there eventually a
hard failure?

Also, for Chef 11, you can use chef-server-ctl tail to see the logs for
all components. Any error messages in there?


Daniel DeLeo


#12

Hi there,

mvaknine@gmail.com writes:

I had some netowrk issues which are now fixed.
All data except the cookbooks are uploaded to the chef11 server correctly.
the cookbooks still fails with this errors

Are you using knife upload to upload your cookbooks? The error trace
from bookshelf looks like a recently found and fixed bookshelf bug that
causes a deadlock when the same checksums (cookbook content) are
uploaded at the same time – something that can easily happen due to the
way that knife upload parallelizes the work.

I suspect you will be able to work around this issue by doing: knife cookbook upload -a rather than making use of knife upload.

We’ve made a number of improvements to bookshelf which will be available
in the next Chef Server release.

  • seth


Seth Falcon | Development Lead | Opscode | @sfalcon


#13

(You can also pass --concurrency 1 to knife upload to achieve the same
effect–the issue occurs when multiple cookbooks get uploaded
simultaneously, and concurrency 1 turns that off.)

On Mon, Oct 21, 2013 at 9:26 AM, Seth Falcon seth@opscode.com wrote:

Hi there,

mvaknine@gmail.com writes:

I had some netowrk issues which are now fixed.
All data except the cookbooks are uploaded to the chef11 server
correctly.
the cookbooks still fails with this errors

Are you using knife upload to upload your cookbooks? The error trace
from bookshelf looks like a recently found and fixed bookshelf bug that
causes a deadlock when the same checksums (cookbook content) are
uploaded at the same time – something that can easily happen due to the
way that knife upload parallelizes the work.

I suspect you will be able to work around this issue by doing: knife cookbook upload -a rather than making use of knife upload.

We’ve made a number of improvements to bookshelf which will be available
in the next Chef Server release.

  • seth


Seth Falcon | Development Lead | Opscode | @sfalcon


#14

Thank you john
concurrency 1 in knife.rb solved the problem
Michael

On Mon, Oct 21, 2013 at 8:58 PM, John Keiser jkeiser@opscode.com wrote:

(You can also pass --concurrency 1 to knife upload to achieve the same
effect–the issue occurs when multiple cookbooks get uploaded
simultaneously, and concurrency 1 turns that off.)

On Mon, Oct 21, 2013 at 9:26 AM, Seth Falcon seth@opscode.com wrote:

Hi there,

mvaknine@gmail.com writes:

I had some netowrk issues which are now fixed.
All data except the cookbooks are uploaded to the chef11 server
correctly.
the cookbooks still fails with this errors

Are you using knife upload to upload your cookbooks? The error trace
from bookshelf looks like a recently found and fixed bookshelf bug that
causes a deadlock when the same checksums (cookbook content) are
uploaded at the same time – something that can easily happen due to the
way that knife upload parallelizes the work.

I suspect you will be able to work around this issue by doing: knife cookbook upload -a rather than making use of knife upload.

We’ve made a number of improvements to bookshelf which will be available
in the next Chef Server release.

  • seth


Seth Falcon | Development Lead | Opscode | @sfalcon