The download failed with an EOF error shortly after I sent the email.
We did a huge ‘spring’ cleaning of all the roles/cookbooks before this last run, we had a lot of cruft that needed cleansing.
What’s odd is that the "chef-server-ctl chef12-upgrade-download” command is just running a “knife download /“ behind the scenes, and that process used 100% of one CPU for the full 7 hours. It appeared to slow down over time, and I noticed that it opens/closes a new https session for every file it downloads.
In comparison, for backups we’re using the knife backup plugin (I think from https://github.com/mdxp/knife-backup) and it can export the entire server in about 1.5 hours. We run a full backup every night and a —latest every few hours. We’re still trying to figure out what the difference is and why one works and the other fails. And if there’s a difference in the data that they provide.
Thanks
Mike
—
Michael Hart
Arctic Wolf Networks
226.388.4773
From: Mark Mzyk
Reply-To: "chef@lists.opscode.commailto:chef@lists.opscode.com"
Date: Friday, March 6, 2015 at 08:58
To: "chef@lists.opscode.commailto:chef@lists.opscode.com"
Subject: [chef] Re: Re: Re: Chef Server 12 upgrade failure
Having it on the same node is fine; that’s the default case the upgrade process was designed for. Hopefully it all goes well this time. If it does not, let us know and we’ll help you get it sorted. The upgrade is taking as long as it is because the process is download all the data/transform it to Chef 12 format/upload all the data to a new server. The more data you have the longer it’ll take (having many cookbook versions is a particular culprit here). It was done this way so that all your data moved through the API so we could ensure it would be in a good format on the Chef 12 server. We recognized that this might take a while, which is why the chef12-upgrade-* commands exist to give you finer control over the process.
[cid:part1.04040905.09050800@chef.io]
Michael Hartmailto:michael.hart@arcticwolf.com
March 5, 2015 at 3:35 PM
Hi Mark,
Thanks for the response. We have chef server 11 and 12 installed on the same node, which is arguably not the best approach but so be it.
So versions:
Chef Server 11.1.1-1
Chef Server 12.0.5-1
Chef Client 11.16.4-1
We trimmed down the amount of data in chef significantly and am running another export again. I dare not say that it’s going well for fear of jinxing it, except that it’s been running for > 6 hours.
Thanks
mike
—
Michael Hart
Arctic Wolf Networks
226.388.4773
From: Mark Mzyk
Reply-To: "chef@lists.opscode.commailto:chef@lists.opscode.com"
Date: Thursday, March 5, 2015 at 11:25
To: "chef@lists.opscode.commailto:chef@lists.opscode.com"
Subject: [chef] Re: Chef Server 12 upgrade failure
Hi Michael,
You’re correct that it’s using knife download. To give you more insight into the process, all the upgrade code is here: https://github.com/chef/opscode-omnibus/tree/master/files/private-chef-ctl-commands
Look at the chef12_upgrade* files and the open_source_chef12_upgrade.rb file.
Specifically, the knife download is run here: https://github.com/chef/opscode-omnibus/blob/master/files/private-chef-ctl-commands/open_source_chef12_upgrade.rb
I don’t know why it would fail during the upgrade but work for you locally. On the upgrade it is using the bundled in chef-client that is used to configure the chef server, so perhaps it is a difference in client version. You can, if you explore the upgrade code, replicate manually what the upgrade code is doing using your local chef-client and then you could proceed on with the other steps.
Can you tell me what version of the chef-client you’re using that works? I think I’ve seen this error crop up for others but I’m not sure what causes it. Also what versions are your Chef 11 server and Chef 12 server package?
Thanks,
Mark
[cid:part1.04040905.09050800@chef.io]
Mark Mzykmailto:mmzyk@chef.io
March 5, 2015 at 11:25 AM
Hi Michael,
You’re correct that it’s using knife download. To give you more insight into the process, all the upgrade code is here: https://github.com/chef/opscode-omnibus/tree/master/files/private-chef-ctl-commands
Look at the chef12_upgrade* files and the open_source_chef12_upgrade.rb file.
Specifically, the knife download is run here: https://github.com/chef/opscode-omnibus/blob/master/files/private-chef-ctl-commands/open_source_chef12_upgrade.rb
I don’t know why it would fail during the upgrade but work for you locally. On the upgrade it is using the bundled in chef-client that is used to configure the chef server, so perhaps it is a difference in client version. You can, if you explore the upgrade code, replicate manually what the upgrade code is doing using your local chef-client and then you could proceed on with the other steps.
Can you tell me what version of the chef-client you’re using that works? I think I’ve seen this error crop up for others but I’m not sure what causes it. Also what versions are your Chef 11 server and Chef 12 server package?
Thanks,
Mark
[cid:part1.04040905.09050800@chef.io]
Michael Hartmailto:michael.hart@arcticwolf.com
March 4, 2015 at 12:46 PM
We are trying to upgrade to Chef Server 12 (from 11.1.1), and ran into this error when running the "chef-server-ctl chef12-upgrade-download” command (which I think is just doing “knife download /“):
ERROR: Chef::ChefFS::FileSystem::NotFoundError: Chef::ChefFS::FileSystem::NotFoundError
knife download failed with pid 9244 exit 100
It was in the middle of downloading a few versions of a cookbook, which I can download successfully using knife cookbook download.
From what we can tell there’s no errors in the chef server log at all.
Any suggestions/ideas?
Thanks
mike
—
Michael Hart
Arctic Wolf Networks
226.388.4773