Hi,
I need to upgrade couple of my servers which are deployed through Chef,
from Ubuntu 12.04 to 14.04. I have both 12.04 and 14.04 packages on my
repository.
Any idea to upgrade the servers?
Mohammad
I’d recreate the servers from your Chef cookbooks, rather than upgrade them. Much cleaner & safer that way. Then you can tear down the old ones when you’re sure the new ones work properly.
–
Jeff Byrnes
@thejeffbyrnes
Lead DevOps Engineer
EverTrue
704.516.4628
On April 27, 2015 at 3:07:02 PM, Mohammad Fattahian (mfattahian@masterfile.com) wrote:
Hi,
I need to upgrade couple of my servers which are deployed through Chef, from Ubuntu 12.04 to 14.04. I have both 12.04 and 14.04 packages on my repository.
Any idea to upgrade the servers?
Mohammad
As far as I know the existing cookbooks and recipes don’t have dependency
to 12.04. I’m thinking to upgrade the servers one by one to 14.04, but
looking for a automated solution, through Chef.
Mohammad
From: Jeff Byrnes [mailto:jeff@evertrue.com]
Sent: Monday, April 27, 2015 3:09 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Upgrade Ubuntu from 12.04 to 14.04
I’d recreate the servers from your Chef cookbooks, rather than upgrade
them. Much cleaner & safer that way. Then you can tear down the old ones
when you’re sure the new ones work properly.
–
Jeff Byrnes
@thejeffbyrnes http://twitter.com/thejeffbyrnes
Lead DevOps Engineer
EverTrue http://www.evertrue.com/
704.516.4628
On April 27, 2015 at 3:07:02 PM, Mohammad Fattahian (
mfattahian@masterfile.com) wrote:
Hi,
I need to upgrade couple of my servers which are deployed through Chef,
from Ubuntu 12.04 to 14.04. I have both 12.04 and 14.04 packages on my
repository.
Any idea to upgrade the servers?
Mohammad
That’s exactly why I recommend re-creating the servers. Upgrading the OS via Chef isn’t something I’m comfortable doing, but we have a number of EC2 instances running 12.04 that we’ve been slowly replacing with 14.04 when the opportunity presents itself.
With quite a few of these, we’ve had to make adjustments to the cookbooks in play, usually due to changes in apt packages, but also due to changes in where config files are located, or how the default configs are in 14.04 vs. 12.04 packages. Apache and PHP are huge offenders here (12.04 has Apache 2.2 & PHP 5.3, while 14.04 has Apache 2.4 & PHP 5.5), but there are lots of others.
–
Jeff Byrnes
@thejeffbyrnes
Lead DevOps Engineer
EverTrue
704.516.4628
On April 27, 2015 at 3:44:37 PM, Mohammad Fattahian (mfattahian@masterfile.com) wrote:
As far as I know the existing cookbooks and recipes don’t have dependency to 12.04. I’m thinking to upgrade the servers one by one to 14.04, but looking for a automated solution, through Chef.
Mohammad
From: Jeff Byrnes [mailto:jeff@evertrue.com]
Sent: Monday, April 27, 2015 3:09 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Upgrade Ubuntu from 12.04 to 14.04
I’d recreate the servers from your Chef cookbooks, rather than upgrade them. Much cleaner & safer that way. Then you can tear down the old ones when you’re sure the new ones work properly.
–
Jeff Byrnes
@thejeffbyrnes
Lead DevOps Engineer
EverTrue
704.516.4628
On April 27, 2015 at 3:07:02 PM, Mohammad Fattahian (mfattahian@masterfile.com) wrote:
Hi,
I need to upgrade couple of my servers which are deployed through Chef, from Ubuntu 12.04 to 14.04. I have both 12.04 and 14.04 packages on my repository.
Any idea to upgrade the servers?
Mohammad
Understood. BTW is there any way to upgrade OS via Chef? Our recipes are
not that much OS oriented. I hope I can do an upgrade via Chef. Also we
don’t have enough resources to recreate the servers with 14.04 from scratch.
Mohammad
From: Jeff Byrnes [mailto:jeff@evertrue.com]
Sent: Monday, April 27, 2015 4:02 PM
To: Mohammad Fattahian; chef@lists.opscode.com
Subject: Re: [chef] RE: Re: Upgrade Ubuntu from 12.04 to 14.04
That’s exactly why I recommend re-creating the servers. Upgrading the OS
via Chef isn’t something I’m comfortable doing, but we have a number of EC2
instances running 12.04 that we’ve been slowly replacing with 14.04 when
the opportunity presents itself.
With quite a few of these, we’ve had to make adjustments to the cookbooks
in play, usually due to changes in apt packages, but also due to changes in
where config files are located, or how the default configs are in 14.04 vs.
12.04 packages. Apache and PHP are huge offenders here (12.04 has Apache
2.2 & PHP 5.3, while 14.04 has Apache 2.4 & PHP 5.5), but there are lots of
others.
–
Jeff Byrnes
@thejeffbyrnes http://twitter.com/thejeffbyrnes
Lead DevOps Engineer
EverTrue http://www.evertrue.com/
704.516.4628
On April 27, 2015 at 3:44:37 PM, Mohammad Fattahian (
mfattahian@masterfile.com) wrote:
As far as I know the existing cookbooks and recipes don’t have dependency
to 12.04. I’m thinking to upgrade the servers one by one to 14.04, but
looking for a automated solution, through Chef.
Mohammad
From: Jeff Byrnes [mailto:jeff@evertrue.com]
Sent: Monday, April 27, 2015 3:09 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Upgrade Ubuntu from 12.04 to 14.04
I’d recreate the servers from your Chef cookbooks, rather than upgrade
them. Much cleaner & safer that way. Then you can tear down the old ones
when you’re sure the new ones work properly.
–
Jeff Byrnes
@thejeffbyrnes http://twitter.com/thejeffbyrnes
Lead DevOps Engineer
EverTrue http://www.evertrue.com/
704.516.4628
On April 27, 2015 at 3:07:02 PM, Mohammad Fattahian (
mfattahian@masterfile.com) wrote:
Hi,
I need to upgrade couple of my servers which are deployed through Chef,
from Ubuntu 12.04 to 14.04. I have both 12.04 and 14.04 packages on my
repository.
Any idea to upgrade the servers?
Mohammad
On Mon, Apr 27, 2015 at 1:12 PM, Mohammad Fattahian
mfattahian@masterfile.com wrote:
Understood. BTW is there any way to upgrade OS via Chef? Our recipes are not
that much OS oriented. I hope I can do an upgrade via Chef. Also we don’t
have enough resources to recreate the servers with 14.04 from scratch.
I'd recommend investing your resources into making your cookbooks able
to reprovision the servers from scratch rather than in trying to make
Chef handle your Ubuntu upgrade. In my experience upgrading Ubuntu (or
any distro) is asking for trouble later from mismatched dependencies
and other driftwood left behind. Also, if your cookbooks can bootstrap
a server from a base OS install, you'll be able to recover from server
failures as an extremely useful benefit.
We recently went through upgrading our cookbooks to be 14.04 friendly,
which was relatively minimal effort. Then we just provisioned new
servers one at a time and took down the old ones.
-J
Never ever ever do in-place OS upgrades. Ever.
This is sysadmin-101 qualification exam level stuff.
Best case scenario, you end up with a machine in an unknown state,
where you find oddly broken things months down the line.
Worst case scenario, you lose your data. I had one incident Long Ago
(sarge to etch?), where a kernel update caused the PCI bus to scan in
the opposite direction, corrupting the software raid.
Re-baseline your machines in a test environment.
-s
On Mon, Apr 27, 2015 at 4:21 PM, Jason J. W. Williams
jasonjwwilliams@gmail.com wrote:
On Mon, Apr 27, 2015 at 1:12 PM, Mohammad Fattahian
mfattahian@masterfile.com wrote:Understood. BTW is there any way to upgrade OS via Chef? Our recipes are not
that much OS oriented. I hope I can do an upgrade via Chef. Also we don’t
have enough resources to recreate the servers with 14.04 from scratch.I'd recommend investing your resources into making your cookbooks able
to reprovision the servers from scratch rather than in trying to make
Chef handle your Ubuntu upgrade. In my experience upgrading Ubuntu (or
any distro) is asking for trouble later from mismatched dependencies
and other driftwood left behind. Also, if your cookbooks can bootstrap
a server from a base OS install, you'll be able to recover from server
failures as an extremely useful benefit.We recently went through upgrading our cookbooks to be 14.04 friendly,
which was relatively minimal effort. Then we just provisioned new
servers one at a time and took down the old ones.-J
I'm not sure that's entirely true, if you can snapshot a VM and do an OS
upgrade, that's usually a lot less effort and we've been doing that for
some time now where I work. Full rebuilds usually only come when they
vastly change things.
--
~~ StormeRider ~~
"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."
(from Smallville Season 6x1: "Zod")
On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS
On Mon, Apr 27, 2015 at 1:27 PM, Sean OMeara someara@chef.io wrote:
Never ever ever do in-place OS upgrades. Ever.
This is sysadmin-101 qualification exam level stuff.
Best case scenario, you end up with a machine in an unknown state,
where you find oddly broken things months down the line.Worst case scenario, you lose your data. I had one incident Long Ago
(sarge to etch?), where a kernel update caused the PCI bus to scan in
the opposite direction, corrupting the software raid.Re-baseline your machines in a test environment.
-s
On Mon, Apr 27, 2015 at 4:21 PM, Jason J. W. Williams
jasonjwwilliams@gmail.com wrote:On Mon, Apr 27, 2015 at 1:12 PM, Mohammad Fattahian
mfattahian@masterfile.com wrote:Understood. BTW is there any way to upgrade OS via Chef? Our recipes
are not
that much OS oriented. I hope I can do an upgrade via Chef. Also we
don’t
have enough resources to recreate the servers with 14.04 from scratch.I'd recommend investing your resources into making your cookbooks able
to reprovision the servers from scratch rather than in trying to make
Chef handle your Ubuntu upgrade. In my experience upgrading Ubuntu (or
any distro) is asking for trouble later from mismatched dependencies
and other driftwood left behind. Also, if your cookbooks can bootstrap
a server from a base OS install, you'll be able to recover from server
failures as an extremely useful benefit.We recently went through upgrading our cookbooks to be 14.04 friendly,
which was relatively minimal effort. Then we just provisioned new
servers one at a time and took down the old ones.-J
VMs should be treated like livestock, not pets. Being able to replace them
quickly when the need arises is very important.
On Mon, Apr 27, 2015 at 4:09 PM, Morgan Blackthorne stormerider@gmail.com
wrote:
I'm not sure that's entirely true, if you can snapshot a VM and do an OS
upgrade, that's usually a lot less effort and we've been doing that for
some time now where I work. Full rebuilds usually only come when they
vastly change things.--
~~ StormeRider ~~"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."(from Smallville Season 6x1: "Zod")
On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS
On Mon, Apr 27, 2015 at 1:27 PM, Sean OMeara someara@chef.io wrote:
Never ever ever do in-place OS upgrades. Ever.
This is sysadmin-101 qualification exam level stuff.
Best case scenario, you end up with a machine in an unknown state,
where you find oddly broken things months down the line.Worst case scenario, you lose your data. I had one incident Long Ago
(sarge to etch?), where a kernel update caused the PCI bus to scan in
the opposite direction, corrupting the software raid.Re-baseline your machines in a test environment.
-s
On Mon, Apr 27, 2015 at 4:21 PM, Jason J. W. Williams
jasonjwwilliams@gmail.com wrote:On Mon, Apr 27, 2015 at 1:12 PM, Mohammad Fattahian
mfattahian@masterfile.com wrote:Understood. BTW is there any way to upgrade OS via Chef? Our recipes
are not
that much OS oriented. I hope I can do an upgrade via Chef. Also we
don’t
have enough resources to recreate the servers with 14.04 from scratch.I'd recommend investing your resources into making your cookbooks able
to reprovision the servers from scratch rather than in trying to make
Chef handle your Ubuntu upgrade. In my experience upgrading Ubuntu (or
any distro) is asking for trouble later from mismatched dependencies
and other driftwood left behind. Also, if your cookbooks can bootstrap
a server from a base OS install, you'll be able to recover from server
failures as an extremely useful benefit.We recently went through upgrading our cookbooks to be 14.04 friendly,
which was relatively minimal effort. Then we just provisioned new
servers one at a time and took down the old ones.-J
"Never ever ever do in-place OS upgrades. Ever. *This is sysadmin-101
qualification exam level stuff." .. *hmmm interesting but haven't came
across this in any form.
For the main question, "I need to upgrade couple of my servers which are
deployed through Chef, from Ubuntu 12.04 to 14.04. I have both 12.04 and
14.04 packages on my repository." - i would chef out a test nodes and try
upgrading them and see how it goes before i draw any conclusion on what
best to do the job. Ubuntu have a neck of upgrade failures in some way or
other
On Mon, Apr 27, 2015 at 1:27 PM, Sean OMeara someara@chef.io wrote:
Never ever ever do in-place OS upgrades. Ever.
This is sysadmin-101 qualification exam level stuff.
Best case scenario, you end up with a machine in an unknown state,
where you find oddly broken things months down the line.Worst case scenario, you lose your data. I had one incident Long Ago
(sarge to etch?), where a kernel update caused the PCI bus to scan in
the opposite direction, corrupting the software raid.Re-baseline your machines in a test environment.
-s
On Mon, Apr 27, 2015 at 4:21 PM, Jason J. W. Williams
jasonjwwilliams@gmail.com wrote:On Mon, Apr 27, 2015 at 1:12 PM, Mohammad Fattahian
mfattahian@masterfile.com wrote:Understood. BTW is there any way to upgrade OS via Chef? Our recipes
are not
that much OS oriented. I hope I can do an upgrade via Chef. Also we
don’t
have enough resources to recreate the servers with 14.04 from scratch.I'd recommend investing your resources into making your cookbooks able
to reprovision the servers from scratch rather than in trying to make
Chef handle your Ubuntu upgrade. In my experience upgrading Ubuntu (or
any distro) is asking for trouble later from mismatched dependencies
and other driftwood left behind. Also, if your cookbooks can bootstrap
a server from a base OS install, you'll be able to recover from server
failures as an extremely useful benefit.We recently went through upgrading our cookbooks to be 14.04 friendly,
which was relatively minimal effort. Then we just provisioned new
servers one at a time and took down the old ones.-J
--
Regards
nirish okram
Okay everyone, lets cool our collective jets with the unhelpful "Don't do that answers.
The best way to do this would probably be a few execute resources running the relevant CLI commands. Unfortunately the officially documented process from Canonical is to use the update-manager tool, which is an interactive curses app if I remember correctly. I would check to see if it has a non-interactive mode. Otherwise you can do it the really old fashioned way of replacing your sources.list and apt-get dist-upgrade.
--Noah
On Apr 27, 2015, at 1:12 PM, Mohammad Fattahian mfattahian@masterfile.com wrote:
Understood. BTW is there any way to upgrade OS via Chef? Our recipes are not that much OS oriented. I hope I can do an upgrade via Chef. Also we don’t have enough resources to recreate the servers with 14.04 from scratch.
Mohammad
From: Jeff Byrnes [mailto:jeff@evertrue.com]
Sent: Monday, April 27, 2015 4:02 PM
To: Mohammad Fattahian; chef@lists.opscode.com
Subject: Re: [chef] RE: Re: Upgrade Ubuntu from 12.04 to 14.04That’s exactly why I recommend re-creating the servers. Upgrading the OS via Chef isn’t something I’m comfortable doing, but we have a number of EC2 instances running 12.04 that we’ve been slowly replacing with 14.04 when the opportunity presents itself.
With quite a few of these, we’ve had to make adjustments to the cookbooks in play, usually due to changes in apt packages, but also due to changes in where config files are located, or how the default configs are in 14.04 vs. 12.04 packages. Apache and PHP are huge offenders here (12.04 has Apache 2.2 & PHP 5.3, while 14.04 has Apache 2.4 & PHP 5.5), but there are lots of others.
--
Jeff Byrnes
@thejeffbyrnes
Lead DevOps Engineer
EverTrue
704.516.4628On April 27, 2015 at 3:44:37 PM, Mohammad Fattahian (mfattahian@masterfile.com) wrote:
As far as I know the existing cookbooks and recipes don’t have dependency to 12.04. I’m thinking to upgrade the servers one by one to 14.04, but looking for a automated solution, through Chef.
Mohammad
From: Jeff Byrnes [mailto:jeff@evertrue.com]
Sent: Monday, April 27, 2015 3:09 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Upgrade Ubuntu from 12.04 to 14.04I’d recreate the servers from your Chef cookbooks, rather than upgrade them. Much cleaner & safer that way. Then you can tear down the old ones when you’re sure the new ones work properly.
--
Jeff Byrnes
@thejeffbyrnes
Lead DevOps Engineer
EverTrue
704.516.4628On April 27, 2015 at 3:07:02 PM, Mohammad Fattahian (mfattahian@masterfile.com) wrote:
Hi,
I need to upgrade couple of my servers which are deployed through Chef, from Ubuntu 12.04 to 14.04. I have both 12.04 and 14.04 packages on my repository.
Any idea to upgrade the servers?
Mohammad
On Mon, Apr 27, 2015 at 3:39 PM, Noah Kantrowitz noah@coderanger.net wrote:
Okay everyone, lets cool our collective jets with the unhelpful "Don't do that answers.
The best way to do this would probably be a few execute resources running the relevant CLI commands. Unfortunately the officially documented process from Canonical is to use the update-manager tool, which is an interactive curses app if I remember correctly. I would check to see if it has a non-interactive mode. Otherwise you can do it the really old fashioned way of replacing your sources.list and apt-get dist-upgrade.
--Noah
Honestly, if Mohammad is dead set on doing an upgrade, I think he'd be
better of cooking up a bash script in a test environment that works
and using some sort of parallel-SSH to apply it.
There's just too many ways for a distro upgrade to go sideways in very
non-idempotent ways for a Chef run to be the right approach to do a
mass upgrade of the OS.
-J
I agree, I don't think Chef is the tool for this. It's certainly not in
existing functionality and it would probably take as much time, if not
more, trying to force this functionality out of Chef instead of doing it
via a script or semi-manually. Once the instances are updated then bring
them back under configuration management.
No matter how you go about it you need to dedicate resources to creating
and re-making instances, whether it's test instances or new production. An
upgrade isn't just something you do without a lot of testing.
On Mon, Apr 27, 2015 at 3:43 PM, Jason J. W. Williams <
jasonjwwilliams@gmail.com> wrote:
On Mon, Apr 27, 2015 at 3:39 PM, Noah Kantrowitz noah@coderanger.net
wrote:Okay everyone, lets cool our collective jets with the unhelpful "Don't
do that answers.The best way to do this would probably be a few execute resources
running the relevant CLI commands. Unfortunately the officially documented
process from Canonical is to use the update-manager tool, which is an
interactive curses app if I remember correctly. I would check to see if it
has a non-interactive mode. Otherwise you can do it the really old
fashioned way of replacing your sources.list and apt-get dist-upgrade.--Noah
Honestly, if Mohammad is dead set on doing an upgrade, I think he'd be
better of cooking up a bash script in a test environment that works
and using some sort of parallel-SSH to apply it.There's just too many ways for a distro upgrade to go sideways in very
non-idempotent ways for a Chef run to be the right approach to do a
mass upgrade of the OS.-J
--
Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com
On Apr 27, 2015, at 3:43 PM, Jason J. W. Williams jasonjwwilliams@gmail.com wrote:
On Mon, Apr 27, 2015 at 3:39 PM, Noah Kantrowitz noah@coderanger.net wrote:
Okay everyone, lets cool our collective jets with the unhelpful "Don't do that answers.
The best way to do this would probably be a few execute resources running the relevant CLI commands. Unfortunately the officially documented process from Canonical is to use the update-manager tool, which is an interactive curses app if I remember correctly. I would check to see if it has a non-interactive mode. Otherwise you can do it the really old fashioned way of replacing your sources.list and apt-get dist-upgrade.
--Noah
Honestly, if Mohammad is dead set on doing an upgrade, I think he'd be
better of cooking up a bash script in a test environment that works
and using some sort of parallel-SSH to apply it.There's just too many ways for a distro upgrade to go sideways in very
non-idempotent ways for a Chef run to be the right approach to do a
mass upgrade of the OS.
Thats fine as an opinion, but multiple people responding to a question with "don't do that" breeds an unwelcoming atmosphere. I agree it is going to be a frustrating use case, but at a minimum let's answer the question to the best of our ability first and then urge caution after that. The "How? Don't!" thing is a cliche for a reason, and I know I personally hate being on the other side of it.
--Noah
Hugs, Noah. Thanks for speaking up with this great response!
--Charles
On April 27, 2015 at 3:50:32 PM, Noah Kantrowitz (noah@coderanger.net) wrote:
On Apr 27, 2015, at 3:43 PM, Jason J. W. Williams jasonjwwilliams@gmail.com wrote:
On Mon, Apr 27, 2015 at 3:39 PM, Noah Kantrowitz noah@coderanger.net wrote:
Okay everyone, lets cool our collective jets with the unhelpful "Don't do that answers.
The best way to do this would probably be a few execute resources running the relevant CLI commands. Unfortunately the officially documented process from Canonical is to use the update-manager tool, which is an interactive curses app if I remember correctly. I would check to see if it has a non-interactive mode. Otherwise you can do it the really old fashioned way of replacing your sources.list and apt-get dist-upgrade.
--Noah
Honestly, if Mohammad is dead set on doing an upgrade, I think he'd be
better of cooking up a bash script in a test environment that works
and using some sort of parallel-SSH to apply it.There's just too many ways for a distro upgrade to go sideways in very
non-idempotent ways for a Chef run to be the right approach to do a
mass upgrade of the OS.
Thats fine as an opinion, but multiple people responding to a question with "don't do that" breeds an unwelcoming atmosphere. I agree it is going to be a frustrating use case, but at a minimum let's answer the question to the best of our ability first and then urge caution after that. The "How? Don't!" thing is a cliche for a reason, and I know I personally hate being on the other side of it.
--Noah