Ohai Chefs!
I am excited to announce the 1.2.0 release of Chef Sugar! This release brings a number of new sugars and helpers that will make your cookbook authoring experience even more awesome! You can view the full CHANGELOG for detailed information, but I have also written a blog post about some of the newest sugars:
https://sethvargo.com/delicious-new-chef-sugars/
Happy cook(book)ing,
Seth
Good morning!
I’m testing some CentOS based systems that need Percona clustering. If I bootstrap the systems and have the ‘recipe[yum::epel],recipe[yumrepo::percona]’ recipes in the bootstrap run list, all is good. If I then configure the mysql settings to require the Percona versions of MySQL components, such as setting the ":mysql => { :server => { packages => “Percona-XtraDB-Cluster-Server” } }’ and other relevant settings, those packages are not visible at the time I do the initial bootstrap, so the bootstrap fails. They’re only visible to chef after some earlier chef run has already enabled the particular yum repositories.
I’m not an expert, and this confuses me. Is there any graceful way to get the yum repositories enabled, and their contents properly detected, for later processing by mysql recipes?
–
Nico Kadel-Garcia
Senior Systems Consultant
Email: nkadelgarcia-consultant@scholastic.com
Cell Phone: +1.339.368.2428
Hi Nico,
I don't know if I correctly caught up your point, but usually, when I want
to be sure a yum repo will be available during my chef run convergence I
force its configuration at compile time [2], using resources method [1]:
This is an example with epel repo:
include_recipe 'yum-epel'
y = resources("yum_repository[epel]")
y.action(:nothing)
y.run_action(:create)
[1] http://docs.opscode.com/dsl_recipe_method_resources.html
[2]
https://wiki.opscode.com/display/chef/Evaluate+and+Run+Resources+at+Compile+Time;jsessionid=BBE750D0DC249823649B3F4F70F24C82
Cheers,
Marco
On Thu, Mar 13, 2014 at 12:43 PM, Kadel-Garcia, Nico <
NKadelGarcia-consultant@scholastic.com> wrote:
Good morning!
I'm testing some CentOS based systems that need Percona clustering. If I
bootstrap the systems and have the
'recipe[yum::epel],recipe[yumrepo::percona]' recipes in the bootstrap run
list, all is good. If I then configure the mysql settings to require the
Percona versions of MySQL components,, such as setting the ":mysql => {
:server => { packages => "Percona-XtraDB-Cluster-Server" } }' and other
relevant settings, those packages are not visible at the time I do the
initial bootstrap, so the bootstrap fails. They're only visible to chef
after some earlier chef run has already enabled the particular yum
repositories.I'm not an expert, and this confuses me. Is there any graceful way to
get the yum repositories enabled, and their contents properly detected, for
later processing by mysql recipes?--
Nico Kadel-Garcia
Senior Systems Consultant
Email: nkadelgarcia-consultant@scholastic.com
Cell Phone: +1.339.368.2428
--
Ing. Marco Betti
RHCE RHEL4 id 804006512121056
I think you correctly caught my point. This seems very workable, if the 'yum-epel" can be reasonably included inside a recipe I’m writing. Getting it into an upstream recipe that hasn’t predetermined what yum repositories I’m personally selecting… may take more work.
But it looks like a solid approach, I’ll try it.
–
Nico Kadel-Garcia
Senior Systems Consultant
Email: nkadelgarcia-consultant@scholastic.com
Cell Phone: +1.339.368.2428
From: Marco Betti m.betti@gmail.com
Sent: Thursday, March 13, 2014 11:58 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Getting local yum repositories enabled before other cookbooks demand them in bootstrap
Hi Nico,
I don’t know if I correctly caught up your point, but usually, when I want to be sure a yum repo will be available during my chef run convergence I force its configuration at compile time [2], using resources method [1]:
This is an example with epel repo:
include_recipe 'yum-epel’
y = resources(“yum_repository[epel]”)
y.action(:nothing)
y.run_action(:create)
[1] http://docs.opscode.com/dsl_recipe_method_resources.html
[2] https://wiki.opscode.com/display/chef/Evaluate+and+Run+Resources+at+Compile+Time;jsessionid=BBE750D0DC249823649B3F4F70F24C82
Cheers,
Marco
On Thu, Mar 13, 2014 at 12:43 PM, Kadel-Garcia, Nico <NKadelGarcia-consultant@scholastic.commailto:NKadelGarcia-consultant@scholastic.com> wrote:
Good morning!
I’m testing some CentOS based systems that need Percona clustering. If I bootstrap the systems and have the ‘recipe[yum::epel],recipe[yumrepo::percona]’ recipes in the bootstrap run list, all is good. If I then configure the mysql settings to require the Percona versions of MySQL components, such as setting the ":mysql => { :server => { packages => “Percona-XtraDB-Cluster-Server” } }’ and other relevant settings, those packages are not visible at the time I do the initial bootstrap, so the bootstrap fails. They’re only visible to chef after some earlier chef run has already enabled the particular yum repositories.
I’m not an expert, and this confuses me. Is there any graceful way to get the yum repositories enabled, and their contents properly detected, for later processing by mysql recipes?
–
Nico Kadel-Garcia
Senior Systems Consultant
Email: nkadelgarcia-consultant@scholastic.commailto:nkadelgarcia-consultant@scholastic.com
Cell Phone: +1.339.368.2428tel:%2B1.339.368.2428
–
Ing. Marco Betti
RHCE RHEL4 id 804006512121056
On Thu, Mar 13, 2014 at 7:43 AM, Kadel-Garcia, Nico
NKadelGarcia-consultant@scholastic.com wrote:
I'm testing some CentOS based systems that need Percona clustering. If I
bootstrap the systems and have the
'recipe[yum::epel],recipe[yumrepo::percona]' recipes in the bootstrap run
list, all is good. If I then configure the mysql settings to require the
Percona versions of MySQL components,, such as setting the ":mysql => {
:server => { packages => "Percona-XtraDB-Cluster-Server" } }' and other
relevant settings, those packages are not visible at the time I do the
initial bootstrap, so the bootstrap fails. They're only visible to chef
after some earlier chef run has already enabled the particular yum
repositories.I'm not an expert, and this confuses me. Is there any graceful way to get
the yum repositories enabled, and their contents properly detected, for
later processing by mysql recipes?
Nico,
I'm a little confused by what you're describing. If you bootstrap a
fresh system with the run_list being something like
'recipe[yum-epel],recipe[yumrepo::percona],recipe[mysql::server]' and
the attribute node['mysql']['server']['packages'] =
['Percona-XtraDB-Cluster-Server'], you're saying it doesn't work?
Isn't the yum repo file created by the yumrepo::percona recipe you're
describing, so that it's available for the mysql::server run?
--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]
As per my understanding, if no recipe in the run list forces any package
installation during compiling phase it is enough to correctly set the
run-list. Sometimes it is not, because some recipe (for example ruby recipe
of mysql cookbook if I well remember) forces packages installation during
compiling phase, but I hadn't really understood if it is Nico's use case 
Cheers!
Marco
Il 15/mar/2014 21:02 "Julian C. Dunn" jdunn@aquezada.com ha scritto:
On Thu, Mar 13, 2014 at 7:43 AM, Kadel-Garcia, Nico
NKadelGarcia-consultant@scholastic.com wrote:I'm testing some CentOS based systems that need Percona clustering. If I
bootstrap the systems and have the
'recipe[yum::epel],recipe[yumrepo::percona]' recipes in the bootstrap run
list, all is good. If I then configure the mysql settings to require the
Percona versions of MySQL components,, such as setting the ":mysql => {
:server => { packages => "Percona-XtraDB-Cluster-Server" } }' and other
relevant settings, those packages are not visible at the time I do the
initial bootstrap, so the bootstrap fails. They're only visible to chef
after some earlier chef run has already enabled the particular yum
repositories.I'm not an expert, and this confuses me. Is there any graceful way to get
the yum repositories enabled, and their contents properly detected, for
later processing by mysql recipes?Nico,
I'm a little confused by what you're describing. If you bootstrap a
fresh system with the run_list being something like
'recipe[yum-epel],recipe[yumrepo::percona],recipe[mysql::server]' and
the attribute node['mysql']['server']['packages'] =
['Percona-XtraDB-Cluster-Server'], you're saying it doesn't work?
Isn't the yum repo file created by the yumrepo::percona recipe you're
describing, so that it's available for the mysql::server run?
- Julian
--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]
This is the classic "compile phase arms race" problem.
Try wrapping any recipes that need to do compilation phase work with the
now cookbook on the community site.
-s
On Sat, Mar 15, 2014 at 4:38 PM, Marco Betti m.betti@gmail.com wrote:
As per my understanding, if no recipe in the run list forces any package
installation during compiling phase it is enough to correctly set the
run-list. Sometimes it is not, because some recipe (for example ruby recipe
of mysql cookbook if I well remember) forces packages installation during
compiling phase, but I hadn't really understood if it is Nico's use caseCheers!
Marco
Il 15/mar/2014 21:02 "Julian C. Dunn" jdunn@aquezada.com ha scritto:On Thu, Mar 13, 2014 at 7:43 AM, Kadel-Garcia, Nico
NKadelGarcia-consultant@scholastic.com wrote:
I'm testing some CentOS based systems that need Percona clustering. If I
bootstrap the systems and have the
'recipe[yum::epel],recipe[yumrepo::percona]' recipes in the bootstrap
run
list, all is good. If I then configure the mysql settings to require the
Percona versions of MySQL components,, such as setting the ":mysql => {
:server => { packages => "Percona-XtraDB-Cluster-Server" } }' and other
relevant settings, those packages are not visible at the time I do the
initial bootstrap, so the bootstrap fails. They're only visible to chef
after some earlier chef run has already enabled the particular yum
repositories.I'm not an expert, and this confuses me. Is there any graceful way to
get
the yum repositories enabled, and their contents properly detected, for
later processing by mysql recipes?Nico,
I'm a little confused by what you're describing. If you bootstrap a
fresh system with the run_list being something like
'recipe[yum-epel],recipe[yumrepo::percona],recipe[mysql::server]' and
the attribute node['mysql']['server']['packages'] =
['Percona-XtraDB-Cluster-Server'], you're saying it doesn't work?
Isn't the yum repo file created by the yumrepo::percona recipe you're
describing, so that it's available for the mysql::server run?
- Julian
--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]
Julian, I’m afraid that the relevant yum repositories, namely for packages such as “socat” from EPEL and the “Percona-XtreDB-Cluster-server-55” from the the Percona repositories, are not detected as available until after the “yum::epel” or more recent “yum-epel” recipes have been successfully run, and after the “yumrepo::percona” recipe has been successfully run. But when bootstrapping a system, those have not been run successfully yet. the installation recipes for the “mysql” or “percona” or any other cookbook winds up using information from an earlier compilation stage before those yum repository recipes have actually been run and enabled the 3rd party repositores.
Because it uses an earlier compiled list of available packages, it reports Percona-XtraDB-Cluster-sever-55 as unavailable and reports errors before any recipe even gets run, so the bootstrap recipe list cannot even run. I can manually pre-run 'chef-client -o ‘recipe[yum::epel],recipe[yumrepo::percona]’. But it’s painful, and breaks push-button deployment via a pure bootstrap procedure.
It’s quite frustrating. I intend to look at the “now” cookbook recommended by Sean O’Mara as a possible solution, and look forward to Chef 12 perhaps being less presumptive that the contents of yum installations can be predicted before the “package” parts of recipes are compiled.
–
Nico Kadel-Garcia
Senior Systems Consultant
Email: nkadelgarcia-consultant@scholastic.com
Cell Phone: +1.339.368.2428
From: Julian C. Dunn jdunn@aquezada.com
Sent: Saturday, March 15, 2014 4:02 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Getting local yum repositories enabled before other cookbooks demand them in bootstrap
Nico,
I’m a little confused by what you’re describing. If you bootstrap a
fresh system with the run_list being something like
’recipe[yum-epel],recipe[yumrepo::percona],recipe[mysql::server]’ and
the attribute node[‘mysql’][‘server’][‘packages’] =
[‘Percona-XtraDB-Cluster-Server’], you’re saying it doesn’t work?
Isn’t the yum repo file created by the yumrepo::percona recipe you’re
describing, so that it’s available for the mysql::server run?
–
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]
On Sun, Mar 16, 2014 at 5:15 PM, Kadel-Garcia, Nico
NKadelGarcia-consultant@scholastic.com wrote:
Julian, I'm afraid that the relevant yum repositories, namely for packages such as "socat" from EPEL
and the "Percona-XtreDB-Cluster-server-55" from the the Percona repositories, are not detected as
available until after the "yum::epel" or more recent "yum-epel" recipes have been successfully run,
and after the "yumrepo::percona" recipe has been successfully run. But when bootstrapping a system,
those have not been run successfully yet. the installation recipes for the "mysql" or "percona" or any
other cookbook winds up using information from an earlier compilation stage before those yum
repository recipes have actually been run and enabled the 3rd party repositores.
That is what I do not understand. Chef does not "use an earlier
compiled list of available packages". There is no such thing. When you
declare a package resource, during its execution phase both the test
("yum -q ") and potentially the action ("yum -y install
") are run.
Everyone who is going on about the "now cookbook" and "having to do
things in the compilation phase" on this thread is wrong, or at least
unnecessarily using the nuclear option without picking up on the fact
that something else is happening under the hood.
To prove that bootstrapping a system, setting up a new Yum repo and
immediately using packages provided by that new repo actually do work,
I fired up Test Kitchen against the MongoDB cookbook
(GitHub - edelight/chef-mongodb: MongoDB Chef cookbook) and ran its
default-10gen-centos-65 suite. This does two things: sets up a new
repo in /etc/yum.repos.d/10gen.repo to MongoDB's repo and then invokes
a package resource to install mongo-10gen-server. This works, in one
fell swoop, out of the box, no extra force required.
In your situation, there is something else going on. Maybe you are
missing a "yum-makecache" somewhere which actually fetches the repo
metadata and makes it available to the 'package' resource?
--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]
Hi,
Apologies for duplicates. It seems I haven't subscribed with my
@getchef.com address yet.
Chef does not "use an earlier compiled list of available packages".
Unfortunately, I believe it does. Chef uses a python script [0] to
dump everything yum knows about packages into an internal cache. The
yum provider has a flush_cache option, however, which can be used to
make sure it reloads the internal cache [1]. Have you tried the
flush_cache options here yet?
Note that even with flush_cache set, you can still run into caching
problems if you have yum itself configured to aggressively cache, but
that usually isn't a problem with a newly added repo.
Cheers,
Steven
On Mon, Mar 17, 2014 at 12:58 PM, Julian C. Dunn jdunn@aquezada.com wrote:
On Sun, Mar 16, 2014 at 5:15 PM, Kadel-Garcia, Nico
NKadelGarcia-consultant@scholastic.com wrote:Julian, I'm afraid that the relevant yum repositories, namely for packages such as "socat" from EPEL
and the "Percona-XtreDB-Cluster-server-55" from the the Percona repositories, are not detected as
available until after the "yum::epel" or more recent "yum-epel" recipes have been successfully run,
and after the "yumrepo::percona" recipe has been successfully run. But when bootstrapping a system,
those have not been run successfully yet. the installation recipes for the "mysql" or "percona" or any
other cookbook winds up using information from an earlier compilation stage before those yum
repository recipes have actually been run and enabled the 3rd party repositores.That is what I do not understand. Chef does not "use an earlier
compiled list of available packages". There is no such thing. When you
declare a package resource, during its execution phase both the test
("yum -q ") and potentially the action ("yum -y install
") are run.Everyone who is going on about the "now cookbook" and "having to do
things in the compilation phase" on this thread is wrong, or at least
unnecessarily using the nuclear option without picking up on the fact
that something else is happening under the hood.To prove that bootstrapping a system, setting up a new Yum repo and
immediately using packages provided by that new repo actually do work,
I fired up Test Kitchen against the MongoDB cookbook
(GitHub - edelight/chef-mongodb: MongoDB Chef cookbook) and ran its
default-10gen-centos-65 suite. This does two things: sets up a new
repo in /etc/yum.repos.d/10gen.repo to MongoDB's repo and then invokes
a package resource to install mongo-10gen-server. This works, in one
fell swoop, out of the box, no extra force required.In your situation, there is something else going on. Maybe you are
missing a "yum-makecache" somewhere which actually fetches the repo
metadata and makes it available to the 'package' resource?
- Julian
--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]