Passing responses back to Chef?

Chef Infra (archive)
1

Folks,

Okay, so I have to be missing something really basic and easy, only I can’t figure out what it is. I was hoping someone here might be able to clue me in.

We’re using a modified version of the mysql cookbook, and trying to detect which version of the MySQL Engine that the user wants to install, and turn that into a full version string that could be passed to the package manager via the package resource.

I know how to access the desired engine version (major.minor.patch), and I figured out a way to determine what the full version string is of the package that might match – e.g., “5.1.70” might translate to “5.1.70-0ubuntu0.10.04.1”.

What I can’t figure out is how to pass that full version string information back to Chef. Setting a shell environment variable doesn’t help, because the environment of the parent process isn’t affected by the child process(es), as explained at http://docs.opscode.com/essentials_environment_variables.html:

From a shell, the source command can be used to reload a given
initialization file; however, since child processes do not affect
their parent’s environment, using a script or execute resource to
run source from inside a recipe will have no effect on the
environment for the chef-client

It also doesn’t help to put that information into a file that could be “sourced” from Chef, because you’d have to go through a bash resource as a child process, as explained at http://docs.opscode.com/resource_execute.html, because this would get done as a child process:

Instead, use the script resource or one of the script-based resources
(bash, csh, perl, python, or ruby). For example:

	bash "foo" do
		code "source /tmp/foo.sh"
	end

So, how else do you get a child process to pass information back to Chef in a useful manner?

–
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

2

This is where you'd want to use mixlib-shellout in your recipes. It captures all the stdout and stderr on a command, you can then process it in Ruby. Take a look at this example.


Thanks,
Matt Ray
Cloud Integrations Product Lead :: Opscode
512.731.2218 :: matt@opscode.com
mattray :: GitHub :: IRC :: Twitter

From: Brad Knowles brad@shub-internet.org
Sent: Tuesday, August 06, 2013 1:16 PM
To: chef@lists.opscode.com
Cc: Brad Knowles
Subject: [chef] Passing responses back to Chef?

Folks,

Okay, so I have to be missing something really basic and easy, only I can't figure out what it is. I was hoping someone here might be able to clue me in.

We're using a modified version of the mysql cookbook, and trying to detect which version of the MySQL Engine that the user wants to install, and turn that into a full version string that could be passed to the package manager via the package resource.

I know how to access the desired engine version (major.minor.patch), and I figured out a way to determine what the full version string is of the package that might match -- e.g., "5.1.70" might translate to "5.1.70-0ubuntu0.10.04.1".

What I can't figure out is how to pass that full version string information back to Chef. Setting a shell environment variable doesn't help, because the environment of the parent process isn't affected by the child process(es), as explained at http://docs.opscode.com/essentials_environment_variables.html:

    From a shell, the source command can be used to reload a given
    initialization file; however, since child processes do not affect
    their parent’s environment, using a script or execute resource to
    run source from inside a recipe will have no effect on the
    environment for the chef-client

It also doesn't help to put that information into a file that could be "sourced" from Chef, because you'd have to go through a bash resource as a child process, as explained at http://docs.opscode.com/resource_execute.html, because this would get done as a child process:

    Instead, use the script resource or one of the script-based resources
    (bash, csh, perl, python, or ruby). For example:

            bash "foo" do
                    code "source /tmp/foo.sh"
            end

So, how else do you get a child process to pass information back to Chef in a useful manner?

--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

3

On Aug 6, 2013, at 2:00 PM, Matt Ray matt@opscode.com wrote:

This is where you'd want to use mixlib-shellout in your recipes. It captures all the stdout and stderr on a command, you can then process it in Ruby. Take a look at this example.
pxe_dust-cookbook/recipes/bootstrap_template.rb at master · jjasghar/pxe_dust-cookbook · GitHub
GitHub - chef/mixlib-shellout: mixin library for subprocess management, output collection

Okay, so now I've got that mostly working, but I'm getting an error when I try to send a notification to install a package resource:

[2013-08-06T15:03:18-05:00] INFO: Processing ruby_block[find closest matching full version for mysql-server to desired version] action create (brad_mysql::server line 53)
[2013-08-06T15:03:18-05:00] DEBUG: aptitude show mysql-server=5.1 | grep '^Version: ' | sed 's/^Version: //' | grep "^5.1"
[2013-08-06T15:03:19-05:00] DEBUG: 5.1.70-0ubuntu0.10.04.1

[2013-08-06T15:03:19-05:00] INFO: ruby_block[find closest matching full version for mysql-server to desired version] called
[2013-08-06T15:03:19-05:00] INFO: ruby_block[find closest matching full version for mysql-server to desired version] sending install action to package[mysql-server] (immediate)
[2013-08-06T15:03:19-05:00] DEBUG: Re-raising exception: NoMethodError - undefined method `run_action' for "package[mysql-server]":String

Is this a feature that is only available in newer versions of chef-client and chef-solo? We're using 10.18.2 at the moment, and I know that the website has been updated to reflect the latest 11.x versions, so I'm not sure if I'm running into a known problem here.

Thanks!

--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

4

On Aug 6, 2013, at 3:07 PM, Brad Knowles brad@shub-internet.org wrote:

[2013-08-06T15:03:18-05:00] DEBUG: aptitude show mysql-server=5.1 | grep '^Version: ' | sed 's/^Version: //' | grep "^5.1"
[2013-08-06T15:03:19-05:00] DEBUG: 5.1.70-0ubuntu0.10.04.1

As it turns out, I appear to be trying to work around what seems to be a bug in the apt provider for chef-client, related to CHEF-2483.

The issue here is that when you select "5.1" as the version that you ask to have installed, the apt provider should find and install version "5.1.70-0ubuntu0.10.04.1" for you. However, if you ask for "5.1.70", that mechanism somehow breaks down and fails -- if you don't give it just "major.minor" you instead have to give it the complete version string, and the middle ground of "major.minor.patch" doesn't work.

I'll work this up as a new ticket, but in the meanwhile I'm trying to figure out how I can bypass this problem.

Of course, we're going to have a large number of different major.minor.patch versions that we might want to install, so we're going to have to run our own apt repo to hold them all. We're okay with that, so long as the apt provider works correctly.

--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

5

On Aug 6, 2013, at 4:10 PM, Brad Knowles brad@shub-internet.org wrote:

I'll work this up as a new ticket, but in the meanwhile I'm trying to figure out how I can bypass this problem.

Okay, I created CHEF-4442 to cover this issue. If anyone has any questions regarding the details of the matter and any improvements that I could make to the ticket to be more useful, please let me know.

Thanks!

--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

6

On Aug 6, 2013, at 11:32 PM, Brad Knowles brad@shub-internet.org wrote:

Okay, I created CHEF-4442 to cover this issue. If anyone has any questions regarding the details of the matter and any improvements that I could make to the ticket to be more useful, please let me know.

At this point, I'm pretty much stumped.

I discovered that I can't send a notify from a ruby_block to a package resource -- that just generates an error. However, I can have the package resource subscribe to the ruby_block, and get notifications that way. But isn't a "subscribe" supposed to just be the opposite of a "notify"? How can it work one way but not the other?

Now I seem to be running into problems about when things are compiled versus when they are run. When the package resource is compiled, the variable/attribute I want to use to set the version to be installed does not yet exist, and even if it did exist, it would not yet contain the information I want to use. So, it seems that a particular code path is chosen and hard-wired based on the information that is available at compile-time, and the other code paths are just thrown away.

By the time the version information is available to me through the ruby_block, and the package resource gets the notification, it doesn't matter because of that hard-wired code path.

I've tried the options shown at http://docs.opscode.com/resource_common.html#run-from-resource-collection to control when things are executed (like, forcing the ruby_block to be executed as early as possible, or forcing the package resource to have a ":delayed" execution), but because everything in the compile phase is done before any part of the execution phase, this still doesn't fix the problem.

Thanks to Matt Ray, I did learn how to use mixlib-shellout to have Chef call a program and get output back from it, which it could then use elsewhere in the recipe. Unfortunately, that doesn't help me if I can't get that information into a package resource to actually install that particular version of the software.

Or should I just skip the package resource and install that version of that software inside of my own ruby_block?

--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

7

On Wednesday, August 7, 2013 at 11:40 AM, Brad Knowles wrote:

On Aug 6, 2013, at 11:32 PM, Brad Knowles <brad@shub-internet.org (mailto:brad@shub-internet.org)> wrote:

Okay, I created CHEF-4442 to cover this issue. If anyone has any questions regarding the details of the matter and any improvements that I could make to the ticket to be more useful, please let me know.

At this point, I'm pretty much stumped.
Can you state more clearly what exactly you're trying to do? I read this whole thread but I can't come up with concrete suggestions because I still don't understand what needs to happen.

You mentioned something about "detect which version of the MySQL Engine the user wants to install." Where does this information come from? Is it available at compile time? Could it be?

Anyway, if you end up needing everything to be really late bound, wrapping your logic into a LWRP makes sense.

--
Daniel DeLeo

8

On Aug 7, 2013, at 1:55 PM, Daniel DeLeo dan@kallistec.com wrote:

Can you state more clearly what exactly you're trying to do? I read this whole thread but I can't come up with concrete suggestions because I still don't understand what needs to happen.

See https://gist.github.com/bknowles/b6d0b8c5d035b5bf95a7. The attribute req_params['engVers'] represents the major.minor or major.minor.patch version that the user has asked us to install. Anywhere you see "engAvail", that's an indication of the corresponding full and complete version string that I'm trying to pass back into the apt_package resource.

We have a modified version of the Opscode mysql cookbook that we're using to install mysql-server, and we need the ability to install any of a wide variety of versions of MySQL -- possibly hundreds of different versions of mysql-server. No one node would be running any more than a single version, but we could be serving any of thousands and thousands of customers in a multi-tenant environment, any of whom might have a requirement to install a very particular version on their node -- we need to be able to support them all.

For now, we worry about MySQL. Once we get this squared away, then we can worry about other database server technologies, and on other platforms.

Does that help?

--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

9

On Wednesday, August 7, 2013 at 12:18 PM, Brad Knowles wrote:

On Aug 7, 2013, at 1:55 PM, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

Can you state more clearly what exactly you're trying to do? I read this whole thread but I can't come up with concrete suggestions because I still don't understand what needs to happen.

See https://gist.github.com/bknowles/b6d0b8c5d035b5bf95a7. The attribute req_params['engVers'] represents the major.minor or major.minor.patch version that the user has asked us to install. Anywhere you see "engAvail", that's an indication of the corresponding full and complete version string that I'm trying to pass back into the apt_package resource.

We have a modified version of the Opscode mysql cookbook that we're using to install mysql-server, and we need the ability to install any of a wide variety of versions of MySQL -- possibly hundreds of different versions of mysql-server. No one node would be running any more than a single version, but we could be serving any of thousands and thousands of customers in a multi-tenant environment, any of whom might have a requirement to install a very particular version on their node -- we need to be able to support them all.

For now, we worry about MySQL. Once we get this squared away, then we can worry about other database server technologies, and on other platforms.

Does that help?
Not really. What I'm getting at is, when is the earliest time in the chef run that you could have enough information to figure out what package you've been asked to install? Or put a different way, what are the prerequisites for figuring this out, and are any of them dependent on steps in the chef run? Looking at your gist, you're using aptitude show to figure this out, but does that rely on a 3rd party repo getting enabled earlier in the chef run?

In an ideal case, you should be able to feed all the information required to configure your system into chef during the compile phase and let chef sort everything out in the converge phase. If that's true for you here, you can just pull the shell out call out of the ruby_block resource and you'll be good to go. Otherwise, the easiest path would be to create a wrapper LWRP, since LWP actions are evaluated during convergence.

--
Brad Knowles <brad@shub-internet.org (mailto:brad@shub-internet.org)>
LinkedIn Profile: http://tinyurl.com/y8kpxu

--
Daniel DeLeo

10

On Aug 7, 2013, at 7:14 PM, Daniel DeLeo dan@kallistec.com wrote:

Not really. What I'm getting at is, when is the earliest time in the chef run that you could have enough information to figure out what package you've been asked to install?

The information is loaded into a data bag by the front-end UI code before Chef is called, so in the recipe as soon as we load the data bag, we know what major.minor.patch that is desired -- 5.1.70, or whatever.

Problem is that the apt provider doesn't work if you try to install this specific version. You can either choose to install 5.1 and you'll get the latest version of 5.1 that is available in the repo, or you can request version 5.1.70-0ubuntu0.10.04.1, but nothing between "5.1" and "5.1.70-0ubuntu0.10.04.1" is accepted. That's why I filed CHEF-4442. Meanwhile, I'm trying to find ways to work around this problem.

Or put a different way, what are the prerequisites for figuring this out, and are any of them dependent on steps in the chef run? Looking at your gist, you're using aptitude show to figure this out, but does that rely on a 3rd party repo getting enabled earlier in the chef run?

I used "aptitude show" because that was a relatively easy way to get the list of versions available returned to me, and then I could do a simple grep anchored to the beginning of the line to find the version(s) that most closely matched the major.minor.patch version that I wanted.

I could have done "apt-cache showpkg" (or whatever), but that would have been a little more difficult for me to parse, even though it would have been a little faster than using aptitude.

In an ideal case, you should be able to feed all the information required to configure your system into chef during the compile phase and let chef sort everything out in the converge phase. If that's true for you here, you can just pull the shell out call out of the ruby_block resource and you'll be good to go.

The apt provider is broken with regard to understanding major.minor.patch versions, which is how we got to where we are now.

Maybe the solution is as simple as updating the version of the apt cookbook that we have? I know what we have now doesn't include the new provider for version pinning, which would be another potential solution to the problem.

Otherwise, the easiest path would be to create a wrapper LWRP, since LWP actions are evaluated during convergence.

If the problem is that the information is not yet available at the time of convergence when the code is being compiled that will install the package when it is called, I don't see how creating a wrapper LWRP around the apt provider is going to help. Can you enlighten me?

--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

11

On Wednesday, August 7, 2013 at 6:33 PM, Brad Knowles wrote:

On Aug 7, 2013, at 7:14 PM, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

Not really. What I'm getting at is, when is the earliest time in the chef run that you could have enough information to figure out what package you've been asked to install?

The information is loaded into a data bag by the front-end UI code before Chef is called, so in the recipe as soon as we load the data bag, we know what major.minor.patch that is desired -- 5.1.70, or whatever.

Problem is that the apt provider doesn't work if you try to install this specific version. You can either choose to install 5.1 and you'll get the latest version of 5.1 that is available in the repo, or you can request version 5.1.70-0ubuntu0.10.04.1, but nothing between "5.1" and "5.1.70-0ubuntu0.10.04.1" is accepted. That's why I filed CHEF-4442. Meanwhile, I'm trying to find ways to work around this problem.

The code you have now is pretty close, then. Just put the mixlib-shellout call at the top level of the recipe with no ruby block. Now it will run during the compile phase, before the package resource is evaluated.

Otherwise, the easiest path would be to create a wrapper LWRP, since LWP actions are evaluated during convergence.

If the problem is that the information is not yet available at the time of convergence when the code is being compiled that will install the package when it is called, I don't see how creating a wrapper LWRP around the apt provider is going to help. Can you enlighten me?
The exact way it works depends on whether you're using use_inline_resources or not, but the provider action of a LWRP is somewhat like a self contained chef run with its own compile phase.

--
Brad Knowles <brad@shub-internet.org (mailto:brad@shub-internet.org)>
LinkedIn Profile: http://tinyurl.com/y8kpxu

--
Daniel DeLeo