RE: Chef Search for Node Discovery

Chef Infra (archive)
1

It seems to me that this problem is fundamental, not specific to chef: you simply can’t discover anything until it is discoverable. Making something discoverable in an automated fashion can happen only by running something. In chef, it’s chef itself. The other option is to have a separate pre-registration process that pre-creates (and pre-populates) the nodes.

That is also quite possible in chef; you’d use knife (possibly from a shell script) to automatically create your nodes.

But in the end, that doesn’t really solve your problem, it just moves it.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang doug.garstang@gmail.com
Sent: Friday 12th June 2015 22:01
To: chef@lists.opscode.com
Subject: [chef] Chef Search for Node Discovery.

I haven’t tried this for a while, so I thought I’d see if there was any progress.

I need to write a cookbook that discovers other chef nodes. The issue with chef search is that nodes don’t report in until they’ve done their first chef client run, which means that the client has to run twice on the one doing the discovering, but… that also applies to the one that wants to be discovered. End result, not automated. This has basically made chef search unusable in every situation for me where it might otherwise be useful.

Is there a better way? I’m on EC2. My other option was to write an ohai plugin to scan ec2 and return instances matching the correct criteria.

Thanks,
Doug

2

Kevin,

My boot process runs a script that calls a knife node add, to add the node
to chef. Since it's been added to chef, it should be discoverable. However,
it's not discoverable until the client has run at least once. That seems
more like an architectural problem with chef, rather than a fundamental
flaw in the universe. :slight_smile:

What prevents me from doing a two pass run is that I would have to take a
process that works fine, and add a hack to make it work when a chef search
is required. Every node would have to run twice, or I would have to create
some criteria, maybe have CloudFormation put a 'needs-chef-hack' EC2 tag on
the instance, and then my scripts check for that. This really isn't very
elegant.

What if I modified my boot scripts to run chef twice. The first run would
essentially be to call a dummy cookbook that does nothing. After this has
run, the node would be discoverable? Plenty of other folks must have this
same issue.

Doug.

On Sat, Jun 13, 2015 at 1:43 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

It seems to me that this problem is fundamental, not specific to chef:
you simply can't discover anything until it is discoverable. Making
something discoverable in an automated fashion can happen only by running
something. In chef, it's chef itself. The other option is to have a
separate pre-registration process that pre-creates (and pre-populates) the
nodes.

That is also quite possible in chef; you'd use knife (possibly from a
shell script) to automatically create your nodes.

But in the end, that doesn't really solve your problem, it just moves it.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang doug.garstang@gmail.com
Sent: Friday 12th June 2015 22:01
To: chef@lists.opscode.com
Subject: [chef] Chef Search for Node Discovery.

I haven't tried this for a while, so I thought I'd see if there was any
progress.

I need to write a cookbook that discovers other chef nodes. The issue with
chef search is that nodes don't report in until they've done their first
chef client run, which means that the client has to run twice on the one
doing the discovering, but... that also applies to the one that wants to be
discovered. End result, not automated. This has basically made chef search
unusable in every situation for me where it might otherwise be useful.

Is there a better way? I'm on EC2. My other option was to write an ohai
plugin to scan ec2 and return instances matching the correct criteria.

Thanks,
Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

3

On Saturday, June 13, 2015 at 2:03 PM, Douglas Garstang wrote:

Kevin,

My boot process runs a script that calls a knife node add, to add the node to chef. Since it's been added to chef, it should be discoverable. However, it's not discoverable until the client has run at least once. That seems more like an architectural problem with chef, rather than a fundamental flaw in the universe. :slight_smile:

As described elsewhere in the thread, the amount of time it takes for an object to hit the search index depends on your configuration and whether or not you’re on Chef Server 12 (which upgrades to Solr 4, which has a “soft commit” feature, reducing indexing time to 1s from default of 60s).

What you’re probably seeing is that chef-client sets some data about its last run into the “automatic” attributes section (where ohai data lives). If your search query queries on these values, you won’t see it until chef-client runs. If you search for things like run list items, node names, etc., you’ll see them when the node is created (after the next Solr commit).

As for whether it’s an architectural issue with chef, sort of. It’s definitely a problem that the node object in Chef represents both desired/aspirational state and last observed state. On the other hand, it’s important that you be able to search for nodes based on one or that other of these. For example, you probably don’t want to add an application server into the load balancer rotation or begin monitoring it as a production system before the application is running. In other cases you definitely do want to find all systems, even if they’re not fully configured yet.

--
Daniel DeLeo

4

Daniel,

I created three new nodes with the "knife node create" command, like this,
for each:

{
"name": "test01.dev.slicetest.com",
"chef_environment": "dev",
"json_class": "Chef::Node",
"automatic": {
},
"normal": {
},
"chef_type": "node",
"default": {
},
"override": {
},
"run_list": [
"role[zookeeper]"
]
}

I've been doing a knife search repeatedly over the past 15 minutes, and all
I get each time for the nodes is this:

Node Name: test01.dev.slicetest.com
Environment: dev
FQDN:
IP:
Run List: role[zookeeper]
Roles:
Recipes:
Platform:
Tags:

How long should I need to wait? The chef_environment is immediately
populated. Why not the role? That comes from the run list and should be
available immediately, just like the chef_environment. I need at least
the environment and the role to be able to do a useful search.

It does not take an instance 15 minutes to boot. Therefore, by the time the
client runs, the data won't be available. Can I reduce the solr refresh
frequency down to a useful value?

Douglas.

On Sun, Jun 14, 2015 at 11:48 AM, Daniel DeLeo dan@kallistec.com wrote:

On Saturday, June 13, 2015 at 2:03 PM, Douglas Garstang wrote:

Kevin,

My boot process runs a script that calls a knife node add, to add the
node to chef. Since it's been added to chef, it should be discoverable.
However, it's not discoverable until the client has run at least once. That
seems more like an architectural problem with chef, rather than a
fundamental flaw in the universe. :slight_smile:

As described elsewhere in the thread, the amount of time it takes for an
object to hit the search index depends on your configuration and whether or
not you’re on Chef Server 12 (which upgrades to Solr 4, which has a “soft
commit” feature, reducing indexing time to 1s from default of 60s).

What you’re probably seeing is that chef-client sets some data about its
last run into the “automatic” attributes section (where ohai data lives).
If your search query queries on these values, you won’t see it until
chef-client runs. If you search for things like run list items, node names,
etc., you’ll see them when the node is created (after the next Solr commit).

As for whether it’s an architectural issue with chef, sort of. It’s
definitely a problem that the node object in Chef represents both
desired/aspirational state and last observed state. On the other hand, it’s
important that you be able to search for nodes based on one or that other
of these. For example, you probably don’t want to add an application server
into the load balancer rotation or begin monitoring it as a production
system before the application is running. In other cases you definitely do
want to find all systems, even if they’re not fully configured yet.

--
Daniel DeLeo

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

5

When you create the node object you will actually need to put all the useful information required that you would like to be able to search. Alternative approach might be to call node.save at the start of your first recipe. ‎I believe that would save all your role/environment attributes right away to the server (searchable after the Solr indexes).

                                                                                                                                                                              ‎<b>From: </b>Douglas Garstang‎<b>Sent: </b>Sunday, June 14, 2015 2:58 PM<b>To: </b>chef@lists.opscode.com<b>Reply To: </b>chef@lists.opscode.com<b>Subject: </b>[chef] Re: Re: Re: RE: Chef Search for Node Discovery.

Daniel,

I created three new nodes with the "knife node create" command, like this, for each:

{ "name": "test01.dev.slicetest.com", "chef_environment": "dev", "json_class": "Chef::Node", "automatic": { }, "normal": { }, "chef_type": "node", "default": { }, "override": { }, "run_list": [ "role[zookeeper]" ]}

I've been doing a knife search repeatedly over the past 15 minutes, and all I get each time for the nodes is this:

Node Name: test01.dev.slicetest.comEnvironment: devFQDN:IP:Run List: role[zookeeper]Roles:Recipes:Platform:Tags:

How long should I need to wait? The chef_environment is immediately populated. Why not the role? That comes from the run list and should be available immediately, just like the chef_environment. I need at least the environment and the role to be able to do a useful search.

It does not take an instance 15 minutes to boot. Therefore, by the time the client runs, the data won't be available. Can I reduce the solr refresh frequency down to a useful value?

Douglas.

On Sun, Jun 14, 2015 at 11:48 AM, Daniel DeLeo <dan@kallistec.com> wrote:

On Saturday, June 13, 2015 at 2:03 PM, Douglas Garstang wrote:

> Kevin,

>

> My boot process runs a script that calls a knife node add, to add the node to chef. Since it's been added to chef, it should be discoverable. However, it's not discoverable until the client has run at least once. That seems more like an architectural problem with chef, rather than a fundamental flaw in the universe. :slight_smile:

As described elsewhere in the thread, the amount of time it takes for an object to hit the search index depends on your configuration and whether or not you’re on Chef Server 12 (which upgrades to Solr 4, which has a “soft commit” feature, reducing indexing time to 1s from default of 60s).

What you’re probably seeing is that chef-client sets some data about its last run into the “automatic” attributes section (where ohai data lives). If your search query queries on these values, you won’t see it until chef-client runs. If you search for things like run list items, node names, etc., you’ll see them when the node is created (after the next Solr commit).

As for whether it’s an architectural issue with chef, sort of. It’s definitely a problem that the node object in Chef represents both desired/aspirational state and last observed state. On the other hand, it’s important that you be able to search for nodes based on one or that other of these. For example, you probably don’t want to add an application server into the load balancer rotation or begin monitoring it as a production system before the application is running. In other cases you definitely do want to find all systems, even if they’re not fully configured yet.

--

Daniel DeLeo

--

Regards,

Douglas Garstang

http://www.linkedin.com/in/garstang

Email: doug.garstang@gmail.com

Cell: +1-805-340-5627

6

What exact search command are you using? I also assume that the other new nodes are called test02 and test03?

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang doug.garstang@gmail.com
Sent: Sunday 14th June 2015 11:58
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Chef Search for Node Discovery.

Daniel,

I created three new nodes with the "knife node create" command, like this, for each:

{
"name": "test01.dev.slicetest.com http://test01.dev.slicetest.com ",
"chef_environment": "dev",
"json_class": "Chef::Node",
"automatic": {
},
"normal": {
},
"chef_type": "node",
"default": {
},
"override": {
},
"run_list": [
"role[zookeeper]"
]
}

I've been doing a knife search repeatedly over the past 15 minutes, and all I get each time for the nodes is this:

Node Name: test01.dev.slicetest.com http://test01.dev.slicetest.com
Environment: dev
FQDN:
IP:
Run List: role[zookeeper]
Roles:
Recipes:
Platform:
Tags:

How long should I need to wait? The chef_environment is immediately populated. Why not the role? That comes from the run list and should be available immediately, just like the chef_environment. I need at least the environment and the role to be able to do a useful search.

It does not take an instance 15 minutes to boot. Therefore, by the time the client runs, the data won't be available. Can I reduce the solr refresh frequency down to a useful value?

Douglas.

On Sun, Jun 14, 2015 at 11:48 AM, Daniel DeLeo <dan@kallistec.com mailto:dan@kallistec.com > wrote:
On Saturday, June 13, 2015 at 2:03 PM, Douglas Garstang wrote:

Kevin,

My boot process runs a script that calls a knife node add, to add the node to chef. Since it's been added to chef, it should be discoverable. However, it's not discoverable until the client has run at least once. That seems more like an architectural problem with chef, rather than a fundamental flaw in the universe. :slight_smile:

As described elsewhere in the thread, the amount of time it takes for an object to hit the search index depends on your configuration and whether or not you’re on Chef Server 12 (which upgrades to Solr 4, which has a “soft commit” feature, reducing indexing time to 1s from default of 60s).

What you’re probably seeing is that chef-client sets some data about its last run into the “automatic” attributes section (where ohai data lives). If your search query queries on these values, you won’t see it until chef-client runs. If you search for things like run list items, node names, etc., you’ll see them when the node is created (after the next Solr commit).

As for whether it’s an architectural issue with chef, sort of. It’s definitely a problem that the node object in Chef represents both desired/aspirational state and last observed state. On the other hand, it’s important that you be able to search for nodes based on one or that other of these. For example, you probably don’t want to add an application server into the load balancer rotation or begin monitoring it as a production system before the application is running. In other cases you definitely do want to find all systems, even if they’re not fully configured yet.

--
Daniel DeLeo

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com mailto:doug.garstang@gmail.com
Cell: +1-805-340-5627

7

Kevin,

I was just doing a "knife search node" and throwing the results into a
file, since I didn't know if my search criteria would yield a result
anyway. Someone just suggested I put node.save and the start of my run
list. I just tried that. It seems to help. I'll move forward with that
approach for now as it's simple. But, now my script that adds the instance
on boot has to also remove it on shutdown, or else I'll have dead nodes
hanging around. The work never ends.... :-\

Doug.

On Sun, Jun 14, 2015 at 12:13 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

What exact search command are you using? I also assume that the other
new nodes are called test02 and test03?

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang doug.garstang@gmail.com
Sent: Sunday 14th June 2015 11:58
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Chef Search for Node Discovery.

Daniel,

I created three new nodes with the "knife node create" command, like this,
for each:

{
"name": "test01.dev.slicetest.com",
"chef_environment": "dev",
"json_class": "Chef::Node",
"automatic": {
},
"normal": {
},
"chef_type": "node",
"default": {
},
"override": {
},
"run_list": [
"role[zookeeper]"
]
}

I've been doing a knife search repeatedly over the past 15 minutes, and
all I get each time for the nodes is this:

Node Name: test01.dev.slicetest.com
Environment: dev
FQDN:
IP:
Run List: role[zookeeper]
Roles:
Recipes:
Platform:
Tags:

How long should I need to wait? The chef_environment is immediately
populated. Why not the role? That comes from the run list and should be
available immediately, just like the chef_environment. I need at least
the environment and the role to be able to do a useful search.

It does not take an instance 15 minutes to boot. Therefore, by the time
the client runs, the data won't be available. Can I reduce the solr refresh
frequency down to a useful value?

Douglas.

On Sun, Jun 14, 2015 at 11:48 AM, Daniel DeLeo dan@kallistec.com wrote:

On Saturday, June 13, 2015 at 2:03 PM, Douglas Garstang wrote:

Kevin,

My boot process runs a script that calls a knife node add, to add the
node to chef. Since it's been added to chef, it should be discoverable.
However, it's not discoverable until the client has run at least once. That
seems more like an architectural problem with chef, rather than a
fundamental flaw in the universe. :slight_smile:

As described elsewhere in the thread, the amount of time it takes for an
object to hit the search index depends on your configuration and whether or
not you’re on Chef Server 12 (which upgrades to Solr 4, which has a “soft
commit” feature, reducing indexing time to 1s from default of 60s).

What you’re probably seeing is that chef-client sets some data about its
last run into the “automatic” attributes section (where ohai data lives).
If your search query queries on these values, you won’t see it until
chef-client runs. If you search for things like run list items, node names,
etc., you’ll see them when the node is created (after the next Solr commit).

As for whether it’s an architectural issue with chef, sort of. It’s
definitely a problem that the node object in Chef represents both
desired/aspirational state and last observed state. On the other hand, it’s
important that you be able to search for nodes based on one or that other
of these. For example, you probably don’t want to add an application server
into the load balancer rotation or begin monitoring it as a production
system before the application is running. In other cases you definitely do
want to find all systems, even if they’re not fully configured yet.

--
Daniel DeLeo

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627