RE: Re: RE: Re: RE: Chef Search for Node Discovery

Chef Infra (archive)
1

Sorry it wasn't helpful, I did misunderstand you. It was actually news to me that search wouldn't work until after the first chef run. I just tried it, and actually search does seem to work as soon as the node is created (at least in Chef 12.0.8 with Chef 12.3.0 client). One thing: populate the FQDN. That appears to make a bit of a difference.

Here is what I did:

knife node create test

knife node search 'node:test'

Works as expected.

knife node search 'node:*'

Did not return the new node in one of my test runs, but did in another.

Now add the fqdn attribute:

knife node edit test

Add the fqdn attribute

Then

knife node search 'node:*'

Now the new node will show up. Whether this was due to adding the FQDN attribute, or coincidence, I don't know. It may simply take a bit of time for chef to add the new node to search indices.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang doug.garstang@gmail.com
Sent: Saturday 13th June 2015 22:19
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: RE: Chef Search for Node Discovery.

Kevin,

Seems we have a slight disconnect. It's been my experience that you can't query anything about a node with chef search, not even the node itself, until after the client has run for the first time. I actually don't need to pull any attributes from the node. This is for zookeeper, so I just need to get a list of host names that match a criteria (role, environment etc).

Alas, I can't use a data bag, or a role. My boot scripts use AWS Dynamodb (and a concatenated role and environment as a key) to pick always incrementing host name numbers. I'd have to start the CF stack, wait until the scripts had queried DynamoDB, had set the Name tag on the instance, eyeball the name in the Amazon console, and then go update the role, upload it to chef etc, before the instance came fully up.

Doug

On Sat, Jun 13, 2015 at 5:46 PM, Kevin Keane Subscription <subscription@kkeane.com mailto:subscription@kkeane.com > wrote:

When you created the node, did you also populate it with whatever attributes your search will later use?

I've run into your problem, too, several times, actually - I'm sure it's common.

For instance, I'm using bacula as a backup solution. The bacula director (server) needs to know about all of the clients, and all of the clients need to know about the director.

So when the clients search for the director, there is a chance they don't find one (yet), and similarly, the director may not (yet) find any clients. Both would lead to errors in the bacula configuration. The system won't be stable until it has run twice.

I don't look as the solution as a "hack" but rather as necessary error handling.

There is one other option: instead of a search, you can use a data bag (or even an attribute) to explicitly store the information you would otherwise get from a search. I use this approach in situations where I may have multiple similar servers, and want to be explicitly in which one gets used - for instance, to control which database is stored on which MySQL server.

Kevin Keane

The NetTech

http://www.4nettech.com http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang <doug.garstang@gmail.com mailto:doug.garstang@gmail.com >
Sent: Saturday 13th June 2015 14:03
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: RE: Chef Search for Node Discovery.

Kevin,

My boot process runs a script that calls a knife node add, to add the node to chef. Since it's been added to chef, it should be discoverable. However, it's not discoverable until the client has run at least once. That seems more like an architectural problem with chef, rather than a fundamental flaw in the universe. :slight_smile:

What prevents me from doing a two pass run is that I would have to take a process that works fine, and add a hack to make it work when a chef search is required. Every node would have to run twice, or I would have to create some criteria, maybe have CloudFormation put a 'needs-chef-hack' EC2 tag on the instance, and then my scripts check for that. This really isn't very elegant.

What if I modified my boot scripts to run chef twice. The first run would essentially be to call a dummy cookbook that does nothing. After this has run, the node would be discoverable? Plenty of other folks must have this same issue.

Doug.

On Sat, Jun 13, 2015 at 1:43 PM, Kevin Keane Subscription <subscription@kkeane.com mailto:subscription@kkeane.com > wrote:

It seems to me that this problem is fundamental, not specific to chef: you simply can't discover anything until it is discoverable. Making something discoverable in an automated fashion can happen only by running something. In chef, it's chef itself. The other option is to have a separate pre-registration process that pre-creates (and pre-populates) the nodes.

That is also quite possible in chef; you'd use knife (possibly from a shell script) to automatically create your nodes.

But in the end, that doesn't really solve your problem, it just moves it.

Kevin Keane

The NetTech

http://www.4nettech.com http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang <doug.garstang@gmail.com mailto:doug.garstang@gmail.com >
Sent: Friday 12th June 2015 22:01
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Chef Search for Node Discovery.

I haven't tried this for a while, so I thought I'd see if there was any progress.

I need to write a cookbook that discovers other chef nodes. The issue with chef search is that nodes don't report in until they've done their first chef client run, which means that the client has to run twice on the one doing the discovering, but... that also applies to the one that wants to be discovered. End result, not automated. This has basically made chef search unusable in every situation for me where it might otherwise be useful.

Is there a better way? I'm on EC2. My other option was to write an ohai plugin to scan ec2 and return instances matching the correct criteria.

Thanks,
Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com mailto:doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com mailto:doug.garstang@gmail.com
Cell: +1-805-340-5627

2
Kevin,

The update frequency for Solr search index is configurable on your chef-server (see below).‎ I think creating the node object first is the correct approach Doug should be attempting. May be worth investigating chef provisioning and the machine resource as it actually creates a node object before start of chef-client run and let's you populate whatever values you desire. At end of chef-client run on vm the client object is created and node object is saved with remaining attributes.

Update Frequency


At the end of every chef-client run, the node object is saved to the Chef server. From the Chef server, each node object is then added to the SOLR search index. This process is asynchronous. By default, node objects are committed to the search index every 60 seconds or per 1000 node objects, whichever occurs first.‎




When data is committed to the Apache Solr index, all incoming updates are blocked. If the duration between updates is too short, it is possible for the rate at which updates are asked to occur to be faster than the rate at which objects can be actually committed.




For Chef server, these settings are configurable:







SettingDescription
opscode_solr['commit_interval']The frequency (in seconds) at which node objects are added to the Apache Solr search index. Default value: ‎60000 (every 60 seconds).opscode_solr['max_commit_docs']The frequency (in documents) at which node objects are added to the Apache Solr search index. Default value: 1000 (every 1000 documents).‎



From: Kevin Keane Subscription‎Sent: Sunday, June 14, 2015 2:12 AMTo: chef@lists.opscode.com‎Reply To: chef@lists.opscode.comSubject: [chef] RE: Re: RE: Re: RE: Chef Search for Node Discovery.‎‎



RE: [chef] Re: RE: Re: RE: Chef Search for Node Discovery.

Sorry it wasn't helpful, I did misunderstand you. It was actually news to me that search wouldn't work until after the first chef run. I just tried it, and actually search *does* seem to work as soon as the node is created (at least in Chef 12.0.8 with Chef 12.3.0 client). One thing: populate the FQDN. That appears to make a bit of a difference.




Here is what I did:




> knife node create test




> knife node search 'node:test'




Works as expected.




> knife node search 'node:*'




Did not return the new node in one of my test runs, but did in another.




Now add the fqdn attribute:




> knife node edit test




Add the fqdn attribute




Then




> knife node search 'node:*'




Now the new node will show up. Whether this was due to adding the FQDN attribute, or coincidence, I don't know. It may simply take a bit of time for chef to add the new node to search indices.




Kevin Keane




The NetTech




http://www.4nettech.com




Our values: Privacy, Liberty, Justice




See https://www.4nettech.com/corp/the-nettech-values.html








-----Original message-----

From: Douglas Garstang <doug.garstang@gmail.com>

Sent: Saturday 13th June 2015 22:19

To: chef@lists.opscode.com

Subject: [chef] Re: RE: Re: RE: Chef Search for Node Discovery.



Kevin,

Seems we have a slight disconnect. It's been my experience that you can't query anything about a node with chef search, not even the node itself, until after the client has run for the first time. I actually don't need to pull any attributes from the node. This is for zookeeper, so I just need to get a list of host names that match a criteria (role, environment etc).

Alas, I can't use a data bag, or a role. My boot scripts use AWS Dynamodb (and a concatenated role and environment as a key) to pick always incrementing host name numbers. I'd have to start the CF stack, wait until the scripts had queried DynamoDB, had set the Name tag on the instance, eyeball the name in the Amazon console, and then go update the role, upload it to chef etc, before the instance came fully up.

Doug



On Sat, Jun 13, 2015 at 5:46 PM, Kevin Keane Subscription <subscription@kkeane.com> wrote:


When you created the node, did you also populate it with whatever attributes your search will later use?




I've run into your problem, too, several times, actually - I'm sure it's common.




For instance, I'm using bacula as a backup solution. The bacula director (server) needs to know about all of the clients, and all of the clients need to know about the director.




So when the clients search for the director, there is a chance they don't find one (yet), and similarly, the director may not (yet) find any clients. Both would lead to errors in the bacula configuration. The system won't be stable until it has run twice.




I don't look as the solution as a "hack" but rather as necessary error handling.




There is one other option: instead of a search, you can use a data bag (or even an attribute) to explicitly store the information you would otherwise get from a search. I use this approach in situations where I may have multiple similar servers, and want to be explicitly in which one gets used - for instance, to control which database is stored on which MySQL server.




Kevin Keane




The NetTech




http://www.4nettech.com




Our values: Privacy, Liberty, Justice




See https://www.4nettech.com/corp/the-nettech-values.html








-----Original message-----

From: Douglas Garstang <doug.garstang@gmail.com>

Sent: Saturday 13th June 2015 14:03

To:chef@lists.opscode.com

Subject: [chef] Re: RE: Chef Search for Node Discovery.



Kevin,



My boot process runs a script that calls a knife node add, to add the node to chef. Since it's been added to chef, it should be discoverable. However, it's not discoverable until the client has run at least once. That seems more like an architectural problem with chef, rather than a fundamental flaw in the universe. :)



What prevents me from doing a two pass run is that I would have to take a process that works fine, and add a hack to make it work when a chef search is required. Every node would have to run twice, or I would have to create some criteria, maybe have CloudFormation put a 'needs-chef-hack' EC2 tag on the instance, and then my scripts check for that. This really isn't very elegant.



What if I modified my boot scripts to run chef twice. The first run would essentially be to call a dummy cookbook that does nothing. After this has run, the node would be discoverable? Plenty of other folks must have this same issue.



Doug.



On Sat, Jun 13, 2015 at 1:43 PM, Kevin Keane Subscription <subscription@kkeane.com> wrote:


It seems to me that this problem is fundamental, not specific to chef: you simply can't discover anything until it is discoverable. Making something discoverable in an automated fashion can happen only by running *something*. In chef, it's chef itself. The other option is to have a separate pre-registration process that pre-creates (and pre-populates) the nodes.




That is also quite possible in chef; you'd use knife (possibly from a shell script) to automatically create your nodes.




But in the end, that doesn't really solve your problem, it just moves it.




Kevin Keane




The NetTech




http://www.4nettech.com




Our values: Privacy, Liberty, Justice




See https://www.4nettech.com/corp/the-nettech-values.html








-----Original message-----

From: Douglas Garstang <doug.garstang@gmail.com>

Sent: Friday 12th June 2015 22:01

To:chef@lists.opscode.com

Subject: [chef] Chef Search for Node Discovery.



I haven't tried this for a while, so I thought I'd see if there was any progress.



I need to write a cookbook that discovers other chef nodes. The issue with chef search is that nodes don't report in until they've done their first chef client run, which means that the client has to run twice on the one doing the discovering, but... that also applies to the one that wants to be discovered. End result, not automated. This has basically made chef search unusable in every situation for me where it might otherwise be useful.



Is there a better way? I'm on EC2. My other option was to write an ohai plugin to scan ec2 and return instances matching the correct criteria.



Thanks,

Doug







--

Regards,



Douglas Garstang

http://www.linkedin.com/in/garstang

Email: doug.garstang@gmail.com

Cell: +1-805-340-5627








--

Regards,



Douglas Garstang

http://www.linkedin.com/in/garstang

Email: doug.garstang@gmail.com

Cell: +1-805-340-5627


3

Kevin,

Thanks for trying that out. I will take another look. I'm on chef 11 though, not 12. I forget the specifics but there was a bunch of issues with 12 that prevented me from moving to it. I sure hope I'm wrong about it not working on 11.

Doug.

Sent from my iPhone

On Jun 13, 2015, at 11:11 PM, Kevin Keane Subscription subscription@kkeane.com wrote:

Sorry it wasn't helpful, I did misunderstand you. It was actually news to me that search wouldn't work until after the first chef run. I just tried it, and actually search does seem to work as soon as the node is created (at least in Chef 12.0.8 with Chef 12.3.0 client). One thing: populate the FQDN. That appears to make a bit of a difference.

Here is what I did:

knife node create test

knife node search 'node:test'

Works as expected.

knife node search 'node:*'

Did not return the new node in one of my test runs, but did in another.

Now add the fqdn attribute:

knife node edit test

Add the fqdn attribute

Then

knife node search 'node:*'

Now the new node will show up. Whether this was due to adding the FQDN attribute, or coincidence, I don't know. It may simply take a bit of time for chef to add the new node to search indices.

Kevin Keane

The NetTech

http://www.4nettech.com
Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang doug.garstang@gmail.com
Sent: Saturday 13th June 2015 22:19
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: RE: Chef Search for Node Discovery.

Kevin,

Seems we have a slight disconnect. It's been my experience that you can't query anything about a node with chef search, not even the node itself, until after the client has run for the first time. I actually don't need to pull any attributes from the node. This is for zookeeper, so I just need to get a list of host names that match a criteria (role, environment etc).

Alas, I can't use a data bag, or a role. My boot scripts use AWS Dynamodb (and a concatenated role and environment as a key) to pick always incrementing host name numbers. I'd have to start the CF stack, wait until the scripts had queried DynamoDB, had set the Name tag on the instance, eyeball the name in the Amazon console, and then go update the role, upload it to chef etc, before the instance came fully up.

Doug

On Sat, Jun 13, 2015 at 5:46 PM, Kevin Keane Subscription subscription@kkeane.com wrote:
When you created the node, did you also populate it with whatever attributes your search will later use?

I've run into your problem, too, several times, actually - I'm sure it's common.

For instance, I'm using bacula as a backup solution. The bacula director (server) needs to know about all of the clients, and all of the clients need to know about the director.

So when the clients search for the director, there is a chance they don't find one (yet), and similarly, the director may not (yet) find any clients. Both would lead to errors in the bacula configuration. The system won't be stable until it has run twice.

I don't look as the solution as a "hack" but rather as necessary error handling.

There is one other option: instead of a search, you can use a data bag (or even an attribute) to explicitly store the information you would otherwise get from a search. I use this approach in situations where I may have multiple similar servers, and want to be explicitly in which one gets used - for instance, to control which database is stored on which MySQL server.

Kevin Keane

The NetTech

http://www.4nettech.com
Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang doug.garstang@gmail.com
Sent: Saturday 13th June 2015 14:03
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Chef Search for Node Discovery.

Kevin,

My boot process runs a script that calls a knife node add, to add the node to chef. Since it's been added to chef, it should be discoverable. However, it's not discoverable until the client has run at least once. That seems more like an architectural problem with chef, rather than a fundamental flaw in the universe. :slight_smile:

What prevents me from doing a two pass run is that I would have to take a process that works fine, and add a hack to make it work when a chef search is required. Every node would have to run twice, or I would have to create some criteria, maybe have CloudFormation put a 'needs-chef-hack' EC2 tag on the instance, and then my scripts check for that. This really isn't very elegant.

What if I modified my boot scripts to run chef twice. The first run would essentially be to call a dummy cookbook that does nothing. After this has run, the node would be discoverable? Plenty of other folks must have this same issue.

Doug.

On Sat, Jun 13, 2015 at 1:43 PM, Kevin Keane Subscription subscription@kkeane.com wrote:
It seems to me that this problem is fundamental, not specific to chef: you simply can't discover anything until it is discoverable. Making something discoverable in an automated fashion can happen only by running something. In chef, it's chef itself. The other option is to have a separate pre-registration process that pre-creates (and pre-populates) the nodes.

That is also quite possible in chef; you'd use knife (possibly from a shell script) to automatically create your nodes.

But in the end, that doesn't really solve your problem, it just moves it.

Kevin Keane

The NetTech

http://www.4nettech.com
Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang doug.garstang@gmail.com
Sent: Friday 12th June 2015 22:01
To: chef@lists.opscode.com
Subject: [chef] Chef Search for Node Discovery.

I haven't tried this for a while, so I thought I'd see if there was any progress.

I need to write a cookbook that discovers other chef nodes. The issue with chef search is that nodes don't report in until they've done their first chef client run, which means that the client has to run twice on the one doing the discovering, but... that also applies to the one that wants to be discovered. End result, not automated. This has basically made chef search unusable in every situation for me where it might otherwise be useful.

Is there a better way? I'm on EC2. My other option was to write an ohai plugin to scan ec2 and return instances matching the correct criteria.

Thanks,
Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627