Chef Search Again

Chef Infra (archive)
1

The battle continues with chef search. I’m trying to use chef search to
discover zookeeper nodes. The very first thing in my run list is node.save,
which seems to cause the Chef server to immediately have access to the data
I need, the role and the environment.

I just did a test, where one window had the result of “knife search node
"chef_environment:dev AND roles:role-zookeeper”. I could see that the data
was populated immediately after the call to node.save. A few minutes later,
when the recipe did it’s search, no nodes were found, as evidenced from an
empty log() statement in the recipe.

I’ve run the test a few times, and basically, the result are unreliable. On
my latest test, two of the nodes discovered all three nodes, and one of the
nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

2

Well, I've tried putting both a 60s and a 90s sleep before the call to chef
search. No improvement. I rerun the chef-client again and the nodes appear.
It appears that node.save() at the start of the run list updates the node
on the server, but it still don't work for searches. :frowning:

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang doug.garstang@gmail.com
wrote:

The battle continues with chef search. I'm trying to use chef search to
discover zookeeper nodes. The very first thing in my run list is node.save,
which seems to cause the Chef server to immediately have access to the data
I need, the role and the environment.

I just did a test, where one window had the result of "knife search node
"chef_environment:dev AND roles:role-zookeeper". I could see that the data
was populated immediately after the call to node.save. A few minutes later,
when the recipe did it's search, no nodes were found, as evidenced from an
empty log() statement in the recipe.

I've run the test a few times, and basically, the result are unreliable.
On my latest test, two of the nodes discovered all three nodes, and one of
the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

3

Ok, I thought, I'll write a new cookbook called nodesave and put JUST
node.save in it. Then, I'll simply call chef twice, the first time passing
just the nodesave cookbook. This should cause the node data to be populated
at the end of the first run, available in the second run, Well trying to do
a node.save on vagrant results in:

"I cannot read /etc/chef/client.pem, which you told me to use to sign
requests!"

Sigh. I guess all I need is an empty cookbook that does nothing as
node.save seems to have no effect anyway. All I need is a dummy chef client
run.

Doug.

On Tue, Jun 16, 2015 at 12:14 PM, Douglas Garstang doug.garstang@gmail.com
wrote:

Well, I've tried putting both a 60s and a 90s sleep before the call to
chef search. No improvement. I rerun the chef-client again and the nodes
appear. It appears that node.save() at the start of the run list updates
the node on the server, but it still don't work for searches. :frowning:

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

The battle continues with chef search. I'm trying to use chef search to
discover zookeeper nodes. The very first thing in my run list is node.save,
which seems to cause the Chef server to immediately have access to the data
I need, the role and the environment.

I just did a test, where one window had the result of "knife search node
"chef_environment:dev AND roles:role-zookeeper". I could see that the data
was populated immediately after the call to node.save. A few minutes later,
when the recipe did it's search, no nodes were found, as evidenced from an
empty log() statement in the recipe.

I've run the test a few times, and basically, the result are unreliable.
On my latest test, two of the nodes discovered all three nodes, and one of
the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

4

Sigh. I guess all I need is an empty cookbook that does nothing as node.save seems to have no effect anyway. All I need is a dummy chef client run.

How are you running this “dummy” chef-client run? If you change the run-list to remove the role role-zookeeper, your node will be persisted without this role and without the attributes associated with it. You’ll have a period of time where the roles attribute of the node does NOT contain what you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang
doug.garstang@gmail.com wrote:

The battle continues with chef search. I'm trying to use chef search to
discover zookeeper nodes. The very first thing in my run list is node.save,
which seems to cause the Chef server to immediately have access to the data
I need, the role and the environment.
I just did a test, where one window had the result of "knife search node
"chef_environment:dev AND roles:role-zookeeper". I could see that the data
was populated immediately after the call to node.save. A few minutes later,
when the recipe did it's search, no nodes were found, as evidenced from an
empty log() statement in the recipe.
I've run the test a few times, and basically, the result are unreliable. On
my latest test, two of the nodes discovered all three nodes, and one of the
nodes discovered only one node (not itself btw).
How can I make this reliable?
Doug

5

Stephen,

Well, I just tried using chef-client -r dummy and that worked. However,
that permanently replaces the run list on the chef server. If I now run
chef-client, hoping to use the zookeeper cookbook formerly associated with
the node, it's been replaced by the dummy one. I am so frustrated. :frowning:

Doug.

On Tue, Jun 16, 2015 at 12:38 PM, Stephen Delano stephen@opscode.com
wrote:

Sigh. I guess all I need is an empty cookbook that does nothing as
node.save seems to have no effect anyway. All I need is a dummy chef client
run.

How are you running this “dummy” chef-client run? If you change the
run-list to remove the role role-zookeeper, your node will be persisted
without this role and without the attributes associated with it. You’ll
have a period of time where the roles attribute of the node does NOT
contain what you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

The battle continues with chef search. I'm trying to use chef search
to discover zookeeper nodes. The very first thing in my run list is
node.save, which seems to cause the Chef server to immediately have access
to the data I need, the role and the environment.

I just did a test, where one window had the result of "knife search node
"chef_environment:dev AND roles:role-zookeeper". I could see that the data
was populated immediately after the call to node.save. A few minutes later,
when the recipe did it's search, no nodes were found, as evidenced from an
empty log() statement in the recipe.

I've run the test a few times, and basically, the result are unreliable.
On my latest test, two of the nodes discovered all three nodes, and one of
the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

6

So, let me summarise this sorry situation...

  1. By default, data about nodes isn't available until after the chef client
    has run at least once.
  2. Node.save at the start of the run list should fix this, but it does not.
  3. Putting a 60s/90s sleep before the knife search, which should allow solr
    to complete a re-index has no effect.
  4. Doing a dummy run of the chef-client and pointing it to a no-op recipe
    to get around (1) above requires passing the -r option which has the side
    effect of updating the run list on the chef server so that the next call to
    chef-client uses the dummy cookbook, not the cookbook this node was
    originally provisioned to use.
  5. Even if node.save did work, it won't run on vagrant and throws an error.

Did someone intentionally design chef server to induce scraping eyes out
with a rusty razor blade? I'm pretty fed up.

Doug.

On Tue, Jun 16, 2015 at 12:51 PM, Douglas Garstang doug.garstang@gmail.com
wrote:

Stephen,

Well, I just tried using chef-client -r dummy and that worked. However,
that permanently replaces the run list on the chef server. If I now run
chef-client, hoping to use the zookeeper cookbook formerly associated with
the node, it's been replaced by the dummy one. I am so frustrated. :frowning:

Doug.

On Tue, Jun 16, 2015 at 12:38 PM, Stephen Delano stephen@opscode.com
wrote:

Sigh. I guess all I need is an empty cookbook that does nothing as
node.save seems to have no effect anyway. All I need is a dummy chef client
run.

How are you running this “dummy” chef-client run? If you change the
run-list to remove the role role-zookeeper, your node will be persisted
without this role and without the attributes associated with it. You’ll
have a period of time where the roles attribute of the node does NOT
contain what you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

The battle continues with chef search. I'm trying to use chef search
to discover zookeeper nodes. The very first thing in my run list is
node.save, which seems to cause the Chef server to immediately have access
to the data I need, the role and the environment.

I just did a test, where one window had the result of "knife search node
"chef_environment:dev AND roles:role-zookeeper". I could see that the data
was populated immediately after the call to node.save. A few minutes later,
when the recipe did it's search, no nodes were found, as evidenced from an
empty log() statement in the recipe.

I've run the test a few times, and basically, the result are unreliable.
On my latest test, two of the nodes discovered all three nodes, and one of
the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

7
  1. By default, data about nodes isn’t available until after the chef client has run at least once.

This is correct.

  1. Node.save at the start of the run list should fix this, but it does not.

Are you running node.save inside of a ruby block or are you running it at compile time? I’m not familiar enough with the Chef Client to know whether one of those is the better option for you or not, but it might be worth printing out node.to_json to see exactly what is getting sent to the Chef Server.

  1. Putting a 60s/90s sleep before the knife search, which should allow solr to complete a re-index has no effect.

Which version of the Chef Server are you running? Chef Server 12 ships with Solr 4 and is configured to make updates to the search index available for searching once per second as opposed to once per minute in Chef Server 11.

  1. Even if node.save did work, it won’t run on vagrant and throws an error.

That error you posted seems to imply that the client had trouble reading /etc/chef/client.pem. This is strictly a client-side issue. Does this file exist? If so, what are the permissions on it? Did you run chef-client as root?

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 12:58 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

So, let me summarise this sorry situation...

  1. By default, data about nodes isn't available until after the chef client
    has run at least once.
  2. Node.save at the start of the run list should fix this, but it does not.
  3. Putting a 60s/90s sleep before the knife search, which should allow solr
    to complete a re-index has no effect.
  4. Doing a dummy run of the chef-client and pointing it to a no-op recipe
    to get around (1) above requires passing the -r option which has the side
    effect of updating the run list on the chef server so that the next call to
    chef-client uses the dummy cookbook, not the cookbook this node was
    originally provisioned to use.
  5. Even if node.save did work, it won't run on vagrant and throws an error.
    Did someone intentionally design chef server to induce scraping eyes out
    with a rusty razor blade? I'm pretty fed up.
    Doug.
    On Tue, Jun 16, 2015 at 12:51 PM, Douglas Garstang doug.garstang@gmail.com
    wrote:

Stephen,

Well, I just tried using chef-client -r dummy and that worked. However,
that permanently replaces the run list on the chef server. If I now run
chef-client, hoping to use the zookeeper cookbook formerly associated with
the node, it's been replaced by the dummy one. I am so frustrated. :frowning:

Doug.

On Tue, Jun 16, 2015 at 12:38 PM, Stephen Delano stephen@opscode.com
wrote:

Sigh. I guess all I need is an empty cookbook that does nothing as
node.save seems to have no effect anyway. All I need is a dummy chef client
run.

How are you running this “dummy” chef-client run? If you change the
run-list to remove the role role-zookeeper, your node will be persisted
without this role and without the attributes associated with it. You’ll
have a period of time where the roles attribute of the node does NOT
contain what you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

The battle continues with chef search. I'm trying to use chef search
to discover zookeeper nodes. The very first thing in my run list is
node.save, which seems to cause the Chef server to immediately have access
to the data I need, the role and the environment.

I just did a test, where one window had the result of "knife search node
"chef_environment:dev AND roles:role-zookeeper". I could see that the data
was populated immediately after the call to node.save. A few minutes later,
when the recipe did it's search, no nodes were found, as evidenced from an
empty log() statement in the recipe.

I've run the test a few times, and basically, the result are unreliable.
On my latest test, two of the nodes discovered all three nodes, and one of
the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,
Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

8

Stephen,

  1. No ruby block. Compile time I guess.

  2. I'm using chef 11, but I tried both a 60s and a 90s sleep and it had no
    effect.

  3. It's vagrant. Chef solo. There is no client.pem file as far as I know.

Doug.

On Tue, Jun 16, 2015 at 1:24 PM, Stephen Delano stephen@opscode.com wrote:

  1. By default, data about nodes isn’t available until after the chef
    client has run at least once.

This is correct.

  1. Node.save at the start of the run list should fix this, but it does
    not.

Are you running node.save inside of a ruby block or are you running it at
compile time? I’m not familiar enough with the Chef Client to know whether
one of those is the better option for you or not, but it might be worth
printing out node.to_json to see exactly what is getting sent to the Chef
Server.

  1. Putting a 60s/90s sleep before the knife search, which should allow
    solr to complete a re-index has no effect.

Which version of the Chef Server are you running? Chef Server 12 ships
with Solr 4 and is configured to make updates to the search index available
for searching once per second as opposed to once per minute in Chef Server
11.

  1. Even if node.save did work, it won’t run on vagrant and throws an
    error.

That error you posted seems to imply that the client had trouble reading
/etc/chef/client.pem. This is strictly a client-side issue. Does this file
exist? If so, what are the permissions on it? Did you run chef-client as
root?

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 12:58 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

So, let me summarise this sorry situation...

  1. By default, data about nodes isn't available until after the chef
    client has run at least once.
  2. Node.save at the start of the run list should fix this, but it does
    not.
  3. Putting a 60s/90s sleep before the knife search, which should allow
    solr to complete a re-index has no effect.
  4. Doing a dummy run of the chef-client and pointing it to a no-op recipe
    to get around (1) above requires passing the -r option which has the side
    effect of updating the run list on the chef server so that the next call to
    chef-client uses the dummy cookbook, not the cookbook this node was
    originally provisioned to use.
  5. Even if node.save did work, it won't run on vagrant and throws an
    error.

Did someone intentionally design chef server to induce scraping eyes out
with a rusty razor blade? I'm pretty fed up.

Doug.

On Tue, Jun 16, 2015 at 12:51 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

Stephen,

Well, I just tried using chef-client -r dummy and that worked. However,
that permanently replaces the run list on the chef server. If I now run
chef-client, hoping to use the zookeeper cookbook formerly associated with
the node, it's been replaced by the dummy one. I am so frustrated. :frowning:

Doug.

On Tue, Jun 16, 2015 at 12:38 PM, Stephen Delano stephen@opscode.com
wrote:

Sigh. I guess all I need is an empty cookbook that does nothing as
node.save seems to have no effect anyway. All I need is a dummy chef client
run.

How are you running this “dummy” chef-client run? If you change the
run-list to remove the role role-zookeeper, your node will be persisted
without this role and without the attributes associated with it. You’ll
have a period of time where the roles attribute of the node does NOT
contain what you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

The battle continues with chef search. I'm trying to use chef
search to discover zookeeper nodes. The very first thing in my run list is
node.save, which seems to cause the Chef server to immediately have access
to the data I need, the role and the environment.

I just did a test, where one window had the result of "knife search
node "chef_environment:dev AND roles:role-zookeeper". I could see that the
data was populated immediately after the call to node.save. A few minutes
later, when the recipe did it's search, no nodes were found, as evidenced
from an empty log() statement in the recipe.

I've run the test a few times, and basically, the result are
unreliable. On my latest test, two of the nodes discovered all three nodes,
and one of the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

9

Welcome to the wonder world of distributed systems!

If you're going to use Chef search as a service discovery mechanism,
you're going to face these limitations.

Saving the node object to the Chef Server at the end of the run is "by
design". A node that's not fully converged is likely not functional
yet. For example, if you has a load balancer searching for web server
nodes to include them in a pool, you don't want a machine that's half
configured and not serving data in your pool yet.

In the general case, I'd recommend using Real Service Discovery
protocol for your service discovery.

However, since you're attempting to do just that, (setting up
zookeeper), something gotta give.

You are, by the very nature of your problem, going to be forced to
write some error handling code, and do multiple converges in this
scenario. This is not a limitation of Chef or Chef Server. This is a
general problem.

-s

On Tue, Jun 16, 2015 at 3:58 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

So, let me summarise this sorry situation...

  1. By default, data about nodes isn't available until after the chef client
    has run at least once.
  2. Node.save at the start of the run list should fix this, but it does not.
  3. Putting a 60s/90s sleep before the knife search, which should allow solr
    to complete a re-index has no effect.
  4. Doing a dummy run of the chef-client and pointing it to a no-op recipe to
    get around (1) above requires passing the -r option which has the side
    effect of updating the run list on the chef server so that the next call to
    chef-client uses the dummy cookbook, not the cookbook this node was
    originally provisioned to use.
  5. Even if node.save did work, it won't run on vagrant and throws an error.

Did someone intentionally design chef server to induce scraping eyes out
with a rusty razor blade? I'm pretty fed up.

Doug.

On Tue, Jun 16, 2015 at 12:51 PM, Douglas Garstang doug.garstang@gmail.com
wrote:

Stephen,

Well, I just tried using chef-client -r dummy and that worked. However,
that permanently replaces the run list on the chef server. If I now run
chef-client, hoping to use the zookeeper cookbook formerly associated with
the node, it's been replaced by the dummy one. I am so frustrated. :frowning:

Doug.

On Tue, Jun 16, 2015 at 12:38 PM, Stephen Delano stephen@opscode.com
wrote:

Sigh. I guess all I need is an empty cookbook that does nothing as
node.save seems to have no effect anyway. All I need is a dummy chef client
run.

How are you running this “dummy” chef-client run? If you change the
run-list to remove the role role-zookeeper, your node will be persisted
without this role and without the attributes associated with it. You’ll have
a period of time where the roles attribute of the node does NOT contain what
you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang
doug.garstang@gmail.com wrote:

The battle continues with chef search. I'm trying to use chef search to
discover zookeeper nodes. The very first thing in my run list is node.save,
which seems to cause the Chef server to immediately have access to the data
I need, the role and the environment.

I just did a test, where one window had the result of "knife search node
"chef_environment:dev AND roles:role-zookeeper". I could see that the data
was populated immediately after the call to node.save. A few minutes later,
when the recipe did it's search, no nodes were found, as evidenced from an
empty log() statement in the recipe.

I've run the test a few times, and basically, the result are unreliable.
On my latest test, two of the nodes discovered all three nodes, and one of
the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

10

On 6/16/15 1:32 PM, Douglas Garstang wrote:

  1. It's vagrant. Chef solo. There is no client.pem file as far as I know.
    Chef-solo doesn't have a server to talk to in order to node.save or post
    results to search. The node.save turns into a NOP transparently in
    chef-solo. You can go down the road of chef-solo-search but then
    eventually data bags will troll you. At some point people wind up
    trying to write a stub chef-server for chef-solo to talk to and replace
    all the APIs which is just going to fight with all the special cases of
    chef-solo to make it NOT talk to a server which are littered around the
    codebase.

The correct way forwards is to use chef-zero instead of chef-solo and
have a normal fully functional chef-client that expects to talk to a
chef server talking to the chef-zero server. Otherwise you will
continue to fight the tool.

+1 on using Chef Server 12 and/or Hosted Chef with Solr 4 which will get
the required sleep down from a couple minutes to a couple seconds.

You can also use an overrride run list instead of -r to run a recipe
without updating the run_list.

11

Thanks all.

Here's my solution...

aws ec2 describe-instances --region us-east-1 --filter
"Name=tag:role,Values=role-zookeeper" "Name=tag:environment,Values=dev"
"Name=instance-state-name,Values=running" --query
'Reservations.Instances.Tags[?Key==Name].Value'

throw that in a ohai plugin, convert the output to json, and I'm good to go.

Doug.

On Tue, Jun 16, 2015 at 2:10 PM, Lamont Granquist lamont@chef.io wrote:

On 6/16/15 1:32 PM, Douglas Garstang wrote:

  1. It's vagrant. Chef solo. There is no client.pem file as far as I know.

Chef-solo doesn't have a server to talk to in order to node.save or post
results to search. The node.save turns into a NOP transparently in
chef-solo. You can go down the road of chef-solo-search but then
eventually data bags will troll you. At some point people wind up trying
to write a stub chef-server for chef-solo to talk to and replace all the
APIs which is just going to fight with all the special cases of chef-solo
to make it NOT talk to a server which are littered around the codebase.

The correct way forwards is to use chef-zero instead of chef-solo and have
a normal fully functional chef-client that expects to talk to a chef server
talking to the chef-zero server. Otherwise you will continue to fight the
tool.

+1 on using Chef Server 12 and/or Hosted Chef with Solr 4 which will get
the required sleep down from a couple minutes to a couple seconds.

You can also use an overrride run list instead of -r to run a recipe
without updating the run_list.

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

12

Lamont,

The node.save doesn't turn into a noop in chef solo. It breaks it with the
error I mentioned earlier.

As for the -r option, it updates the run list on the node on the chef
server. It's not a local thing.

Doug.

On Tue, Jun 16, 2015 at 2:10 PM, Lamont Granquist lamont@chef.io wrote:

On 6/16/15 1:32 PM, Douglas Garstang wrote:

  1. It's vagrant. Chef solo. There is no client.pem file as far as I know.

Chef-solo doesn't have a server to talk to in order to node.save or post
results to search. The node.save turns into a NOP transparently in
chef-solo. You can go down the road of chef-solo-search but then
eventually data bags will troll you. At some point people wind up trying
to write a stub chef-server for chef-solo to talk to and replace all the
APIs which is just going to fight with all the special cases of chef-solo
to make it NOT talk to a server which are littered around the codebase.

The correct way forwards is to use chef-zero instead of chef-solo and have
a normal fully functional chef-client that expects to talk to a chef server
talking to the chef-zero server. Otherwise you will continue to fight the
tool.

+1 on using Chef Server 12 and/or Hosted Chef with Solr 4 which will get
the required sleep down from a couple minutes to a couple seconds.

You can also use an overrride run list instead of -r to run a recipe
without updating the run_list.

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

13

Doug,

This is another reminder that the Chef community guidelines apply to all posts in this list, and that “be respectful” is part of those guidelines. Thoughtful, passionate human beings work hard on the Chef server. If you need a refresher, the guidelines are available for review here: Chef Contributor Covenant Code of Conduct . I strongly suggest you familiarize yourself with these guidelines as quickly as possible, and adhere to them strictly in the future.

While it is clear that you are fed up, and your frustration is completely understandable, saying things like “this sorry situation" and "scraping eyes out with a rusty razor blade” shuts down conversation instead of inviting response. It’s possible to be open and critical when sharing your frustrations around the functionality of the Chef server, without being dismissive of the work that has gone and continues to go into creating it.

This is not the first time someone has said something to you about your rhetoric on this list. Unless you are able to show that you can understand the difference between honesty and candor, it will be the last.

Thanks,
--Charles

On June 16, 2015 at 12:59:12 PM, Douglas Garstang (doug.garstang@gmail.com) wrote:

So, let me summarise this sorry situation...

  1. By default, data about nodes isn't available until after the chef client has run at least once.
  2. Node.save at the start of the run list should fix this, but it does not.
  3. Putting a 60s/90s sleep before the knife search, which should allow solr to complete a re-index has no effect.
  4. Doing a dummy run of the chef-client and pointing it to a no-op recipe to get around (1) above requires passing the -r option which has the side effect of updating the run list on the chef server so that the next call to chef-client uses the dummy cookbook, not the cookbook this node was originally provisioned to use.
  5. Even if node.save did work, it won't run on vagrant and throws an error.

Did someone intentionally design chef server to induce scraping eyes out with a rusty razor blade? I'm pretty fed up.

Doug.

On Tue, Jun 16, 2015 at 12:51 PM, Douglas Garstang doug.garstang@gmail.com wrote:
Stephen,

Well, I just tried using chef-client -r dummy and that worked. However, that permanently replaces the run list on the chef server. If I now run chef-client, hoping to use the zookeeper cookbook formerly associated with the node, it's been replaced by the dummy one. I am so frustrated. :frowning:

Doug.

On Tue, Jun 16, 2015 at 12:38 PM, Stephen Delano stephen@opscode.com wrote:

Sigh. I guess all I need is an empty cookbook that does nothing as node.save seems to have no effect anyway. All I need is a dummy chef client run.

How are you running this “dummy” chef-client run? If you change the run-list to remove the role role-zookeeper, your node will be persisted without this role and without the attributes associated with it. You’ll have a period of time where the roles attribute of the node does NOT contain what you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang doug.garstang@gmail.com wrote:

The battle continues with chef search. I'm trying to use chef search to discover zookeeper nodes. The very first thing in my run list is node.save, which seems to cause the Chef server to immediately have access to the data I need, the role and the environment.

I just did a test, where one window had the result of "knife search node "chef_environment:dev AND roles:role-zookeeper". I could see that the data was populated immediately after the call to node.save. A few minutes later, when the recipe did it's search, no nodes were found, as evidenced from an empty log() statement in the recipe.

I've run the test a few times, and basically, the result are unreliable. On my latest test, two of the nodes discovered all three nodes, and one of the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

14

Charles,

Your response viewed. After viewing the community guidelines, I see nothing
there about using colorful language to explain a personal frustrating
situation being against the rules. I do however see obvious things like
personal threats against others, sexual harassment etc. Having said this, I
realise I risk being removed for speaking my mind. If that's what
eventuates, then let it stand as a matter of public record that I
politefully disagree with your assesment.

Perhaps the community guidelines need to be updated to be more restrictive
in nature?

Douglas.

On Tue, Jun 16, 2015 at 3:34 PM, Charles Johnson charles@chef.io wrote:

Doug,

This is another reminder that the Chef community guidelines apply to all
posts in this list, and that “be respectful” is part of those guidelines.
Thoughtful, passionate human beings work hard on the Chef server. If you
need a refresher, the guidelines are available for review here:
Chef Contributor Covenant Code of Conduct . I strongly suggest you
familiarize yourself with these guidelines as quickly as possible, and
adhere to them strictly in the future.

While it is clear that you are fed up, and your frustration is completely
understandable, saying things like “this sorry situation" and "scraping
eyes out with a rusty razor blade” shuts down conversation instead of
inviting response. It’s possible to be open and critical when sharing your
frustrations around the functionality of the Chef server, without being
dismissive of the work that has gone and continues to go into creating it.

This is not the first time someone has said something to you about your
rhetoric on this list. Unless you are able to show that you can understand
the difference between honesty and candor, it will be the last.
Thanks,
--Charles

On June 16, 2015 at 12:59:12 PM, Douglas Garstang (doug.garstang@gmail.com)
wrote:

So, let me summarise this sorry situation...

  1. By default, data about nodes isn't available until after the chef
    client has run at least once.
  2. Node.save at the start of the run list should fix this, but it does not.
  3. Putting a 60s/90s sleep before the knife search, which should allow
    solr to complete a re-index has no effect.
  4. Doing a dummy run of the chef-client and pointing it to a no-op recipe
    to get around (1) above requires passing the -r option which has the side
    effect of updating the run list on the chef server so that the next call to
    chef-client uses the dummy cookbook, not the cookbook this node was
    originally provisioned to use.
  5. Even if node.save did work, it won't run on vagrant and throws an error.

Did someone intentionally design chef server to induce scraping eyes out
with a rusty razor blade? I'm pretty fed up.

Doug.

On Tue, Jun 16, 2015 at 12:51 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

Stephen,

Well, I just tried using chef-client -r dummy and that worked. However,
that permanently replaces the run list on the chef server. If I now run
chef-client, hoping to use the zookeeper cookbook formerly associated with
the node, it's been replaced by the dummy one. I am so frustrated. :frowning:

Doug.

On Tue, Jun 16, 2015 at 12:38 PM, Stephen Delano stephen@opscode.com
wrote:

Sigh. I guess all I need is an empty cookbook that does nothing as
node.save seems to have no effect anyway. All I need is a dummy chef client
run.

How are you running this “dummy” chef-client run? If you change the
run-list to remove the role role-zookeeper, your node will be persisted
without this role and without the attributes associated with it. You’ll
have a period of time where the roles attribute of the node does NOT
contain what you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

The battle continues with chef search. I'm trying to use chef
search to discover zookeeper nodes. The very first thing in my run list is
node.save, which seems to cause the Chef server to immediately have access
to the data I need, the role and the environment.

I just did a test, where one window had the result of "knife search
node "chef_environment:dev AND roles:role-zookeeper". I could see that the
data was populated immediately after the call to node.save. A few minutes
later, when the recipe did it's search, no nodes were found, as evidenced
from an empty log() statement in the recipe.

I've run the test a few times, and basically, the result are
unreliable. On my latest test, two of the nodes discovered all three nodes,
and one of the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

15

Douglas,

Your message (chef - [chef] Re: Re: Chef Search Again)
includes language that is not considerate, respectful, or professional.

"Did someone intentionally design chef server to induce scraping eyes out
with a rusty razor blade?" goes beyond describing your personal
frustrations. Clearly, you are frustrated and are having difficulty
getting Chef to behave the way you'd like.

Our Community Guidelines require that you:

"Be careful in the words that you choose. Be kind to others. Do not insult
or put down others."

and

  • Be considerate.
  • Be respectful.
  • Be professional.
  • Be careful in the words that you choose.

Eventually, your questions to the list will be ignored unless this behavior
improves. Additionally, continuing to post messages that are not in
keeping with our community guidelines will lead to corrective action which
will likely mean banning you from the community mailing list. I don't
really think anyone wants that.

I would welcome a live discussion via phone or video chat to answer any
additional questions you have. Please let me know what time would work
best for you.

Thank you,
Nathen Harvey
Community Director, Chef
202.368.7264

On Tue, Jun 16, 2015 at 6:41 PM, Douglas Garstang doug.garstang@gmail.com
wrote:

Charles,

Your response viewed. After viewing the community guidelines, I see
nothing there about using colorful language to explain a personal
frustrating situation being against the rules. I do however see obvious
things like personal threats against others, sexual harassment etc. Having
said this, I realise I risk being removed for speaking my mind. If that's
what eventuates, then let it stand as a matter of public record that I
politefully disagree with your assesment.

Perhaps the community guidelines need to be updated to be more restrictive
in nature?

Douglas.

On Tue, Jun 16, 2015 at 3:34 PM, Charles Johnson charles@chef.io wrote:

Doug,

This is another reminder that the Chef community guidelines apply to all
posts in this list, and that “be respectful” is part of those guidelines.
Thoughtful, passionate human beings work hard on the Chef server. If you
need a refresher, the guidelines are available for review here:
Chef Contributor Covenant Code of Conduct . I strongly suggest you
familiarize yourself with these guidelines as quickly as possible, and
adhere to them strictly in the future.

While it is clear that you are fed up, and your frustration is completely
understandable, saying things like “this sorry situation" and "scraping
eyes out with a rusty razor blade” shuts down conversation instead of
inviting response. It’s possible to be open and critical when sharing your
frustrations around the functionality of the Chef server, without being
dismissive of the work that has gone and continues to go into creating it.

This is not the first time someone has said something to you about your
rhetoric on this list. Unless you are able to show that you can understand
the difference between honesty and candor, it will be the last.
Thanks,
--Charles

On June 16, 2015 at 12:59:12 PM, Douglas Garstang (
doug.garstang@gmail.com) wrote:

So, let me summarise this sorry situation...

  1. By default, data about nodes isn't available until after the chef
    client has run at least once.
  2. Node.save at the start of the run list should fix this, but it does
    not.
  3. Putting a 60s/90s sleep before the knife search, which should allow
    solr to complete a re-index has no effect.
  4. Doing a dummy run of the chef-client and pointing it to a no-op recipe
    to get around (1) above requires passing the -r option which has the side
    effect of updating the run list on the chef server so that the next call to
    chef-client uses the dummy cookbook, not the cookbook this node was
    originally provisioned to use.
  5. Even if node.save did work, it won't run on vagrant and throws an
    error.

Did someone intentionally design chef server to induce scraping eyes out
with a rusty razor blade? I'm pretty fed up.

Doug.

On Tue, Jun 16, 2015 at 12:51 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

Stephen,

Well, I just tried using chef-client -r dummy and that worked. However,
that permanently replaces the run list on the chef server. If I now run
chef-client, hoping to use the zookeeper cookbook formerly associated with
the node, it's been replaced by the dummy one. I am so frustrated. :frowning:

Doug.

On Tue, Jun 16, 2015 at 12:38 PM, Stephen Delano stephen@opscode.com
wrote:

Sigh. I guess all I need is an empty cookbook that does nothing as
node.save seems to have no effect anyway. All I need is a dummy chef client
run.

How are you running this “dummy” chef-client run? If you change the
run-list to remove the role role-zookeeper, your node will be persisted
without this role and without the attributes associated with it. You’ll
have a period of time where the roles attribute of the node does NOT
contain what you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

The battle continues with chef search. I'm trying to use chef
search to discover zookeeper nodes. The very first thing in my run list is
node.save, which seems to cause the Chef server to immediately have access
to the data I need, the role and the environment.

I just did a test, where one window had the result of "knife search
node "chef_environment:dev AND roles:role-zookeeper". I could see that the
data was populated immediately after the call to node.save. A few minutes
later, when the recipe did it's search, no nodes were found, as evidenced
from an empty log() statement in the recipe.

I've run the test a few times, and basically, the result are
unreliable. On my latest test, two of the nodes discovered all three nodes,
and one of the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

16

For us, we define our zookeeper ensemble as a hash, at our environment
layer. I know, it's not chef-search and you'd like to use it, but I
consider ZK to be an extremely mission-critical piece, and I like to
minimize the elasticness/reconfiguration of him as much as possible, just
to err on the side of caution as it can be an extremely touchy beast. All
I need is a service outage triggered by a bombed chef-search, which
triggered a reconfigure/restart of my ZK ensemble, and either: left me in a
completely broken state, or potentially a split-brain scenario at 0300. I
feel more comfortable using chef-search in places where it might be ok I
missed one single node, it's not going to create a full on outage for me,
and that node will be picked up on the next chef-client run. We get mixed
delays after a node has converged of when Solr will have its indexes
updated, and properly returning me the results of the updated node
attributes. Solr4 appears to be a great improvement to these delays. +1
to an upgrade to Chef12/Hosted chef if this is the primary cause of your
frustrations. I love chef search, but I understand this limitation though,
and instead architect my solutions to deal with these types of limits.

Alternatively, we've reduced the role ZK plays in our org, by letting ZK
drive our application configurations, and moved service discovery to
Consul. This has been a really great win for us in terms of ease of
discovery, and greatly smoothed out the elastic scaling up/down of our
applications with no human intervention, and we can still use it in unison
with Zookeeper. I'm sorry you're feeling frustrated, but if you're
experiencing frustrations with chef-search just to get Zookeeper ensemble
setup, you have a long road to travel of debugging Zookeeper in failure
scenarios, they will be much more frustrating than chef-search. Just my .02

HTH,

-Dennis

On Wed, Jun 17, 2015 at 7:35 AM, Nathen Harvey nharvey@chef.io wrote:

Douglas,

Your message (
chef - [chef] Re: Re: Chef Search Again) includes
language that is not considerate, respectful, or professional.

"Did someone intentionally design chef server to induce scraping eyes out
with a rusty razor blade?" goes beyond describing your personal
frustrations. Clearly, you are frustrated and are having difficulty
getting Chef to behave the way you'd like.

Our Community Guidelines require that you:

"Be careful in the words that you choose. Be kind to others. Do not insult
or put down others."

and

  • Be considerate.
  • Be respectful.
  • Be professional.
  • Be careful in the words that you choose.

Eventually, your questions to the list will be ignored unless this
behavior improves. Additionally, continuing to post messages that are not
in keeping with our community guidelines will lead to corrective action
which will likely mean banning you from the community mailing list. I
don't really think anyone wants that.

I would welcome a live discussion via phone or video chat to answer any
additional questions you have. Please let me know what time would work
best for you.

Thank you,
Nathen Harvey
Community Director, Chef
202.368.7264

On Tue, Jun 16, 2015 at 6:41 PM, Douglas Garstang <doug.garstang@gmail.com

wrote:

Charles,

Your response viewed. After viewing the community guidelines, I see
nothing there about using colorful language to explain a personal
frustrating situation being against the rules. I do however see obvious
things like personal threats against others, sexual harassment etc. Having
said this, I realise I risk being removed for speaking my mind. If that's
what eventuates, then let it stand as a matter of public record that I
politefully disagree with your assesment.

Perhaps the community guidelines need to be updated to be more
restrictive in nature?

Douglas.

On Tue, Jun 16, 2015 at 3:34 PM, Charles Johnson charles@chef.io wrote:

Doug,

This is another reminder that the Chef community guidelines apply to all
posts in this list, and that “be respectful” is part of those guidelines.
Thoughtful, passionate human beings work hard on the Chef server. If you
need a refresher, the guidelines are available for review here:
Chef Contributor Covenant Code of Conduct . I strongly suggest you
familiarize yourself with these guidelines as quickly as possible, and
adhere to them strictly in the future.

While it is clear that you are fed up, and your frustration is
completely understandable, saying things like “this sorry situation" and
"scraping eyes out with a rusty razor blade” shuts down conversation
instead of inviting response. It’s possible to be open and critical when
sharing your frustrations around the functionality of the Chef server,
without being dismissive of the work that has gone and continues to go into
creating it.

This is not the first time someone has said something to you about your
rhetoric on this list. Unless you are able to show that you can understand
the difference between honesty and candor, it will be the last.
Thanks,
--Charles

On June 16, 2015 at 12:59:12 PM, Douglas Garstang (
doug.garstang@gmail.com) wrote:

So, let me summarise this sorry situation...

  1. By default, data about nodes isn't available until after the chef
    client has run at least once.
  2. Node.save at the start of the run list should fix this, but it does
    not.
  3. Putting a 60s/90s sleep before the knife search, which should allow
    solr to complete a re-index has no effect.
  4. Doing a dummy run of the chef-client and pointing it to a no-op
    recipe to get around (1) above requires passing the -r option which has the
    side effect of updating the run list on the chef server so that the next
    call to chef-client uses the dummy cookbook, not the cookbook this node was
    originally provisioned to use.
  5. Even if node.save did work, it won't run on vagrant and throws an
    error.

Did someone intentionally design chef server to induce scraping eyes out
with a rusty razor blade? I'm pretty fed up.

Doug.

On Tue, Jun 16, 2015 at 12:51 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

Stephen,

Well, I just tried using chef-client -r dummy and that worked. However,
that permanently replaces the run list on the chef server. If I now run
chef-client, hoping to use the zookeeper cookbook formerly associated with
the node, it's been replaced by the dummy one. I am so frustrated. :frowning:

Doug.

On Tue, Jun 16, 2015 at 12:38 PM, Stephen Delano stephen@opscode.com
wrote:

Sigh. I guess all I need is an empty cookbook that does nothing as
node.save seems to have no effect anyway. All I need is a dummy chef client
run.

How are you running this “dummy” chef-client run? If you change the
run-list to remove the role role-zookeeper, your node will be persisted
without this role and without the attributes associated with it. You’ll
have a period of time where the roles attribute of the node does NOT
contain what you’re looking for and searches will fail.

—
Stephen Delano - Engineering Lead, Chef

On Tue, Jun 16, 2015 at 11:50 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

The battle continues with chef search. I'm trying to use chef
search to discover zookeeper nodes. The very first thing in my run list is
node.save, which seems to cause the Chef server to immediately have access
to the data I need, the role and the environment.

I just did a test, where one window had the result of "knife search
node "chef_environment:dev AND roles:role-zookeeper". I could see that the
data was populated immediately after the call to node.save. A few minutes
later, when the recipe did it's search, no nodes were found, as evidenced
from an empty log() statement in the recipe.

I've run the test a few times, and basically, the result are
unreliable. On my latest test, two of the nodes discovered all three nodes,
and one of the nodes discovered only one node (not itself btw).

How can I make this reliable?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627