Push jobs vs SSH

Chef Infra (archive)
1

What’s the difference between chef push jobs and knife ssh?

I’m researching a use case for chef in which we want highly controlled
deployments to orchestrate across several different systems. The
traditional “pull” doesn’t fit the bill well because it is difficult to
get that to coordinate properly across systems.

From what I’ve read so far, chef push jobs are essentially a daemon
running on the remote servers that allow arbitrary (but
pre-defined/whitelisted) execution of commands. Aside from perhaps a
cleaner white-listing concept (over forced-ssh), how is this different
(better) than just using knife ssh?

I’m failing to see the benefits or use cases of chef push jobs over
knife ssh. The documentation is lacking in terms of how it is intended
to be used. Are push jobs better suited for different situations (and
what are those situations?)

Thanks for the help!

-Eric

2

One of the big benefits is speed.

Knife ssh uses a parallel ssh method. Chef push jobs use a message bus ( zeromq I believe ).

Here's a video that demonstrates the push job feature:

I also believe that push jobs are the way of the future over knife ssh.

sh

On Feb 6, 2015, at 5:31 AM, Eric Horne eric+opscode@box11.org wrote:

What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly controlled deployments to orchestrate across several different systems. The traditional "pull" doesn't fit the bill well because it is difficult to get that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon running on the remote servers that allow arbitrary (but pre-defined/whitelisted) execution of commands. Aside from perhaps a cleaner white-listing concept (over forced-ssh), how is this different (better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over knife ssh. The documentation is lacking in terms of how it is intended to be used. Are push jobs better suited for different situations (and what are those situations?)

Thanks for the help!

-Eric

3

Thanks Steven (and Hi!) I saw the video, but there isn't a comparison to
ssh that I remember (maybe I should watch it again). I was sold on pushy
being the next awesomeness, but now that I'm playing with it... it just
looks like another ssh client (yes I know it's not using ssh at all)
with some additional easy-to-use controls and reporting.

The video appears aimed at showing that the technology works as opposed
to presenting use cases in which it is superior to other solutions. When
is pushy better than knife ssh or an ssh call integrated into a recipe?

My expectations for pushy were a little more lofty, I guess. I was
hoping for more integrated orchestration, the ability to run a single
recipe on a system as opposed to just running chef-client to get
everything (so I guess something more like ansible, but integrated into
Chef). Something like: "push out THIS change" as opposed to "sync all
changes now".

So.. all that said -- this isn't intended to be a critque of push; I
honestly think I'm missing something somewhere along the line.

(on a side note, doesn't Chef already use rabbitmq?)

-Eric

Hajducko, Steven mailto:Steven_Hajducko@intuit.com
Friday, February 06, 2015 7:10 AM
One of the big benefits is speed.

Knife ssh uses a parallel ssh method. Chef push jobs use a message bus
( zeromq I believe ).

Here's a video that demonstrates the push job feature:

https://m.youtube.com/watch?v=yHub6E4DNvg

I also believe that push jobs are the way of the future over knife ssh.

sh

Eric Horne mailto:eric+opscode@box11.org
Friday, February 06, 2015 5:34 AM
What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly controlled
deployments to orchestrate across several different systems. The
traditional "pull" doesn't fit the bill well because it is difficult
to get that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon
running on the remote servers that allow arbitrary (but
pre-defined/whitelisted) execution of commands. Aside from perhaps a
cleaner white-listing concept (over forced-ssh), how is this different
(better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over
knife ssh. The documentation is lacking in terms of how it is intended
to be used. Are push jobs better suited for different situations (and
what are those situations?)

Thanks for the help!

-Eric

4

Hi Eric,

The reason your expectations aren't being met is that push jobs is
intended to be the building block (or primitive) on which more advanced
features are built. Therefore it's going to feel more bare bones than
you might otherwise expect. This is why it's open source and use cases
are still being explored.

That said, and others will have to correct me if I'm wrong, as I'm not
familiar with all of the push job capabilities, I believe you can do
things like define groups, so then you can push to just certain machines
by specifying the group. As far as running a single recipe, chef-apply
gives you this capability, so you could trigger this with knife ssh or
pushy to achieve that result.

Does that explanation help?

On your side note: yes, Chef server does use rabbitmq. Push Jobs uses
zeromq instead, as I believe the desire was for a lighter weight message
bus with less overhead than rabbitmq.

  • Mark Mzyk

Eric Horne mailto:eric+opscode@box11.org
February 6, 2015 at 10:45 AM
Thanks Steven (and Hi!) I saw the video, but there isn't a comparison
to ssh that I remember (maybe I should watch it again). I was sold on
pushy being the next awesomeness, but now that I'm playing with it...
it just looks like another ssh client (yes I know it's not using ssh
at all) with some additional easy-to-use controls and reporting.

The video appears aimed at showing that the technology works as
opposed to presenting use cases in which it is superior to other
solutions. When is pushy better than knife ssh or an ssh call
integrated into a recipe?

My expectations for pushy were a little more lofty, I guess. I was
hoping for more integrated orchestration, the ability to run a single
recipe on a system as opposed to just running chef-client to get
everything (so I guess something more like ansible, but integrated
into Chef). Something like: "push out THIS change" as opposed to "sync
all changes now".

So.. all that said -- this isn't intended to be a critque of push; I
honestly think I'm missing something somewhere along the line.

(on a side note, doesn't Chef already use rabbitmq?)

-Eric

Hajducko, Steven mailto:Steven_Hajducko@intuit.com
February 6, 2015 at 10:10 AM
One of the big benefits is speed.

Knife ssh uses a parallel ssh method. Chef push jobs use a message bus
( zeromq I believe ).

Here's a video that demonstrates the push job feature:

https://m.youtube.com/watch?v=yHub6E4DNvg

I also believe that push jobs are the way of the future over knife ssh.

sh

Eric Horne mailto:eric+opscode@box11.org
February 6, 2015 at 8:34 AM
What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly controlled
deployments to orchestrate across several different systems. The
traditional "pull" doesn't fit the bill well because it is difficult
to get that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon
running on the remote servers that allow arbitrary (but
pre-defined/whitelisted) execution of commands. Aside from perhaps a
cleaner white-listing concept (over forced-ssh), how is this different
(better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over
knife ssh. The documentation is lacking in terms of how it is intended
to be used. Are push jobs better suited for different situations (and
what are those situations?)

Thanks for the help!

-Eric

5

Hi Mark. That explains a lot. Thank you for that perspective.

-Eric

Mark Mzyk mailto:mmzyk@chef.io
Friday, February 06, 2015 7:49 AM
Hi Eric,

The reason your expectations aren't being met is that push jobs is
intended to be the building block (or primitive) on which more
advanced features are built. Therefore it's going to feel more bare
bones than you might otherwise expect. This is why it's open source
and use cases are still being explored.

That said, and others will have to correct me if I'm wrong, as I'm not
familiar with all of the push job capabilities, I believe you can do
things like define groups, so then you can push to just certain
machines by specifying the group. As far as running a single recipe,
chef-apply gives you this capability, so you could trigger this with
knife ssh or pushy to achieve that result.

Does that explanation help?

On your side note: yes, Chef server does use rabbitmq. Push Jobs uses
zeromq instead, as I believe the desire was for a lighter weight
message bus with less overhead than rabbitmq.

  • Mark Mzyk
    Eric Horne mailto:eric+opscode@box11.org
    Friday, February 06, 2015 7:45 AM
    Thanks Steven (and Hi!) I saw the video, but there isn't a comparison
    to ssh that I remember (maybe I should watch it again). I was sold on
    pushy being the next awesomeness, but now that I'm playing with it...
    it just looks like another ssh client (yes I know it's not using ssh
    at all) with some additional easy-to-use controls and reporting.

The video appears aimed at showing that the technology works as
opposed to presenting use cases in which it is superior to other
solutions. When is pushy better than knife ssh or an ssh call
integrated into a recipe?

My expectations for pushy were a little more lofty, I guess. I was
hoping for more integrated orchestration, the ability to run a single
recipe on a system as opposed to just running chef-client to get
everything (so I guess something more like ansible, but integrated
into Chef). Something like: "push out THIS change" as opposed to "sync
all changes now".

So.. all that said -- this isn't intended to be a critque of push; I
honestly think I'm missing something somewhere along the line.

(on a side note, doesn't Chef already use rabbitmq?)

-Eric

Hajducko, Steven mailto:Steven_Hajducko@intuit.com
Friday, February 06, 2015 7:10 AM
One of the big benefits is speed.

Knife ssh uses a parallel ssh method. Chef push jobs use a message bus
( zeromq I believe ).

Here's a video that demonstrates the push job feature:

https://m.youtube.com/watch?v=yHub6E4DNvg

I also believe that push jobs are the way of the future over knife ssh.

sh

Eric Horne mailto:eric+opscode@box11.org
Friday, February 06, 2015 5:34 AM
What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly controlled
deployments to orchestrate across several different systems. The
traditional "pull" doesn't fit the bill well because it is difficult
to get that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon
running on the remote servers that allow arbitrary (but
pre-defined/whitelisted) execution of commands. Aside from perhaps a
cleaner white-listing concept (over forced-ssh), how is this different
(better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over
knife ssh. The documentation is lacking in terms of how it is intended
to be used. Are push jobs better suited for different situations (and
what are those situations?)

Thanks for the help!

-Eric

6

On 2/6/15 5:34 AM, Eric Horne wrote:

What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly controlled
deployments to orchestrate across several different systems. The
traditional "pull" doesn't fit the bill well because it is difficult
to get that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon
running on the remote servers that allow arbitrary (but
pre-defined/whitelisted) execution of commands. Aside from perhaps a
cleaner white-listing concept (over forced-ssh), how is this different
(better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over
knife ssh. The documentation is lacking in terms of how it is intended
to be used. Are push jobs better suited for different situations (and
what are those situations?)

Thanks for the help!

-Eric
Scaling is a big problem with ssh. Doing jobs over ssh really buckles
when you start hitting a thousand servers as the target for one job.
The protocol is slow and cpu hungry, at scale it can take hours to hit
all the servers just from the ssh client overhead on the central box,
which leads to having to build fan out to distribute ssh connections
over multiple source servers. Its also unreliable. You have to wrap it
with your own failure checks and timeouts and then sometimes it just
fails for no reason because its designed as an interactive login
protocol first and foremost and its reliability as an "RPC" mechanism is
poor -- so you need to detect that and retry individual failed hosts or
else just re-run the job you're doing multiple times. The way that ssh
trust test to grow in an organization also tends to result in really bad
security holes. I've seen bidirectional full meshes of ssh trust
constructed across 25,000 servers so that any compromise on any one
system would lead to login access on all the other servers (at a company
with otherwise really good security -- but ssh trust was way too 'easy'
and 'useful').

At a small scale of 400 servers or so it can work fine, you don't see
the anomalous failures often enough, the runs are short enough (and
probably faster with current horsepower and lots of cores), and its
generally contained enough that you can stay on top of the security
issues. Scale it out, though, and you'll eventually find where its
really not designed to do that. And 'knife ssh' would need a bunch more
work to make it more reliable (I don't know of any other tool that wraps
ssh that does it any better, though, since mostly people find that
inflection point where ssh starts to be a really poor tool for the job
and then ditch the protocol).

And that's also having said that I don't know enough about what we've
built to comment on what layers we've added on top. The ability to set
a job to run and report back success if a quorum of servers succeed (but
failure if not enough servers succeed) is a vital higher layer that
knife ssh certainly doesn't provide and becomes important when you can
just about guarantee that one or two of your servers may be down, but
that's fine (if you're pushing software to 2,500 targets all at once,
you'll find that linux itself often just isn't reliable enough to
guarantee that they're all up), yet you want to know if half your
deployments fail because that's really bad.

7

I had the same lofty expectation on push jobs, only to realize that Mark says the truth, that is, Chef push jobs functionality is at its primitive stage.

Chris

From: Mark Mzyk [mailto:mmzyk@chef.io]
Sent: Friday, February 06, 2015 10:49 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Push jobs vs SSH

Hi Eric,

The reason your expectations aren’t being met is that push jobs is intended to be the building block (or primitive) on which more advanced features are built. Therefore it’s going to feel more bare bones than you might otherwise expect. This is why it’s open source and use cases are still being explored.

That said, and others will have to correct me if I’m wrong, as I’m not familiar with all of the push job capabilities, I believe you can do things like define groups, so then you can push to just certain machines by specifying the group. As far as running a single recipe, chef-apply gives you this capability, so you could trigger this with knife ssh or pushy to achieve that result.

Does that explanation help?

On your side note: yes, Chef server does use rabbitmq. Push Jobs uses zeromq instead, as I believe the desire was for a lighter weight message bus with less overhead than rabbitmq.

  • Mark Mzyk

[cid:image001.jpg@01D0422D.1AEB1580]
Eric Hornemailto:eric+opscode@box11.org
February 6, 2015 at 10:45 AM
Thanks Steven (and Hi!) I saw the video, but there isn’t a comparison to ssh that I remember (maybe I should watch it again). I was sold on pushy being the next awesomeness, but now that I’m playing with it… it just looks like another ssh client (yes I know it’s not using ssh at all) with some additional easy-to-use controls and reporting.

The video appears aimed at showing that the technology works as opposed to presenting use cases in which it is superior to other solutions. When is pushy better than knife ssh or an ssh call integrated into a recipe?

My expectations for pushy were a little more lofty, I guess. I was hoping for more integrated orchestration, the ability to run a single recipe on a system as opposed to just running chef-client to get everything (so I guess something more like ansible, but integrated into Chef). Something like: “push out THIS change” as opposed to “sync all changes now”.

So… all that said – this isn’t intended to be a critque of push; I honestly think I’m missing something somewhere along the line.

(on a side note, doesn’t Chef already use rabbitmq?)

-Eric
[cid:image002.jpg@01D0422D.1AEB1580]
Hajducko, Stevenmailto:Steven_Hajducko@intuit.com
February 6, 2015 at 10:10 AM
One of the big benefits is speed.

Knife ssh uses a parallel ssh method. Chef push jobs use a message bus ( zeromq I believe ).

Here’s a video that demonstrates the push job feature:

I also believe that push jobs are the way of the future over knife ssh.

sh
[cid:image001.jpg@01D0422D.1AEB1580]
Eric Hornemailto:eric+opscode@box11.org
February 6, 2015 at 8:34 AM
What’s the difference between chef push jobs and knife ssh?

I’m researching a use case for chef in which we want highly controlled deployments to orchestrate across several different systems. The traditional “pull” doesn’t fit the bill well because it is difficult to get that to coordinate properly across systems.

From what I’ve read so far, chef push jobs are essentially a daemon running on the remote servers that allow arbitrary (but pre-defined/whitelisted) execution of commands. Aside from perhaps a cleaner white-listing concept (over forced-ssh), how is this different (better) than just using knife ssh?

I’m failing to see the benefits or use cases of chef push jobs over knife ssh. The documentation is lacking in terms of how it is intended to be used. Are push jobs better suited for different situations (and what are those situations?)

Thanks for the help!

-Eric

8

We tried it where I worked but gave up. We still have some x86 systems, and
Windows boxes, and some Oracle Linux boxes without gcc/etc installed. And
we hit intermittent service starting issues on nodes that it could install
on, which led to annoyances when bootstrapping new vms. Eventually we
decided that our network was too heterogeneous to truly be useful, and the
bootstrap errors just weren't worth it.

We might reevaluate this down the line when you don't insist on using runit
(the main thing that needed compile tools IIRC), and support Windows
clients as well.
On Feb 6, 2015 1:51 PM, "Fouts, Chris" Chris.Fouts@sensus.com wrote:

I had the same lofty expectation on push jobs, only to realize that Mark
says the truth, that is, Chef push jobs functionality is at its primitive
stage.

Chris

From: Mark Mzyk [mailto:mmzyk@chef.io]
Sent: Friday, February 06, 2015 10:49 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Push jobs vs SSH

Hi Eric,

The reason your expectations aren't being met is that push jobs is
intended to be the building block (or primitive) on which more advanced
features are built. Therefore it's going to feel more bare bones than you
might otherwise expect. This is why it's open source and use cases are
still being explored.

That said, and others will have to correct me if I'm wrong, as I'm not
familiar with all of the push job capabilities, I believe you can do things
like define groups, so then you can push to just certain machines by
specifying the group. As far as running a single recipe, chef-apply gives
you this capability, so you could trigger this with knife ssh or pushy to
achieve that result.

Does that explanation help?

On your side note: yes, Chef server does use rabbitmq. Push Jobs uses
zeromq instead, as I believe the desire was for a lighter weight message
bus with less overhead than rabbitmq.

February 6, 2015 at 10:45 AM

Thanks Steven (and Hi!) I saw the video, but there isn't a comparison to
ssh that I remember (maybe I should watch it again). I was sold on pushy
being the next awesomeness, but now that I'm playing with it... it just
looks like another ssh client (yes I know it's not using ssh at all) with
some additional easy-to-use controls and reporting.

The video appears aimed at showing that the technology works as opposed to
presenting use cases in which it is superior to other solutions. When is
pushy better than knife ssh or an ssh call integrated into a recipe?

My expectations for pushy were a little more lofty, I guess. I was hoping
for more integrated orchestration, the ability to run a single recipe on a
system as opposed to just running chef-client to get everything (so I guess
something more like ansible, but integrated into Chef). Something like:
"push out THIS change" as opposed to "sync all changes now".

So.. all that said -- this isn't intended to be a critque of push; I
honestly think I'm missing something somewhere along the line.

(on a side note, doesn't Chef already use rabbitmq?)

-Eric

Hajducko, Steven Steven_Hajducko@intuit.com

February 6, 2015 at 10:10 AM

One of the big benefits is speed.

Knife ssh uses a parallel ssh method. Chef push jobs use a message bus (
zeromq I believe ).

Here's a video that demonstrates the push job feature:

https://m.youtube.com/watch?v=yHub6E4DNvg

I also believe that push jobs are the way of the future over knife ssh.

sh

Eric Horne eric+opscode@box11.org

February 6, 2015 at 8:34 AM

What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly controlled
deployments to orchestrate across several different systems. The
traditional "pull" doesn't fit the bill well because it is difficult to get
that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon
running on the remote servers that allow arbitrary (but
pre-defined/whitelisted) execution of commands. Aside from perhaps a
cleaner white-listing concept (over forced-ssh), how is this different
(better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over knife
ssh. The documentation is lacking in terms of how it is intended to be
used. Are push jobs better suited for different situations (and what are
those situations?)

Thanks for the help!

-Eric

9

On Feb 6, 2015, at 9:45 AM, Eric Horne eric+opscode@box11.org wrote:

My expectations for pushy were a little more lofty, I guess. I was hoping for more integrated orchestration, the ability to run a single recipe on a system as opposed to just running chef-client to get everything (so I guess something more like ansible, but integrated into Chef). Something like: "push out THIS change" as opposed to "sync all changes now”.

So why not schedule a chef-client run with a specified recipe to be executed?

IMO, the problem with tools like Ansible is that they don’t force you to re-think your approach to solving problems, as you try to scale up. You still think of it as shell scripts and ssh to run those scripts.

From the perspective of managing scalable systems, you really do want to define what you want the overall system to look like, and then you need to let the system maintain itself. Don’t ssh out to a bunch of machines to run a shell script to edit files in place, because you’re already starting from an unknown state that just takes you one step further down the line of being further and further un-maintainable.

Instead, take files that are properly version controlled and push them out and replace whatever might be present that doesn’t match what’s in your known good copy of the file from your version control system. And if you do actually need to make changes to what configuration gets put on a given server, then make sure that you properly version control the file and the code that makes that happen.

Chef doesn’t completely and totally force you to work this way, but this is the easiest way to do things in Chef, and that’s a part of what makes it more scalable.

I’m getting involved in a Puppet job for a customer, and while there are some things that I am finding I like about being able to easily apply a puppet manifest locally to a vagrant VM, I am now starting to have some questions. Like, I have no idea whatsoever how to take what I’ve done with local manual operations that amount to “puppet module install xyzzy” and “puppet module install dependency” followed by “puppet apply xyzzy/manifests/init.pp” and take that to the next level of being able to automatically apply that process to nodes as they are created.

--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

10

Thank you for this. I didn't realize knife ssh doesn't scale well.

I'm familiar with ansible, it does a pretty nice job wrapping ssh (I
only mention this because you said you weren't familiar with another
tool that wrapped ssh and worked at scale). But I'm not managing 25,000
nodes either. :slight_smile:

-Eric

Lamont Granquist mailto:lamont@chef.io
Friday, February 06, 2015 9:18 AM

Scaling is a big problem with ssh. Doing jobs over ssh really buckles
when you start hitting a thousand servers as the target for one job.
The protocol is slow and cpu hungry, at scale it can take hours to hit
all the servers just from the ssh client overhead on the central box,
which leads to having to build fan out to distribute ssh connections
over multiple source servers. Its also unreliable. You have to wrap
it with your own failure checks and timeouts and then sometimes it
just fails for no reason because its designed as an interactive login
protocol first and foremost and its reliability as an "RPC" mechanism
is poor -- so you need to detect that and retry individual failed
hosts or else just re-run the job you're doing multiple times. The
way that ssh trust test to grow in an organization also tends to
result in really bad security holes. I've seen bidirectional full
meshes of ssh trust constructed across 25,000 servers so that any
compromise on any one system would lead to login access on all the
other servers (at a company with otherwise really good security -- but
ssh trust was way too 'easy' and 'useful').

At a small scale of 400 servers or so it can work fine, you don't see
the anomalous failures often enough, the runs are short enough (and
probably faster with current horsepower and lots of cores), and its
generally contained enough that you can stay on top of the security
issues. Scale it out, though, and you'll eventually find where its
really not designed to do that. And 'knife ssh' would need a bunch
more work to make it more reliable (I don't know of any other tool
that wraps ssh that does it any better, though, since mostly people
find that inflection point where ssh starts to be a really poor tool
for the job and then ditch the protocol).

And that's also having said that I don't know enough about what we've
built to comment on what layers we've added on top. The ability to
set a job to run and report back success if a quorum of servers
succeed (but failure if not enough servers succeed) is a vital higher
layer that knife ssh certainly doesn't provide and becomes important
when you can just about guarantee that one or two of your servers may
be down, but that's fine (if you're pushing software to 2,500 targets
all at once, you'll find that linux itself often just isn't reliable
enough to guarantee that they're all up), yet you want to know if half
your deployments fail because that's really bad.

Eric Horne mailto:eric+opscode@box11.org
Friday, February 06, 2015 5:34 AM
What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly controlled
deployments to orchestrate across several different systems. The
traditional "pull" doesn't fit the bill well because it is difficult
to get that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon
running on the remote servers that allow arbitrary (but
pre-defined/whitelisted) execution of commands. Aside from perhaps a
cleaner white-listing concept (over forced-ssh), how is this different
(better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over
knife ssh. The documentation is lacking in terms of how it is intended
to be used. Are push jobs better suited for different situations (and
what are those situations?)

Thanks for the help!

-Eric

11

How do you schedule a chef-client run without logging into the box to
schedule it (.. because if you have to log into the box, you may as well
just run it yourself at that point).

The thought around using knife ssh was to launch chef-client with it..
not to abandon chef altogether with scripts.

A question remains, though. How do people in the real world use chef
push to orchestrate a change among different servers with different roles?

Slightly off topic here.. My thinking right now is to control releases
with attributes assigned to either nodes or environments and node tags.
This way chef-client can just run and install what is desired for the
node and still support the A/B testing we do in production. Hosts
foo001-010 get version 2.5 and foo010-019 get 2.6. Push! The only
problem there is setting the variables for each node. hmm.. I wonder if
I could use a databag and a version matrix. anyway.. back on topic.

-Eric

Brad Knowles mailto:brad@shub-internet.org
Friday, February 06, 2015 2:54 PM

So why not schedule a chef-client run with a specified recipe to be
executed?

IMO, the problem with tools like Ansible is that they don’t force you
to re-think your approach to solving problems, as you try to scale up.
You still think of it as shell scripts and ssh to run those scripts.

From the perspective of managing scalable systems, you really do want
to define what you want the overall system to look like, and then you
need to let the system maintain itself. Don’t ssh out to a bunch of
machines to run a shell script to edit files in place, because you’re
already starting from an unknown state that just takes you one step
further down the line of being further and further un-maintainable.

Instead, take files that are properly version controlled and push them
out and replace whatever might be present that doesn’t match what’s in
your known good copy of the file from your version control system. And
if you do actually need to make changes to what configuration gets put
on a given server, then make sure that you properly version control
the file and the code that makes that happen.

Chef doesn’t completely and totally force you to work this way, but
this is the easiest way to do things in Chef, and that’s a part of
what makes it more scalable.

I’m getting involved in a Puppet job for a customer, and while there
are some things that I am finding I like about being able to easily
apply a puppet manifest locally to a vagrant VM, I am now starting to
have some questions. Like, I have no idea whatsoever how to take what
I’ve done with local manual operations that amount to “puppet module
install xyzzy” and “puppet module install dependency” followed by
“puppet apply xyzzy/manifests/init.pp” and take that to the next level
of being able to automatically apply that process to nodes as they are
created.

Eric Horne mailto:eric+opscode@box11.org
Friday, February 06, 2015 7:45 AM
Thanks Steven (and Hi!) I saw the video, but there isn't a comparison
to ssh that I remember (maybe I should watch it again). I was sold on
pushy being the next awesomeness, but now that I'm playing with it...
it just looks like another ssh client (yes I know it's not using ssh
at all) with some additional easy-to-use controls and reporting.

The video appears aimed at showing that the technology works as
opposed to presenting use cases in which it is superior to other
solutions. When is pushy better than knife ssh or an ssh call
integrated into a recipe?

My expectations for pushy were a little more lofty, I guess. I was
hoping for more integrated orchestration, the ability to run a single
recipe on a system as opposed to just running chef-client to get
everything (so I guess something more like ansible, but integrated
into Chef). Something like: "push out THIS change" as opposed to "sync
all changes now".

So.. all that said -- this isn't intended to be a critque of push; I
honestly think I'm missing something somewhere along the line.

(on a side note, doesn't Chef already use rabbitmq?)

-Eric

Hajducko, Steven mailto:Steven_Hajducko@intuit.com
Friday, February 06, 2015 7:10 AM
One of the big benefits is speed.

Knife ssh uses a parallel ssh method. Chef push jobs use a message bus
( zeromq I believe ).

Here's a video that demonstrates the push job feature:

https://m.youtube.com/watch?v=yHub6E4DNvg

I also believe that push jobs are the way of the future over knife ssh.

sh

Eric Horne mailto:eric+opscode@box11.org
Friday, February 06, 2015 5:34 AM
What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly controlled
deployments to orchestrate across several different systems. The
traditional "pull" doesn't fit the bill well because it is difficult
to get that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon
running on the remote servers that allow arbitrary (but
pre-defined/whitelisted) execution of commands. Aside from perhaps a
cleaner white-listing concept (over forced-ssh), how is this different
(better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over
knife ssh. The documentation is lacking in terms of how it is intended
to be used. Are push jobs better suited for different situations (and
what are those situations?)

Thanks for the help!

-Eric

12

We use rundeck for this.

Sent from my iPhone

On Feb 7, 2015, at 9:06 AM, Eric Horne eric+opscode@box11.org wrote:

How do you schedule a chef-client run without logging into the box to schedule it (.. because if you have to log into the box, you may as well just run it yourself at that point).

The thought around using knife ssh was to launch chef-client with it.. not to abandon chef altogether with scripts.

A question remains, though. How do people in the real world use chef push to orchestrate a change among different servers with different roles?

Slightly off topic here.. My thinking right now is to control releases with attributes assigned to either nodes or environments and node tags. This way chef-client can just run and install what is desired for the node and still support the A/B testing we do in production. Hosts foo001-010 get version 2.5 and foo010-019 get 2.6. Push! The only problem there is setting the variables for each node. hmm.. I wonder if I could use a databag and a version matrix. anyway.. back on topic.

-Eric

<postbox-contact.jpg> Brad Knowles Friday, February 06, 2015 2:54 PM

So why not schedule a chef-client run with a specified recipe to be executed?

IMO, the problem with tools like Ansible is that they don’t force you to re-think your approach to solving problems, as you try to scale up. You still think of it as shell scripts and ssh to run those scripts.

From the perspective of managing scalable systems, you really do want to define what you want the overall system to look like, and then you need to let the system maintain itself. Don’t ssh out to a bunch of machines to run a shell script to edit files in place, because you’re already starting from an unknown state that just takes you one step further down the line of being further and further un-maintainable.

Instead, take files that are properly version controlled and push them out and replace whatever might be present that doesn’t match what’s in your known good copy of the file from your version control system. And if you do actually need to make changes to what configuration gets put on a given server, then make sure that you properly version control the file and the code that makes that happen.

Chef doesn’t completely and totally force you to work this way, but this is the easiest way to do things in Chef, and that’s a part of what makes it more scalable.

I’m getting involved in a Puppet job for a customer, and while there are some things that I am finding I like about being able to easily apply a puppet manifest locally to a vagrant VM, I am now starting to have some questions. Like, I have no idea whatsoever how to take what I’ve done with local manual operations that amount to “puppet module install xyzzy” and “puppet module install dependency” followed by “puppet apply xyzzy/manifests/init.pp” and take that to the next level of being able to automatically apply that process to nodes as they are created.

<compose-unknown-contact.jpg> Eric Horne Friday, February 06, 2015 7:45 AM
Thanks Steven (and Hi!) I saw the video, but there isn't a comparison to ssh that I remember (maybe I should watch it again). I was sold on pushy being the next awesomeness, but now that I'm playing with it... it just looks like another ssh client (yes I know it's not using ssh at all) with some additional easy-to-use controls and reporting.

The video appears aimed at showing that the technology works as opposed to presenting use cases in which it is superior to other solutions. When is pushy better than knife ssh or an ssh call integrated into a recipe?

My expectations for pushy were a little more lofty, I guess. I was hoping for more integrated orchestration, the ability to run a single recipe on a system as opposed to just running chef-client to get everything (so I guess something more like ansible, but integrated into Chef). Something like: "push out THIS change" as opposed to "sync all changes now".

So.. all that said -- this isn't intended to be a critque of push; I honestly think I'm missing something somewhere along the line.

(on a side note, doesn't Chef already use rabbitmq?)

-Eric

<postbox-contact.jpg> Hajducko, Steven Friday, February 06, 2015 7:10 AM
One of the big benefits is speed.

Knife ssh uses a parallel ssh method. Chef push jobs use a message bus ( zeromq I believe ).

Here's a video that demonstrates the push job feature:

https://m.youtube.com/watch?v=yHub6E4DNvg

I also believe that push jobs are the way of the future over knife ssh.

sh

<compose-unknown-contact.jpg> Eric Horne Friday, February 06, 2015 5:34 AM
What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly controlled deployments to orchestrate across several different systems. The traditional "pull" doesn't fit the bill well because it is difficult to get that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon running on the remote servers that allow arbitrary (but pre-defined/whitelisted) execution of commands. Aside from perhaps a cleaner white-listing concept (over forced-ssh), how is this different (better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over knife ssh. The documentation is lacking in terms of how it is intended to be used. Are push jobs better suited for different situations (and what are those situations?)

Thanks for the help!

-Eric

13

Yep, I'm familiar with Ansible :slight_smile:

On 2/7/15 5:47 AM, Eric Horne wrote:

Thank you for this. I didn't realize knife ssh doesn't scale well.

I'm familiar with ansible, it does a pretty nice job wrapping ssh (I
only mention this because you said you weren't familiar with another
tool that wrapped ssh and worked at scale). But I'm not managing
25,000 nodes either. :slight_smile:

-Eric

Lamont Granquist mailto:lamont@chef.io
Friday, February 06, 2015 9:18 AM

Scaling is a big problem with ssh. Doing jobs over ssh really
buckles when you start hitting a thousand servers as the target for
one job. The protocol is slow and cpu hungry, at scale it can take
hours to hit all the servers just from the ssh client overhead on the
central box, which leads to having to build fan out to distribute ssh
connections over multiple source servers. Its also unreliable. You
have to wrap it with your own failure checks and timeouts and then
sometimes it just fails for no reason because its designed as an
interactive login protocol first and foremost and its reliability as
an "RPC" mechanism is poor -- so you need to detect that and retry
individual failed hosts or else just re-run the job you're doing
multiple times. The way that ssh trust test to grow in an
organization also tends to result in really bad security holes. I've
seen bidirectional full meshes of ssh trust constructed across 25,000
servers so that any compromise on any one system would lead to login
access on all the other servers (at a company with otherwise really
good security -- but ssh trust was way too 'easy' and 'useful').

At a small scale of 400 servers or so it can work fine, you don't see
the anomalous failures often enough, the runs are short enough (and
probably faster with current horsepower and lots of cores), and its
generally contained enough that you can stay on top of the security
issues. Scale it out, though, and you'll eventually find where its
really not designed to do that. And 'knife ssh' would need a bunch
more work to make it more reliable (I don't know of any other tool
that wraps ssh that does it any better, though, since mostly people
find that inflection point where ssh starts to be a really poor tool
for the job and then ditch the protocol).

And that's also having said that I don't know enough about what we've
built to comment on what layers we've added on top. The ability to
set a job to run and report back success if a quorum of servers
succeed (but failure if not enough servers succeed) is a vital higher
layer that knife ssh certainly doesn't provide and becomes important
when you can just about guarantee that one or two of your servers may
be down, but that's fine (if you're pushing software to 2,500 targets
all at once, you'll find that linux itself often just isn't reliable
enough to guarantee that they're all up), yet you want to know if
half your deployments fail because that's really bad.

Eric Horne mailto:eric+opscode@box11.org
Friday, February 06, 2015 5:34 AM
What's the difference between chef push jobs and knife ssh?

I'm researching a use case for chef in which we want highly
controlled deployments to orchestrate across several different
systems. The traditional "pull" doesn't fit the bill well because it
is difficult to get that to coordinate properly across systems.

From what I've read so far, chef push jobs are essentially a daemon
running on the remote servers that allow arbitrary (but
pre-defined/whitelisted) execution of commands. Aside from perhaps a
cleaner white-listing concept (over forced-ssh), how is this
different (better) than just using knife ssh?

I'm failing to see the benefits or use cases of chef push jobs over
knife ssh. The documentation is lacking in terms of how it is
intended to be used. Are push jobs better suited for different
situations (and what are those situations?)

Thanks for the help!

-Eric