Chef-Client wait until successful ssh connection ruby block

[aravind]

Hello,

I need chef to wait until it has been able to successfully SSH into a remote box/node.

I know that I can use the following logic to determine whether I am able to SSH from Node A/user A to Node B/user B:

ssh -q userB@NodeB exit
If ? / Return code is 0 then ssh is successful Else IF ? / Return code is 255 then ssh is unsuccessful.

So I need to know how can I implement this in a resource/Ruby block within chef:

    Step1: Execute test SSH connection
    Step2: UNTIL (return code is 0)
                      wait 60 seconds
                      print a info log message to chef::log
                     GO TO Step1
           END
    Step3: Proceed to next chef resource in the recipe only when you come out of the above wait block, no matter how much time it takes

Thanks for your help

Regards,
Aravind

[aravind]

bump* (hope thats not against forum rules)

[Mark_Harrison]

That’s weird - I tried to find your topic yesterday and got a 404 when clicking view topic, but it’s showing up now.

What you want is something like this:

execute "Wait until SSH is up" do
   command "ssh -q userB@nodeB exit"
   retries 10
   retry_delay 60
   timeout 10
end

This will exit the chef run with an error if you can’t ssh after 10 times, and it will print to the log each time it tries.

[Rilindo_Foster]

Arvin,

It looks like the question was already answered, so if I may ask, what were you trying to accomplish with this? Does an application deploy depend on an event that occurred after you login into another box?

[aravind]

Hi Mark,

Thanks for the response.

I think I can tweak the retries option to a higher number and achieve a indefinite wait.

If you are wondering why I can’t afford to just error out, fix the problem and retry chef-client execution, its because I have quite a lot of resources “preceeding” this test SSH connection resource, which have to do with generating certificates, adding the certificate to the trust-store etc. (>100 resources) i.e. I cannot simply abort in the middle, fix the problem and retry because the keystores etc. would already have been created and cert import commands fail because a certificate already exists. Therefore, the only way I can do this is to wipe the VM and start clean - something I don’t want to waste lot of time on during my chef code development. (I know I can take VM snapshots, but wont go into that here for some other reasons)

Rilindo >> Yes, effectively I am installing 2 parts of an application on different VMs, and these 2 parts talk to each other over SSL/keystore/truststore on a bi-directional basis. Both these VMs are built by 2 different cookbooks. I need one cookbook, running on VM #2, to wait until it can successfully copy the root cert/public key/trust certs from VM #1 - for which I need to setup SSH keys in the background so that I can do a SFTP via Chef.

You may ask why I cant setup the SSH Keys via Chef itself i.e. import the public key from VM#2 into authorized_keys file of VM#1, but Chef itself is creating the application user account (OS) and therefore since passoword auth is disabled, I need to do this “enable SFTP via SSH” activity manually via a different OS user ID.

Thank you for the help.

[Tensibai]

I cannot simply abort in the middle, fix the problem and retry because the keystores etc. would already have been created and cert import commands fail because a certificate already exists.

Then your recipe is not idempotent (not testing if there’s something to do) and thus should be improved.

Keep in mind chef is designed to be run periodically and success at each run, enforcing your described state at each run.
If you wish to have a bunch of sequential commands, chef is not the right tool IMHO. If you wan to do a system configuration management, you MUST take into account it may have to be done in two or three pass sometimes, and write your recipes accordingly using guards/writing custom resources when really necessary, reviewing how you plan it at all.

Considering ssh keys, there’s no reason you can’t generate them before even creating the VMs, so you should be able to rethink it, generating each machine key, provisioning each machine and setting it up with the keys generated beforehand.

[aravind]

Then your recipe is not idempotent (not testing if there’s something to do) and thus should be improved.

You are absolutely right.

I think I am realizing that when I need to check something before I DO something within Chef, I run into having to know a bit of Ruby syntax. Trying to learn that now.

For example, BEFORE I tell Chef to run a execute block, I know that I can use the guard not_if to check for something.
e.g.

execute "unzip #{folder}/installer.zip -d #{node[:app][:installer_dir]}" do
  user node[:app][:user]
  group node[:app][:group]
  not_if { File.exists?("#{node[:app][:installer_dir]}/app") }
end

What I need to figure out is some examples and “possible options that I can use” for the not_if section.

For example, if I can run a bash command and grep for a particular exit code or output AS PART OF the not_if block, I can then determine if the cert already exists and if so, not execute the “import cert” execute block and that way make the “import cert” execute block idempotent.

Regards
Aravind

[brianhatfield]

In your example, it may be better to set that execute to action :nothing and then notify it from the place where you download
installer.zip, so that it will re-unzip files in the case where the zip
had changed. Your not_if approach will do nothing in that circumstance.
As an aside, a system package rather than a zip is going to be more
idempotent-while-supporting-interesting-changes approach.

[yoshiwaan]

So a couple of points on guards which might help you out:

1. There’s only_if as well as not_if, which is the reverse
2. You can execute a local shell command instead of ruby by omitting the
   curly braces in a guard, i.e. only_if ‘grep “myhost.example.com
   http://myhost.example.com” /etc/hosts’**
3. Looking for a specific exit code from a guard isn’t supported. Write to
   your guard to pass or fail

** Note that you are shelling out here, which is slower than using pure
ruby, so avoid it when you know how to do it in ruby. That being said if
you’re getting frustrated and just want something to work, or your guard is
getting too big then it’s an option.

Having looked at a few of your posts in the last few days I think a bit of
a mentality shift on your part would be useful. Think of the code you’re
writing in Chef as describing a state for your infrastructure, rather than
a series of chained ad-hoc tasks. As the author of the code you should be
aware of the dependencies of your resources and factor that into the code
so that all the cogs turn nicely, dependencies are created before they’re
are needed and your script could fail at any step and be re-run without
incident.
As mentioned earlier idempotency is also important to your code, and for
the sake of idempotency (and the speed related to idempotency checks) you
might want to spent a bit of time thinking about how you implement.
As you’re probably noticing Chef has a bit of a learning curve, it’s
definitely a technology that shines when you take a deep dive and learn as
much as you can about it (and ruby), but when you do the reward is that it
becomes quite easy to do pretty much anything.

[Tensibai]

To complement @yoshiwaan’s answer above, here is the Guard documentation, with some examples, as well as details on how they behave (return 0 to trigger the guard,etc).

According to your example, the ark cookbook could be of use too, taking care of getting a remote file, unzip it, do simlinks, etc.

You can have a look at it for example too.

[aravind]

Thanks All.

I think I am at that point of the Chef learning lifecycle where after getting to grip with key resources (bash, execute, template, remote_file etc.) used in a “vanilla” sort-of-way, now I am slowing moving into idempotency/Guards. I need to next further improve the cookbooks using notifications.

May I ask if any of you know of existing Sublime Text 3 snippets for all chef resources (with ALL possible options included) - I am finding writing my own snippets does cut down on the programming time quite a lot, in the absence of a dedicated IDE for Chef scripting.

I think as @yoshiwaan pointed out, it does require a bit of analyzing things in a different way - it feels almost as though Chef is intended to work in non-sequential way (which is NOT what I am used to with Bash scripting etc. where one thing leads to another methodically, usually).

If indeed Chef is meant to work non-sequentially and in sort of neural network way, I am also surprised to not find multi-threading capabilities where for example i should have to ability to spawn multiple resources (especially ones that take long time like downloading or unzipping something) and then make use of notifications to then trigger dependent resources. Sure, I suppose I could do this via a execute block and nohup & a command but then I lose the ability to track that PID’s exit status within Chef.

Interesting times, though, and definitely something to look forward to in the New Year.

[Tensibai]

it feels almost as though Chef is intended to work in non-sequential way

Absolutely not, chef will run (converge) resources in the order specified in the recipe (minus the compile vs converge times, but that’s another story and there’s a lot of pages on it), and the recipes in the order specified in the run list. You may get a more complex ‘expanded runlist’ with roles runlist and all include_recipe calls (search ‘wrapper cookbook’ for clues here)

What does really change between a bash script and chef is that Chef is meant to be run in a regular schedule and enforce the state described, the main difference is here, you describe a state more than a path to a desired state.

Because of this, you should never assume the state anything will be in when chef run, the goal is to end the run in the desired state, whatever was the starting state. This involve doing things in an idempotent manner, do something only if necessary, and if there’s nothing to do, don’t do anything (or for worst cases do a no-op thing, like executing something which will have no effect if the configuration is already ok, but will correct it if not)

I’ve the feeling you should take a day or two to go through https://learn.chef.io to get the hands on with this principal change and how to tackle things with Chef. As a rule of thumb, using script (bash or else) or execute resources is usually a sign you’re doing something wrong.

there’s side cases really precise where it’s unavoidable to use bash/script or execute, but 90% of time you can take advantage of existing cookbooks (for their LWRP/custom resources) or even base chef resources.

[yoshiwaan]

So as a bit of context as to WHY you should use resources other than
execute, instead of just “you’re doing it wrong”

Pretty much all of the core resources in Chef other than execute/script
have inbuilt logic to deal with current and desired state (and thus
idempotency) as well as clear interfaces (in the form of attributes to the
resource) for managing the resource.

When reading a recipe it’s a lot easier to see a file resource and know
that it manages a file, and see the parameters of that resource there in an
understandable format than to have grok an execute statement.
When the code converges the logic of those resources obtains the current
state of the resource and can work out any changes to make to bring it to
the desired state (which also allows notifications from one resource to
another on state change). It can also implement those changes. This is done
without shelling out (which slows down the Chef run) and without guards, as
the resource logic handles whether to take actions or not.

An execute/script resource is triggered by Chef but the action that takes
isn’t understood by the Chef run, thus the need for guards and the fact
execute resources are often specified with action :nothing and triggered
via notification.

So it’s always worth asking the question “can I do what this execute/script
resource using other Chef resources (including custom resources from
cookbooks)?” because if the answer is yes you’ll have a faster, cleaner
result with more clarity.

[Tensibai]

instead of just “you’re doing it wrong”

I apologize, if it gets quoted, it’s that it sounds rude.

My goal was to highlight the fact @yoshiwaan explained very well that you should think twice before using those kind of resources, but leaving the why as an exercise.

Sorry again if it was rude, it was not the intention.

[yoshiwaan]

Well, I didn’t write that because it sounded rude. More for someone new
coming along and reading through this thread and wondering why it’s
wrong, which is something I know I struggled with when I was learning.

[aravind]

I’ve the feeling you should take a day or two to go through https://learn.chef.io to get the hands on with this principal change and how to tackle things with Chef.

Thank you, yes I have been through this.

Also, for the benefit of new forum members/new chefers (is that the right term) who may not have done this already, I also suggest requesting training material from https://www.chef.io/contact/open-training/ - they send you a Box shared folder link which has very useful presentations on Basics and Intermediary topics of Chef which I found very useful. I felt its also a bit simpler to understand and structured in an easy to understand way, especially for newbies like me.

Keep cooking & sharing!

Aravind