How to prevent tests to be cached on remote targets?

#1

Hello,

we have implemented some controls where we check if a file exists on a remote machine and checks its content.
This file doesn't appear straight away after the machine starts up though.

control 'test-01' do
  describe file('/test-ok') do
    it('should exist') { should exist }
    its('content') { should eq "something" }
  end
end
end

If we execute this control right after the machine started up, it's going to fail. Ideally, we are looking for a way to retry the describe() block for a while until it succeeds (or fails if it's too long).

We tried the rspec/retry gem with this type of configuration:

it('should exist', retry: 60, retry_wait: 5) { should exist }

This works when running InSpec on the target machine itself, but when running on a remote target (using inspec exec --host a.b.c.d test-profile), it looks like the file existence / content is check once, then the result seems somehow "cached" and never rechecked. This makes the retry configuration to execute until the end and then to fail, although the file arrives in between.

We opened a similar issue for the http() resource but it looks like this is a behavior similar across other resources as well (it's the case for file and service, for what we tried.)

Is there any known solution for that problem, beside adding a huge sleep at the very beginning, or duplicating the existence of the resources tested externally before executing the InSpec profile?

#2

Hi,

I think this is a good occasion to use describe.one, which allows you to write tests where you have multiple complaint states but only one (or more) needs to be correct to pass. https://www.inspec.io/docs/reference/dsl_inspec/#advanced-concepts. There is only_if, which is also documented on that page, but I think describe.one is a bit neater here as you have two distinct cases.

If I'm understanding your requirement you have two states that you would consider compliant:

  • No file exists with that name, because the process which generates it has not run yet.
  • The file exists and has an expected content.

So something like the following should work.

describe.one do
  describe file( '/test-ok' ) do
    it { should exist }
    its('content') { should cmp 'something' }
  end

  describe file('/test-ok') do
    it { should_not exist }
  end
end

There is a case here that isn't covered when the process which creates the file can fail in a way that doesn't generate the test-ok file. If that could happen the above will show as a pass and you would need to find another resource to test which would test the state of the process.

Best regards,

Chris

#3

Hi Chris,

thanks for your reply!

tl;dr: I realized I asked this question a bit as a XY problem. :confused:
My real problem is that I'd like to be able to retry failing tests for a while, but it has already been reported on GitHub.

If I'm understanding your requirement you have two states that you would consider compliant:

  • No file exists with that name, because the process which generates it has not run yet.
  • The file exists and has an expected content.

I have these two states but I don't consider the first one compliant - what I would like to test in that case, is if a machine has been configured to initialize correctly, which can take ~30 seconds after the machine gets the network - or fails completely if there's a problem.
There's a laps of time between the moment the machine becomes reachable and the moment it gets the configuration (if it ever gets it.), which is fine and I just would like to wait a bit and retry the InSpec control.

The problem here is that when InSpec manages to connect during that laps of time, it sees the file is not there and then, however I try to turn the problem around, InSpec never forgets the file wasn't there.

I tried different type of retries (loops around the tests , rspec-retry, which is often mentioned when search the Internet for "inspec + retry", I tried to write a custom InSpec plugin to had core mechanism for retries), but I hit this behavior where InSpec, in remote mode only, executes a test once and then never retries it (apparently).

So, if you have:

require 'rspec/retry'

control 'something' do
  describe file('/foo/bar') do
    it('should exist', retry: 60, retry_wait: 5) { should exist }
    # or using `before` from rspec
  end
end

and executes this with inspec exec --host $REMOTE_HOST my-control.rb and the file is not there, the test will never pass, even if the file arrives during one of the retry loop.

#4

By default InSpec remembers each command result and when you repeat commands it just gives you the cached result. There's a --no-backend-cache option that turns this cache off.

#5

@stocksy Wow, I'm not sure how we missed that command ... and it looks like it fixes most of our retrying problems! :heart: :heart: :heart: :heart:

It seems it doesn't work for all the resources though? That definitely fixes it for the file resource, but I tried to have a service() resource as well (which was checking that the "initialization service" that is setting up the files was indeed running) and it didn't detect that that service went from "not running"é to "running".
Do you know if it's considered as a bug and/or missing implementation for some resources?

#6

It's not an outright bug because --no-backend-cache is a low-level option for the backend train connection - it's not designed to work at the level of resources.
The --no-backend-cache option does not affect the service resource because that resource keeps its own result cache.
It would be a valid suggestion to ask that the service resource does not try to cache data, and leaves caching to train

Edit: I filed https://github.com/inspec/inspec/issues/4831 - please comment or outright close that issue if you feel it's not right.

#7

@stocksy OK, understood, thanks for the ticket also, I'll link it back to the one we found for the http resource :+1: