I heard a rumor that InSpec now supports referencing Chef node attributes within InSpec test files run by Test Kitchen. But I can’t find any documentation on it.
I’m trying to write an InSpec test for a Chef cookbook. The test will be run by Test Kitchen. The cookbook sets an attribute foo to a certain value depending on environmental factors. I want my InSpec test to access the value of that foo attribute. But I can’t figure out how to access node attributes from within my InSpec test file.
This doesn’t work for me (although it worked in Serverspec test files):
foo = node['foo']
describe file(foo) do
. . .
When I do that, I get this error from InSpec:
./test/smoke/default/default_test.rb:14:in `load_with_context': undefined local
variable or method `node' for #<#<Class:0x000000036afb28>:0x000000036af420>
(NameError)
I’d appreciate any info on this. It seems like a basic capability to me. The fact that I’ve had such a hard time finding info about it makes me think I’m just not understanding something fundamental about how InSpec is supposed to be used to write integration tests for cookbooks.
Thank you!
This approach seems to work for me, for Linux Test Kitchen instances. But it seems so heavy-handed. Do any regular InSpec-using Chefs out there do anything remotely similar?
Example file COOKBOOK_NAME/test/smoke/default/default_test.rb:
cb = 'COOKBOOK_NAME' # This cookbook
#---
# Get node object
#---
nodes_dir = '/tmp/kitchen/nodes'
tk_instance = command("ls #{nodes_dir}").stdout.strip
node_json = "#{nodes_dir}/#{tk_instance}"
node = json(node_json).params
#---
# Get node attributes we need
#---
test_file = node['default'][cb]['test_file']
#---
# Run tests
#---
describe file(test_file) do
it { should be_file }
its('content') { should match(/Hello, World!/) }
end
That “hack” is how I have seen it done, I am personally torn on whether or not it should be more easily accessible.
On the one hand if (in|server)spec had access to node objects it would be easier to write dynamic tests with. On the other hand in many cases you lose the fact that these are integration tests and you want to test not only did you not break anything in your code but also that upgrading chef-client does not break your code. If you rely on chef to tell you chef is working then it’s not very reliable or useful test.
I recently had a particularly interesting scenario this year where in chef-client they made a change that blocked me from abusing immutable mashes in a way which I have used since chef 10 and was removed in chef 12.x (some minor version, can’t recall off the top of my head) there was no release notes letting me know that it was removed. As I had to write more static tests we discovered the issue, engaged chef engineering as we felt it was a regression, prevented an outage, and fixed our code.
Reading the node object directly in our tests would have masked the failure.
While I am decidedly in the same hesitant camp as @majormoses as to whether it’s a Good Thing to Do with regard to accessing node attributes in a test, we do have a slightly more official version of the hack in the Audit cookbook
Also the code above would only work in test-kitchen and not anywhere else so you definitely don’t want to use /tmp/kitchen/nodes. You just want to dump the node object directly in the recipe or via a helper like Audit does here.
Thank you for your comments, @majormoses and @cheeseplus. I appreciate your inputs. I’m slowing getting beter understanding about InSpec’s purpose and limitations as I work with it and engage in good dialog with help from folks like you.
While we’re on the topic, may I get your opinion on this: Is my basic assumption even sound that InSpec is intended to support two distinct types of verification, which for now (not knowing the real terms) I’ll call “testing” and “auditing”:
-
Testing: When you write functional tests of Chef cookbook recipes in InSpec, and execute them via Test Kitchen’s “kitchen verify” command, in order to test that your recipe does what you want under different sets of environmental conditions.
-
Auditing: When you run an InSpec-based Chef Compliance Profile on a given server to see that it meets certain criteria, without any pre-knowledge of how that server was actually configured. (Maybe the server is not even managed by Chef.)
I ask because the hesitancy you both express, and which I’ve seen elsewhere in the InSpec community, about exposing the node object within InSpec test files, makes perfect sense to me in the context of the “auditing” use case for InSpec above; but I don’t get it in the context of the “testing” use case.
For cookbooks that our users “wrap” in order to provide non-defafult cookbook attribute values, the cookbook attributes seem like “inputs” to me, and the cookbook recipe a “function.” Just like when I writes tests for code in any programming language, I want to verify that the “function” (i.e., the recipe) does what it should do when executed with different “inputs” (i.e., cookbook node attribute values). Is InSpec not really intended for this type of cookbook testing? If not, maybe I should stick with Serverspec for that.
Thanks for your time in reading this, sorry for the long wind.
You’re very much correct that InSpec supports two distinct but sometimes overlapping use cases and how you describe them is pretty close to how I would. When it comes to the question as to knowing about the node object in a testing sense that same pre-knowledge problem comes into play. Even before InSpec came along and we were using ServerSpec more broadly, we’ve always emphasized a notion somewhat akin to “black box” testing.
These “verifiers” should assert the state of the target node as it is, not how it got to be that way - I often refer to this akin to having the answers to your test beforehand. The problem with this is seemingly subtle but what happens is that you’re only testing things you already know should work and at worst introducing tests that are testing the config mgmt framework, not the system state itself. Additionally, the same tests you wrote for your cookbook in kitchen could be used ad-hoc directly against nodes.
To be clear, this advice remains constant for ServerSpec, InSpec, or any other testing framework. Applying the “hack” in ServerSpec is nigh on identical. If you are concerned more with the recipe logic itself, then a unit testing tool like ChefSpec should be used. I’d also highly advocate reading this post by @coderanger.
Is there anything inherently bad about knowing about the node object in a test?
Maybe
Is using the dumped node object potentially in tests potentially erroneous and/or brittle?
Very Possibly
Is it ok to dump the node object and use it in tests like many of our customers and users?
Definitely
Best practice, especially in this specific instance, is on a spectrum of idealism to what works for the end-user.