We’re running Chef 11 here, and we have environment restrictions in place so that new cookbook versions don’t get rolled to production outside of our Jenkins pipeline.
The problem is that we didn’t anticipate people playing dirty pool and (intentionally or not uploading new cookbook recipe versions WITHOUT bumping the metadata.rb version number.
Does anyone have a solution in place that handles this? We’re thinking about some combination of Github hooks and Jenkins jobs but I wanted to see what solutions others have built before rolling our own.
Before I go into how I fixed this I’ll say that although it’s definitely a bad practice the negative effect isn’t a bad as you’d initially think as a node won’t pull in that changed code if it’s already running the existing version, i.e. if cookbook version 1.2.3 is uploaded and a node updates itself to 1.2.3, if someone overwrites 1.2.3 on the Chef server any nodes already at 1.2.3 own’t change. So having a reasonably frequent chef-client run schedule at least mitigates some of the damage short term.
My solution to this problem was to use Jenkins not just as the way to control environment locks but also to deploy cookbooks to Chef server. I wrote a generic build script for testing cookbooks (i.e. check syntax -> foodcritic -> chefspec -> serverspec) and if all tests passed then the new version would upload to the Chef server. One of the first checks in the build script was to check that the current version being tested did not already exist on the Chef server.
There are plugins for Jenkins that allow you to deploy Chef credentials during the build, so you can use a specific Jenkins user/client. If you’re still finding you have troubles you can alter the permissions on the Chef server so that only the Jenkins user/client has permission to modify cookbooks.
Here’s the script for reference. It’s nothing fancy and note that if you use the environment cookbook pattern you’ll need a separate build script for those to handle Berksfile.lock.
If you have a mono-repo you’ll probably have to test all cookbooks as a single unit and batch upload all new versions at once.
set -e
JOB_STATUS=1
BERKS_BIN='/opt/chefdk/bin/berks'
CHEFSPEC_BIN='/opt/chefdk/embedded/bin/rspec'
FOODCRITIC_BIN='/opt/chefdk/bin/foodcritic'
KITCHEN_BIN='/opt/chefdk/bin/kitchen'
KNIFE_BIN='/opt/chefdk/bin/knife'
banner() {
content=`echo $1 | tr '[:lower:]' '[:upper:]'`
echo "########################################################################"
echo "${content}"
echo "########################################################################"
}
# TODO: use rubocop
function check_ruby_syntax {
SUCCESS=0
set +e
for file in `find . -type f -name '*.rb'`
do
ruby -c $file > /dev/null
if [ $? -ne 0 ]; then
SUCCESS=1
fi
done
set -e
return $SUCCESS
}
finish() {
banner 'cleaning up'
$KITCHEN_BIN destroy
if [ $JOB_STATUS -eq 0 ]; then
RESULT='success'
else
RESULT='failure'
fi
banner "JOB RESULT: ${RESULT}"
exit $JOB_STATUS
}
trap finish EXIT
COOKBOOK_NAME=`cat metadata.rb | grep ^name | awk '{print $2}' | sed "s/'//g"`
COOKBOOK_VERSION=`cat metadata.rb | grep ^version | awk '{print $2}' | sed "s/'//g"`
banner "begin cookbook build job"
echo "Cookbook: ${COOKBOOK_NAME}"
echo "Version: ${COOKBOOK_VERSION}"
echo "Fetching git tags"
git fetch --tags
if git tag | grep "${COOKBOOK_VERSION}"; then
echo "ERROR: Tag for this version already exists in git"
exit 3
else
echo "No tag found for this version in github"
fi
set +e
SERVER_COOKBOOK=`${KNIFE_BIN} cookbook show ${COOKBOOK_NAME}`
KNIFE_EXIT=$?
if [ $KNIFE_EXIT -eq 0 ]; then
SERVER_COOKBOOK_VERSION=`echo ${SERVER_COOKBOOK} | awk '{print $2}'`
if [ "${SERVER_COOKBOOK_VERSION}" = "${COOKBOOK_VERSION}" ]; then
echo "ERROR: Current cookbook version exists on chef server, metadata was not incremented"
exit 2
else
echo 'Found older versions of cookbook on chef server'
fi
elif [ $KNIFE_EXIT -eq 100 ]; then
echo "Cookbook does not exist on chef server"
else
echo "ERROR: Knife returned an error when checking cookbook version on chef server"
exit $KNIFE_EXIT
fi
set -e
echo "Obtaining latest data from chef server"
rm -rf test/shared/chef
mkdir -p test/shared/chef
$KNIFE_BIN download --chef-repo-path test/shared/chef /roles
$KNIFE_BIN download --chef-repo-path test/shared/chef /data_bags
banner 'checking ruby syntax of components'
check_ruby_syntax
banner 'installing cookbook dependencies (BERKSHELF)'
rm -f Berksfile.lock
$BERKS_BIN install
banner 'running linting tool (foodcritic)'
$FOODCRITIC_BIN -f any -X spec -X test .
banner 'running unit tests (chefspec)'
$CHEFSPEC_BIN --color
banner 'running integration tests (test kitchen + serverspec)'
set +e
$KITCHEN_BIN destroy
rm -rf .kitchen
set -e
$KITCHEN_BIN test
banner 'uploading new version of cookbook and dependencies to chef server'
$BERKS_BIN upload
banner 'creating git tag'
git tag ${COOKBOOK_VERSION}
git push origin --tags
JOB_STATUS=0
This isn't the case. On each run, Chef Client pulls down the list of files and their checksums, and will update any files that are different.
This is the optimal way to do it.
Though Chef Delivery is a proprietary product, much of the work it does is handled by cookbooks, many of which are open source. The delivery-truck and delivery-sugar cookbooks have a means of detecting which cookbooks changes by looking at git and then ensuring the version number was bumped if anything changed. See:
I think you have a people/management problem, not a technical problem. I’ve seen these types of things happen quite frequently - sometimes pushed by management, and with disastrous consequences, sometimes for very good reason (somebody may need to deploy a critical bug fix or address a major security concern quickly), and sometimes simple innocent mistakes.
When people circumvent measures you have put in place, you have to look at the root cause - otherwise, all you are going to do is aggravate people, and cause them to circumvent anything else you put in place, too.
You have put a certain policy in place and enforce it through technical means. When people circumvent that, you have to look at a few things:
Are these innocent mistakes? In that case, a gatekeeper may be your solution. Don’t use your development chef server as a production chef server.
Why do people circumvent it? They have a pressing business need that your existing policy does not account for. You may have to relax your policies, or come up with other mechanisms to allow people what they need to do.
How does management feel about this? I found that quite frequently, these types of breaches of practices originate with management. If management pushes for such breaches, and does so consistently, there is really nothing much you can do. The worst case scenario is that management may start perceiving chef or the devops approach as part of the problem holding the business back.
There is a --freeze flag that you could use when uploading with knife. If
used, you’ll need to use --force to override a version.
Berkshelf sets this automatically iirc.
I stand corrected then, however I always found I had to manually delete the cookbook cache when trying to pull code down of the same version. Perhaps it was another reason.
You may want to check your cache_type and cache_options settings in knife.rb. I’m assuming there are similar settings in client.rb as well, although I didn’t see those.
It seems that those settings control whether or not you use checksums, although the documentation is fairly vague.
You might also want to check the system time and time zone. Those shouldn’t affect the check sums, but may affect which one “wins”.
uploading new cookbook recipe versions WITHOUT bumping the metadata.rb version number.