Strategies for debugging with dependency resolution issues?

Hello everyone,

As we’ve started to build up our cookbook library and the dependencies have grown, we’re running into a lot of dependency resolution issues. Since Chef has two dependency resolution mechanisms (Berkshelf and chef-client), we have two different errors.

We’ll have something like this happen when we do a chef-client run:

$ sudo chef-client
Starting Chef Client, version 12.4.3
resolving cookbooks for run list: ["base_cookbook::recipe", "cookbookB::recipe"]
 
================================================================================
Error Resolving Cookbooks for Run List:
================================================================================
 
Cookbook dependency resolution error:
-------------------------------------
Error message: unable to solve dependencies in alotted time
You might be able to resolve this issue with:
  1-) Removing cookbook versions that depend on deleted cookbooks.
  2-) Removing unused cookbook versions.
  3-) Pinning exact cookbook versions using environments.
 
Expanded Run List:
------------------
* base_cookbook::recipe
* cookbookB::recipe
 
 
Running handlers:
[2015-11-24T14:24:59-05:00] ERROR: Running exception handlers
Running handlers complete
[2015-11-24T14:24:59-05:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 11.843015546 seconds
[2015-11-24T14:24:59-05:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2015-11-24T14:24:59-05:00] ERROR: 412 "Precondition Failed"
[2015-11-24T14:24:59-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Given this node has only two items on the run list (a base cookbook and the node specific cookbook), we brought the base cookbook into cookbookB's metadata.rb and tried to do a Berkshelf resolution. That hung for a long time and comes back with:

$ berks update
Resolving cookbook dependencies...
Fetching 'cookbookB' from source at .
Fetching cookbook index from https://supermarket.chef.io...
Fetching cookbook index from http://local.supermarket...
The dependency constraints could not be solved in the time allotted.
Unable to find a solution for demands: test-helper-cookbook (~> 2.1), cookbookB (1.3.0)

In both scenarios, the error messages are not clear as to what’s causing the conflicting dependency resolution. The approach that we usually take in this case is to take a look at the Berksfile.lock in both cookbooks to try and find (by :eyes:) where the conflicts are, which is not ideal. In this particular case we actually haven’t found the issue yet.

We’ve tried bumping up the dependency resolution times on the Chef Server, but in these cases the dependencies are actually never resolved, and so chef converges just hung for 5 minutes. We’ve since reduced the time back to the default.

Am I missing something? Is there a trick that people have? Are there some other tools we can use to make this debugging easier?

P.S. I’ve read a bit about Policyfiles, do I understand correctly that this won’t happen there? At least we can catch these dependency resolution issues locally before they even make it to the server?

Thank you in advance!

Hi @amaltson,

I would recommend trying DEBUG_RESOLVER=1 berks install -d. Additionally, if you want to try a newer version of Berkshelf that includes this and this, you could try using the :ruby resolver instead of the :gecode one, and you might get some more information back to help you diagnose the problem.

You could also try moving the current Berksfile.lock so that Berkshelf doesn’t start its dependency resolution with some versions already locked.

Hope this helps,

Martin

1 Like

Hey @martinb3,

Thanks for recommendations, I just gave it a whirl. Unfortunately, the DEBUG_RESOLVER=1 flag doesn’t seem to provide any information on top of berks install -d. We still see the Starting resolution... and nothing really after that.

D, [2015-11-30T15:07:53.399562 #25156] DEBUG -- : Using 'chefignore' at '/Users/arthur/work/chef/cookbookB/chefignore'
D, [2015-11-30T15:07:53.426414 #25156] DEBUG -- :   Creating a resolver
Fetching cookbook index from https://supermarket.chef.io...
Fetching cookbook index from http://local.supermarket...
D, [2015-11-30T15:07:53.498954 #25156] DEBUG -- : ==> parsing Chef response body as JSON
D, [2015-11-30T15:07:59.373128 #25156] DEBUG -- : ==> parsing Chef response body as JSON
D, [2015-11-30T15:07:59.628007 #25156] DEBUG -- : Using 'chefignore' at '/Users/arthur/work/chef/cookbookB/chefignore'
D, [2015-11-30T15:07:59.638490 #25156] DEBUG -- :   Adding explicit dependency on cookbookB (1.9.0) '/Users/arthur/work/chef/cookbookB'
D, [2015-11-30T15:07:59.638630 #25156] DEBUG -- :   Starting resolution...
D, [2015-11-30T15:07:59.662137 #25156] DEBUG -- : Using 'chefignore' at '/Users/arthur/.berkshelf/cookbooks/elasticsearch-0.3.14/chefignore'
D, [2015-11-30T15:07:59.712579 #25156] DEBUG -- : Using 'chefignore' at '/Users/arthur/.berkshelf/cookbooks/helper_cookbook-2.1.1/chefignore'
D, [2015-11-30T15:07:59.717878 #25156] DEBUG -- : Using 'chefignore' at '/Users/arthur/.berkshelf/cookbooks/cookbookA-0.1.0/chefignore'
D, [2015-11-30T15:07:59.728012 #25156] DEBUG -- : Using 'chefignore' at '/Users/arthur/.berkshelf/cookbooks/internal_redis-1.0.3/chefignore'
The dependency constraints could not be solved in the time allotted.
Unable to find a solution for demands: helper_cookbook (~> 2.1), cookbookB (1.9.0)
        /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/resolver.rb:83:in `rescue in resolve'
        /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/resolver.rb:73:in `resolve'
        /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/installer.rb:175:in `install_from_universe'
        /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/installer.rb:39:in `run'
        /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/berksfile.rb:374:in `install'
        /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/cli.rb:143:in `install'
        /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
        /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
        /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
        /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/cli.rb:52:in `dispatch'
        /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
        /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/cli.rb:27:in `execute!'
        /opt/chefdk/embedded/apps/berkshelf/bin/berks:5:in `<top (required)>'
        /opt/chefdk/bin/berks:44:in `load'
        /opt/chefdk/bin/berks:44:in `<main>'

For this cookbook where there are no public Supermarket dependencies, I can actually remove public Supermarket from the Berksfile, it resolves instantly and correctly.

source 'http://supermarket.local'
- source 'https://supermarket.chef.io'

metadata

group :test do
  cookbook 'helper_cookbook', '~> 2.1'
end

metadata.rb

...
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '1.8.0'

depends 'cookbookA', '0.1.0'

The problem is a bunch of our cookbooks depend on both internal Private Supermarket dependencies and Public Supermarket dependencies.

Should I maybe create a new post given this new insight? Thanks again!

You might also try the latest Berkshelf and solve gems which contain some fixes, e.g.:

Gemfile:

gem 'berkshelf', git: 'git@github.com:berkshelf/berkshelf.git'
gem 'solve', git: 'git@github.com:berkshelf/solve.git'

If you did that, you could try forcing Berkshelf to use the ruby resolver in your Berksfile:

solver :ruby, :required

That might give you some additional clues. But otherwise, not sure how to suggest you debug the gecode solver, and it sounds like you can’t give us an example since they’re internal cookbooks.

1 Like

Hey @martinb3,

Very cool, didn’t know Berkshelf was working on a new solver. The good news is the Ruby solver works very well and solves the dependencies without any issues. It also respects the DEBUG_RESOLVER=1 environment variable, so I get more information on the resolution.

The bad news is the Chef Server dependency resolution still fails because Chef Server uses dep-selector. Is there anything we can do short of going to Policyfiles? Do Policyfiles use the new Ruby solver or the older dep-selector?

Thanks!

1 Like

If possible, you can try to clean up unused cookbook versions from the server that might be complicating the dependency solver problem. Unfortunately, there is no automated way to do this, because the only way to determine which cookbook versions are in use is to repeatedly query the dependency solver endpoint and combine the results.

Another way to go would be to start using environments pretty aggressively, always using equality pins. If you need to have different cookbook versions active at the same time, you’d need to use the environment cookbook pattern ( http://blog.vialstudios.com/the-environment-cookbook-pattern/ see the part about berks apply). That basically gets you a poor man’s Policyfile.

And finally, you can switch to Policyfiles. The chef command has been using the ruby solver for the past few releases, and using policyfiles entirely bypasses the server’s dependency solver, so if it solves locally it will work on the server.

HTH,

Dan

2 Likes

Thanks @kallistec, just gave that a try but it didn’t work unfortunately.

We kind of do this already, but not for the whole run list and that’s why we got into trouble. I tried equality pinning when trying to replicate the dep resolution issues in Berkshelf, and with everything equality pinned the resolution worked. I guess Policyfiles replace the environment cookbook pattern anyway, so better to go there.

Very happy to hear Policyfiles use the Ruby solver, that’s great. I see Policyfiles are still pre 1.0, any idea when the big 1.0 release might be? I see the don’t use in production label has been removed, which I guess is a good thing. I think we’ll have to rapidly transition here.

Thanks again for your help!

The core functionality that we’re targeting for a 1.0 release is working and stable. The stuff we’re working on now is integration with other tools in the ecosystem and documentation. For example:

  • Chef Zero doesn’t yet implement all of the APIs that you need for “real” policyfile use. Instead it uses a hacky “compatibility mode” which works transparently most of the time, but can have edge cases if you’re reaching into chef internals, and also has slightly different configuration (though the chef export command creates a working config file for you). Work on this is in progress, but Chef Zero has a lot of modes of operation so there’s a lot more work to do than you’d first guess.
  • ChefDK ships with a Test Kitchen provisioner, which works for basic use cases. But you have to configure it separately, which you don’t have to do with, e.g., berks. Also, the currently released version doesn’t support named run lists, which is the correct way to test different run lists via different test kitchen suites (this feature is in master and will ship in the next release, though). We want to put policyfile integration into Test Kitchen core so it’s equally nice to use as berks, but we’re waiting until the Chef Zero stuff is done to prevent having to do re-work to update it for any changes there.
  • Some LearnChef content is in-progress, but we don’t have much canonical reference material on docs.chef.io

Longer term, we’re planning to implement a mechanism for composing policyfiles from smaller pieces, which is often considered a blocker for folks who are using roles heavily or who have multiple teams pushing Chef code at different rates.

Overall, the way I see it is, many people have one or two specific must-have features which aren’t done yet, and for them Policyfiles will be “obviously not production-ready,” but if you don’t need those things, it’s safe to use in production. Though the letter of SemVer-law says we can make breaking changes any time before 1.0, folks are using it production and we’re avoiding breaking changes unless they are very minor and low impact (this is the only one I can remember over the past few months, FWIW: https://github.com/chef/chef-dk/commit/d8eb1c96d4a3e4ee7f1870ddd2e0fb4f4bd11f34 ).

HTH,

Dan