Docker_container dependency hell

Chef Infra (archive)
1

I have two docker containers. Container 1 links to container 2 (target). Consequently, container2 needs to be created and started before container1.

What I am trying to accomplish is this:

  • if the containers do not exist, they will be run in the correct sequence, even if the images already exist.

  • If an image for container1 is updated, container 1 is redeployed.

  • If an image for container2 is updated, containers2 is redeployed, and then container1 is redeployed.

Here is what I am currently doing (simplified for readability)

docker_image image1 do

action :nothing

built only based on notifications

end

docker_container container2 do

action :run_if_missing

subscribe :redeploy, “docker_image[image2]”

end

And in another cookbook:

docker_image image2 do

action :nothing

built only based on notifications

end

docker_container container1 do

action :run_if_missing

subscribe :redeploy, "docker_image[image1]

subscribe :redeploy, "docker_container[container2]

end

Problems with this approach:

  • The containers are often built twice, first by the run_if_missing action, and then again by the redeploy action

  • The run_if_missing action does not observe dependencies, so container1’s run_if_missing action is actually invoked before container2 even exists, causing a build failure.

Obviously, I could circumvent this by changing the default action to :nothing, but then the containers aren’t built at all if they are missing.

Also, using :immediate for the subscriptions is not an option (because then the redeploy action may get invoked too early and multiple times, and various other problems).

What is the best way to resolve this?

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

2

Remarks inline

Here is what I am currently doing (simplified for readability)

docker_image image1 do

action :nothing

built only based on notifications

end

docker_container container2 do

action :run_if_missing

subscribe :redeploy, "docker_image[image2]"

end

And in another cookbook:

docker_image image2 do

action :nothing

built only based on notifications

end
docker_container container1 do

action :run_if_missing

subscribe :redeploy, "docker_image[image1]

subscribe :redeploy, "docker_container[container2]

end

Problems with this approach:

  • The containers are often built twice, first by the run_if_missing action, and then again by the redeploy action

  • The run_if_missing action does not observe dependencies, so container1's run_if_missing action is actually invoked before container2 even exists, causing a build failure.

Don't you have a problem in your recipes order ? Out of notifications,
you should ensure the run_list is coherent with the timeline of your
provisionning, if the container2 was before container1 in the resource
appearance order, it will be built before.

Another option could be to set a flag 'already_run' to define the
subscribe attributes' only after first run, something along:

docker_container container1 do

action :run_if_missing

subscribe :redeploy, "docker_image[image1] if node.tagged?('flag')

subscribe :redeploy, "docker_container[container2] if
node.tagged?('flag')

end

At end of recipe (could be improved to handle failure by doing it in a

ruby_block and using notifications.)

node.tag('flag') # Flag could be generic or something named by container
for a more precise approch

Obviously, I could circumvent this by changing the default action to :nothing, but then the containers aren't built at all if they are missing.

Also, using :immediate for the subscriptions is not an option (because then the redeploy action may get invoked too early and multiple times, and various other problems).

What is the best way to resolve this?

3

I believe this gist is what you're looking for:

A few things..

  1. you probably want :run instead of :run_if_missing if you're launching a
    long running service. It's safe across multiple runs and will repair the
    container in the event of a crash or out-of-band administrative actions.

  2. Remember, :delayed notify/subscribes are queued until after the end of
    the chef-client converge.

  3. Avoid spreading resources with notify/subscribe relationships across
    multiple cookbooks. It creates a semantic dependency relationship between
    them that's extremely difficult to reason about and test. The same recipe
    is best.

-s

On Mon, Sep 7, 2015 at 12:10 AM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

I have two docker containers. Container 1 links to container 2 (target).
Consequently, container2 needs to be created and started before container1.

What I am trying to accomplish is this:

  • if the containers do not exist, they will be run in the correct
    sequence, even if the images already exist.

  • If an image for container1 is updated, container 1 is redeployed.

  • If an image for container2 is updated, containers2 is redeployed, and
    then container1 is redeployed.

Here is what I am currently doing (simplified for readability)

docker_image image1 do

action :nothing

built only based on notifications

end

docker_container container2 do

action :run_if_missing

subscribe :redeploy, "docker_image[image2]"

end

And in another cookbook:

docker_image image2 do

action :nothing

built only based on notifications

end
docker_container container1 do

action :run_if_missing

subscribe :redeploy, "docker_image[image1]

subscribe :redeploy, "docker_container[container2]

end

Problems with this approach:

  • The containers are often built twice, first by the run_if_missing
    action, and then again by the redeploy action

  • The run_if_missing action does not observe dependencies, so container1's
    run_if_missing action is actually invoked before container2 even exists,
    causing a build failure.

Obviously, I could circumvent this by changing the default action to
:nothing, but then the containers aren't built at all if they are missing.

Also, using :immediate for the subscriptions is not an option (because
then the redeploy action may get invoked too early and multiple times, and
various other problems).

What is the best way to resolve this?

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html