RE: Re: Docker_container dependency hell

kkeane · September 7, 2015, 10:07pm

I think this gist is pretty much what I am doing right now, and it actually illustrates the problem:

The docker_images are built twice, first with the build_if_missing action, and then with the subscribes action. In the gist, the this will happen in the correct sequence, but if the actions are spread over several cookbooks, that can’t be guaranteed, and the build_if_missing action may actually fire before the templates do.

The same problem exists with the docker container: the run action will fire, and then subscribes will stop and remove the just-created container, before re-creating it.

Also, the docker_container action run will fire on every chef run. Shouldn’t it at least be run_if_missing?

Agreed in principle on spreading over multiple cookbooks, but that’s not always realistic. For instance, In my actual scenario, I have a database container, a Web server container, a PHP FPM container, and one container (data only) for each hosted Web site. The cookbook for the hosted Web sites shouldn’t have to be involved with the database container.

Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Sean OMeara someara@chef.io
Sent: Monday 7th September 2015 14:13
To: chef@lists.opscode.com
Subject: [chef] Re: Docker_container dependency hell

I believe this gist is what you’re looking for:

gist.github.com

https://gist.github.com/someara/24edc8b969207fb55d7b

gistfile1.rb

# container A
template '/some/Dockerfile' do
  action :create
end

template '/some/file' do
  action :create
end

docker_image 'databass' do

This file has been truncated. show original

A few things…

you probably want :run instead of :run_if_missing if you’re launching a long running service. It’s safe across multiple runs and will repair the container in the event of a crash or out-of-band administrative actions.
Remember, :delayed notify/subscribes are queued until after the end of the chef-client converge.
Avoid spreading resources with notify/subscribe relationships across multiple cookbooks. It creates a semantic dependency relationship between them that’s extremely difficult to reason about and test. The same recipe is best.

-s

On Mon, Sep 7, 2015 at 12:10 AM, Kevin Keane Subscription <subscription@kkeane.com mailto:subscription@kkeane.com > wrote:

I have two docker containers. Container 1 links to container 2 (target). Consequently, container2 needs to be created and started before container1.

What I am trying to accomplish is this:

if the containers do not exist, they will be run in the correct sequence, even if the images already exist.
If an image for container1 is updated, container 1 is redeployed.
If an image for container2 is updated, containers2 is redeployed, and then container1 is redeployed.

Here is what I am currently doing (simplified for readability)

docker_image image1 do

action :nothing

built only based on notifications

end

docker_container container2 do

action :run_if_missing

subscribe :redeploy, “docker_image[image2]”

end

And in another cookbook:

docker_image image2 do

action :nothing

built only based on notifications

end

docker_container container1 do

action :run_if_missing

subscribe :redeploy, "docker_image[image1]

subscribe :redeploy, "docker_container[container2]

end

Problems with this approach:

The containers are often built twice, first by the run_if_missing action, and then again by the redeploy action
The run_if_missing action does not observe dependencies, so container1’s run_if_missing action is actually invoked before container2 even exists, causing a build failure.

Obviously, I could circumvent this by changing the default action to :nothing, but then the containers aren’t built at all if they are missing.

Also, using :immediate for the subscriptions is not an option (because then the redeploy action may get invoked too early and multiple times, and various other problems).

What is the best way to resolve this?

Kevin Keane

The NetTech

http://www.4nettech.com http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

someara · September 7, 2015, 11:09pm

I've added a working test recipe so you can play with it in test-kitchen.

github.com

sous-chefs/docker/blob/7a19c4937d60bd8ffb403f40b7dd70322ab685a0/test/cookbooks/docker_test/recipes/notifications.rb

# service
include_recipe 'docker_test::default'

# container A
directory '/alice' do
  action :create
end

file '/alice/Dockerfile' do
  content <<-EOF
  FROM alpine
  MAINTAINER alice@computers.biz
  COPY file /
  EOF
  action :create
end

file '/alice/file' do
  content 'alice was here\n'
  action :create

This file has been truncated. show original

kitchen converge notifications-ubuntu-1504

During the first converge, it does indeed restart the containers at the end
of the chef client run..

write some file
starting alice
write some file
starting bob
starting alice
starting bob

But that's okay, the end state of the converge is what's important, not the
journey.

Run test-kitchen again
kitchen converge notifications-ubuntu-1504

During the second converge, nothing happens. the docker_containers are
already in the "Up" state (docker ps -a), so there is nothing to do. Had
the commands on the containers exited (ls -la / for example), :run would
have taken action to converge them to their "Up" state.

The relevant line in the provider is here:

github.com

sous-chefs/docker/blob/7a19c4937d60bd8ffb403f40b7dd70322ab685a0/libraries/provider_docker_container.rb#L183


      
            converge_by "creating #{new_resource.container_name}" do
              create_container
            end
            new_resource.updated_by_last_action(true)
          end
          
          action :start do
            api_timeouts
            c = Docker::Container.get(new_resource.container_name)
            next if c.info['State']['Restarting']
            next if c.info['State']['Running']
            converge_by "starting #{new_resource.container_name}" do
              begin
                tries ||= new_resource.api_retries
          
                if new_resource.detach
                  new_resource.attach_stdin false
                  new_resource.attach_stdout false
                  new_resource.attach_stderr false
                  new_resource.stdin_once false
                  c.start

Edit line 19 of test/cookbooks/docker_test/recipes/notifications.rb, and
change "alice was here" to "alice was there".

Run test-kitchen again.
kitchen converge notifications-ubuntu-1504

The only things that happen are:

'file[/alice/file]' is repaired
'docker_image[alice]' is rebuilt
'docker_container[alice]' is redeployed
'docker_container[bob]' is redeployed

The restarts during the initial chef-client run are the price of admission
for getting subsequent updates chained together with the notification
system.

-s

On Tue, Sep 8, 2015 at 12:07 AM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

I think this gist is pretty much what I am doing right now, and it
actually illustrates the problem:

The docker_images are built twice, first with the build_if_missing action,
and then with the subscribes action. In the gist, the this will happen in
the correct sequence, but if the actions are spread over several cookbooks,
that can't be guaranteed, and the build_if_missing action may actually fire
before the templates do.

The same problem exists with the docker container: the run action will
fire, and then subscribes will stop and remove the just-created container,
before re-creating it.

Also, the docker_container action run will fire on every chef run.
Shouldn't it at least be run_if_missing?

Agreed in principle on spreading over multiple cookbooks, but that's not
always realistic. For instance, In my actual scenario, I have a database
container, a Web server container, a PHP FPM container, and one container
(data only) for each hosted Web site. The cookbook for the hosted Web sites
shouldn't have to be involved with the database container.

Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Sean OMeara someara@chef.io
Sent: Monday 7th September 2015 14:13
To: chef@lists.opscode.com
Subject: [chef] Re: Docker_container dependency hell

I believe this gist is what you're looking for:

gist:24edc8b969207fb55d7b · GitHub

A few things..

you probably want :run instead of :run_if_missing if you're launching a
long running service. It's safe across multiple runs and will repair the
container in the event of a crash or out-of-band administrative actions.

Remember, :delayed notify/subscribes are queued until after the end of
the chef-client converge.

Avoid spreading resources with notify/subscribe relationships across
multiple cookbooks. It creates a semantic dependency relationship between
them that's extremely difficult to reason about and test. The same recipe
is best.

-s

On Mon, Sep 7, 2015 at 12:10 AM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

I have two docker containers. Container 1 links to container 2 (target).
Consequently, container2 needs to be created and started before container1.

What I am trying to accomplish is this:

if the containers do not exist, they will be run in the correct
sequence, even if the images already exist.

If an image for container1 is updated, container 1 is redeployed.

If an image for container2 is updated, containers2 is redeployed, and
then container1 is redeployed.

Here is what I am currently doing (simplified for readability)

docker_image image1 do

action :nothing

built only based on notifications

end

docker_container container2 do

action :run_if_missing

subscribe :redeploy, "docker_image[image2]"

end

And in another cookbook:

docker_image image2 do

action :nothing

built only based on notifications

end
docker_container container1 do

action :run_if_missing

subscribe :redeploy, "docker_image[image1]

subscribe :redeploy, "docker_container[container2]

end

Problems with this approach:

The containers are often built twice, first by the run_if_missing
action, and then again by the redeploy action

The run_if_missing action does not observe dependencies, so
container1's run_if_missing action is actually invoked before container2
even exists, causing a build failure.

Obviously, I could circumvent this by changing the default action to
:nothing, but then the containers aren't built at all if they are missing.

Also, using :immediate for the subscriptions is not an option (because
then the redeploy action may get invoked too early and multiple times, and
various other problems).

What is the best way to resolve this?

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

Topic		Replies	Views
RE: Re: RE: Re: Docker_container dependency hell Chef Infra (archive)	1	616	September 8, 2015
RE: Re: Docker_container dependency hell Chef Infra (archive)	1	292	September 8, 2015
Docker_container dependency hell Chef Infra (archive)	2	284	September 7, 2015
RE: Re: RE: Re: Docker_container dependency hell Chef Infra (archive)	0	267	September 8, 2015
RE: Re: docker cookbook and idempotency Chef Infra (archive)	3	343	September 1, 2015

RE: Re: Docker_container dependency hell

built only based on notifications

built only based on notifications

built only based on notifications

built only based on notifications

Related Topics