Occasionally client fails with 404 when copying a file

Chef Infra (archive)
1

hi all,

i’m often plagued by a chef client run failing due to it getting a 404 on a file
it was trying to copy into place as declared in a recipe. is this known?

here’s how it goes down:

  1. i update file like site-cookbooks/foo/files/default/usr/local/bin/my-cool-tool.
  2. some linux host kicks off its chef client run, which includes a recipe that
    calls remote_directory to copy the contents of site-cookbooks/foo/files/default/usr/local/bin/
    to the local filesystem
  3. i run “knife cookbook upload foo” just after the above client has kicked off
    its run
  4. the chef client run fails because when it tries to copy the file
    site-cookbooks/foo/files/default/usr/local/bin/my-cool-tool it gets a 404
    for it. if i run chef client again right after this, it succeeds.

i can repro it pretty consistently. is this known? a known bug? is there a
known workaround? today, in one of the many places in my recipes where i copy
a file, where this 404 problem occasionally plagues us, i put “retries 3” into
the resource block. i’m betting that’ll help, but i’m hoping there’s a better
fix or work around. if there’s not, i’ll have to add retries in a lot of places.

chef client details: 10.24, Amazon Linux 2012.09 (RHEL6-based)
chef server details: 10.18, Ubuntu 12.04.2 LTS

thanks!
kallen

2

I used to get that periodically on my chef 10.16 and 10.18 servers, but
haven't seen it at all in chef 11... have you considered upgrading just the
server? (i'm guessing there's another reason you're waiting to upgrade the
clients).

On Thu, Apr 4, 2013 at 9:26 PM, kallen@groknaut.net wrote:

hi all,

i'm often plagued by a chef client run failing due to it getting a 404 on
a file
it was trying to copy into place as declared in a recipe. is this known?

here's how it goes down:

  1. i update file like
    site-cookbooks/foo/files/default/usr/local/bin/my-cool-tool.
  2. some linux host kicks off its chef client run, which includes a recipe
    that
    calls remote_directory to copy the contents of
    site-cookbooks/foo/files/default/usr/local/bin/
    to the local filesystem
  3. i run "knife cookbook upload foo" just after the above client has
    kicked off
    its run
  4. the chef client run fails because when it tries to copy the file
    site-cookbooks/foo/files/default/usr/local/bin/my-cool-tool it gets a
    404
    for it. if i run chef client again right after this, it succeeds.

i can repro it pretty consistently. is this known? a known bug? is there a
known workaround? today, in one of the many places in my recipes where i
copy
a file, where this 404 problem occasionally plagues us, i put "retries 3"
into
the resource block. i'm betting that'll help, but i'm hoping there's a
better
fix or work around. if there's not, i'll have to add retries in a lot of
places.

chef client details: 10.24, Amazon Linux 2012.09 (RHEL6-based)
chef server details: 10.18, Ubuntu 12.04.2 LTS

thanks!
kallen

3

thanks jesse --

i'm assuming at this point, that moving us from chef 10.x server to chef 11
will be a non-trivial amount of work. but i haven't assessed it deeply yet. i
do want to move to chef 11, but, Priorities[TM]...

does anyone know if chef 11 is the one and only answer to this?

On Thu, 04 Apr 2013, Jesse Campbell wrote:

I used to get that periodically on my chef 10.16 and 10.18 servers, but
haven't seen it at all in chef 11... have you considered upgrading just the
server? (i'm guessing there's another reason you're waiting to upgrade the
clients).

On Thu, Apr 4, 2013 at 9:26 PM, kallen@groknaut.net wrote:

hi all,

i'm often plagued by a chef client run failing due to it getting a 404 on
a file
it was trying to copy into place as declared in a recipe. is this known?

here's how it goes down:

  1. i update file like
    site-cookbooks/foo/files/default/usr/local/bin/my-cool-tool.
  2. some linux host kicks off its chef client run, which includes a recipe
    that
    calls remote_directory to copy the contents of
    site-cookbooks/foo/files/default/usr/local/bin/
    to the local filesystem
  3. i run "knife cookbook upload foo" just after the above client has
    kicked off
    its run
  4. the chef client run fails because when it tries to copy the file
    site-cookbooks/foo/files/default/usr/local/bin/my-cool-tool it gets a
    404
    for it. if i run chef client again right after this, it succeeds.

i can repro it pretty consistently. is this known? a known bug? is there a
known workaround? today, in one of the many places in my recipes where i
copy
a file, where this 404 problem occasionally plagues us, i put "retries 3"
into
the resource block. i'm betting that'll help, but i'm hoping there's a
better
fix or work around. if there's not, i'll have to add retries in a lot of
places.

chef client details: 10.24, Amazon Linux 2012.09 (RHEL6-based)
chef server details: 10.18, Ubuntu 12.04.2 LTS

thanks!
kallen

4

On Thursday, April 4, 2013 at 6:53 PM, kallen@groknaut.net wrote:

thanks jesse --

i'm assuming at this point, that moving us from chef 10.x server to chef 11
will be a non-trivial amount of work. but i haven't assessed it deeply yet. i
do want to move to chef 11, but, Priorities[TM]...

does anyone know if chef 11 is the one and only answer to this?
I'm pretty sure that if you bump your cookbook's version number this problem will go away.

--
Daniel DeLeo

5

another thing i found that helped back in chef 10 land, was to run a knife
cookbook bulk delete .* --purge, then reload all cookbooks

On Fri, Apr 5, 2013 at 1:52 PM, Daniel DeLeo dan@kallistec.com wrote:

On Thursday, April 4, 2013 at 6:53 PM, kallen@groknaut.net wrote:

thanks jesse --

i'm assuming at this point, that moving us from chef 10.x server to chef 11
will be a non-trivial amount of work. but i haven't assessed it deeply
yet. i
do want to move to chef 11, but, Priorities[TM]...

does anyone know if chef 11 is the one and only answer to this?

I'm pretty sure that if you bump your cookbook's version number this
problem will go away.

--
Daniel DeLeo