Ark cookbook resources are not idempotent?


#1

I’ve started looking at using ark to do the download, unzip,
configure, make, make install dance.

It seems like every time Chef runs, make, make install is executed again.

ark ‘ghostscript’ do
url 'http://downloads.ghostscript.com/public/ghostscript-9.07.tar.gz
version '9.07’
checksum '44800d004c53f13192d1b5db413119198ddfc8a11c4d2a030aac2f2fda822ebf’
action [:configure, :install_with_make]
end

From logs:

  • ark[ghostscript] action install_with_makeRecipe:

  • directory[/usr/local/ghostscript-9.07] action create (up to date)

  • bash[build with make] action run

    • execute “bash” “/tmp/chef-script20130520-15167-13q0blu”
  • link[/usr/local/ghostscript] action create (up to date)

  • bash[make install] action run

    • execute “bash” “/tmp/chef-script20130520-15167-ys05zc”

Chef 11.4.4, Omnibus build, ark 0.1.0

Further evidenced by the timestamp on every binary added getting a new
timestamp.

Is this just me? I hope not…
-M


#2

Hi Mike,

Can you try with the master branch of the ark cookbook git repository? We’ve recently refactored it pretty heavily to use inline resources in the provider.

See COOK-2520 for more information.

On May 20, 2013, at 6:16 AM, Mike miketheman@gmail.com wrote:

I’ve started looking at using ark to do the download, unzip,
configure, make, make install dance.

It seems like every time Chef runs, make, make install is executed again.

ark ‘ghostscript’ do
url 'http://downloads.ghostscript.com/public/ghostscript-9.07.tar.gz
version '9.07’
checksum '44800d004c53f13192d1b5db413119198ddfc8a11c4d2a030aac2f2fda822ebf’
action [:configure, :install_with_make]
end

From logs:

  • ark[ghostscript] action install_with_makeRecipe:

  • directory[/usr/local/ghostscript-9.07] action create (up to date)

  • bash[build with make] action run

    • execute “bash” “/tmp/chef-script20130520-15167-13q0blu”
  • link[/usr/local/ghostscript] action create (up to date)

  • bash[make install] action run

    • execute “bash” “/tmp/chef-script20130520-15167-ys05zc”

Chef 11.4.4, Omnibus build, ark 0.1.0

Further evidenced by the timestamp on every binary added getting a new
timestamp.

Is this just me? I hope not…
-M


#3

Trying ark 0.1.1 from master.

I can confirm that the behavior I was unhappy with seems to be
resolved, thanks Joshua & Sean!

This issue may predate the current state, but I found it nonetheless:

ark resource downloads source file despite being installed, probably
due to the temp dir being devoid of the file.
How do we prevent a download of something that’s already been installed?
Maybe use other attributes provided to determine if the
’package-version’ is already there and not empty?

-M

On Mon, May 20, 2013 at 12:56 PM, Joshua Timberman joshua@opscode.com wrote:

Hi Mike,

Can you try with the master branch of the ark cookbook git repository? We’ve recently refactored it pretty heavily to use inline resources in the provider.

See COOK-2520 for more information.

On May 20, 2013, at 6:16 AM, Mike miketheman@gmail.com wrote:

I’ve started looking at using ark to do the download, unzip,
configure, make, make install dance.

It seems like every time Chef runs, make, make install is executed again.

ark ‘ghostscript’ do
url 'http://downloads.ghostscript.com/public/ghostscript-9.07.tar.gz
version '9.07’
checksum '44800d004c53f13192d1b5db413119198ddfc8a11c4d2a030aac2f2fda822ebf’
action [:configure, :install_with_make]
end

From logs:

  • ark[ghostscript] action install_with_makeRecipe:

  • directory[/usr/local/ghostscript-9.07] action create (up to date)

  • bash[build with make] action run

    • execute “bash” “/tmp/chef-script20130520-15167-13q0blu”
  • link[/usr/local/ghostscript] action create (up to date)

  • bash[make install] action run

    • execute “bash” “/tmp/chef-script20130520-15167-ys05zc”

Chef 11.4.4, Omnibus build, ark 0.1.0

Further evidenced by the timestamp on every binary added getting a new
timestamp.

Is this just me? I hope not…
-M


#4

hrm… maybe we can dedicate a temp directory for ark?

Personally, I’d rather see the arks being downloaded and checksummed with
an extraction trigger than using a “if the directory exists” test because
its stronger.

The real solution to this is “use packages”

-s

On Mon, May 20, 2013 at 8:52 PM, Mike miketheman@gmail.com wrote:

Trying ark 0.1.1 from master.

I can confirm that the behavior I was unhappy with seems to be
resolved, thanks Joshua & Sean!

This issue may predate the current state, but I found it nonetheless:

ark resource downloads source file despite being installed, probably
due to the temp dir being devoid of the file.
How do we prevent a download of something that’s already been installed?
Maybe use other attributes provided to determine if the
’package-version’ is already there and not empty?

-M

On Mon, May 20, 2013 at 12:56 PM, Joshua Timberman joshua@opscode.com
wrote:

Hi Mike,

Can you try with the master branch of the ark cookbook git repository?
We’ve recently refactored it pretty heavily to use inline resources in the
provider.

See COOK-2520 for more information.

On May 20, 2013, at 6:16 AM, Mike miketheman@gmail.com wrote:

I’ve started looking at using ark to do the download, unzip,
configure, make, make install dance.

It seems like every time Chef runs, make, make install is executed
again.

ark ‘ghostscript’ do
url 'http://downloads.ghostscript.com/public/ghostscript-9.07.tar.gz
version ‘9.07’
checksum
’44800d004c53f13192d1b5db413119198ddfc8a11c4d2a030aac2f2fda822ebf’

action [:configure, :install_with_make]
end

From logs:

  • ark[ghostscript] action install_with_makeRecipe:

  • directory[/usr/local/ghostscript-9.07] action create (up to date)

  • bash[build with make] action run

    • execute “bash” “/tmp/chef-script20130520-15167-13q0blu”
  • link[/usr/local/ghostscript] action create (up to date)

  • bash[make install] action run

    • execute “bash” “/tmp/chef-script20130520-15167-ys05zc”

Chef 11.4.4, Omnibus build, ark 0.1.0

Further evidenced by the timestamp on every binary added getting a new
timestamp.

Is this just me? I hope not…
-M


#5

The temp dir should probably be /var/cache/chef or /var/chef/cache
(why are there two???).

It probably already is, however in my vagrant box it’s
/tmp/vagrant-chef-1/ so I don’t expect it to be there all the time.

Thanks for thinking about it.

On Tue, May 21, 2013 at 6:21 AM, Sean OMeara someara@gmail.com wrote:

hrm… maybe we can dedicate a temp directory for ark?

Personally, I’d rather see the arks being downloaded and checksummed with an
extraction trigger than using a “if the directory exists” test because its
stronger.

The real solution to this is “use packages”

-s

On Mon, May 20, 2013 at 8:52 PM, Mike miketheman@gmail.com wrote:

Trying ark 0.1.1 from master.

I can confirm that the behavior I was unhappy with seems to be
resolved, thanks Joshua & Sean!

This issue may predate the current state, but I found it nonetheless:

ark resource downloads source file despite being installed, probably
due to the temp dir being devoid of the file.
How do we prevent a download of something that’s already been installed?
Maybe use other attributes provided to determine if the
’package-version’ is already there and not empty?

-M

On Mon, May 20, 2013 at 12:56 PM, Joshua Timberman joshua@opscode.com
wrote:

Hi Mike,

Can you try with the master branch of the ark cookbook git repository?
We’ve recently refactored it pretty heavily to use inline resources in the
provider.

See COOK-2520 for more information.

On May 20, 2013, at 6:16 AM, Mike miketheman@gmail.com wrote:

I’ve started looking at using ark to do the download, unzip,
configure, make, make install dance.

It seems like every time Chef runs, make, make install is executed
again.

ark ‘ghostscript’ do
url 'http://downloads.ghostscript.com/public/ghostscript-9.07.tar.gz
version '9.07’
checksum
’44800d004c53f13192d1b5db413119198ddfc8a11c4d2a030aac2f2fda822ebf’
action [:configure, :install_with_make]
end

From logs:

  • ark[ghostscript] action install_with_makeRecipe:

  • directory[/usr/local/ghostscript-9.07] action create (up to date)

  • bash[build with make] action run

    • execute “bash” “/tmp/chef-script20130520-15167-13q0blu”
  • link[/usr/local/ghostscript] action create (up to date)

  • bash[make install] action run

    • execute “bash” “/tmp/chef-script20130520-15167-ys05zc”

Chef 11.4.4, Omnibus build, ark 0.1.0

Further evidenced by the timestamp on every binary added getting a new
timestamp.

Is this just me? I hope not…
-M


#6

RIght now its using Chef::Config[:file_cache_path]

… send a patch it maybe?

I don’t have bandwidth this week and feel funny filing a ticket against
something thats not released yet as a reminder to myself to fix it =)

-s

On Tue, May 21, 2013 at 8:07 AM, Mike miketheman@gmail.com wrote:

The temp dir should probably be /var/cache/chef or /var/chef/cache
(why are there two???).

It probably already is, however in my vagrant box it’s
/tmp/vagrant-chef-1/ so I don’t expect it to be there all the time.

Thanks for thinking about it.

On Tue, May 21, 2013 at 6:21 AM, Sean OMeara someara@gmail.com wrote:

hrm… maybe we can dedicate a temp directory for ark?

Personally, I’d rather see the arks being downloaded and checksummed
with an
extraction trigger than using a “if the directory exists” test because
its
stronger.

The real solution to this is “use packages”

-s

On Mon, May 20, 2013 at 8:52 PM, Mike miketheman@gmail.com wrote:

Trying ark 0.1.1 from master.

I can confirm that the behavior I was unhappy with seems to be
resolved, thanks Joshua & Sean!

This issue may predate the current state, but I found it nonetheless:

ark resource downloads source file despite being installed, probably
due to the temp dir being devoid of the file.
How do we prevent a download of something that’s already been installed?
Maybe use other attributes provided to determine if the
’package-version’ is already there and not empty?

-M

On Mon, May 20, 2013 at 12:56 PM, Joshua Timberman joshua@opscode.com
wrote:

Hi Mike,

Can you try with the master branch of the ark cookbook git repository?
We’ve recently refactored it pretty heavily to use inline resources
in the

provider.

See COOK-2520 for more information.

On May 20, 2013, at 6:16 AM, Mike miketheman@gmail.com wrote:

I’ve started looking at using ark to do the download, unzip,
configure, make, make install dance.

It seems like every time Chef runs, make, make install is
executed

again.

ark ‘ghostscript’ do
url ‘
http://downloads.ghostscript.com/public/ghostscript-9.07.tar.gz

version '9.07’
checksum
’44800d004c53f13192d1b5db413119198ddfc8a11c4d2a030aac2f2fda822ebf’
action [:configure, :install_with_make]
end

From logs:

  • ark[ghostscript] action install_with_makeRecipe:

  • directory[/usr/local/ghostscript-9.07] action create (up to date)

  • bash[build with make] action run

    • execute “bash” “/tmp/chef-script20130520-15167-13q0blu”
  • link[/usr/local/ghostscript] action create (up to date)

  • bash[make install] action run

    • execute “bash” “/tmp/chef-script20130520-15167-ys05zc”

Chef 11.4.4, Omnibus build, ark 0.1.0

Further evidenced by the timestamp on every binary added getting a
new

timestamp.

Is this just me? I hope not…
-M


#7

I wonder if there’s a way to make it easier for the checksum check to
happen? Like have an option to store the checksum as a node attribute
for subsequent runs if you know the download is static vs dynamic. For
example, a ruby tarball vs an app deployment war. That would at least
prevent Chef from downloading the file every time to verify that
checksums are indeed the same.

Of course, that’s no help if someone deletes the downloaded artifact. I
tried setting a flag in a couple different ways at one point to work
with the idea that first level help desk/ops whatever will go through
and delete everything they can find when they get disk full
notifications. Unfortunately my implementation was flawed and it just
left people confused as to why re-downloading didn’t trigger when there
were problems.

This is why I also like a “creates” or “not_if File.exists?” or
"create_if_missing." It prevents chef run slowdowns when dealing with
unpackaged files.

For the record, I also strongly prefer packages and repos for static
software, but for those caught up in a less than optimal system,
sometimes it can’t be an immediate reality. Even then I prefer to take
the time and create a package and put it wherever I store static software.

Artifactory has an sha256 checksum plugin that you can use to
programmatically retrieve checksums to avoid downloading. I wonder if
we can get something like that for other places people are dumping
tarballs, like Nexus and Artifactory?

Sean OMeara wrote:

hrm… maybe we can dedicate a temp directory for ark?

Personally, I’d rather see the arks being downloaded and checksummed
with an extraction trigger than using a “if the directory exists” test
because its stronger.

The real solution to this is “use packages”

-s

On Mon, May 20, 2013 at 8:52 PM, Mike <miketheman@gmail.com
mailto:miketheman@gmail.com> wrote:

Trying ark 0.1.1 from master.

I can confirm that the behavior I was unhappy with seems to be
resolved, thanks Joshua & Sean!

This issue may predate the current state, but I found it nonetheless:

ark resource downloads source file despite being installed, probably
due to the temp dir being devoid of the file.
How do we prevent a download of something that's already been
installed?
Maybe use other attributes provided to determine if the
'package-version' is already there and not empty?

-M


On Mon, May 20, 2013 at 12:56 PM, Joshua Timberman
<joshua@opscode.com <mailto:joshua@opscode.com>> wrote:
> Hi Mike,
>
> Can you try with the master branch of the ark cookbook git
repository? We've recently refactored it pretty heavily to use
inline resources in the provider.
>
> See COOK-2520 for more information.
>
> On May 20, 2013, at 6:16 AM, Mike <miketheman@gmail.com
<mailto:miketheman@gmail.com>> wrote:
>
>> I've started looking at using ark to do the download, unzip,
>> configure, make, make install dance.
>>
>> It seems like every time Chef runs, `make`, `make install` is
executed again.
>>
>> ark 'ghostscript' do
>>  url
'http://downloads.ghostscript.com/public/ghostscript-9.07.tar.gz'
>>  version '9.07'
>>  checksum
'44800d004c53f13192d1b5db413119198ddfc8a11c4d2a030aac2f2fda822ebf'
>>  action [:configure, :install_with_make]
>> end
>>
>> From logs:
>>
>>  * ark[ghostscript] action install_with_makeRecipe: <Dynamically
>> Defined Resource>
>>  * directory[/usr/local/ghostscript-9.07] action create (up to
date)
>>  * bash[build with make] action run
>>    - execute "bash"  "/tmp/chef-script20130520-15167-13q0blu"
>>
>>  * link[/usr/local/ghostscript] action create (up to date)
>>  * bash[make install] action run
>>    - execute "bash"  "/tmp/chef-script20130520-15167-ys05zc"
>>
>> Chef 11.4.4, Omnibus build, ark 0.1.0
>>
>> Further evidenced by the timestamp on every binary added
getting a new
>> timestamp.
>>
>> Is this just me? I hope not...
>> -M
>

#8

Ohai,

On Tuesday, May 21, 2013 at 6:07, Mike wrote:

The temp dir should probably be /var/cache/chef or /var/chef/cache
(why are there two???).

As Sean said it should be the Chef::Config[:file_cache_path], wherever that may be configured. These two exist as a matter if config standards.

/var/chef/cache is the default and follows hier(7) on OSX or BSD systems.

/var/cache/chef is used for Debian packages and follows the “FHS” for Linux systems.

It probably already is, however in my vagrant box it’s
/tmp/vagrant-chef-1/ so I don’t expect it to be there all the time.

Which is why using the chef config value is better as wherever that points will exist because chef creates it.


#9

One (partially related) side note here is that using the cache directories has a possible issue in some scenarios; one of my recipes is failing because of SELinux issues with the cache directory. I haven’t tracked down the details yet, so I can’t really say what the exact issue is.

-----Original Message-----
From: Mike [mailto:miketheman@gmail.com]
Sent: Tuesday, May 21, 2013 5:08 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: ark cookbook resources are not idempotent?

The temp dir should probably be /var/cache/chef or /var/chef/cache (why are there two???).

It probably already is, however in my vagrant box it’s /tmp/vagrant-chef-1/ so I don’t expect it to be there all the time.

Thanks for thinking about it.

On Tue, May 21, 2013 at 6:21 AM, Sean OMeara someara@gmail.com wrote:

hrm… maybe we can dedicate a temp directory for ark?

Personally, I’d rather see the arks being downloaded and checksummed
with an extraction trigger than using a “if the directory exists” test
because its stronger.

The real solution to this is “use packages”

-s

On Mon, May 20, 2013 at 8:52 PM, Mike miketheman@gmail.com wrote:

Trying ark 0.1.1 from master.

I can confirm that the behavior I was unhappy with seems to be
resolved, thanks Joshua & Sean!

This issue may predate the current state, but I found it nonetheless:

ark resource downloads source file despite being installed, probably
due to the temp dir being devoid of the file.
How do we prevent a download of something that’s already been installed?
Maybe use other attributes provided to determine if the
’package-version’ is already there and not empty?

-M

On Mon, May 20, 2013 at 12:56 PM, Joshua Timberman
joshua@opscode.com
wrote:

Hi Mike,

Can you try with the master branch of the ark cookbook git repository?
We’ve recently refactored it pretty heavily to use inline resources
in the provider.

See COOK-2520 for more information.

On May 20, 2013, at 6:16 AM, Mike miketheman@gmail.com wrote:

I’ve started looking at using ark to do the download, unzip,
configure, make, make install dance.

It seems like every time Chef runs, make, make install is
executed again.

ark ‘ghostscript’ do
url 'http://downloads.ghostscript.com/public/ghostscript-9.07.tar.gz
version '9.07’
checksum
’44800d004c53f13192d1b5db413119198ddfc8a11c4d2a030aac2f2fda822ebf’
action [:configure, :install_with_make] end

From logs:

  • ark[ghostscript] action install_with_makeRecipe:

  • directory[/usr/local/ghostscript-9.07] action create (up to
    date)

  • bash[build with make] action run

    • execute “bash” “/tmp/chef-script20130520-15167-13q0blu”
  • link[/usr/local/ghostscript] action create (up to date)

  • bash[make install] action run

    • execute “bash” “/tmp/chef-script20130520-15167-ys05zc”

Chef 11.4.4, Omnibus build, ark 0.1.0

Further evidenced by the timestamp on every binary added getting a
new timestamp.

Is this just me? I hope not…
-M