RE: Re: Re: Re: Re: Chef Client 12.1.0 Released


#1

Actually, there was one other way to do it prior to 12.1.0: create your own repo. I’m extremely glad that workaround is no longer needed, though. Thank you, thank you, thank you!

As for the problem of manually maintaining indices, agreed, but there is a fairly simple workaround. Instead of an array, use a hash, just as Ameir suggested earlier, and build the arrays right before building the resource.

packageversions = {“php-pecl-imagick” => “3.1.2-2.el6”, “ImageMagick-last-libs” => “6.8.7.4-1.el6”, “other-package” => nil}

pkgs = packageversions.keys

packageversions.values may or may not use the same sequence as keys.

To guarantee the same sequence, we must iterate over pkgs rather than over packageversions.

versions = packageversions.map{ |pkg| packageversions[pkg] }

package pkgs do

versions packageversions

end

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Lamont Granquist lamont@chef.io
Sent: Saturday 7th March 2015 9:52
To: chef@lists.opscode.com
Cc: chef-dev@lists.opscode.com; Ameir Abdeldayem ameirh@gmail.com
Subject: [chef] Re: Re: Re: Re: Chef Client 12.1.0 Released

Also, in some cases its impossible to install software outside of installing two RPMs at the same time:

These two commands may both error out (even if you swap the order) because they have mutual dependencies on each other:

yum install foo-part1
yum install foo-part2

While this may succeed because yum treats it as one transaction and can see that they both satisfy each other dependencies:

yum install foo-part1 foo-part2

Prior to Chef 12.1.0 there was no way of having Chef install packages like this other than writing your own shell_out or arguing with the package author that their packaging is shitty (arguably true, but if the author is not responsive that is not something that is likely to make you happy about your job).

On 3/7/15 9:43 AM, Lamont Granquist wrote:

Speed. One resource and one command issued and one transaction, vs. many:

time ./apply-new.rb

Recipe: (chef-apply cookbook)::(chef-apply recipe)

  • apt_package[bc, zsh, vim, curl, strace, lsof, systemtap, telnet, mtr, subversion, nmap, gdb, traceroute, tcpdump, htop, iptraf, tmux, nuttcp, s3cmd, source-highlight, sysbench] action install
    • install version 1.06.95-8ubuntu1 of package bc
    • install version 7.35.0-1ubuntu2.3 of package curl
    • install version 4.8-1ubuntu5 of package strace
    • install version 2.3-1ubuntu1 of package systemtap
    • install version 0.17-36build2 of package telnet
    • install version 0.85-2 of package mtr
    • install version 1.8.8-1ubuntu3.1 of package subversion
    • install version 6.40-0.2ubuntu1 of package nmap
    • install version 1:2.0.20-0ubuntu0.1 of package traceroute
    • install version 4.5.1-2ubuntu1.1 of package tcpdump
    • install version 1.0.2-3 of package htop
    • install version 3.0.0-8.1 of package iptraf
    • install version 1.8-5 of package tmux
    • install version 6.1.2-4 of package nuttcp
    • install version 1.1.0~beta3-2 of package s3cmd
    • install version 0.4.12-1.1 of package sysbench
      ./apply-new.rb 4.01s user 5.63s system 68% cpu 14.007 total

time ./apply-old.rb

Recipe: (chef-apply cookbook)::(chef-apply recipe)

  • apt_package[bc] action install
    • install version 1.06.95-8ubuntu1 of package bc
  • apt_package[zsh] action install (up to date)
  • apt_package[vim] action install (up to date)
  • apt_package[curl] action install
    • install version 7.35.0-1ubuntu2.3 of package curl
  • apt_package[strace] action install
    • install version 4.8-1ubuntu5 of package strace
  • apt_package[lsof] action install (up to date)
  • apt_package[systemtap] action install
    • install version 2.3-1ubuntu1 of package systemtap
  • apt_package[telnet] action install
    • install version 0.17-36build2 of package telnet
  • apt_package[mtr] action install
    • install version 0.85-2 of package mtr
  • apt_package[subversion] action install
    • install version 1.8.8-1ubuntu3.1 of package subversion
  • apt_package[nmap] action install
    • install version 6.40-0.2ubuntu1 of package nmap
  • apt_package[gdb] action install (up to date)
  • apt_package[traceroute] action install
    • install version 1:2.0.20-0ubuntu0.1 of package traceroute
  • apt_package[tcpdump] action install
    • install version 4.5.1-2ubuntu1.1 of package tcpdump
  • apt_package[htop] action install
    • install version 1.0.2-3 of package htop
  • apt_package[iptraf] action install
    • install version 3.0.0-8.1 of package iptraf
  • apt_package[tmux] action install
    • install version 1.8-5 of package tmux
  • apt_package[nuttcp] action install
    • install version 6.1.2-4 of package nuttcp
  • apt_package[s3cmd] action install
    • install version 1.1.0~beta3-2 of package s3cmd
  • apt_package[source-highlight] action install (up to date)
  • apt_package[sysbench] action install
    • install version 0.4.12-1.1 of package sysbench
      ./apply-old.rb 18.95s user 16.84s system 78% cpu 45.515 total

4 seconds vs 19 seconds to install those packages – and yum_package is even slower per-install than apt_package and the gains are larger there.

On 3/7/15 4:43 AM, Ameir Abdeldayem wrote:

Regarding the multi-package support, what benefits does it bring to the table over something like this?:

{“php-pecl-imagick” => “3.1.2-2.el6”, “ImageMagick-last-libs” => “6.8.7.4-1.el6”, “other-package” => nil}

.each do |package, version|

package package do

version version

end

end

I agree with the others that having to keep track of indices is a poor approach.

Also, most (all?) resources currently apply to a single target (a user account, a directory, a file, etc.). Will the package resource be the special exception, or will other resources follow suit with the array format? e.g., will we be seeing things like directory ["/dir1", "/dir2"] down the road?

On Fri, Mar 6, 2015 at 1:37 PM, Charles Johnson charles@chef.io wrote:

+1 on this, having people manually keep array indices in sync isn’t safe.

I’ve created https://github.com/chef/chef/issues/3040 to track this.

–Charles

On March 4, 2015 at 10:58:49 AM, Michael Fischer (mfischer@zendesk.com) wrote:

I did some background reading on the list support to the package provider.

While I’m very pleased that you can now do package installations much faster, I’m concerned about the syntax. In particular, my concern is the way names and versions are mapped using a pair of Arrays instead of a more logical Hash map. I’d expect something like:

packages “group” do

list { “foo” => “1.0.0”, “bar”, => “2.1.7” }

end

Instead we got something like this:

package [“foo”, “bar”] do

version [“1.0.0”, “2.1.7”]

end

For reasonably short lists, this is fine, but once you get past 5 or so, having to iterate the lists by hand to match the name and version will become extremely difficult and error prone. If the above couldn’t be done using the package provider, I’d suggest that a new resource should have been created instead so that the semantics could be better defined.

Best regards,

–Michael

On Tue, Mar 3, 2015 at 10:43 AM, Jay Mundrawala jdm@getchef.com wrote:

Hi Chefs,

We’ve just released Chef 12.1.0. You can read more about it at https://www.chef.io/blog/2015/03/03/chef-12-1-0-released/.

Thanks,

Jay


#2

Of course you can work around the provider’s new brain damage. But that
doesn’t mean that it shouldn’t be fixed.

–Michael

On Sun, Mar 8, 2015 at 3:01 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

Actually, there was one other way to do it prior to 12.1.0: create your
own repo. I’m extremely glad that workaround is no longer needed,
though. Thank you, thank you, thank you!

As for the problem of manually maintaining indices, agreed, but there is a
fairly simple workaround. Instead of an array, use a hash, just as Ameir
suggested earlier, and build the arrays right before building the resource.

packageversions = {“php-pecl-imagick” => “3.1.2-2.el6”,
“ImageMagick-last-libs” => “6.8.7.4-1.el6”, “other-package” => nil}

pkgs = packageversions.keys

packageversions.values may or may not use the same sequence as keys.

To guarantee the same sequence, we must iterate over pkgs rather than

over packageversions.

versions = packageversions.map{ |pkg| packageversions[pkg] }

package pkgs do

versions packageversions

end

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Lamont Granquist lamont@chef.io
Sent: Saturday 7th March 2015 9:52
To: chef@lists.opscode.com
Cc: chef-dev@lists.opscode.com; Ameir Abdeldayem ameirh@gmail.com
Subject: [chef] Re: Re: Re: Re: Chef Client 12.1.0 Released

Also, in some cases its impossible to install software outside of
installing two RPMs at the same time:

These two commands may both error out (even if you swap the order) because
they have mutual dependencies on each other:

yum install foo-part1
yum install foo-part2

While this may succeed because yum treats it as one transaction and can
see that they both satisfy each other dependencies:

yum install foo-part1 foo-part2

Prior to Chef 12.1.0 there was no way of having Chef install packages like
this other than writing your own shell_out or arguing with the package
author that their packaging is shitty (arguably true, but if the author is
not responsive that is not something that is likely to make you happy about
your job).

On 3/7/15 9:43 AM, Lamont Granquist wrote:

Speed. One resource and one command issued and one transaction, vs. many:

time ./apply-new.rb

Recipe: (chef-apply cookbook)::(chef-apply recipe)

  • apt_package[bc, zsh, vim, curl, strace, lsof, systemtap, telnet, mtr,
    subversion, nmap, gdb, traceroute, tcpdump, htop, iptraf, tmux, nuttcp,
    s3cmd, source-highlight, sysbench] action install
    • install version 1.06.95-8ubuntu1 of package bc
    • install version 7.35.0-1ubuntu2.3 of package curl
    • install version 4.8-1ubuntu5 of package strace
    • install version 2.3-1ubuntu1 of package systemtap
    • install version 0.17-36build2 of package telnet
    • install version 0.85-2 of package mtr
    • install version 1.8.8-1ubuntu3.1 of package subversion
    • install version 6.40-0.2ubuntu1 of package nmap
    • install version 1:2.0.20-0ubuntu0.1 of package traceroute
    • install version 4.5.1-2ubuntu1.1 of package tcpdump
    • install version 1.0.2-3 of package htop
    • install version 3.0.0-8.1 of package iptraf
    • install version 1.8-5 of package tmux
    • install version 6.1.2-4 of package nuttcp
    • install version 1.1.0~beta3-2 of package s3cmd
    • install version 0.4.12-1.1 of package sysbench
      ./apply-new.rb 4.01s user 5.63s system 68% cpu 14.007 total

time ./apply-old.rb

Recipe: (chef-apply cookbook)::(chef-apply recipe)

  • apt_package[bc] action install
    • install version 1.06.95-8ubuntu1 of package bc
  • apt_package[zsh] action install (up to date)
  • apt_package[vim] action install (up to date)
  • apt_package[curl] action install
    • install version 7.35.0-1ubuntu2.3 of package curl
  • apt_package[strace] action install
    • install version 4.8-1ubuntu5 of package strace
  • apt_package[lsof] action install (up to date)
  • apt_package[systemtap] action install
    • install version 2.3-1ubuntu1 of package systemtap
  • apt_package[telnet] action install
    • install version 0.17-36build2 of package telnet
  • apt_package[mtr] action install
    • install version 0.85-2 of package mtr
  • apt_package[subversion] action install
    • install version 1.8.8-1ubuntu3.1 of package subversion
  • apt_package[nmap] action install
    • install version 6.40-0.2ubuntu1 of package nmap
  • apt_package[gdb] action install (up to date)
  • apt_package[traceroute] action install
    • install version 1:2.0.20-0ubuntu0.1 of package traceroute
  • apt_package[tcpdump] action install
    • install version 4.5.1-2ubuntu1.1 of package tcpdump
  • apt_package[htop] action install
    • install version 1.0.2-3 of package htop
  • apt_package[iptraf] action install
    • install version 3.0.0-8.1 of package iptraf
  • apt_package[tmux] action install
    • install version 1.8-5 of package tmux
  • apt_package[nuttcp] action install
    • install version 6.1.2-4 of package nuttcp
  • apt_package[s3cmd] action install
    • install version 1.1.0~beta3-2 of package s3cmd
  • apt_package[source-highlight] action install (up to date)
  • apt_package[sysbench] action install
    • install version 0.4.12-1.1 of package sysbench
      ./apply-old.rb 18.95s user 16.84s system 78% cpu 45.515 total

4 seconds vs 19 seconds to install those packages – and yum_package is
even slower per-install than apt_package and the gains are larger there.

On 3/7/15 4:43 AM, Ameir Abdeldayem wrote:

Regarding the multi-package support, what benefits does it bring to the
table over something like this?:

{“php-pecl-imagick” => “3.1.2-2.el6”, “ImageMagick-last-libs” =>
“6.8.7.4-1.el6”, “other-package” => nil}
.each do |package, version|
package package do
version version
end
end

I agree with the others that having to keep track of indices is a poor
approach.

Also, most (all?) resources currently apply to a single target (a user
account, a directory, a file, etc.). Will the package resource be the
special exception, or will other resources follow suit with the array
format? e.g., will we be seeing things like directory ["/dir1", "/dir2"]
down the road?

On Fri, Mar 6, 2015 at 1:37 PM, Charles Johnson charles@chef.io wrote:

+1 on this, having people manually keep array indices in sync isn’t
safe.

I’ve created https://github.com/chef/chef/issues/3040 to track this.

–Charles

On March 4, 2015 at 10:58:49 AM, Michael Fischer (mfischer@zendesk.com)
wrote:

I did some background reading on the list support to the package
provider.

While I’m very pleased that you can now do package installations much
faster, I’m concerned about the syntax. In particular, my concern is the
way names and versions are mapped using a pair of Arrays instead of a more
logical Hash map. I’d expect something like:

packages “group” do
list { “foo” => “1.0.0”, “bar”, => “2.1.7” }
end

Instead we got something like this:

package [“foo”, “bar”] do
version [“1.0.0”, “2.1.7”]
end

For reasonably short lists, this is fine, but once you get past 5 or
so, having to iterate the lists by hand to match the name and version will
become extremely difficult and error prone. If the above couldn’t be done
using the package provider, I’d suggest that a new resource should have
been created instead so that the semantics could be better defined.

Best regards,

–Michael

On Tue, Mar 3, 2015 at 10:43 AM, Jay Mundrawala jdm@getchef.com wrote:

Hi Chefs,
We’ve just released Chef 12.1.0. You can read more about it at
https://www.chef.io/blog/2015/03/03/chef-12-1-0-released/.

Thanks,
Jay


#3

I’d like to apologize for this remark. It was unprofessional and you all
deserve more helpful contributions.

My regrets,

–Michael

On Wed, Mar 11, 2015 at 7:10 PM, Michael Fischer mfischer@zendesk.com
wrote:

Of course you can work around the provider’s new brain damage. But that
doesn’t mean that it shouldn’t be fixed.

–Michael

On Sun, Mar 8, 2015 at 3:01 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

Actually, there was one other way to do it prior to 12.1.0: create your
own repo. I’m extremely glad that workaround is no longer needed,
though. Thank you, thank you, thank you!

As for the problem of manually maintaining indices, agreed, but there is
a fairly simple workaround. Instead of an array, use a hash, just as Ameir
suggested earlier, and build the arrays right before building the resource.

packageversions = {“php-pecl-imagick” => “3.1.2-2.el6”,
“ImageMagick-last-libs” => “6.8.7.4-1.el6”, “other-package” => nil}

pkgs = packageversions.keys

packageversions.values may or may not use the same sequence as keys.

To guarantee the same sequence, we must iterate over pkgs rather than

over packageversions.

versions = packageversions.map{ |pkg| packageversions[pkg] }

package pkgs do

versions packageversions

end

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Lamont Granquist lamont@chef.io
Sent: Saturday 7th March 2015 9:52
To: chef@lists.opscode.com
Cc: chef-dev@lists.opscode.com; Ameir Abdeldayem ameirh@gmail.com
Subject: [chef] Re: Re: Re: Re: Chef Client 12.1.0 Released

Also, in some cases its impossible to install software outside of
installing two RPMs at the same time:

These two commands may both error out (even if you swap the order)
because they have mutual dependencies on each other:

yum install foo-part1
yum install foo-part2

While this may succeed because yum treats it as one transaction and can
see that they both satisfy each other dependencies:

yum install foo-part1 foo-part2

Prior to Chef 12.1.0 there was no way of having Chef install packages
like this other than writing your own shell_out or arguing with the package
author that their packaging is shitty (arguably true, but if the author is
not responsive that is not something that is likely to make you happy about
your job).

On 3/7/15 9:43 AM, Lamont Granquist wrote:

Speed. One resource and one command issued and one transaction, vs. many:

time ./apply-new.rb

Recipe: (chef-apply cookbook)::(chef-apply recipe)

  • apt_package[bc, zsh, vim, curl, strace, lsof, systemtap, telnet, mtr,
    subversion, nmap, gdb, traceroute, tcpdump, htop, iptraf, tmux, nuttcp,
    s3cmd, source-highlight, sysbench] action install
    • install version 1.06.95-8ubuntu1 of package bc
    • install version 7.35.0-1ubuntu2.3 of package curl
    • install version 4.8-1ubuntu5 of package strace
    • install version 2.3-1ubuntu1 of package systemtap
    • install version 0.17-36build2 of package telnet
    • install version 0.85-2 of package mtr
    • install version 1.8.8-1ubuntu3.1 of package subversion
    • install version 6.40-0.2ubuntu1 of package nmap
    • install version 1:2.0.20-0ubuntu0.1 of package traceroute
    • install version 4.5.1-2ubuntu1.1 of package tcpdump
    • install version 1.0.2-3 of package htop
    • install version 3.0.0-8.1 of package iptraf
    • install version 1.8-5 of package tmux
    • install version 6.1.2-4 of package nuttcp
    • install version 1.1.0~beta3-2 of package s3cmd
    • install version 0.4.12-1.1 of package sysbench
      ./apply-new.rb 4.01s user 5.63s system 68% cpu 14.007 total

time ./apply-old.rb

Recipe: (chef-apply cookbook)::(chef-apply recipe)

  • apt_package[bc] action install
    • install version 1.06.95-8ubuntu1 of package bc
  • apt_package[zsh] action install (up to date)
  • apt_package[vim] action install (up to date)
  • apt_package[curl] action install
    • install version 7.35.0-1ubuntu2.3 of package curl
  • apt_package[strace] action install
    • install version 4.8-1ubuntu5 of package strace
  • apt_package[lsof] action install (up to date)
  • apt_package[systemtap] action install
    • install version 2.3-1ubuntu1 of package systemtap
  • apt_package[telnet] action install
    • install version 0.17-36build2 of package telnet
  • apt_package[mtr] action install
    • install version 0.85-2 of package mtr
  • apt_package[subversion] action install
    • install version 1.8.8-1ubuntu3.1 of package subversion
  • apt_package[nmap] action install
    • install version 6.40-0.2ubuntu1 of package nmap
  • apt_package[gdb] action install (up to date)
  • apt_package[traceroute] action install
    • install version 1:2.0.20-0ubuntu0.1 of package traceroute
  • apt_package[tcpdump] action install
    • install version 4.5.1-2ubuntu1.1 of package tcpdump
  • apt_package[htop] action install
    • install version 1.0.2-3 of package htop
  • apt_package[iptraf] action install
    • install version 3.0.0-8.1 of package iptraf
  • apt_package[tmux] action install
    • install version 1.8-5 of package tmux
  • apt_package[nuttcp] action install
    • install version 6.1.2-4 of package nuttcp
  • apt_package[s3cmd] action install
    • install version 1.1.0~beta3-2 of package s3cmd
  • apt_package[source-highlight] action install (up to date)
  • apt_package[sysbench] action install
    • install version 0.4.12-1.1 of package sysbench
      ./apply-old.rb 18.95s user 16.84s system 78% cpu 45.515 total

4 seconds vs 19 seconds to install those packages – and yum_package is
even slower per-install than apt_package and the gains are larger there.

On 3/7/15 4:43 AM, Ameir Abdeldayem wrote:

Regarding the multi-package support, what benefits does it bring to the
table over something like this?:

{“php-pecl-imagick” => “3.1.2-2.el6”, “ImageMagick-last-libs” =>
“6.8.7.4-1.el6”, “other-package” => nil}
.each do |package, version|
package package do
version version
end
end

I agree with the others that having to keep track of indices is a poor
approach.

Also, most (all?) resources currently apply to a single target (a user
account, a directory, a file, etc.). Will the package resource be the
special exception, or will other resources follow suit with the array
format? e.g., will we be seeing things like directory ["/dir1", "/dir2"]
down the road?

On Fri, Mar 6, 2015 at 1:37 PM, Charles Johnson charles@chef.io wrote:

+1 on this, having people manually keep array indices in sync isn’t
safe.

I’ve created https://github.com/chef/chef/issues/3040 to track this.

–Charles

On March 4, 2015 at 10:58:49 AM, Michael Fischer (mfischer@zendesk.com)
wrote:

I did some background reading on the list support to the package
provider.

While I’m very pleased that you can now do package installations much
faster, I’m concerned about the syntax. In particular, my concern is the
way names and versions are mapped using a pair of Arrays instead of a more
logical Hash map. I’d expect something like:

packages “group” do
list { “foo” => “1.0.0”, “bar”, => “2.1.7” }
end

Instead we got something like this:

package [“foo”, “bar”] do
version [“1.0.0”, “2.1.7”]
end

For reasonably short lists, this is fine, but once you get past 5 or
so, having to iterate the lists by hand to match the name and version will
become extremely difficult and error prone. If the above couldn’t be done
using the package provider, I’d suggest that a new resource should have
been created instead so that the semantics could be better defined.

Best regards,

–Michael

On Tue, Mar 3, 2015 at 10:43 AM, Jay Mundrawala jdm@getchef.com wrote:

Hi Chefs,
We’ve just released Chef 12.1.0. You can read more about it at
https://www.chef.io/blog/2015/03/03/chef-12-1-0-released/.

Thanks,
Jay