Sorting bug?

So we set up a new Chef client (10.16.2) from scratch and we’ve noticed
some interesting behavior:

Any cookbook that relies on iteration over a list of any sort now returns
random ordering of the list between runs with no change in the underlying
data.

For example the following data:

default[:testapp][:languages] = {“US” => “ASCII”,
“JP” => “utf-8”}

When inserted into a file with this template code:
[Languages]
<% for language in node[:testapp][:languages].keys-%>
<%=language%>:<%=node[:testappanguages][language]%>
<% end-%>

Sometimes generates:

[Languages]
JP:utf-8
US:ASCII

And other times generates:

[Languages]
US:ASCII
JP:utf-8

Again nothing has changed between runs in the underlying data, cookbooks or
roles. This doesn’t affect any of our existing nodes (we noticed it but
multiple apps on the new node were intermittently and restarting for no
reason). Also downgrading the affected node to Chef 0.10.8 or 10.14.2 (the
versions we run on the other nodes) doesn’t help.

Ruby version is 1.8.7 across the board. Unaffected nodes are running Ubuntu
10.04.3 LTS, the affected node is running 10.04.4 LTS.

Any ideas what may be causing this?

-J

Hashes don't preserve ordering in Ruby 1.8.7

I'd suggest installing and using Chef with the Omnibus Installers, this
will give you a modern Ruby environment (1.9.3-p194 or similar)

Cheers,

AJ

On 8 November 2012 16:03, Jason J. W. Williams jasonjwwilliams@gmail.comwrote:

So we set up a new Chef client (10.16.2) from scratch and we've noticed
some interesting behavior:

Any cookbook that relies on iteration over a list of any sort now returns
random ordering of the list between runs with no change in the underlying
data.

For example the following data:

default[:testapp][:languages] = {"US" => "ASCII",
"JP" => "utf-8"}

When inserted into a file with this template code:
[Languages]
<% for language in node[:testapp][:languages].keys-%>
<%=language%>:<%=node[:testappanguages][language]%>
<% end-%>

Sometimes generates:

[Languages]
JP:utf-8
US:ASCII

And other times generates:

[Languages]
US:ASCII
JP:utf-8

Again nothing has changed between runs in the underlying data, cookbooks
or roles. This doesn't affect any of our existing nodes (we noticed it but
multiple apps on the new node were intermittently and restarting for no
reason). Also downgrading the affected node to Chef 0.10.8 or 10.14.2 (the
versions we run on the other nodes) doesn't help.

Ruby version is 1.8.7 across the board. Unaffected nodes are running
Ubuntu 10.04.3 LTS, the affected node is running 10.04.4 LTS.

Any ideas what may be causing this?

-J

They may not preserve ordering when adding/removing items are involved, but
absent anything changing in the hash, we haven't seen them change. These
cookbooks have been running for months with no problem (and continuing with
no problem on the other nodes).

-J

On Wed, Nov 7, 2012 at 7:20 PM, AJ Christensen aj@junglist.gen.nz wrote:

Hashes don't preserve ordering in Ruby 1.8.7

I'd suggest installing and using Chef with the Omnibus Installers, this
will give you a modern Ruby environment (1.9.3-p194 or similar)

Cheers,

AJ

On 8 November 2012 16:03, Jason J. W. Williams jasonjwwilliams@gmail.comwrote:

So we set up a new Chef client (10.16.2) from scratch and we've noticed
some interesting behavior:

Any cookbook that relies on iteration over a list of any sort now returns
random ordering of the list between runs with no change in the underlying
data.

For example the following data:

default[:testapp][:languages] = {"US" => "ASCII",
"JP" => "utf-8"}

When inserted into a file with this template code:
[Languages]
<% for language in node[:testapp][:languages].keys-%>
<%=language%>:<%=node[:testappanguages][language]%>
<% end-%>

Sometimes generates:

[Languages]
JP:utf-8
US:ASCII

And other times generates:

[Languages]
US:ASCII
JP:utf-8

Again nothing has changed between runs in the underlying data, cookbooks
or roles. This doesn't affect any of our existing nodes (we noticed it but
multiple apps on the new node were intermittently and restarting for no
reason). Also downgrading the affected node to Chef 0.10.8 or 10.14.2 (the
versions we run on the other nodes) doesn't help.

Ruby version is 1.8.7 across the board. Unaffected nodes are running
Ubuntu 10.04.3 LTS, the affected node is running 10.04.4 LTS.

Any ideas what may be causing this?

-J

We're also seeing it in another template that's converting an array to a
string:

mynetworks = <%= @my_networks.join(", ") %>

The code generating mynetworks is here: gist:4036561 · GitHub

Again, we've never had this issue before with this code either. I'm curious
why we're only seeing this problem now on a new node, and not when we were
developing these cookbooks, or for the months they've been running.

-J

On Wed, Nov 7, 2012 at 7:25 PM, Jason J. W. Williams <
jasonjwwilliams@gmail.com> wrote:

They may not preserve ordering when adding/removing items are involved,
but absent anything changing in the hash, we haven't seen them change.
These cookbooks have been running for months with no problem (and
continuing with no problem on the other nodes).

-J

On Wed, Nov 7, 2012 at 7:20 PM, AJ Christensen aj@junglist.gen.nz wrote:

Hashes don't preserve ordering in Ruby 1.8.7

I'd suggest installing and using Chef with the Omnibus Installers, this
will give you a modern Ruby environment (1.9.3-p194 or similar)

Cheers,

AJ

On 8 November 2012 16:03, Jason J. W. Williams <jasonjwwilliams@gmail.com

wrote:

So we set up a new Chef client (10.16.2) from scratch and we've noticed
some interesting behavior:

Any cookbook that relies on iteration over a list of any sort now
returns random ordering of the list between runs with no change in the
underlying data.

For example the following data:

default[:testapp][:languages] = {"US" => "ASCII",
"JP" => "utf-8"}

When inserted into a file with this template code:
[Languages]
<% for language in node[:testapp][:languages].keys-%>
<%=language%>:<%=node[:testappanguages][language]%>
<% end-%>

Sometimes generates:

[Languages]
JP:utf-8
US:ASCII

And other times generates:

[Languages]
US:ASCII
JP:utf-8

Again nothing has changed between runs in the underlying data, cookbooks
or roles. This doesn't affect any of our existing nodes (we noticed it but
multiple apps on the new node were intermittently and restarting for no
reason). Also downgrading the affected node to Chef 0.10.8 or 10.14.2 (the
versions we run on the other nodes) doesn't help.

Ruby version is 1.8.7 across the board. Unaffected nodes are running
Ubuntu 10.04.3 LTS, the affected node is running 10.04.4 LTS.

Any ideas what may be causing this?

-J

Jason,

The key order in Ruby 1.8 depends on the hashes of the keys. But as of Ruby
1.8.7-p357, the algorithm for computing hashes was changed. The algorithm
used to be constant, but now it is seeded with a new seed computed each
time the Ruby VM starts up. See
https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2011-4815 .

Cheers,
Jay

On Wed, Nov 7, 2012 at 10:39 PM, Jason J. W. Williams <
jasonjwwilliams@gmail.com> wrote:

We're also seeing it in another template that's converting an array to a
string:

mynetworks = <%= @my_networks.join(", ") %>

The code generating mynetworks is here: gist:4036561 · GitHub

Again, we've never had this issue before with this code either. I'm
curious why we're only seeing this problem now on a new node, and not when
we were developing these cookbooks, or for the months they've been running.

-J

On Wed, Nov 7, 2012 at 7:25 PM, Jason J. W. Williams <
jasonjwwilliams@gmail.com> wrote:

They may not preserve ordering when adding/removing items are involved,
but absent anything changing in the hash, we haven't seen them change.
These cookbooks have been running for months with no problem (and
continuing with no problem on the other nodes).

-J

On Wed, Nov 7, 2012 at 7:20 PM, AJ Christensen aj@junglist.gen.nzwrote:

Hashes don't preserve ordering in Ruby 1.8.7

I'd suggest installing and using Chef with the Omnibus Installers, this
will give you a modern Ruby environment (1.9.3-p194 or similar)

Cheers,

AJ

On 8 November 2012 16:03, Jason J. W. Williams <
jasonjwwilliams@gmail.com> wrote:

So we set up a new Chef client (10.16.2) from scratch and we've
noticed some interesting behavior:

Any cookbook that relies on iteration over a list of any sort now
returns random ordering of the list between runs with no change in the
underlying data.

For example the following data:

default[:testapp][:languages] = {"US" => "ASCII",
"JP" => "utf-8"}

When inserted into a file with this template code:
[Languages]
<% for language in node[:testapp][:languages].keys-%>
<%=language%>:<%=node[:testappanguages][language]%>
<% end-%>

Sometimes generates:

[Languages]
JP:utf-8
US:ASCII

And other times generates:

[Languages]
US:ASCII
JP:utf-8

Again nothing has changed between runs in the underlying data,
cookbooks or roles. This doesn't affect any of our existing nodes (we
noticed it but multiple apps on the new node were intermittently and
restarting for no reason). Also downgrading the affected node to Chef
0.10.8 or 10.14.2 (the versions we run on the other nodes) doesn't help.

Ruby version is 1.8.7 across the board. Unaffected nodes are running
Ubuntu 10.04.3 LTS, the affected node is running 10.04.4 LTS.

Any ideas what may be causing this?

-J

Thanks Jay.

That foots with the repro I just did with a Ruby 2-liner. Only box affected is the Ubuntu 10.04.4 which claims to have the same ruby -v output but is clearly incorporating the code you referenced.

Interestingly 1.9.2 doesn't seem to have the problem. It sorts complete differently but consistently with identical hashes. I appreciate you identifying the cause.

-J

Sent via iPhone

Is your email Premiere?

On Nov 7, 2012, at 20:13, Jay Feldblum y_feldblum@yahoo.com wrote:

Jason,

The key order in Ruby 1.8 depends on the hashes of the keys. But as of Ruby 1.8.7-p357, the algorithm for computing hashes was changed. The algorithm used to be constant, but now it is seeded with a new seed computed each time the Ruby VM starts up. See https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2011-4815 .

Cheers,
Jay

On Wed, Nov 7, 2012 at 10:39 PM, Jason J. W. Williams jasonjwwilliams@gmail.com wrote:

We're also seeing it in another template that's converting an array to a string:

mynetworks = <%= @my_networks.join(", ") %>

The code generating mynetworks is here: gist:4036561 · GitHub

Again, we've never had this issue before with this code either. I'm curious why we're only seeing this problem now on a new node, and not when we were developing these cookbooks, or for the months they've been running.

-J

On Wed, Nov 7, 2012 at 7:25 PM, Jason J. W. Williams jasonjwwilliams@gmail.com wrote:

They may not preserve ordering when adding/removing items are involved, but absent anything changing in the hash, we haven't seen them change. These cookbooks have been running for months with no problem (and continuing with no problem on the other nodes).

-J

On Wed, Nov 7, 2012 at 7:20 PM, AJ Christensen aj@junglist.gen.nz wrote:

Hashes don't preserve ordering in Ruby 1.8.7

I'd suggest installing and using Chef with the Omnibus Installers, this will give you a modern Ruby environment (1.9.3-p194 or similar)

Cheers,

AJ

On 8 November 2012 16:03, Jason J. W. Williams jasonjwwilliams@gmail.com wrote:

So we set up a new Chef client (10.16.2) from scratch and we've noticed some interesting behavior:

Any cookbook that relies on iteration over a list of any sort now returns random ordering of the list between runs with no change in the underlying data.

For example the following data:

default[:testapp][:languages] = {"US" => "ASCII",
"JP" => "utf-8"}

When inserted into a file with this template code:
[Languages]
<% for language in node[:testapp][:languages].keys-%>
<%=language%>:<%=node[:testappanguages][language]%>
<% end-%>

Sometimes generates:

[Languages]
JP:utf-8
US:ASCII

And other times generates:

[Languages]
US:ASCII
JP:utf-8

Again nothing has changed between runs in the underlying data, cookbooks or roles. This doesn't affect any of our existing nodes (we noticed it but multiple apps on the new node were intermittently and restarting for no reason). Also downgrading the affected node to Chef 0.10.8 or 10.14.2 (the versions we run on the other nodes) doesn't help.

Ruby version is 1.8.7 across the board. Unaffected nodes are running Ubuntu 10.04.3 LTS, the affected node is running 10.04.4 LTS.

Any ideas what may be causing this?

-J

Ruby 1.9 has ordered hashes, I am pretty sure this was not back ported to 1.8.7.

IMO, everyone should have yanked 1.8.7 out a long time ago ;), but I understand that's not as easy.
One reason embedded chef (Omnibus) is soo awesome.

John

On Wednesday, November 7, 2012 at 8:53 PM, Jason J. W. Williams wrote:

Thanks Jay.

That foots with the repro I just did with a Ruby 2-liner. Only box affected is the Ubuntu 10.04.4 which claims to have the same ruby -v output but is clearly incorporating the code you referenced.

Interestingly 1.9.2 doesn't seem to have the problem. It sorts complete differently but consistently with identical hashes. I appreciate you identifying the cause.

-J

Sent via iPhone

Is your email Premiere?

On Nov 7, 2012, at 20:13, Jay Feldblum <y_feldblum@yahoo.com (mailto:y_feldblum@yahoo.com)> wrote:

Jason,

The key order in Ruby 1.8 depends on the hashes of the keys. But as of Ruby 1.8.7-p357, the algorithm for computing hashes was changed. The algorithm used to be constant, but now it is seeded with a new seed computed each time the Ruby VM starts up. See 750564 – (CVE-2011-4815) CVE-2011-4815 ruby: hash table collisions CPU usage DoS (oCERT-2011-003) .

Cheers,
Jay

On Wed, Nov 7, 2012 at 10:39 PM, Jason J. W. Williams <jasonjwwilliams@gmail.com (mailto:jasonjwwilliams@gmail.com)> wrote:

We're also seeing it in another template that's converting an array to a string:

mynetworks = <%= @my_networks.join(", ") %>

The code generating mynetworks is here: gist:4036561 · GitHub

Again, we've never had this issue before with this code either. I'm curious why we're only seeing this problem now on a new node, and not when we were developing these cookbooks, or for the months they've been running.

-J

On Wed, Nov 7, 2012 at 7:25 PM, Jason J. W. Williams <jasonjwwilliams@gmail.com (mailto:jasonjwwilliams@gmail.com)> wrote:

They may not preserve ordering when adding/removing items are involved, but absent anything changing in the hash, we haven't seen them change. These cookbooks have been running for months with no problem (and continuing with no problem on the other nodes).

-J

On Wed, Nov 7, 2012 at 7:20 PM, AJ Christensen <aj@junglist.gen.nz (mailto:aj@junglist.gen.nz)> wrote:

Hashes don't preserve ordering in Ruby 1.8.7

I'd suggest installing and using Chef with the Omnibus Installers, this will give you a modern Ruby environment (1.9.3-p194 or similar)

Cheers,

AJ

On 8 November 2012 16:03, Jason J. W. Williams <jasonjwwilliams@gmail.com (mailto:jasonjwwilliams@gmail.com)> wrote:

So we set up a new Chef client (10.16.2) from scratch and we've noticed some interesting behavior:

Any cookbook that relies on iteration over a list of any sort now returns random ordering of the list between runs with no change in the underlying data.

For example the following data:

default[:testapp][:languages] = {"US" => "ASCII",
"JP" => "utf-8"}

When inserted into a file with this template code:
[Languages]
<% for language in node[:testapp][:languages].keys-%>
<%=language%>:<%=node[:testappanguages][language]%>
<% end-%>

Sometimes generates:

[Languages]
JP:utf-8
US:ASCII

And other times generates:

[Languages]
US:ASCII

JP:utf-8

Again nothing has changed between runs in the underlying data, cookbooks or roles. This doesn't affect any of our existing nodes (we noticed it but multiple apps on the new node were intermittently and restarting for no reason). Also downgrading the affected node to Chef 0.10.8 or 10.14.2 (the versions we run on the other nodes) doesn't help.

Ruby version is 1.8.7 across the board. Unaffected nodes are running Ubuntu 10.04.3 LTS, the affected node is running 10.04.4 LTS.

Any ideas what may be causing this?

-J

Deb/Ubuntu folks probably pulled the patch in manually since it was considered a high-profile security issue (related to worst-case forced hash collisions in web frameworks).

--Noah

On Nov 7, 2012, at 8:53 PM, Jason J. W. Williams wrote:

Thanks Jay.

That foots with the repro I just did with a Ruby 2-liner. Only box affected is the Ubuntu 10.04.4 which claims to have the same ruby -v output but is clearly incorporating the code you referenced.

Interestingly 1.9.2 doesn't seem to have the problem. It sorts complete differently but consistently with identical hashes. I appreciate you identifying the cause.

-J

Sent via iPhone

Is your email Premiere?

On Nov 7, 2012, at 20:13, Jay Feldblum y_feldblum@yahoo.com wrote:

Jason,

The key order in Ruby 1.8 depends on the hashes of the keys. But as of Ruby 1.8.7-p357, the algorithm for computing hashes was changed. The algorithm used to be constant, but now it is seeded with a new seed computed each time the Ruby VM starts up. See 750564 – (CVE-2011-4815) CVE-2011-4815 ruby: hash table collisions CPU usage DoS (oCERT-2011-003) .

Cheers,
Jay

On Wed, Nov 7, 2012 at 10:39 PM, Jason J. W. Williams jasonjwwilliams@gmail.com wrote:
We're also seeing it in another template that's converting an array to a string:

mynetworks = <%= @my_networks.join(", ") %>

The code generating mynetworks is here: gist:4036561 · GitHub

Again, we've never had this issue before with this code either. I'm curious why we're only seeing this problem now on a new node, and not when we were developing these cookbooks, or for the months they've been running.

-J

On Wed, Nov 7, 2012 at 7:25 PM, Jason J. W. Williams jasonjwwilliams@gmail.com wrote:
They may not preserve ordering when adding/removing items are involved, but absent anything changing in the hash, we haven't seen them change. These cookbooks have been running for months with no problem (and continuing with no problem on the other nodes).

-J

On Wed, Nov 7, 2012 at 7:20 PM, AJ Christensen aj@junglist.gen.nz wrote:
Hashes don't preserve ordering in Ruby 1.8.7

I'd suggest installing and using Chef with the Omnibus Installers, this will give you a modern Ruby environment (1.9.3-p194 or similar)

Cheers,

AJ

On 8 November 2012 16:03, Jason J. W. Williams jasonjwwilliams@gmail.com wrote:
So we set up a new Chef client (10.16.2) from scratch and we've noticed some interesting behavior:

Any cookbook that relies on iteration over a list of any sort now returns random ordering of the list between runs with no change in the underlying data.

For example the following data:

default[:testapp][:languages] = {"US" => "ASCII",
"JP" => "utf-8"}

When inserted into a file with this template code:
[Languages]
<% for language in node[:testapp][:languages].keys-%>
<%=language%>:<%=node[:testappanguages][language]%>
<% end-%>

Sometimes generates:

[Languages]
JP:utf-8
US:ASCII

And other times generates:

[Languages]
US:ASCII
JP:utf-8

Again nothing has changed between runs in the underlying data, cookbooks or roles. This doesn't affect any of our existing nodes (we noticed it but multiple apps on the new node were intermittently and restarting for no reason). Also downgrading the affected node to Chef 0.10.8 or 10.14.2 (the versions we run on the other nodes) doesn't help.

Ruby version is 1.8.7 across the board. Unaffected nodes are running Ubuntu 10.04.3 LTS, the affected node is running 10.04.4 LTS.

Any ideas what may be causing this?

-J