PostgreSQL cookbook questions / issues

Hello all,

I have been messing around with Chef this week with a view to using it
to manage our Linux servers.

Generally things have been going well but I have run into a few problems
which I would like to ask more experienced chef'ers (Is there a proper
term for chef users?) about.

My goal is to install PostgreSQL 9.2 on CentOS 6.4. I wish to use the
"Database" cookbook to manage parts of the database therefore I need to
install the pg gem in chef. To this end I have the following .json file
and "master" cookbook recipe: https://gist.github.com/rurounijones/5596193

After much trial and tribulation I have managed to get PostgreSQL
installed BUT I have had to modify various recipes in the PostgreSQL
cookbook which seems...strange since I cannot believe I am the first
person who has tried the above.

Therefore I would like to ask if anyone can comment on the following and
let me know if I am missing something:

  1. PostgreSQL 9.2 on CentOS 6.4

As far as I can tell, using opscode's cookbook it is impossible to
install anything other than PostgreSQL 8.4 on CentOS >= 6.0 without
having t modify lots of things like package names, service names etc. I
have submitted a pull request regarding this:

  1. postgresql::ruby recipe

There appears to be a redundant chunk of code in the postgresql::ruby
recipe at
https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb#L33
which was causing me problems. The line in question installs the client
packages but in the line right before it it includes the
postgresl::client recipe which... installs the client packages. I had
various problems with this chunk of code as it appeared to be running
before the postgresql::client code which resulted in it trying to
install the packages before the pgdg repository had been enabled.
Removing the package installation code from this file solved the issue.

  1. No libpqxx-devel

On all CentOS systems I have ever set-up with ruby and the pg gem I have
needed to install the libpqxx-devel library. However I can see no
mention of this package in any of the PostgreSQL chef scripts, I am
therefore puzzled how this has worked up until now, in my copy of the
cookbook I added this to the client packages.

  1. Installing the pg gem itself.

In relation to the above, on all CentOS systems I have ever used with
the libpqxx-devel library I have had to include the pg bin into the PATH
so that, during installation, the pg gem can find the pg_config binary.

The pg gem installation code in the
https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb
is pretty complex, and do be honest, I am not entirely sure what it is
doing. However, on a CentOS 6.4 install I was never able to get this
working (I am afraid I did not save the error message, apologies). What
I did instead was replace the chef_gem method
(https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb#L40)
with
the following:

 bash 'install_chef_pg_gem' do
   code <<-EOH
     PATH=$PATH:/usr/pgsql-9.2/bin /opt/chef/embedded/bin/gem install pg
   EOH
 end

It may not be pretty but it works for me (caveat emptor).

Now as I said before, with the above modifications, I have managed to
install PostgreSQL, setup database management etc. so I am quite happy
but I am shocked that I had to make the above changes, it rather implies
that I am "doing it wrong" or misunderstanding something.

What I would like is for a Chef'er to have a look at my findings and
tell me if I am doing things wrong. If it IS possible to install
PostgreSQL 9.2 on CentOS 6.4 with the pg gem installed into chef without
having to modify the postgresql cookbook as I have done, would it be
possible for some nice person to put up a sample of how it would be done?

(If you have Vagrant then I am using the following box:
http://developer.nrel.gov/downloads/vagrant-boxes/CentOS-6.4-x86_64-v20130309.box
)

Cheers,

Jeff

https://github.com/opscode-cookbooks/postgresql/pull/59

After much trial and tribulation I have managed to get PostgreSQL
installed BUT I have had to modify various recipes in the PostgreSQL
cookbook which seems...strange since I cannot believe I am the first
person who has tried the above.

It's not that hard to believe. The number of people like yourself
who would have flailed like you've been dealing with, unfortunately,
and then come for advice with a pull request in hand... is very
small.

  1. No libpqxx-devel

On all CentOS systems I have ever set-up with ruby and the pg gem I have
needed to install the libpqxx-devel library. However I can see no
mention of this package in any of the PostgreSQL chef scripts, I am
therefore puzzled how this has worked up until now, in my copy of the
cookbook I added this to the client packages.

It's 'postgresql-devel' and is installed by the postgresql::client
recipe which is called from the postgresql::server recipe.

In attributes/default.rb:

when "redhat", "centos", "scientific", "oracle"
if node['platform_version'].to_f >= 6.0
default['postgresql']['client']['packages'] = %w{postgresql-devel}
#...
else
default['postgresql']['client']['packages'] =
["postgresql#{node['postgresql']['version'].split('.').join}-devel"]
#...
end
#...
end

The postgresql-devel package has its own dependency on the postgres
package (the clienty binaries, etc).

Sorry that that is all I can assist with.

On 17/05/13 10:02, Jeff Blaine wrote:

After much trial and tribulation I have managed to get PostgreSQL
installed BUT I have had to modify various recipes in the PostgreSQL
cookbook which seems...strange since I cannot believe I am the first
person who has tried the above.

It's not that hard to believe. The number of people like yourself
who would have flailed like you've been dealing with, unfortunately,
and then come for advice with a pull request in hand... is very
small.

  1. No libpqxx-devel

On all CentOS systems I have ever set-up with ruby and the pg gem I have
needed to install the libpqxx-devel library. However I can see no
mention of this package in any of the PostgreSQL chef scripts, I am
therefore puzzled how this has worked up until now, in my copy of the
cookbook I added this to the client packages.

It's 'postgresql-devel' and is installed by the postgresql::client
recipe which is called from the postgresql::server recipe.

Well bugger me with a fishfork. I could have sworn that I had to install
libpqxx-devel in the past and that not having it this time around was
causing trouble but I have just removed it from the package list,
re-created the VM (God I love Vagrant) and it all worked (as long as I
had the PATH variable set to include the path to pg_config as I
mentioned in point 4).

Thank you for that, consider number 3 answered!

In attributes/default.rb:

when "redhat", "centos", "scientific", "oracle"
if node['platform_version'].to_f >= 6.0
default['postgresql']['client']['packages'] = %w{postgresql-devel}
#...
else
default['postgresql']['client']['packages'] =
["postgresql#{node['postgresql']['version'].split('.').join}-devel"]
#...
end
#...
end

The postgresql-devel package has its own dependency on the postgres
package (the clienty binaries, etc).

Sorry that that is all I can assist with.

On Thursday, May 16, 2013 at 5:39 PM, Jeffrey Jones wrote:

Hello all,

I have been messing around with Chef this week with a view to using it to manage our Linux servers.

Generally things have been going well but I have run into a few problems which I would like to ask more experienced chef'ers (Is there a proper term for chef users?) about.

My goal is to install PostgreSQL 9.2 on CentOS 6.4. I wish to use the "Database" cookbook to manage parts of the database therefore I need to install the pg gem in chef. To this end I have the following .json file and "master" cookbook recipe: Chef json and master cookbook recipe. · GitHub

After much trial and tribulation I have managed to get PostgreSQL installed BUT I have had to modify various recipes in the PostgreSQL cookbook which seems...strange since I cannot believe I am the first person who has tried the above.

Therefore I would like to ask if anyone can comment on the following and let me know if I am missing something:

  1. PostgreSQL 9.2 on CentOS 6.4

As far as I can tell, using opscode's cookbook it is impossible to install anything other than PostgreSQL 8.4 on CentOS >= 6.0 without having t modify lots of things like package names, service names etc. I have submitted a pull request regarding this: [COOK-2954] Support Installing non-standard versions on CentOS 6 by rurounijones · Pull Request #59 · sous-chefs/postgresql · GitHub

Woot, thanks. Did you file a ticket on http://tickets.opscode.com/secure/Dashboard.jspa ? I know it feels redundant, but we use jira to keep track of what goes in each release and we got heavily invested in Jira before github had any features that met our needs. Creating a ticket on tickets.opscode.com and setting to "fix provided" with a link to your pull request will ensure your contribution comes up during code review meetings (these are held on G+ hangouts if you're interested in participating).

  1. postgresql::ruby recipe

There appears to be a redundant chunk of code in the postgresql::ruby recipe at https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb#L33 which was causing me problems. The line in question installs the client packages but in the line right before it it includes the postgresl::client recipe which... installs the client packages. I had various problems with this chunk of code as it appeared to be running before the postgresql::client code which resulted in it trying to install the packages before the pgdg repository had been enabled. Removing the package installation code from this file solved the issue.
The code there is forcing the packages to get installed during the cookbook compile phase so the C library deps of the pg gem are available when the gem gets installed, which is itself required to be available early for database resources to work. It may be possible to rework the cookbook to avoid this hackery, I'm not too familiar with it so I can't say.

  1. No libpqxx-devel

On all CentOS systems I have ever set-up with ruby and the pg gem I have needed to install the libpqxx-devel library. However I can see no mention of this package in any of the PostgreSQL chef scripts, I am therefore puzzled how this has worked up until now, in my copy of the cookbook I added this to the client packages.

  1. Installing the pg gem itself.

In relation to the above, on all CentOS systems I have ever used with the libpqxx-devel library I have had to include the pg bin into the PATH so that, during installation, the pg gem can find the pg_config binary.

The pg gem installation code in the https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb is pretty complex, and do be honest, I am not entirely sure what it is doing. However, on a CentOS 6.4 install I was never able to get this working (I am afraid I did not save the error message, apologies). What I did instead was replace the chef_gem method (https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb#L40) with
the following:

bash 'install_chef_pg_gem' do
  code <<-EOH
    PATH=$PATH:/usr/pgsql-9.2/bin /opt/chef/embedded/bin/gem install pg
  EOH
end

It may not be pretty but it works for me (caveat emptor).
IIRC, in some cases there have been conflicts between the OpenSSL included with the omnibus package (that is linked by the embedded ruby) and the one installed from packages. That code is a bunch of hackery to workaround the issue. Again, I'm not personally familiar with the constraints on the solution.

Now as I said before, with the above modifications, I have managed to install PostgreSQL, setup database management etc. so I am quite happy but I am shocked that I had to make the above changes, it rather implies that I am "doing it wrong" or misunderstanding something.

What I would like is for a Chef'er to have a look at my findings and tell me if I am doing things wrong. If it IS possible to install PostgreSQL 9.2 on CentOS 6.4 with the pg gem installed into chef without having to modify the postgresql cookbook as I have done, would it be possible for some nice person to put up a sample of how it would be done?

(If you have Vagrant then I am using the following box: Download CentOS-6.4-x86_64-v20130309.box (NREL Vagrant Base Boxes) )

Cheers,

Jeff

Hope you find that a bit helpful.

Daniel DeLeo

On 17/05/13 10:55, Daniel DeLeo wrote:

On Thursday, May 16, 2013 at 5:39 PM, Jeffrey Jones wrote:

Hello all,

I have been messing around with Chef this week with a view to using
it to manage our Linux servers.

Generally things have been going well but I have run into a few
problems which I would like to ask more experienced chef'ers (Is
there a proper term for chef users?) about.

My goal is to install PostgreSQL 9.2 on CentOS 6.4. I wish to use the
"Database" cookbook to manage parts of the database therefore I need
to install the pg gem in chef. To this end I have the following .json
file and "master" cookbook recipe:
Chef json and master cookbook recipe. · GitHub

After much trial and tribulation I have managed to get PostgreSQL
installed BUT I have had to modify various recipes in the PostgreSQL
cookbook which seems...strange since I cannot believe I am the first
person who has tried the above.

Therefore I would like to ask if anyone can comment on the following
and let me know if I am missing something:

  1. PostgreSQL 9.2 on CentOS 6.4

As far as I can tell, using opscode's cookbook it is impossible to
install anything other than PostgreSQL 8.4 on CentOS >= 6.0 without
having t modify lots of things like package names, service names etc.
I have submitted a pull request regarding this:
[COOK-2954] Support Installing non-standard versions on CentOS 6 by rurounijones · Pull Request #59 · sous-chefs/postgresql · GitHub

Woot, thanks. Did you file a ticket on
http://tickets.opscode.com/secure/Dashboard.jspa ? I know it feels
redundant, but we use jira to keep track of what goes in each release
and we got heavily invested in Jira before github had any features
that met our needs. Creating a ticket on tickets.opscode.com and
setting to "fix provided" with a link to your pull request will ensure
your contribution comes up during code review meetings (these are held
on G+ hangouts if you're interested in participating).

I created a ticket http://tickets.opscode.com/browse/COOK-2954
[COOK-2954] But I could not figure out how to set the "fix provided"
status so I just added a comment with the link. If someone could give me
a hint (or set it themselves) then that would be grand.

Code review sounds interesting, not sure about timezone issues (I am in
Japan) and G+ registration requirements. Is there a page somewhere with
information on when these are held?

  1. postgresql::ruby recipe

There appears to be a redundant chunk of code in the postgresql::ruby
recipe at
https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb#L33
which was causing me problems. The line in question installs the
client packages but in the line right before it it includes the
postgresl::client recipe which... installs the client packages. I had
various problems with this chunk of code as it appeared to be running
before the postgresql::client code which resulted in it trying to
install the packages before the pgdg repository had been enabled.
Removing the package installation code from this file solved the issue.
The code there is forcing the packages to get installed during the
cookbook compile phase so the C library deps of the pg gem are
available when the gem gets installed, which is itself required to be
available early for database resources to work. It may be possible to
rework the cookbook to avoid this hackery, I'm not too familiar with
it so I can't say.
Ok, that makes sense'ish but it seems that it is trying to install the
packages in the compile phase but not setting up the pgdg repository,
this is fine when using standard PostgreSQL 8.4 but PostgreSQL 9.2 needs
the pgdg repository setup. This sounds like an edge-case bug but I would
be very hesitant to make any changes since I do not really understand
what is going on there. If I would like someone to look at it should I
make a ticket?

  1. No libpqxx-devel

On all CentOS systems I have ever set-up with ruby and the pg gem I
have needed to install the libpqxx-devel library. However I can see
no mention of this package in any of the PostgreSQL chef scripts, I
am therefore puzzled how this has worked up until now, in my copy of
the cookbook I added this to the client packages.

  1. Installing the pg gem itself.

In relation to the above, on all CentOS systems I have ever used with
the libpqxx-devel library I have had to include the pg bin into the
PATH so that, during installation, the pg gem can find the pg_config
binary.

The pg gem installation code in the
https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb
is pretty complex, and do be honest, I am not entirely sure what it
is doing. However, on a CentOS 6.4 install I was never able to get
this working (I am afraid I did not save the error message,
apologies). What I did instead was replace the chef_gem method
(https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb#L40)
with
the following:

bash 'install_chef_pg_gem' do
  code <<-EOH
    PATH=$PATH:/usr/pgsql-9.2/bin /opt/chef/embedded/bin/gem 

install pg
EOH
end

It may not be pretty but it works for me (caveat emptor).
IIRC, in some cases there have been conflicts between the OpenSSL
included with the omnibus package (that is linked by the embedded
ruby) and the one installed from packages. That code is a bunch of
hackery to workaround the issue. Again, I'm not personally familiar
with the constraints on the solution.
Yea, I did a bit of reading but again I cannot really follow the
hackery. All I can say is that it failed to install no matter what I
tried with chef_gem, I cannot remember if I tried it with the default
packages (must give that a go) or if it is only failing because I am
using PostgreSQL 9.2. I need to check that but time is a bit tight. The
bash workaround was all I could come up with that worked.

Now as I said before, with the above modifications, I have managed to
install PostgreSQL, setup database management etc. so I am quite
happy but I am shocked that I had to make the above changes, it
rather implies that I am "doing it wrong" or misunderstanding something.

What I would like is for a Chef'er to have a look at my findings and
tell me if I am doing things wrong. If it IS possible to install
PostgreSQL 9.2 on CentOS 6.4 with the pg gem installed into chef
without having to modify the postgresql cookbook as I have done,
would it be possible for some nice person to put up a sample of how
it would be done?

(If you have Vagrant then I am using the following box:
Download CentOS-6.4-x86_64-v20130309.box (NREL Vagrant Base Boxes)
)

Cheers,

Jeff

Hope you find that a bit helpful.

Daniel DeLeo
Many thanks, very useful explanations, I need to do some checking to see
if the problems are a result of my not using the default CentOS version
or not.

Cheers

Jeff

Ohai,

On May 16, 2013, at 6:39 PM, Jeffrey Jones jjones@toppan-f.co.jp wrote:

  1. PostgreSQL 9.2 on CentOS 6.4

As far as I can tell, using opscode's cookbook it is impossible to install anything other than PostgreSQL 8.4 on CentOS >= 6.0 without having t modify lots of things like package names, service names etc. I have submitted a pull request regarding this: [COOK-2954] Support Installing non-standard versions on CentOS 6 by rurounijones · Pull Request #59 · sous-chefs/postgresql · GitHub

Are you using Chef 11+?

One of the things that Chef 11's node attributes fixes from earlier versions is that you can override attributes that are used in the cookbook's attributes file. Prior to Chef 11, they weren't loaded in the proper order.

"Computing attributes from attributes"

http://docs.opscode.com/breaking_changes_chef_11.html#role-and-environment-attribute-changes

  1. postgresql::ruby recipe

There appears to be a redundant chunk of code in the postgresql::ruby recipe at https://github.com/opscode-cookbooks/postgresql/blob/master/recipes/ruby.rb#L33 which was causing me problems. The line in question installs the client packages but in the line right before it it includes the postgresl::client recipe which... installs the client packages. I had various problems with this chunk of code as it appeared to be running before the postgresql::client code which resulted in it trying to install the packages before the pgdg repository had been enabled. Removing the package installation code from this file solved the issue.

This recipe is intentional, and it is really complicated and nuanced.

The purpose of this recipe is to get the 'pg' gem installed into Chef's Ruby environment. In order to do that, when using Chef from the Omnibus install packages, we need to do the things in that recipe so all the proper libraries are available for the compilation of the native extensions in the pg gem. This must happen during the "compile" phase of the Chef run, which is why the "client" packages are handled like that. It also predates the pgdg repository, and may need to be updated to account for that.

Further details are recorded in this ticket:

http://tickets.opscode.com/browse/COOK-1406