Rails db:migrate fails via knife, succeeds when run directly

Chef Infra (archive)
1

I’m using the deploy cookbook to deploy a Rails app (nginx + passenger).
I’m running “bundle exec rake db:migrate --trace” during the migration
step, after the database is created. Many times this step will fail with:

ERROR: deploy_revision[/var/www] (_site::deploy line 39) had an error:
Chef::Exceptions::Exec: bundle exec rake db:migrate --trace returned 1,
expected 0

It is very frustrating because it will sometimes succeed but often will
fail. It seems like more often, when I run via knife ssh ("knife ssh
’fqdn:xxx*’ ‘sudo chef-client’), it will fail at the above step, then I ssh
into the actual box, start a shell as root (“sudo su -”) then run
chef-client and it will succeed.

I see that the unix error code “1” means a permissions issue; is that what
I’m facing here or is rake (er, bundle I guess) just using this as a
generic error return code? Any way I can get a more specific reason as to
why the migrate is failing?

The chef-stacktrace.out file doesn’t give any more specific info about the
actual db:migrate failure.

2

This is permission issue.

Or may be environment variables do not setup correctly.

I don't know application cookbook. But I has a question: where is the ruby
installed with it (in home or system ruby)?

2014-02-18 6:51 GMT+04:00 Toby Johnson irontoby@gmail.com:

I'm using the deploy cookbook to deploy a Rails app (nginx + passenger).
I'm running "bundle exec rake db:migrate --trace" during the migration
step, after the database is created. Many times this step will fail with:

ERROR: deploy_revision[/var/www] (_site::deploy line 39) had an error:
Chef::Exceptions::Exec: bundle exec rake db:migrate --trace returned 1,
expected 0

It is very frustrating because it will sometimes succeed but often will
fail. It seems like more often, when I run via knife ssh ("knife ssh
'fqdn:xxx*' 'sudo chef-client'), it will fail at the above step, then I ssh
into the actual box, start a shell as root ("sudo su -") then run
chef-client and it will succeed.

I see that the unix error code "1" means a permissions issue; is that what
I'm facing here or is rake (er, bundle I guess) just using this as a
generic error return code? Any way I can get a more specific reason as to
why the migrate is failing?

The chef-stacktrace.out file doesn't give any more specific info about the
actual db:migrate failure.

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

3

Thanks Vladimir; my apologies, I meant to mention that Ruby is installed
with rbenv, but using a "global" setup at /opt/rbenv and not in a home
directory.

I am setting up RAILS_ENV to production using the "environment" command in
the deploy_revision resource.

My database.yml is a symlink to the "shared" one
at /var/www/shared/config/database.yml. That file has restrictive
permissions (owner root:nginx with ug+r; or 0440), while all directories
from /var/www below have owner root:nginx and permissions u+rwx,g+rx; or
0750).

One point of confusion I have, is which user actually runs the "rake
db:migrate" command? I didn't see anywhere to set this using the deploy
cookbook.

On Mon, Feb 17, 2014 at 11:20 PM, Vladimir Skubriev <skubriev@cvisionlab.com

wrote:

This is permission issue.

Or may be environment variables do not setup correctly.

I don't know application cookbook. But I has a question: where is the ruby
installed with it (in home or system ruby)?

2014-02-18 6:51 GMT+04:00 Toby Johnson irontoby@gmail.com:

I'm using the deploy cookbook to deploy a Rails app (nginx + passenger).

I'm running "bundle exec rake db:migrate --trace" during the migration
step, after the database is created. Many times this step will fail with:

ERROR: deploy_revision[/var/www] (_site::deploy line 39) had an error:
Chef::Exceptions::Exec: bundle exec rake db:migrate --trace returned 1,
expected 0

It is very frustrating because it will sometimes succeed but often will
fail. It seems like more often, when I run via knife ssh ("knife ssh
'fqdn:xxx*' 'sudo chef-client'), it will fail at the above step, then I ssh
into the actual box, start a shell as root ("sudo su -") then run
chef-client and it will succeed.

I see that the unix error code "1" means a permissions issue; is that
what I'm facing here or is rake (er, bundle I guess) just using this as a
generic error return code? Any way I can get a more specific reason as to
why the migrate is failing?

The chef-stacktrace.out file doesn't give any more specific info about
the actual db:migrate failure.

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

4

To be honest I do not know.

2014-02-18 9:15 GMT+04:00 Toby Johnson irontoby@gmail.com:

Thanks Vladimir; my apologies, I meant to mention that Ruby is installed
with rbenv, but using a "global" setup at /opt/rbenv and not in a home
directory.

I am setting up RAILS_ENV to production using the "environment" command in
the deploy_revision resource.

My database.yml is a symlink to the "shared" one
at /var/www/shared/config/database.yml. That file has restrictive
permissions (owner root:nginx with ug+r; or 0440), while all directories
from /var/www below have owner root:nginx and permissions u+rwx,g+rx; or
0750).

One point of confusion I have, is which user actually runs the "rake
db:migrate" command? I didn't see anywhere to set this using the deploy
cookbook.

On Mon, Feb 17, 2014 at 11:20 PM, Vladimir Skubriev <
skubriev@cvisionlab.com> wrote:

This is permission issue.

Or may be environment variables do not setup correctly.

I don't know application cookbook. But I has a question: where is the
ruby installed with it (in home or system ruby)?

2014-02-18 6:51 GMT+04:00 Toby Johnson irontoby@gmail.com:

I'm using the deploy cookbook to deploy a Rails app (nginx + passenger).

I'm running "bundle exec rake db:migrate --trace" during the migration
step, after the database is created. Many times this step will fail with:

ERROR: deploy_revision[/var/www] (_site::deploy line 39) had an error:
Chef::Exceptions::Exec: bundle exec rake db:migrate --trace returned 1,
expected 0

It is very frustrating because it will sometimes succeed but often will
fail. It seems like more often, when I run via knife ssh ("knife ssh
'fqdn:xxx*' 'sudo chef-client'), it will fail at the above step, then I ssh
into the actual box, start a shell as root ("sudo su -") then run
chef-client and it will succeed.

I see that the unix error code "1" means a permissions issue; is that
what I'm facing here or is rake (er, bundle I guess) just using this as a
generic error return code? Any way I can get a more specific reason as to
why the migrate is failing?

The chef-stacktrace.out file doesn't give any more specific info about
the actual db:migrate failure.

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

5

Hey Toby,

You might need to load rbenv to the shell before running bundle. The
application-rails cookbook makes this easy for you:

application "my_app" do
...
rails do
bundle_command ". /etc/profile.d/rbenv.sh &&
RAILS_ENV=#{node.chef_environment} bundle"
...
end
end

That's how I used to do. The path to the rbenv init script might not be
right -- I was using chruby --, so adjust to your environment.

On 18 February 2014 07:35, Vladimir Skubriev skubriev@cvisionlab.comwrote:

To be honest I do not know.

2014-02-18 9:15 GMT+04:00 Toby Johnson irontoby@gmail.com:

Thanks Vladimir; my apologies, I meant to mention that Ruby is installed

with rbenv, but using a "global" setup at /opt/rbenv and not in a home
directory.

I am setting up RAILS_ENV to production using the "environment" command
in the deploy_revision resource.

My database.yml is a symlink to the "shared" one
at /var/www/shared/config/database.yml. That file has restrictive
permissions (owner root:nginx with ug+r; or 0440), while all directories
from /var/www below have owner root:nginx and permissions u+rwx,g+rx; or
0750).

One point of confusion I have, is which user actually runs the "rake
db:migrate" command? I didn't see anywhere to set this using the deploy
cookbook.

On Mon, Feb 17, 2014 at 11:20 PM, Vladimir Skubriev <
skubriev@cvisionlab.com> wrote:

This is permission issue.

Or may be environment variables do not setup correctly.

I don't know application cookbook. But I has a question: where is the
ruby installed with it (in home or system ruby)?

2014-02-18 6:51 GMT+04:00 Toby Johnson irontoby@gmail.com:

I'm using the deploy cookbook to deploy a Rails app (nginx + passenger).

I'm running "bundle exec rake db:migrate --trace" during the migration
step, after the database is created. Many times this step will fail with:

ERROR: deploy_revision[/var/www] (_site::deploy line 39) had an error:
Chef::Exceptions::Exec: bundle exec rake db:migrate --trace returned 1,
expected 0

It is very frustrating because it will sometimes succeed but often will
fail. It seems like more often, when I run via knife ssh ("knife ssh
'fqdn:xxx*' 'sudo chef-client'), it will fail at the above step, then I ssh
into the actual box, start a shell as root ("sudo su -") then run
chef-client and it will succeed.

I see that the unix error code "1" means a permissions issue; is that
what I'm facing here or is rake (er, bundle I guess) just using this as a
generic error return code? Any way I can get a more specific reason as to
why the migrate is failing?

The chef-stacktrace.out file doesn't give any more specific info about
the actual db:migrate failure.

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

6

Okay Cassiano, that makes sense, thanks for the pointer. I'll give it a try.

I hadn't heard of the application-rails cookbook (though a Google search
suggests maybe you meant application-ruby?) and since I'm using nginx I
don't know if it would apply to me but I can certainly make the same call
using the deploy cookbook too.

On Tue, Feb 18, 2014 at 10:31 AM, Cassiano Leal cassianoleal@gmail.comwrote:

Hey Toby,

You might need to load rbenv to the shell before running bundle. The
application-rails cookbook makes this easy for you:

application "my_app" do
...
rails do
bundle_command ". /etc/profile.d/rbenv.sh &&
RAILS_ENV=#{node.chef_environment} bundle"
...
end
end

That's how I used to do. The path to the rbenv init script might not be
right -- I was using chruby --, so adjust to your environment.

On 18 February 2014 07:35, Vladimir Skubriev skubriev@cvisionlab.comwrote:

To be honest I do not know.

2014-02-18 9:15 GMT+04:00 Toby Johnson irontoby@gmail.com:

Thanks Vladimir; my apologies, I meant to mention that Ruby is installed

with rbenv, but using a "global" setup at /opt/rbenv and not in a home
directory.

I am setting up RAILS_ENV to production using the "environment" command
in the deploy_revision resource.

My database.yml is a symlink to the "shared" one
at /var/www/shared/config/database.yml. That file has restrictive
permissions (owner root:nginx with ug+r; or 0440), while all directories
from /var/www below have owner root:nginx and permissions u+rwx,g+rx; or
0750).

One point of confusion I have, is which user actually runs the "rake
db:migrate" command? I didn't see anywhere to set this using the deploy
cookbook.

On Mon, Feb 17, 2014 at 11:20 PM, Vladimir Skubriev <
skubriev@cvisionlab.com> wrote:

This is permission issue.

Or may be environment variables do not setup correctly.

I don't know application cookbook. But I has a question: where is the
ruby installed with it (in home or system ruby)?

2014-02-18 6:51 GMT+04:00 Toby Johnson irontoby@gmail.com:

I'm using the deploy cookbook to deploy a Rails app (nginx +

passenger). I'm running "bundle exec rake db:migrate --trace" during the
migration step, after the database is created. Many times this step will
fail with:

ERROR: deploy_revision[/var/www] (_site::deploy line 39) had an error:
Chef::Exceptions::Exec: bundle exec rake db:migrate --trace returned 1,
expected 0

It is very frustrating because it will sometimes succeed but often
will fail. It seems like more often, when I run via knife ssh ("knife ssh
'fqdn:xxx*' 'sudo chef-client'), it will fail at the above step, then I ssh
into the actual box, start a shell as root ("sudo su -") then run
chef-client and it will succeed.

I see that the unix error code "1" means a permissions issue; is that
what I'm facing here or is rake (er, bundle I guess) just using this as a
generic error return code? Any way I can get a more specific reason as to
why the migrate is failing?

The chef-stacktrace.out file doesn't give any more specific info about
the actual db:migrate failure.

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

7

Well I just gave your suggestion a try and it seems to have worked! Many
thanks... looking at the recipe again, it makes sense, because it's not
using "rbenv_execute" like for the "bundle install" step. I assume
rbenv_execute takes care of the rbenv.sh automatically but for
migration_command I need to do it myself. I guess I could also get rid of
the "built-in" migration step and just issue a second rbenv_execute myself.

environment 'RAILS_ENV' => app[:rails_env]
migration_command '. /etc/profile.d/rbenv.sh && bundle exec rake
db:migrate --trace'

user app[:owner]
group app[:group]

before_migrate do
current_release = release_path

rbenv_execute 'bundle install' do
  environment 'RAILS_ENV' => app[:rails_env]
  command 'bundle install'
  cwd current_release
end

end

On Tue, Feb 18, 2014 at 9:05 PM, Toby Johnson irontoby@gmail.com wrote:

Okay Cassiano, that makes sense, thanks for the pointer. I'll give it a
try.

I hadn't heard of the application-rails cookbook (though a Google search
suggests maybe you meant application-ruby?) and since I'm using nginx I
don't know if it would apply to me but I can certainly make the same call
using the deploy cookbook too.

On Tue, Feb 18, 2014 at 10:31 AM, Cassiano Leal cassianoleal@gmail.comwrote:

Hey Toby,

You might need to load rbenv to the shell before running bundle. The
application-rails cookbook makes this easy for you:

application "my_app" do
...
rails do
bundle_command ". /etc/profile.d/rbenv.sh &&
RAILS_ENV=#{node.chef_environment} bundle"
...
end
end

That's how I used to do. The path to the rbenv init script might not be
right -- I was using chruby --, so adjust to your environment.

On 18 February 2014 07:35, Vladimir Skubriev skubriev@cvisionlab.comwrote:

To be honest I do not know.

2014-02-18 9:15 GMT+04:00 Toby Johnson irontoby@gmail.com:

Thanks Vladimir; my apologies, I meant to mention that Ruby is installed

with rbenv, but using a "global" setup at /opt/rbenv and not in a home
directory.

I am setting up RAILS_ENV to production using the "environment" command
in the deploy_revision resource.

My database.yml is a symlink to the "shared" one
at /var/www/shared/config/database.yml. That file has restrictive
permissions (owner root:nginx with ug+r; or 0440), while all directories
from /var/www below have owner root:nginx and permissions u+rwx,g+rx; or
0750).

One point of confusion I have, is which user actually runs the "rake
db:migrate" command? I didn't see anywhere to set this using the deploy
cookbook.

On Mon, Feb 17, 2014 at 11:20 PM, Vladimir Skubriev <
skubriev@cvisionlab.com> wrote:

This is permission issue.

Or may be environment variables do not setup correctly.

I don't know application cookbook. But I has a question: where is the
ruby installed with it (in home or system ruby)?

2014-02-18 6:51 GMT+04:00 Toby Johnson irontoby@gmail.com:

I'm using the deploy cookbook to deploy a Rails app (nginx +

passenger). I'm running "bundle exec rake db:migrate --trace" during the
migration step, after the database is created. Many times this step will
fail with:

ERROR: deploy_revision[/var/www] (_site::deploy line 39) had an
error: Chef::Exceptions::Exec: bundle exec rake db:migrate --trace returned
1, expected 0

It is very frustrating because it will sometimes succeed but often
will fail. It seems like more often, when I run via knife ssh ("knife ssh
'fqdn:xxx*' 'sudo chef-client'), it will fail at the above step, then I ssh
into the actual box, start a shell as root ("sudo su -") then run
chef-client and it will succeed.

I see that the unix error code "1" means a permissions issue; is that
what I'm facing here or is rake (er, bundle I guess) just using this as a
generic error return code? Any way I can get a more specific reason as to
why the migrate is failing?

The chef-stacktrace.out file doesn't give any more specific info
about the actual db:migrate failure.

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

8

Yes, I meant the application_ruby cb.

Glad you solved it!

On 19 February 2014 02:22, Toby Johnson irontoby@gmail.com wrote:

Well I just gave your suggestion a try and it seems to have worked! Many
thanks... looking at the recipe again, it makes sense, because it's not
using "rbenv_execute" like for the "bundle install" step. I assume
rbenv_execute takes care of the rbenv.sh automatically but for
migration_command I need to do it myself. I guess I could also get rid of
the "built-in" migration step and just issue a second rbenv_execute myself.

environment 'RAILS_ENV' => app[:rails_env]
migration_command '. /etc/profile.d/rbenv.sh && bundle exec rake
db:migrate --trace'

user app[:owner]
group app[:group]

before_migrate do
current_release = release_path

rbenv_execute 'bundle install' do
  environment 'RAILS_ENV' => app[:rails_env]
  command 'bundle install'
  cwd current_release
end

end

On Tue, Feb 18, 2014 at 9:05 PM, Toby Johnson irontoby@gmail.com wrote:

Okay Cassiano, that makes sense, thanks for the pointer. I'll give it a
try.

I hadn't heard of the application-rails cookbook (though a Google search
suggests maybe you meant application-ruby?) and since I'm using nginx I
don't know if it would apply to me but I can certainly make the same call
using the deploy cookbook too.

On Tue, Feb 18, 2014 at 10:31 AM, Cassiano Leal cassianoleal@gmail.comwrote:

Hey Toby,

You might need to load rbenv to the shell before running bundle. The
application-rails cookbook makes this easy for you:

application "my_app" do
...
rails do
bundle_command ". /etc/profile.d/rbenv.sh &&
RAILS_ENV=#{node.chef_environment} bundle"
...
end
end

That's how I used to do. The path to the rbenv init script might not be
right -- I was using chruby --, so adjust to your environment.

On 18 February 2014 07:35, Vladimir Skubriev skubriev@cvisionlab.comwrote:

To be honest I do not know.

2014-02-18 9:15 GMT+04:00 Toby Johnson irontoby@gmail.com:

Thanks Vladimir; my apologies, I meant to mention that Ruby is

installed with rbenv, but using a "global" setup at /opt/rbenv and not in a
home directory.

I am setting up RAILS_ENV to production using the "environment"
command in the deploy_revision resource.

My database.yml is a symlink to the "shared" one
at /var/www/shared/config/database.yml. That file has restrictive
permissions (owner root:nginx with ug+r; or 0440), while all directories
from /var/www below have owner root:nginx and permissions u+rwx,g+rx; or
0750).

One point of confusion I have, is which user actually runs the "rake
db:migrate" command? I didn't see anywhere to set this using the deploy
cookbook.

On Mon, Feb 17, 2014 at 11:20 PM, Vladimir Skubriev <
skubriev@cvisionlab.com> wrote:

This is permission issue.

Or may be environment variables do not setup correctly.

I don't know application cookbook. But I has a question: where is the
ruby installed with it (in home or system ruby)?

2014-02-18 6:51 GMT+04:00 Toby Johnson irontoby@gmail.com:

I'm using the deploy cookbook to deploy a Rails app (nginx +

passenger). I'm running "bundle exec rake db:migrate --trace" during the
migration step, after the database is created. Many times this step will
fail with:

ERROR: deploy_revision[/var/www] (_site::deploy line 39) had an
error: Chef::Exceptions::Exec: bundle exec rake db:migrate --trace returned
1, expected 0

It is very frustrating because it will sometimes succeed but often
will fail. It seems like more often, when I run via knife ssh ("knife ssh
'fqdn:xxx*' 'sudo chef-client'), it will fail at the above step, then I ssh
into the actual box, start a shell as root ("sudo su -") then run
chef-client and it will succeed.

I see that the unix error code "1" means a permissions issue; is
that what I'm facing here or is rake (er, bundle I guess) just using this
as a generic error return code? Any way I can get a more specific reason as
to why the migrate is failing?

The chef-stacktrace.out file doesn't give any more specific info
about the actual db:migrate failure.

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev

--
Faithfully yours,

CVision Lab System Administrator
Vladmir Skubriev