Supervisord service not using correct env variables

Chef Infra (archive)
1

Dear Chefs,

I am using the application_python cookbook to set up the
readthedocs.org application.

I need the services to run with a number of special environment variables

i have this in my configuration for the rtfd-celeryd service file
/etc/supervisor

user=docs
environment=HOME=’/home/docs’,USER=‘docs’,PIP_DOWNLOAD_CACHE=’/tmp/pip’,DJANGO_SETTINGS_MODULE=‘settings’,PYTHONPATH="$PYTHONPATH:/opt/rtfd/"

unfortunately, supervisord doesn’t seem to pick them up and persists
w/ the env vars from the root user

here is a dump of the environment variables from within my celery service

(Pdb) environment
{’_’: ‘/usr/bin/sudo’, ‘MP_FORK_LOGFILE’: ‘’, ‘CELERY_LOG_REDIRECT’:
‘1’, ‘LOGNAME’: ‘root’, ‘USER’: ‘root’, ‘PATH’:
’/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin’,
‘HOME’: ‘/home/ubuntu’, ‘LANG’: ‘en_US.UTF-8’, ‘TERM’: ‘xterm’,
‘SHELL’: ‘/bin/bash’, ‘TZ’: ‘America/Chicago’, ‘MP_FORK_LOGFORMAT’:
’[%(asctime)s: %(levelname)s/%(processName)s] %(message)s’, ‘SHLVL’:
‘1’, ‘SUPERVISOR_ENABLED’: ‘1’, ‘CELERY_LOG_FILE’: ‘’,
‘DJANGO_PROJECT_DIR’:
’/opt/rtfd/releases/39376186d93dd861e091b68ef0031984d84e979a/readthedocs’,
‘SUDO_USER’: ‘ubuntu’, ‘USERNAME’: ‘root’, ‘READTHEDOCS’: ‘True’,
‘SUDO_UID’: ‘1000’, ‘SUPERVISOR_PROCESS_NAME’: ‘rtfd-celeryd’,
‘SUPERVISOR_SERVER_URL’: ‘unix:///var/run/supervisor.sock’, ‘LC_ALL’:
‘C’, ‘CELERY_LOG_REDIRECT_LEVEL’: ‘WARNING’, ‘SUDO_COMMAND’:
’/usr/bin/chef-solo --config /tmp/kitchen-chef-solo/solo.rb
–json-attributes /tmp/kitchen-chef-solo/dna.json --log_level info’,
‘SUDO_GID’: ‘1000’, ‘CELERY_LOADER’: ‘djcelery.loaders.DjangoLoader’,
‘SSH_TTY’: ‘/dev/pts/0’, ‘CELERY_LOG_LEVEL’: ‘30’, ‘PWD’: ‘/’,
‘MP_FORK_LOGLEVEL’: ‘30’, ‘MAIL’: ‘/var/mail/ubuntu’,
‘SUPERVISOR_GROUP_NAME’: ‘rtfd-celeryd’}

anyone have any advice here? thanks in advance

the application_python cookboook has been great so far and would hate
to have to shortcircuit it just because of the supervisord issues

2

On the machine, if you run supervisorctl reread does it list any files as
updated?

On Wed, Sep 4, 2013 at 10:43 AM, Bryan Berry bryan.berry@gmail.com wrote:

Dear Chefs,

I am using the application_python cookbook to set up the
readthedocs.org application.

I need the services to run with a number of special environment variables

i have this in my configuration for the rtfd-celeryd service file
/etc/supervisor

user=docs

environment=HOME='/home/docs',USER='docs',PIP_DOWNLOAD_CACHE='/tmp/pip',DJANGO_SETTINGS_MODULE='settings',PYTHONPATH="$PYTHONPATH:/opt/rtfd/"

unfortunately, supervisord doesn't seem to pick them up and persists
w/ the env vars from the root user

here is a dump of the environment variables from within my celery service

(Pdb) environment
{'_': '/usr/bin/sudo', 'MP_FORK_LOGFILE': '', 'CELERY_LOG_REDIRECT':
'1', 'LOGNAME': 'root', 'USER': 'root', 'PATH':
'/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin',
'HOME': '/home/ubuntu', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm',
'SHELL': '/bin/bash', 'TZ': 'America/Chicago', 'MP_FORK_LOGFORMAT':
'[%(asctime)s: %(levelname)s/%(processName)s] %(message)s', 'SHLVL':
'1', 'SUPERVISOR_ENABLED': '1', 'CELERY_LOG_FILE': '',
'DJANGO_PROJECT_DIR':
'/opt/rtfd/releases/39376186d93dd861e091b68ef0031984d84e979a/readthedocs',
'SUDO_USER': 'ubuntu', 'USERNAME': 'root', 'READTHEDOCS': 'True',
'SUDO_UID': '1000', 'SUPERVISOR_PROCESS_NAME': 'rtfd-celeryd',
'SUPERVISOR_SERVER_URL': 'unix:///var/run/supervisor.sock', 'LC_ALL':
'C', 'CELERY_LOG_REDIRECT_LEVEL': 'WARNING', 'SUDO_COMMAND':
'/usr/bin/chef-solo --config /tmp/kitchen-chef-solo/solo.rb
--json-attributes /tmp/kitchen-chef-solo/dna.json --log_level info',
'SUDO_GID': '1000', 'CELERY_LOADER': 'djcelery.loaders.DjangoLoader',
'SSH_TTY': '/dev/pts/0', 'CELERY_LOG_LEVEL': '30', 'PWD': '/',
'MP_FORK_LOGLEVEL': '30', 'MAIL': '/var/mail/ubuntu',
'SUPERVISOR_GROUP_NAME': 'rtfd-celeryd'}

anyone have any advice here? thanks in advance

the application_python cookboook has been great so far and would hate
to have to shortcircuit it just because of the supervisord issues

3

it does list my service file as updated, however i don't see any
change in the environment variables

On Wed, Sep 4, 2013 at 4:49 PM, Andrew Gross andrew@yipit.com wrote:

On the machine, if you run supervisorctl reread does it list any files as
updated?

On Wed, Sep 4, 2013 at 10:43 AM, Bryan Berry bryan.berry@gmail.com wrote:

Dear Chefs,

I am using the application_python cookbook to set up the
readthedocs.org application.

I need the services to run with a number of special environment variables

i have this in my configuration for the rtfd-celeryd service file
/etc/supervisor

user=docs

environment=HOME='/home/docs',USER='docs',PIP_DOWNLOAD_CACHE='/tmp/pip',DJANGO_SETTINGS_MODULE='settings',PYTHONPATH="$PYTHONPATH:/opt/rtfd/"

unfortunately, supervisord doesn't seem to pick them up and persists
w/ the env vars from the root user

here is a dump of the environment variables from within my celery service

(Pdb) environment
{'_': '/usr/bin/sudo', 'MP_FORK_LOGFILE': '', 'CELERY_LOG_REDIRECT':
'1', 'LOGNAME': 'root', 'USER': 'root', 'PATH':
'/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin',
'HOME': '/home/ubuntu', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm',
'SHELL': '/bin/bash', 'TZ': 'America/Chicago', 'MP_FORK_LOGFORMAT':
'[%(asctime)s: %(levelname)s/%(processName)s] %(message)s', 'SHLVL':
'1', 'SUPERVISOR_ENABLED': '1', 'CELERY_LOG_FILE': '',
'DJANGO_PROJECT_DIR':
'/opt/rtfd/releases/39376186d93dd861e091b68ef0031984d84e979a/readthedocs',
'SUDO_USER': 'ubuntu', 'USERNAME': 'root', 'READTHEDOCS': 'True',
'SUDO_UID': '1000', 'SUPERVISOR_PROCESS_NAME': 'rtfd-celeryd',
'SUPERVISOR_SERVER_URL': 'unix:///var/run/supervisor.sock', 'LC_ALL':
'C', 'CELERY_LOG_REDIRECT_LEVEL': 'WARNING', 'SUDO_COMMAND':
'/usr/bin/chef-solo --config /tmp/kitchen-chef-solo/solo.rb
--json-attributes /tmp/kitchen-chef-solo/dna.json --log_level info',
'SUDO_GID': '1000', 'CELERY_LOADER': 'djcelery.loaders.DjangoLoader',
'SSH_TTY': '/dev/pts/0', 'CELERY_LOG_LEVEL': '30', 'PWD': '/',
'MP_FORK_LOGLEVEL': '30', 'MAIL': '/var/mail/ubuntu',
'SUPERVISOR_GROUP_NAME': 'rtfd-celeryd'}

anyone have any advice here? thanks in advance

the application_python cookboook has been great so far and would hate
to have to shortcircuit it just because of the supervisord issues

4

When supervisor starts it reads the files from disk in to memory. Changing
the files on disk does nothing until you run the reread command (or
restart supervisor. To get supervisor to apply the changes from disk to
the active programs, you need to run supervisorctl update.

In our case when we make changes we just pair the commands together with

supervisorctl reread && supervisorctl update

Currently we use our own hacked together supervisor recipe instead of the
opscode one, but we have the same underlying issues, but we use envdir for
our environments instead of templating into supervisor.

On Wed, Sep 4, 2013 at 11:02 AM, Bryan Berry bryan.berry@gmail.com wrote:

it does list my service file as updated, however i don't see any
change in the environment variables

On Wed, Sep 4, 2013 at 4:49 PM, Andrew Gross andrew@yipit.com wrote:

On the machine, if you run supervisorctl reread does it list any files
as
updated?

On Wed, Sep 4, 2013 at 10:43 AM, Bryan Berry bryan.berry@gmail.com
wrote:

Dear Chefs,

I am using the application_python cookbook to set up the
readthedocs.org application.

I need the services to run with a number of special environment
variables

i have this in my configuration for the rtfd-celeryd service file
/etc/supervisor

user=docs

environment=HOME='/home/docs',USER='docs',PIP_DOWNLOAD_CACHE='/tmp/pip',DJANGO_SETTINGS_MODULE='settings',PYTHONPATH="$PYTHONPATH:/opt/rtfd/"

unfortunately, supervisord doesn't seem to pick them up and persists
w/ the env vars from the root user

here is a dump of the environment variables from within my celery
service

(Pdb) environment
{'_': '/usr/bin/sudo', 'MP_FORK_LOGFILE': '', 'CELERY_LOG_REDIRECT':
'1', 'LOGNAME': 'root', 'USER': 'root', 'PATH':
'/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin',
'HOME': '/home/ubuntu', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm',
'SHELL': '/bin/bash', 'TZ': 'America/Chicago', 'MP_FORK_LOGFORMAT':
'[%(asctime)s: %(levelname)s/%(processName)s] %(message)s', 'SHLVL':
'1', 'SUPERVISOR_ENABLED': '1', 'CELERY_LOG_FILE': '',
'DJANGO_PROJECT_DIR':

'/opt/rtfd/releases/39376186d93dd861e091b68ef0031984d84e979a/readthedocs',

'SUDO_USER': 'ubuntu', 'USERNAME': 'root', 'READTHEDOCS': 'True',
'SUDO_UID': '1000', 'SUPERVISOR_PROCESS_NAME': 'rtfd-celeryd',
'SUPERVISOR_SERVER_URL': 'unix:///var/run/supervisor.sock', 'LC_ALL':
'C', 'CELERY_LOG_REDIRECT_LEVEL': 'WARNING', 'SUDO_COMMAND':
'/usr/bin/chef-solo --config /tmp/kitchen-chef-solo/solo.rb
--json-attributes /tmp/kitchen-chef-solo/dna.json --log_level info',
'SUDO_GID': '1000', 'CELERY_LOADER': 'djcelery.loaders.DjangoLoader',
'SSH_TTY': '/dev/pts/0', 'CELERY_LOG_LEVEL': '30', 'PWD': '/',
'MP_FORK_LOGLEVEL': '30', 'MAIL': '/var/mail/ubuntu',
'SUPERVISOR_GROUP_NAME': 'rtfd-celeryd'}

anyone have any advice here? thanks in advance

the application_python cookboook has been great so far and would hate
to have to shortcircuit it just because of the supervisord issues

5

tks for the tip Andrew, that seems to do it

On Wed, Sep 4, 2013 at 5:06 PM, Andrew Gross andrew@yipit.com wrote:

When supervisor starts it reads the files from disk in to memory. Changing
the files on disk does nothing until you run the reread command (or
restart supervisor. To get supervisor to apply the changes from disk to the
active programs, you need to run supervisorctl update.

In our case when we make changes we just pair the commands together with

supervisorctl reread && supervisorctl update

Currently we use our own hacked together supervisor recipe instead of the
opscode one, but we have the same underlying issues, but we use envdir for
our environments instead of templating into supervisor.

On Wed, Sep 4, 2013 at 11:02 AM, Bryan Berry bryan.berry@gmail.com wrote:

it does list my service file as updated, however i don't see any
change in the environment variables

On Wed, Sep 4, 2013 at 4:49 PM, Andrew Gross andrew@yipit.com wrote:

On the machine, if you run supervisorctl reread does it list any files
as
updated?

On Wed, Sep 4, 2013 at 10:43 AM, Bryan Berry bryan.berry@gmail.com
wrote:

Dear Chefs,

I am using the application_python cookbook to set up the
readthedocs.org application.

I need the services to run with a number of special environment
variables

i have this in my configuration for the rtfd-celeryd service file
/etc/supervisor

user=docs

environment=HOME='/home/docs',USER='docs',PIP_DOWNLOAD_CACHE='/tmp/pip',DJANGO_SETTINGS_MODULE='settings',PYTHONPATH="$PYTHONPATH:/opt/rtfd/"

unfortunately, supervisord doesn't seem to pick them up and persists
w/ the env vars from the root user

here is a dump of the environment variables from within my celery
service

(Pdb) environment
{'_': '/usr/bin/sudo', 'MP_FORK_LOGFILE': '', 'CELERY_LOG_REDIRECT':
'1', 'LOGNAME': 'root', 'USER': 'root', 'PATH':
'/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin',
'HOME': '/home/ubuntu', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm',
'SHELL': '/bin/bash', 'TZ': 'America/Chicago', 'MP_FORK_LOGFORMAT':
'[%(asctime)s: %(levelname)s/%(processName)s] %(message)s', 'SHLVL':
'1', 'SUPERVISOR_ENABLED': '1', 'CELERY_LOG_FILE': '',
'DJANGO_PROJECT_DIR':

'/opt/rtfd/releases/39376186d93dd861e091b68ef0031984d84e979a/readthedocs',
'SUDO_USER': 'ubuntu', 'USERNAME': 'root', 'READTHEDOCS': 'True',
'SUDO_UID': '1000', 'SUPERVISOR_PROCESS_NAME': 'rtfd-celeryd',
'SUPERVISOR_SERVER_URL': 'unix:///var/run/supervisor.sock', 'LC_ALL':
'C', 'CELERY_LOG_REDIRECT_LEVEL': 'WARNING', 'SUDO_COMMAND':
'/usr/bin/chef-solo --config /tmp/kitchen-chef-solo/solo.rb
--json-attributes /tmp/kitchen-chef-solo/dna.json --log_level info',
'SUDO_GID': '1000', 'CELERY_LOADER': 'djcelery.loaders.DjangoLoader',
'SSH_TTY': '/dev/pts/0', 'CELERY_LOG_LEVEL': '30', 'PWD': '/',
'MP_FORK_LOGLEVEL': '30', 'MAIL': '/var/mail/ubuntu',
'SUPERVISOR_GROUP_NAME': 'rtfd-celeryd'}

anyone have any advice here? thanks in advance

the application_python cookboook has been great so far and would hate
to have to shortcircuit it just because of the supervisord issues