Emergency help

Chef Infra (archive)
1

Hey Guys,

My Chef Server attained a new IP address; therefore, none of my clients can
connect to chef server and I can’t even access the interface for chef
anymore. Any help?

–

  • Nikhil Shah */ System Administrator

nshah@theorchard.com

The Orchard® / www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com http://www.dailyrindblog.com/

Facebook http://www.facebook.com/theorchard /
@orchtweetshttp://www.twitter.com/orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are
protected by the Electronic Communications Privacy Act (18 USC §§
2510-2521), (b) may contain confidential and/or legally privileged
information, and © are for the sole use of the intended recipient named
above. If you have received this electronic message in error, please notify
the sender and delete the electronic message. Any disclosure, copying,
distribution, or use of the contents of the information received in error
is strictly prohibited

2

Are your clients referencing an IP address or hostname? Look in your
client config (/etc/chef/client.rb). Send verbose output of a chef-client
run so that we can see what is happening.

On Wed, Sep 18, 2013 at 11:17 AM, Nikhil Shah nshah@theorchard.com wrote:

Hey Guys,

My Chef Server attained a new IP address; therefore, none of my clients
can connect to chef server and I can't even access the interface for chef
anymore. Any help?

--

  • Nikhil Shah */ System Administrator

nshah@theorchard.com

The Orchard® / www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com http://www.dailyrindblog.com/

Facebook http://www.facebook.com/theorchard / @orchtweetshttp://www.twitter.com/orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are
protected by the Electronic Communications Privacy Act (18 USC §§
2510-2521), (b) may contain confidential and/or legally privileged
information, and (c) are for the sole use of the intended recipient named
above. If you have received this electronic message in error, please notify
the sender and delete the electronic message. Any disclosure, copying,
distribution, or use of the contents of the information received in error
is strictly prohibited

--
Charles Sullivan
charlie.sullivan@gmail.com

3

If you are using DNS: change the A record in DNS to point to the new IP address. Everything should start working again within the next few hours (depending on how long your TTL is). Depending on your setup, some clients may start working again while others will still be failing. You can speed up the process by restarting your DNS server and/or flushing DNS caches.

If you are using the IP address directly, you will have no choice but to go around to each machine and update it.

Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Nikhil Shah nshah@theorchard.com
Sent: Wednesday 18th September 2013 10:17
To: chef@lists.opscode.com
Subject: [chef] emergency help

Hey Guys,

My Chef Server attained a new IP address; therefore, none of my clients can connect to chef server and I can’t even access the interface for chef anymore. Any help?

–
Nikhil Shah / System Administrator

nshah@theorchard.com http://theorchard.com

The Orchard® / www.theorchard.com http://www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com http://www.dailyrindblog.com/

Facebook http://www.facebook.com/theorchard / @orchtweets http://www.twitter.com/orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are protected by the Electronic Communications Privacy Act (18 USC §§ 2510-2521), (b) may contain confidential and/or legally privileged information, and © are for the sole use of the intended recipient named above. If you have received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of the information received in error is strictly prohibited

4

now what about the chef server itself. I can't seem to access the
interface. I keep getting a gateway timed out, I tried to
run chef-server-ctl start to start all of the services

On Wed, Sep 18, 2013 at 1:25 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

**

If you are using DNS: change the A record in DNS to point to the new IP
address. Everything should start working again within the next few hours
(depending on how long your TTL is). Depending on your setup, some clients
may start working again while others will still be failing. You can speed
up the process by restarting your DNS server and/or flushing DNS caches.

If you are using the IP address directly, you will have no choice but to
go around to each machine and update it.

Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Nikhil Shah nshah@theorchard.com
Sent: Wednesday 18th September 2013 10:17
To: chef@lists.opscode.com
Subject: [chef] emergency help

Hey Guys,

My Chef Server attained a new IP address; therefore, none of my clients
can connect to chef server and I can't even access the interface for chef
anymore. Any help?

--

  • Nikhil Shah */ System Administrator

nshah@theorchard.com

The Orchard® / www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com http://www.dailyrindblog.com/

Facebook http://www.facebook.com/theorchard / @orchtweetshttp://www.twitter.com/orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are
protected by the Electronic Communications Privacy Act (18 USC §§
2510-2521), (b) may contain confidential and/or legally privileged
information, and (c) are for the sole use of the intended recipient named
above. If you have received this electronic message in error, please notify
the sender and delete the electronic message. Any disclosure, copying,
distribution, or use of the contents of the information received in error
is strictly prohibited

--

  • Nikhil Shah */ System Administrator

nshah@theorchard.com

The Orchard® / www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com http://www.dailyrindblog.com/

Facebook http://www.facebook.com/theorchard /
@orchtweetshttp://www.twitter.com/orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are
protected by the Electronic Communications Privacy Act (18 USC §§
2510-2521), (b) may contain confidential and/or legally privileged
information, and (c) are for the sole use of the intended recipient named
above. If you have received this electronic message in error, please notify
the sender and delete the electronic message. Any disclosure, copying,
distribution, or use of the contents of the information received in error
is strictly prohibited

5

Hey Nikhil,

I'm assuming you have your own private chef server ( of some version ) and a million more guesses and assumptions .....

... tell us a little more about what is the nature of new IP address .... public IP?, private IP? ...

Do you control the IP address or is the "loss" part of a change over do to changing hosting or ??? that was out of your control ....

Was the IP mapping via DNS records to something ... or....

More details will give more room to speculate on remediation or a temporary forwarding the traffic to the right place ( IP address .... )

Security and Mapping of the DNS records are in order....

Terry

Terry J Fundak
Systems Engineer
Network Design and Security Solutions for SMBs
Tech Support - Client and Server Systems

TJSoftworks
1834 Chadwick Court
Santa Rosa, CA 95401
(707) 849-1000
(415) 877-1783
terry@tjsoftworks.com

On Sep 18, 2013, at 10:17, Nikhil Shah wrote:

Hey Guys,

My Chef Server attained a new IP address; therefore, none of my clients can connect to chef server and I can't even access the interface for chef anymore. Any help?

--
Nikhil Shah / System Administrator

nshah@theorchard.com

The Orchard® / www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com

Facebook / @orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are protected by the Electronic Communications Privacy Act (18 USC §§ 2510-2521), (b) may contain confidential and/or legally privileged information, and (c) are for the sole use of the intended recipient named above. If you have received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of the information received in error is strictly prohibited

6

Is this a VM or a physical machine? What happened to the old IP address? If it's a VM you could just attach another virtual NIC and assign the old IP address assuming it isn't being used elsewhere.

If you have console access (assuming you do if you can run commands), make sure to check the routes/iptables to ensure the new IP will allow access.

On Sep 18, 2013, at 1:29 PM, Nikhil Shah nshah@theorchard.com wrote:

now what about the chef server itself. I can't seem to access the interface. I keep getting a gateway timed out, I tried to run chef-server-ctl start to start all of the services

On Wed, Sep 18, 2013 at 1:25 PM, Kevin Keane Subscription subscription@kkeane.com wrote:

If you are using DNS: change the A record in DNS to point to the new IP address. Everything should start working again within the next few hours (depending on how long your TTL is). Depending on your setup, some clients may start working again while others will still be failing. You can speed up the process by restarting your DNS server and/or flushing DNS caches.

If you are using the IP address directly, you will have no choice but to go around to each machine and update it.

Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Nikhil Shah nshah@theorchard.com
Sent: Wednesday 18th September 2013 10:17
To: chef@lists.opscode.com
Subject: [chef] emergency help

Hey Guys,

My Chef Server attained a new IP address; therefore, none of my clients can connect to chef server and I can't even access the interface for chef anymore. Any help?

--
Nikhil Shah / System Administrator

nshah@theorchard.com

The Orchard® / www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com

Facebook / @orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are protected by the Electronic Communications Privacy Act (18 USC §§ 2510-2521), (b) may contain confidential and/or legally privileged information, and (c) are for the sole use of the intended recipient named above. If you have received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of the information received in error is strictly prohibited

--
Nikhil Shah / System Administrator

nshah@theorchard.com

The Orchard® / www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com

Facebook / @orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are protected by the Electronic Communications Privacy Act (18 USC §§ 2510-2521), (b) may contain confidential and/or legally privileged information, and (c) are for the sole use of the intended recipient named above. If you have received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of the information received in error is strictly prohibited

7

Actually, I did get it working...It was running off of an ESX server and
seems like when i restarted the services again, I Was able to log back in.

Thank You!

On Wed, Sep 18, 2013 at 1:30 PM, Terry J Fundak terry@tjsoftworks.comwrote:

Hey Nikhil,

I'm assuming you have your own private chef server ( of some version ) and
a million more guesses and assumptions .....

... tell us a little more about what is the nature of new IP address ....
public IP?, private IP? ...

Do you control the IP address or is the "loss" part of a change over do to
changing hosting or ??? that was out of your control ....

Was the IP mapping via DNS records to something ... or....

More details will give more room to speculate on remediation or a
temporary forwarding the traffic to the right place ( IP address .... )

Security and Mapping of the DNS records are in order....

Terry

Terry J Fundak
Systems Engineer
Network Design and Security Solutions for SMBs
Tech Support - Client and Server Systems

TJSoftworks
1834 Chadwick Court
Santa Rosa, CA 95401
(707) 849-1000
(415) 877-1783
terry@tjsoftworks.com

On Sep 18, 2013, at 10:17, Nikhil Shah wrote:

Hey Guys,

My Chef Server attained a new IP address; therefore, none of my clients
can connect to chef server and I can't even access the interface for chef
anymore. Any help?

--

  • Nikhil Shah */ System Administrator

nshah@theorchard.com http://theorchard.com/

The Orchard® / www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com http://www.dailyrindblog.com/

Facebook http://www.facebook.com/theorchard / @orchtweetshttp://www.twitter.com/orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are
protected by the Electronic Communications Privacy Act (18 USC §§
2510-2521), (b) may contain confidential and/or legally privileged
information, and (c) are for the sole use of the intended recipient named
above. If you have received this electronic message in error, please notify
the sender and delete the electronic message. Any disclosure, copying,
distribution, or use of the contents of the information received in error
is strictly prohibited

--

  • Nikhil Shah */ System Administrator

nshah@theorchard.com

The Orchard® / www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com http://www.dailyrindblog.com/

Facebook http://www.facebook.com/theorchard /
@orchtweetshttp://www.twitter.com/orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are
protected by the Electronic Communications Privacy Act (18 USC §§
2510-2521), (b) may contain confidential and/or legally privileged
information, and (c) are for the sole use of the intended recipient named
above. If you have received this electronic message in error, please notify
the sender and delete the electronic message. Any disclosure, copying,
distribution, or use of the contents of the information received in error
is strictly prohibited

8

“Gateway timed out” means that you have a proxy server (probably a reverse proxy in front of the Chef server), and the proxy server can’t talk to the back end server.

When you restarted all your services, that flushed the DNS caches, and forced reloading with the new information immediately That’s why things started working again for you.

Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Nikhil Shah nshah@theorchard.com
Sent: Wednesday 18th September 2013 10:29
To: chef@lists.opscode.com
Subject: [chef] Re: RE: emergency help

now what about the chef server itself. I can’t seem to access the interface. I keep getting a gateway timed out, I tried to run chef-server-ctl start to start all of the services

On Wed, Sep 18, 2013 at 1:25 PM, Kevin Keane Subscription <subscription@kkeane.com mailto:subscription@kkeane.com > wrote:

If you are using DNS: change the A record in DNS to point to the new IP address. Everything should start working again within the next few hours (depending on how long your TTL is). Depending on your setup, some clients may start working again while others will still be failing. You can speed up the process by restarting your DNS server and/or flushing DNS caches.

If you are using the IP address directly, you will have no choice but to go around to each machine and update it.

Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Nikhil Shah <nshah@theorchard.com mailto:nshah@theorchard.com >
Sent: Wednesday 18th September 2013 10:17
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] emergency help

Hey Guys,

My Chef Server attained a new IP address; therefore, none of my clients can connect to chef server and I can’t even access the interface for chef anymore. Any help?

–
Nikhil Shah / System Administrator

nshah@theorchard.com http://theorchard.com

The Orchard® / www.theorchard.com http://www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com http://www.dailyrindblog.com/

Facebook http://www.facebook.com/theorchard / @orchtweets http://www.twitter.com/orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are protected by the Electronic Communications Privacy Act (18 USC §§ 2510-2521), (b) may contain confidential and/or legally privileged information, and © are for the sole use of the intended recipient named above. If you have received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of the information received in error is strictly prohibited

–
Nikhil Shah / System Administrator

nshah@theorchard.com http://theorchard.com

The Orchard® / www.theorchard.com http://www.theorchard.com

t (+1) 212.308.5648 / f (+1) 212.201.9203
23 E. 4th St., 3rd Fl / New York, NY 10003

The Daily Rind™ / www.dailyrindblog.com http://www.dailyrindblog.com/

Facebook http://www.facebook.com/theorchard / @orchtweets http://www.twitter.com/orchtweets

Privileged And Confidential Communication.

This electronic transmission, and any documents attached hereto, (a) are protected by the Electronic Communications Privacy Act (18 USC §§ 2510-2521), (b) may contain confidential and/or legally privileged information, and © are for the sole use of the intended recipient named above. If you have received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of the information received in error is strictly prohibited