RabbitMQ and Logstash

Chef Infra (archive)
1

Can anyone point me to the most up to date / canonical rabbitmq or logstash
cookbook repositories? Hoping to not have to reinvent the wheel.

Thanks,
-Kevin

2

On Dec 20, 2011, at 10:08 AM, Kevin Nuckolls wrote:

Can anyone point me to the most up to date / canonical rabbitmq or logstash cookbook repositories? Hoping to not have to reinvent the wheel.

We don't use logstash, but we're using the current rabbitmq cookbook from the opscode community site. Works for us.

--
Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1

3

Hey Kevin,

i used that one some while ago, but i'm not sure right now if it was my
last choice.

https://github.com/jtimberman/logstash-cookbook/blob/master/recipes/default.rb

i think used Joshuas recipe, but i just used it for the client provisioning.

Greetings,
Sebastian

2011/12/20 Brad Knowles bknowles@ihiji.com

On Dec 20, 2011, at 10:08 AM, Kevin Nuckolls wrote:

Can anyone point me to the most up to date / canonical rabbitmq or
logstash cookbook repositories? Hoping to not have to reinvent the wheel.

We don't use logstash, but we're using the current rabbitmq cookbook from
the opscode community site. Works for us.

--
Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1

--

Mit freundlichen Grüßen,
Sebastian Wendel.

//////////////////////////////////////////////////////////////////////////////////////

SourceIndex IT-Services, Wohlwillstraße 2, 20359 Hamburg

Office: +49(0)40-209 344 900, Fax: +49(0)40-209 344 909
Mobil: +49(0)160 150 3154, Email: s.wendel@sourceindex.de

Web: http://www.sourceindex.de
Xing: Sebastian Wendel - Fullstack Developer & Platform Engineer - REBOOT IN PROGRESS | XING

WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

4

Thanks. I'll dig through all of these. :slight_smile:

On Tue, Dec 20, 2011 at 10:49 AM, Sebastian Wendel
s.wendel@sourceindex.dewrote:

Hey Kevin,

i used that one some while ago, but i'm not sure right now if it was my
last choice.

https://github.com/jtimberman/logstash-cookbook/blob/master/recipes/default.rb

i think used Joshuas recipe, but i just used it for the client
provisioning.

GitHub - heavywater/chef-logstash: "Installs/Configures logstash"

Greetings,
Sebastian

2011/12/20 Brad Knowles bknowles@ihiji.com

On Dec 20, 2011, at 10:08 AM, Kevin Nuckolls wrote:

Can anyone point me to the most up to date / canonical rabbitmq or
logstash cookbook repositories? Hoping to not have to reinvent the wheel.

We don't use logstash, but we're using the current rabbitmq cookbook from
the opscode community site. Works for us.

--
Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1

--

Mit freundlichen Grüßen,
Sebastian Wendel.

//////////////////////////////////////////////////////////////////////////////////////

SourceIndex IT-Services, Wohlwillstraße 2, 20359 Hamburg

Office: +49(0)40-209 344 900, Fax: +49(0)40-209 344 909
Mobil: +49(0)160 150 3154, Email: s.wendel@sourceindex.de

Web: http://www.sourceindex.de
Xing: Sebastian Wendel - Fullstack Developer & Platform Engineer - REBOOT IN PROGRESS | XING

WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

5

On Wed, Dec 21, 2011 at 11:44 AM, Kevin Nuckolls
kevin.nuckolls@gmail.comwrote:

i used that one some while ago, but i'm not sure right now if it was my

last choice.

https://github.com/jtimberman/logstash-cookbook/blob/master/recipes/default.rb

i think used Joshuas recipe, but i just used it for the client
provisioning.

GitHub - heavywater/chef-logstash: "Installs/Configures logstash"

I just went through this same investigation. I ended up with recipes for
rabbitmq, elasticsearch and logstash (with grok) and then apache to
terminate SSL and proxy requests to the logstash web interface. My goal was
to have a an N node cluster of the above application stack consuming logs
from various sources.

Most of the Cookbooks I used were the Opscode supported ones. I added some
things I wanted (more for trying my hand and writing / understanding chef).
For example, I added auto-clustering and ssl support for rabbitmq and a
RedHat/Centos-friendly init for elasticsearch. I modified Joshua's logstash
one to have RedHat/Centos friendly init and made attributes for the
directories and such. I've passed most of it back as pull requests, but
frankly, I'm new to Chef. So there's no telling if I did it right.

I've experienced most of the pain recently so it's fresh in my mind if you
want to ping me to share ideas, I'd be happy to chat. I'm interested in
seeing if my conclusions about things are in the ballpark.

6

Harlan,

We're currently working on a similar project. If you want to contribute
your changes back / post them somewhere, we're happy to build on it and do
the same.

James

On Wed, Dec 21, 2011 at 9:35 PM, Harlan Barnes hbarnes@pobox.com wrote:

On Wed, Dec 21, 2011 at 11:44 AM, Kevin Nuckolls <kevin.nuckolls@gmail.com

wrote:

i used that one some while ago, but i'm not sure right now if it was my

last choice.

https://github.com/jtimberman/logstash-cookbook/blob/master/recipes/default.rb

i think used Joshuas recipe, but i just used it for the client
provisioning.

GitHub - heavywater/chef-logstash: "Installs/Configures logstash"

I just went through this same investigation. I ended up with recipes for
rabbitmq, elasticsearch and logstash (with grok) and then apache to
terminate SSL and proxy requests to the logstash web interface. My goal was
to have a an N node cluster of the above application stack consuming logs
from various sources.

Most of the Cookbooks I used were the Opscode supported ones. I added some
things I wanted (more for trying my hand and writing / understanding chef).
For example, I added auto-clustering and ssl support for rabbitmq and a
RedHat/Centos-friendly init for elasticsearch. I modified Joshua's logstash
one to have RedHat/Centos friendly init and made attributes for the
directories and such. I've passed most of it back as pull requests, but
frankly, I'm new to Chef. So there's no telling if I did it right.

I've experienced most of the pain recently so it's fresh in my mind if you
want to ping me to share ideas, I'd be happy to chat. I'm interested in
seeing if my conclusions about things are in the ballpark.

7

Messed up and only sent to James the first time.

On Tue, Jan 10, 2012 at 3:01 PM, James js@aegisco.com wrote:

We're currently working on a similar project. If you want to contribute
your changes back / post them somewhere, we're happy to build on it and do
the same.

Hi James - Sure, here's where I am. (I've been sidetracked with another
project, but I've solved what I think are hard problems. I've tested
everything individually but not all together.)

My plan was two parts A) to have logstash act as an agent/shipper on of our
nodes producing logs with a simple input from the local file to the
rabbitmq cluster and B) have X number of logstash instances listening to
the same queue in hopes of spreading the consuming out across multiple
nodes. The logstash instances in B would do heavy filtering (grok'ing each
event type to parse out interesting information to index in elasticsearch).

The log agent side ("A" above) is pretty easy. I modified Joshua
Timerbman's cookbook to:

  • Changed the agent startup scripts so that it will use Jordan's
    "directory" functionality instead of the explicit script. (The plan was to
    have other recipes dump their configurations into the directory as needed
    and let the agent glob them up.)
  • To not install the agent.conf unless you tell it to (so it doesn't
    interfere with my own agent configuration above)
  • I also changed the web startup scripts so that it didn't enforce
    'localhost' as the elasticsearch backend (it's an attribute now).
  • Added Centos (init.d+daemonize) service support
  • Here's the pull request I sent Joshua (and of course, it references back
    my version of it): https://github.com/jtimberman/logstash-cookbook/pull/1

On the server side ("B" above), my goal was have each one of the main
components "clustered" (or at least stateless so they could be load
balanced as is the case with the logstash web interface). I also wanted to
be able to run a node having each "component" installed ... or run each
component on a node by itself. Here's what I ended up with:

There's more work to do (probably in a cookbook that implements the other
cookbooks):

  • Setup the users and queues on rabbitmq (There's a LWRP in the rabbitmq
    cookbook for that.)
  • Write the logstash configuration into files or templates to drop in the
    configuration directory.

So that's it. I'm still relatively new to chef, so I'm sure there's lots of
"wrong stuff." :slight_smile:

8

I've ditched rabbitmq completely in favor of zeromq. apparently the
bunny gem doesn't perform so well, and zeromq both alleviate the need
for a broker and give great performance.

Regards,
Avishai

On 01/11/2012 07:27 PM, Harlan Barnes wrote:

Messed up and only sent to James the first time.

On Tue, Jan 10, 2012 at 3:01 PM, James <js@aegisco.com
mailto:js@aegisco.com> wrote:

We're currently working on a similar project. If you want to
contribute your changes back / post them somewhere, we're happy to
build on it and do the same.

Hi James - Sure, here's where I am. (I've been sidetracked with
another project, but I've solved what I think are hard problems. I've
tested everything individually but not all together.)

My plan was two parts A) to have logstash act as an agent/shipper on
of our nodes producing logs with a simple input from the local file to
the rabbitmq cluster and B) have X number of logstash instances
listening to the same queue in hopes of spreading the consuming out
across multiple nodes. The logstash instances in B would do heavy
filtering (grok'ing each event type to parse out interesting
information to index in elasticsearch).

The log agent side ("A" above) is pretty easy. I modified Joshua
Timerbman's cookbook to:

  • Changed the agent startup scripts so that it will use Jordan's
    "directory" functionality instead of the explicit script. (The plan
    was to have other recipes dump their configurations into the directory
    as needed and let the agent glob them up.)
  • To not install the agent.conf unless you tell it to (so it doesn't
    interfere with my own agent configuration above)
  • I also changed the web startup scripts so that it didn't enforce
    'localhost' as the elasticsearch backend (it's an attribute now).
  • Added Centos (init.d+daemonize) service support
  • Here's the pull request I sent Joshua (and of course, it references
    back my version of it):
    https://github.com/jtimberman/logstash-cookbook/pull/1

On the server side ("B" above), my goal was have each one of the main
components "clustered" (or at least stateless so they could be load
balanced as is the case with the logstash web interface). I also
wanted to be able to run a node having each "component" installed ...
or run each component on a node by itself. Here's what I ended up with:

There's more work to do (probably in a cookbook that implements the
other cookbooks):

  • Setup the users and queues on rabbitmq (There's a LWRP in the
    rabbitmq cookbook for that.)
  • Write the logstash configuration into files or templates to drop in
    the configuration directory.

So that's it. I'm still relatively new to chef, so I'm sure there's
lots of "wrong stuff." :slight_smile: