Feeding chef with Inventory Database


#1

Hi guys!!

We have an IT ivnentory app, where all our machines are registered. We want
to chef get info from that app via API getting json files and use that json
files as attributes.

Is there any way to do this “right”, i.e: Caching that results in order to
not overload the inventory web app with queries on each node…

thanks!


Si necesitas una máquina para hacer algo y no la compras al final te darás
cuenta de que has pagado lo mismo y no tienes la máquina.–Henry Ford

Alberto


#2

was too crazy the question? ^^


Si necesitas una máquina para hacer algo y no la compras al final te darás
cuenta de que has pagado lo mismo y no tienes la máquina.–Henry Ford

Alberto

On Fri, Nov 15, 2013 at 1:47 PM, aL. ocholetrasaleatorias@gmail.com wrote:

Hi guys!!

We have an IT ivnentory app, where all our machines are registered. We
want to chef get info from that app via API getting json files and use that
json files as attributes.

Is there any way to do this “right”, i.e: Caching that results in order to
not overload the inventory web app with queries on each node…

thanks!


Si necesitas una máquina para hacer algo y no la compras al final te darás
cuenta de que has pagado lo mismo y no tienes la máquina.–Henry Ford

Alberto


#3

Hi,

It sort of depends on;

  • the amount of data requested from the inventory server per node
  • the number of nodes in your chef infrastructure
  • the frequency of converging
  • the reliability of the inventory app

Initially we had a similar situation except that we merged data in
from N different external sources. Today we only have one sources of
configuration data - chef data bags. What we tend to do now is
synchronise data from the external applications into the chef server
as data bag items. The advantage of this that the chef runs no longer
rely on external services that may or may not meet reliability or
performance requirements. It does add a bit more effort to synchronize
data in but it is well worth it. About the only painful thing is that
we have to take special care to ensure that the data appears in the
search index before kicking of a chef run if it is part of an
orchestration workflow

On Mon, Nov 18, 2013 at 7:26 PM, aL. ocholetrasaleatorias@gmail.com wrote:

was too crazy the question? ^^


Si necesitas una máquina para hacer algo y no la compras al final te darás
cuenta de que has pagado lo mismo y no tienes la máquina.–Henry Ford

Alberto

On Fri, Nov 15, 2013 at 1:47 PM, aL. ocholetrasaleatorias@gmail.com wrote:

Hi guys!!

We have an IT ivnentory app, where all our machines are registered. We
want to chef get info from that app via API getting json files and use that
json files as attributes.

Is there any way to do this “right”, i.e: Caching that results in order to
not overload the inventory web app with queries on each node…

thanks!


Si necesitas una máquina para hacer algo y no la compras al final te darás
cuenta de que has pagado lo mismo y no tienes la máquina.–Henry Ford

Alberto


Cheers,

Peter Donald


#4

Thanks for your reply Peter!

Looks like a comon sense way, was my first thougth but then i started to
think about keeping data sync, cron schedules to write those databags and
updating the data bags on the chef-server…

I see two ways of updating those data bags.

Sync: post-save action / hook on yout inventory app.
Async: cron’s scheduled

On both ways, there are a question mark: How often commit that changes on
databags to the chef-repo? each update? each XX minutes? manually?

And on the async way: workstation dedicated to that cron’s tasks? do the
chef-server make workstation tasks with those cron’s schedules?

I am just thinking out loud, to share knowledge and choose the least bad
way to do this!

:slight_smile:


Si necesitas una máquina para hacer algo y no la compras al final te darás
cuenta de que has pagado lo mismo y no tienes la máquina.–Henry Ford

Alberto

On Mon, Nov 18, 2013 at 9:36 AM, Peter Donald peter@realityforge.orgwrote:

Hi,

It sort of depends on;

  • the amount of data requested from the inventory server per node
  • the number of nodes in your chef infrastructure
  • the frequency of converging
  • the reliability of the inventory app

Initially we had a similar situation except that we merged data in
from N different external sources. Today we only have one sources of
configuration data - chef data bags. What we tend to do now is
synchronise data from the external applications into the chef server
as data bag items. The advantage of this that the chef runs no longer
rely on external services that may or may not meet reliability or
performance requirements. It does add a bit more effort to synchronize
data in but it is well worth it. About the only painful thing is that
we have to take special care to ensure that the data appears in the
search index before kicking of a chef run if it is part of an
orchestration workflow

On Mon, Nov 18, 2013 at 7:26 PM, aL. ocholetrasaleatorias@gmail.com
wrote:

was too crazy the question? ^^


Si necesitas una máquina para hacer algo y no la compras al final te
darás
cuenta de que has pagado lo mismo y no tienes la máquina.–Henry Ford

Alberto

On Fri, Nov 15, 2013 at 1:47 PM, aL. ocholetrasaleatorias@gmail.com
wrote:

Hi guys!!

We have an IT ivnentory app, where all our machines are registered. We
want to chef get info from that app via API getting json files and use
that

json files as attributes.

Is there any way to do this “right”, i.e: Caching that results in order
to

not overload the inventory web app with queries on each node…

thanks!


Si necesitas una máquina para hacer algo y no la compras al final te
darás

cuenta de que has pagado lo mismo y no tienes la máquina.–Henry Ford

Alberto


Cheers,

Peter Donald


#5

Hi,

On Mon, Nov 18, 2013 at 9:20 PM, aL. ocholetrasaleatorias@gmail.com wrote:

I see two ways of updating those data bags.

Sync: post-save action / hook on yout inventory app.
Async: cron’s scheduled

On both ways, there are a question mark: How often commit that changes on
databags to the chef-repo? each update? each XX minutes? manually?

We ended up to doing it on a case by case basis - ultimately it
depends on the volatility of the external configuration data and how
important it is to align with the chef infrastructure. For example our
"release" process basically updates version in external system which
triggers a synchronize to chef which triggers an orchestration flow
(which may trigger a rollback to previous version in the external
system if the flow fails). While some of our user data is synchronized
once per night.


Cheers,

Peter Donald