# API Interaction from node

**URL:** <https://discourse.chef.io/t/api-interaction-from-node/3479>\
**Category:** Chef Infra (archive)\
**Created:** [January 8, 2013, 9:49pm UTC](https://discourse.chef.io/t/api-interaction-from-node/3479 "2013-01-08T21:49:24Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anthony\_Goddard](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/anthony_goddard/32/622_2.png) [@Anthony\_Goddard](https://discourse.chef.io/u/Anthony_Goddard)\
**Post date:** [January 8, 2013, 9:49pm UTC](https://discourse.chef.io/t/api-interaction-from-node/3479/1 "2013-01-08T21:49:24Z")

</div>

Hi all,  
I’m writing a little script to populate a databag each time it’s run - the purpose is to store mysql bin log data in a databag when a slave is backed up. I was originally thinking of using the API from the node and just using the local client.pem for authentication, but while I can read databags this way, I get a 403 when writing to them, so I’m looking for a breakdown of what objects the client is able to modify.  
I could also just change where I’m storing the data, or if that doesn’t work, some middleware with another key will work, but I’d be interested in hearing about any other approaches if I’m Doing It Wrong™.

fwiw, the script that’s interacting with chef is run from cron.

Cheers,  
Ant

---

<div class="post-metadata">

**Author:** ![AJ\_Christensen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/aj_christensen/32/525_2.png) [@AJ\_Christensen](https://discourse.chef.io/u/AJ_Christensen)\
**Post date:** [January 8, 2013, 9:53pm UTC](https://discourse.chef.io/t/api-interaction-from-node/3479/2 "2013-01-08T21:53:45Z")

</div>

Yo Ant,

You won't be able to update data bag items already created with a node like  
that, at least with the default permissions? I do believe you will be able  
to create _new_ data-bag items inside of a data-bag.

Hosted chef or OSS?

On OSS, you can make the node an admin. Hosted/Private, you can use RBAC to  
allow create items in that d-bag. knife-acls may help with automating this.

Cheers,

AJ

On 9 January 2013 10:49, Anthony Goddard [agoddard@mbl.edu](mailto:agoddard@mbl.edu) wrote:

> Hi all,  
> I'm writing a little script to populate a databag each time it's run - the  
> purpose is to store mysql bin log data in a databag when a slave is backed  
> up. I was originally thinking of using the API from the node and just using  
> the local client.pem for authentication, but while I can read databags this  
> way, I get a 403 when writing to them, so I'm looking for a breakdown of  
> what objects the client is able to modify.  
> I could also just change where I'm storing the data, or if that doesn't  
> work, some middleware with another key will work, but I'd be interested in  
> hearing about any other approaches if I'm Doing It Wrong™.
> 
> fwiw, the script that's interacting with chef is run from cron.
> 
> Cheers,  
> Ant

---

<div class="post-metadata">

**Author:** ![Anthony\_Goddard](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/anthony_goddard/32/622_2.png) [@Anthony\_Goddard](https://discourse.chef.io/u/Anthony_Goddard)\
**Post date:** [January 8, 2013, 10:36pm UTC](https://discourse.chef.io/t/api-interaction-from-node/3479/3 "2013-01-08T22:36:12Z")

</div>

Thanks AJ,

OSS chef - I got 403's back when adding a new data bag item though. I think the admin node will be a good workaround, and I'll write some middleware in front of it.

Cheers,  
Ant

On Jan 8, 2013, at 4:53 PM, AJ Christensen [aj@junglist.gen.nz](mailto:aj@junglist.gen.nz) wrote:

> Yo Ant,
> 
> You won't be able to update data bag items already created with a node like that, at least with the default permissions? I do believe you will be able to create _new_ data-bag items inside of a data-bag.
> 
> Hosted chef or OSS?
> 
> On OSS, you can make the node an admin. Hosted/Private, you can use RBAC to allow create items in that d-bag. knife-acls may help with automating this.
> 
> Cheers,
> 
> AJ
> 
> On 9 January 2013 10:49, Anthony Goddard [agoddard@mbl.edu](mailto:agoddard@mbl.edu) wrote:  
> Hi all,  
> I'm writing a little script to populate a databag each time it's run - the purpose is to store mysql bin log data in a databag when a slave is backed up. I was originally thinking of using the API from the node and just using the local client.pem for authentication, but while I can read databags this way, I get a 403 when writing to them, so I'm looking for a breakdown of what objects the client is able to modify.  
> I could also just change where I'm storing the data, or if that doesn't work, some middleware with another key will work, but I'd be interested in hearing about any other approaches if I'm Doing It Wrong™.
> 
> fwiw, the script that's interacting with chef is run from cron.
> 
> Cheers,  
> Ant

---

<div class="post-metadata">

**Author:** ![Jay\_Feldblum](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@Jay\_Feldblum](https://discourse.chef.io/u/Jay_Feldblum)\
**Post date:** [January 9, 2013, 1:54am UTC](https://discourse.chef.io/t/api-interaction-from-node/3479/4 "2013-01-09T01:54:05Z")

</div>

Anthony,

I would say "You're Doing It Wrong."

Chef is for configuration, not bulk storage. Storing backups in chef-server  
is Doing It Wrong. There are plenty of other approaches. You can run  
backups on a schedule (or on demand) separately from chef-client runs.  
Chef-client can provision the backup script and everything necessary to get  
it running so that it's a one-liner in the crontab or a one-liner from  
SSH. If the script is copying data to remote storage, it should not be to  
chef-server. Instead, chef can provision a remote storage node with  
replicated disks and can provision the slave's backup script to know about  
it, or you can use a storage service like S3 and chef can provision the  
slave's backup script to know about that instead.

Cheers,  
Jay

On Tue, Jan 8, 2013 at 4:49 PM, Anthony Goddard [agoddard@mbl.edu](mailto:agoddard@mbl.edu) wrote:

> Hi all,  
> I'm writing a little script to populate a databag each time it's run - the  
> purpose is to store mysql bin log data in a databag when a slave is backed  
> up. I was originally thinking of using the API from the node and just using  
> the local client.pem for authentication, but while I can read databags this  
> way, I get a 403 when writing to them, so I'm looking for a breakdown of  
> what objects the client is able to modify.  
> I could also just change where I'm storing the data, or if that doesn't  
> work, some middleware with another key will work, but I'd be interested in  
> hearing about any other approaches if I'm Doing It Wrong™.
> 
> fwiw, the script that's interacting with chef is run from cron.
> 
> Cheers,  
> Ant

---

<div class="post-metadata">

**Author:** ![AJ\_Christensen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/aj_christensen/32/525_2.png) [@AJ\_Christensen](https://discourse.chef.io/u/AJ_Christensen)\
**Post date:** [January 9, 2013, 1:57am UTC](https://discourse.chef.io/t/api-interaction-from-node/3479/5 "2013-01-09T01:57:13Z")

</div>

I think he means storing the bin log position in a data-bag (quite a common  
approach to allowing slaves to find the correct position)

No?

--AJ

On 9 January 2013 14:54, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:

> Anthony,
> 
> I would say "You're Doing It Wrong."
> 
> Chef is for configuration, not bulk storage. Storing backups in  
> chef-server is Doing It Wrong. There are plenty of other approaches. You  
> can run backups on a schedule (or on demand) separately from chef-client  
> runs. Chef-client can provision the backup script and everything necessary  
> to get it running so that it's a one-liner in the crontab or a one-liner  
> from SSH. If the script is copying data to remote storage, it should not be  
> to chef-server. Instead, chef can provision a remote storage node with  
> replicated disks and can provision the slave's backup script to know about  
> it, or you can use a storage service like S3 and chef can provision the  
> slave's backup script to know about that instead.
> 
> Cheers,  
> Jay
> 
> On Tue, Jan 8, 2013 at 4:49 PM, Anthony Goddard [agoddard@mbl.edu](mailto:agoddard@mbl.edu) wrote:
> 
> > Hi all,  
> > I'm writing a little script to populate a databag each time it's run -  
> > the purpose is to store mysql bin log data in a databag when a slave is  
> > backed up. I was originally thinking of using the API from the node and  
> > just using the local client.pem for authentication, but while I can read  
> > databags this way, I get a 403 when writing to them, so I'm looking for a  
> > breakdown of what objects the client is able to modify.  
> > I could also just change where I'm storing the data, or if that doesn't  
> > work, some middleware with another key will work, but I'd be interested in  
> > hearing about any other approaches if I'm Doing It Wrong™.
> > 
> > fwiw, the script that's interacting with chef is run from cron.
> > 
> > Cheers,  
> > Ant

---

<div class="post-metadata">

**Author:** ![Adam\_Jacob](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/adam_jacob/32/292_2.png) [@Adam\_Jacob](https://discourse.chef.io/u/Adam_Jacob)\
**Post date:** [January 9, 2013, 4:54am UTC](https://discourse.chef.io/t/api-interaction-from-node/3479/6 "2013-01-09T04:54:00Z")

</div>

Man I hope so.

Be kind of awesome if he's packing actual binlogs into a JSON value,  
though.

Adam

On 1/8/13 5:57 PM, "AJ Christensen" [aj@junglist.gen.nz](mailto:aj@junglist.gen.nz) wrote:

> I think he means storing the bin log position in a data-bag (quite a  
> common approach to allowing slaves to find the correct position)
> 
> No?
> 
> --AJ
> 
> On 9 January 2013 14:54, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:
> 
> Anthony,
> 
> I would say "You're Doing It Wrong."
> 
> Chef is for configuration, not bulk storage. Storing backups in  
> chef-server is Doing It Wrong. There are plenty of other approaches. You  
> can run backups on a schedule (or on demand) separately from chef-client  
> runs. Chef-client can provision the backup  
> script and everything necessary to get it running so that it's a  
> one-liner in the crontab or a one-liner from SSH. If the script is  
> copying data to remote storage, it should not be to chef-server. Instead,  
> chef can provision a remote storage node with replicated  
> disks and can provision the slave's backup script to know about it, or  
> you can use a storage service like S3 and chef can provision the slave's  
> backup script to know about that instead.
> 
> Cheers,  
> Jay
> 
> On Tue, Jan 8, 2013 at 4:49 PM, Anthony Goddard  
> [agoddard@mbl.edu](mailto:agoddard@mbl.edu) wrote:
> 
> Hi all,  
> I'm writing a little script to populate a databag each time it's run -  
> the purpose is to store mysql bin log data in a databag when a slave is  
> backed up. I was originally thinking of using the API from the node and  
> just using the local client.pem for authentication,  
> but while I can read databags this way, I get a 403 when writing to  
> them, so I'm looking for a breakdown of what objects the client is able  
> to modify.  
> I could also just change where I'm storing the data, or if that doesn't  
> work, some middleware with another key will work, but I'd be interested  
> in hearing about any other approaches if I'm Doing It Wrong.
> 
> fwiw, the script that's interacting with chef is run from cron.
> 
> Cheers,  
> Ant

---

<div class="post-metadata">

**Author:** ![AJ\_Christensen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/aj_christensen/32/525_2.png) [@AJ\_Christensen](https://discourse.chef.io/u/AJ_Christensen)\
**Post date:** [January 9, 2013, 4:56am UTC](https://discourse.chef.io/t/api-interaction-from-node/3479/7 "2013-01-09T04:56:15Z")

</div>

In re-reading, maybe I understood. Ant, care to confirm? You aren't storing  
bloody big fuck off binary logs in JSON on the chef-server in a data-bag,  
are you?

--AJ

On 9 January 2013 17:54, Adam Jacob [adam@opscode.com](mailto:adam@opscode.com) wrote:

> Man I hope so.
> 
> Be kind of awesome if he's packing actual binlogs into a JSON value,  
> though.
> 
> Adam
> 
> On 1/8/13 5:57 PM, "AJ Christensen" [aj@junglist.gen.nz](mailto:aj@junglist.gen.nz) wrote:
> 
> > I think he means storing the bin log position in a data-bag (quite a  
> > common approach to allowing slaves to find the correct position)
> > 
> > No?
> > 
> > --AJ
> > 
> > On 9 January 2013 14:54, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:
> > 
> > Anthony,
> > 
> > I would say "You're Doing It Wrong."
> > 
> > Chef is for configuration, not bulk storage. Storing backups in  
> > chef-server is Doing It Wrong. There are plenty of other approaches. You  
> > can run backups on a schedule (or on demand) separately from chef-client  
> > runs. Chef-client can provision the backup  
> > script and everything necessary to get it running so that it's a  
> > one-liner in the crontab or a one-liner from SSH. If the script is  
> > copying data to remote storage, it should not be to chef-server. Instead,  
> > chef can provision a remote storage node with replicated  
> > disks and can provision the slave's backup script to know about it, or  
> > you can use a storage service like S3 and chef can provision the slave's  
> > backup script to know about that instead.
> > 
> > Cheers,  
> > Jay
> > 
> > On Tue, Jan 8, 2013 at 4:49 PM, Anthony Goddard  
> > [agoddard@mbl.edu](mailto:agoddard@mbl.edu) wrote:
> > 
> > Hi all,  
> > I'm writing a little script to populate a databag each time it's run -  
> > the purpose is to store mysql bin log data in a databag when a slave is  
> > backed up. I was originally thinking of using the API from the node and  
> > just using the local client.pem for authentication,  
> > but while I can read databags this way, I get a 403 when writing to  
> > them, so I'm looking for a breakdown of what objects the client is able  
> > to modify.  
> > I could also just change where I'm storing the data, or if that doesn't  
> > work, some middleware with another key will work, but I'd be interested  
> > in hearing about any other approaches if I'm Doing It Wrong .
> > 
> > fwiw, the script that's interacting with chef is run from cron.
> > 
> > Cheers,  
> > Ant

---

<div class="post-metadata">

**Author:** ![Adam\_Jacob](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/adam_jacob/32/292_2.png) [@Adam\_Jacob](https://discourse.chef.io/u/Adam_Jacob)\
**Post date:** [January 9, 2013, 4:58am UTC](https://discourse.chef.io/t/api-interaction-from-node/3479/8 "2013-01-09T04:58:46Z")

</div>

And if you are, you get a beer for most bold use of JSON in 2013 so far. 🙂

Adam

On 1/8/13 8:56 PM, "AJ Christensen" [aj@junglist.gen.nz](mailto:aj@junglist.gen.nz) wrote:

> In re-reading, maybe I understood. Ant, care to confirm? You aren't  
> storing bloody big fuck off binary logs in JSON on the chef-server in a  
> data-bag, are you?
> 
> --AJ
> 
> On 9 January 2013 17:54, Adam Jacob [adam@opscode.com](mailto:adam@opscode.com) wrote:
> 
> Man I hope so.
> 
> Be kind of awesome if he's packing actual binlogs into a JSON value,  
> though.
> 
> Adam
> 
> On 1/8/13 5:57 PM, "AJ Christensen" [aj@junglist.gen.nz](mailto:aj@junglist.gen.nz) wrote:
> 
> > I think he means storing the bin log position in a data-bag (quite a  
> > common approach to allowing slaves to find the correct position)
> > 
> > No?
> > 
> > --AJ
> > 
> > On 9 January 2013 14:54, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:
> > 
> > Anthony,
> > 
> > I would say "You're Doing It Wrong."
> > 
> > Chef is for configuration, not bulk storage. Storing backups in  
> > chef-server is Doing It Wrong. There are plenty of other approaches. You  
> > can run backups on a schedule (or on demand) separately from chef-client  
> > runs. Chef-client can provision the backup  
> > script and everything necessary to get it running so that it's a  
> > one-liner in the crontab or a one-liner from SSH. If the script is  
> > copying data to remote storage, it should not be to chef-server. Instead,  
> > chef can provision a remote storage node with replicated  
> > disks and can provision the slave's backup script to know about it, or  
> > you can use a storage service like S3 and chef can provision the slave's  
> > backup script to know about that instead.
> > 
> > Cheers,  
> > Jay
> > 
> > On Tue, Jan 8, 2013 at 4:49 PM, Anthony Goddard  
> > [agoddard@mbl.edu](mailto:agoddard@mbl.edu) wrote:
> > 
> > Hi all,  
> > I'm writing a little script to populate a databag each time it's run -  
> > the purpose is to store mysql bin log data in a databag when a slave is  
> > backed up. I was originally thinking of using the API from the node and  
> > just using the local client.pem for authentication,  
> > but while I can read databags this way, I get a 403 when writing to  
> > them, so I'm looking for a breakdown of what objects the client is able  
> > to modify.  
> > I could also just change where I'm storing the data, or if that doesn't  
> > work, some middleware with another key will work, but I'd be interested  
> > in hearing about any other approaches if I'm Doing It Wrong .
> > 
> > fwiw, the script that's interacting with chef is run from cron.
> > 
> > Cheers,  
> > Ant

---

<div class="post-metadata">

**Author:** ![Anthony\_Goddard](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/anthony_goddard/32/622_2.png) [@Anthony\_Goddard](https://discourse.chef.io/u/Anthony_Goddard)\
**Post date:** [January 9, 2013, 5:37am UTC](https://discourse.chef.io/t/api-interaction-from-node/3479/9 "2013-01-09T05:37:25Z")

</div>

lmao, that'd be awesome 😉  
Fortunately AJ is right, I'm only storing the bin-log position (and filename) in the databag

process is:

- backup starts (using backup gem) on a backup slave
- pre-backup hook stops replication, dumps the current bin-log position and filename in a data bag
- backup completes, replication resumes
- a new slave can now get spun up automagically and know what position to begin replication from by doing a query to the data bag that matches the backup ID

Ant

On Jan 8, 2013, at 11:58 PM, Adam Jacob [adam@opscode.com](mailto:adam@opscode.com) wrote:

> And if you are, you get a beer for most bold use of JSON in 2013 so far. 🙂
> 
> Adam
> 
> On 1/8/13 8:56 PM, "AJ Christensen" [aj@junglist.gen.nz](mailto:aj@junglist.gen.nz) wrote:
> 
> > In re-reading, maybe I understood. Ant, care to confirm? You aren't  
> > storing bloody big fuck off binary logs in JSON on the chef-server in a  
> > data-bag, are you?
> > 
> > --AJ
> > 
> > On 9 January 2013 17:54, Adam Jacob [adam@opscode.com](mailto:adam@opscode.com) wrote:
> > 
> > Man I hope so.
> > 
> > Be kind of awesome if he's packing actual binlogs into a JSON value,  
> > though.
> > 
> > Adam
> > 
> > On 1/8/13 5:57 PM, "AJ Christensen" [aj@junglist.gen.nz](mailto:aj@junglist.gen.nz) wrote:
> > 
> > > I think he means storing the bin log position in a data-bag (quite a  
> > > common approach to allowing slaves to find the correct position)
> > > 
> > > No?
> > > 
> > > --AJ
> > > 
> > > On 9 January 2013 14:54, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:
> > > 
> > > Anthony,
> > > 
> > > I would say "You're Doing It Wrong."
> > > 
> > > Chef is for configuration, not bulk storage. Storing backups in  
> > > chef-server is Doing It Wrong. There are plenty of other approaches. You  
> > > can run backups on a schedule (or on demand) separately from chef-client  
> > > runs. Chef-client can provision the backup  
> > > script and everything necessary to get it running so that it's a  
> > > one-liner in the crontab or a one-liner from SSH. If the script is  
> > > copying data to remote storage, it should not be to chef-server. Instead,  
> > > chef can provision a remote storage node with replicated  
> > > disks and can provision the slave's backup script to know about it, or  
> > > you can use a storage service like S3 and chef can provision the slave's  
> > > backup script to know about that instead.
> > > 
> > > Cheers,  
> > > Jay
> > > 
> > > On Tue, Jan 8, 2013 at 4:49 PM, Anthony Goddard  
> > > [agoddard@mbl.edu](mailto:agoddard@mbl.edu) wrote:
> > > 
> > > Hi all,  
> > > I'm writing a little script to populate a databag each time it's run -  
> > > the purpose is to store mysql bin log data in a databag when a slave is  
> > > backed up. I was originally thinking of using the API from the node and  
> > > just using the local client.pem for authentication,  
> > > but while I can read databags this way, I get a 403 when writing to  
> > > them, so I'm looking for a breakdown of what objects the client is able  
> > > to modify.  
> > > I could also just change where I'm storing the data, or if that doesn't  
> > > work, some middleware with another key will work, but I'd be interested  
> > > in hearing about any other approaches if I'm Doing It Wrong .
> > > 
> > > fwiw, the script that's interacting with chef is run from cron.
> > > 
> > > Cheers,  
> > > Ant