Bronto’s engineering organization subscribes to the idea of microservices. For all the benefits microservices provides, upfront tooling costs are required to continue operations at scale. For us, the first hurdle to overcome was the need for a more standardized and robust way to deploy services than our existing methods could provide.
We solved this in part with Trebuchet, an internally developed stack for automating deploys. We examined configuration as part of that stack and exposed a need to provide both common configuration and sensitive configuration data, such as passwords, to a service.
In this article, I will talk about Bronto’s approach to providing sensitive data to microservices, how we store that data, and how we leverage that same system to audit and authorize access to execute operational tasks.
LaFours and Developer Self-Service
For our at rest storage mechanism of secrets, we decided to use Vault, for its token management mechanisms and its out-of-the-box satisfaction of auditability requirements. It’s all fine and well to have a secure secret storage mechanism, but if the secrets are cordoned away behind privileged tokens that someone must maintain, operations scalability goes out the window.
We needed a facade for Vault to automate common operational tasks and to template Vault data structures between services. We also wanted to create a means to control developers’ operational access to a service in ways that do not necessarily align with other team organizational structures, like Lightweight Directory Access Protocol (LDAP) groups. At the same time, we wanted to expose that data in an API that other tools could consume. To do this, we created LaFours with some unique features that build on Vault’s storage and authorization mechanisms to manage user access.
Vault uses the concept of policies to act as access control lists (ACLs) to its datastore. Policies can be used to grant permissions ranging from basic data access to the creation of additional policies and associate them with authenticated users. Extending this, Lafours has a concept of two user roles: A Maintainer has the ability to see and modify secrets, and an Owner has the privileges of a Maintainer but can add and remove a user’s service roles.
We settled on a standard set of policies for every service in LaFours, including:
- A read-only policy to provide tools like Trebuchet the means to retrieve secrets from Vault for a service.
- A read-write policy for use in reading and writing secrets for a service to Vault. This policy is granted to both Owners and Maintainers.
- A read-write policy for use in reading and writing to the storage of Role to Service mappings, effectively giving the user the ability to add and remove Owners and Maintainers from a service. This policy is granted to Owners only.
Each of these policies is individually tied with a group that users can be associated with in order to grant them those permissions. On service creation, LaFours uses a manually allocated token capable solely of creating policies, associating groups with those policies and associating users with groups in order to create a set of these policies and groups specific to the new service. On top of all this, there is a single basic policy granted to all authenticated LDAP users to create and read, but not update, the storage of Role to Service mappings aimed to bootstrap the service creation process in LaFours explained below.
Since LaFours only has the power to create templated policy group pairs and grant users access to them, the groups with which a user is associated become the true strength of LaFours. In terms of security, we did not want to create a service with unlimited privileges to orchestrate all possible Vault interactions. Instead, we wanted to leverage Vault’s LDAP authentication backend to allow users to authenticate using their own accounts and execute reads and writes to a service’s secrets.
When a user issues a request to LaFours to create a new service, their credentials are used to issue a “create” in the storage of Role-to-Service mappings, recording themselves as the first owner. Because no other user can update that same entry without the Access policy belonging to owners, no other user can usurp the original owner by creating a new service with the same name. At that point, LaFours creates the template set of policies and groups specific to that new service. The Access and Maintainer group is granted to the creating user, giving them full access to manage secrets for the service and to modify the record they initially created to add more users to the service.
When an Owner makes a request to add a user, they leverage their Access group policy to update that service’s Role-to-Service mapping and LaFours again grants that new user the appropriate group memberships. In that way, any data manipulation performed is done by an authenticated user and not LaFours itself. LaFours only has the power to grant user permissions, not read secrets. Even a compromised LaFours instance would only be capable of granting additional permissions to an existing LDAP user. To access secrets, that LDAP user would still be forced to do an audited read through LaFours or Vault directly.
In my next article, I will write about the integration between Vault, LaFours and Trebuchet and how we can give hosts the ability to exchange secrets, but without the need to grant secret access to a system-level user.