In my previous article, Tooling Microservices for Scale and Access, I explained how Bronto leverages Vault and LaFours to manage our secrets and who has access to them. In this article, I will share how we integrated our deployment platform, which we named Trebuchet, with Vault and LaFours to allow auditable deployments with limited secret visibility.
There were a few important requirements we laid out when approaching the integration of Vault, Trebuchet and Lafours. First, we wanted to make sure only users sporting the role of Owner or Maintainer were able to initiate deploy operations, restarts or any variation of those actions that could impact a production environment.
Second, we wanted the Trebuchet Agent, which runs on all Orchestrated hosts, to remain unprivileged by default, meaning no credentials of any sort are bundled with the Agent. Third, we wanted to ensure that through a deployment, no part of the deployment stack, including the user initiating it, has access to secret data or credentials other than the Agent starting the service.
I’ll start by introducing changes we’ve made in Trebuchet services since we developed it to deploy microservices, then I can explain the interactions that occur between them all during a deploy operation.
The Trebuchet Orchestrator’s requirements have changed slightly since we began development. Trebuchet Orchestrator is still aimed to be the brains behind a deploy operation, and we evolved some of its security requirements to fit Bronto’s operational uses. The Orchestrator is designed to drive each Agent through its REST API to execute a deploy operation while at the same time handling more administrative tasks related to Agent wrangling. The Orchestrator itself is controlled via a principally declarative REST API. Launching a deploy operation is a matter of telling the Orchestrator what service build and configuration you expect to be running, and letting it go about doing the minimum required steps to make that a reality.
The Trebuchet Agent is still the heavy lifter of the deployment stack, responsible for all interactions with the host and the service itself. In addition to the existing service configuration provided on the environment, the Agent was extended to merge in additional configuration values served out of Vault. It does this by identifying if a configuration key represents a Vault secret based on prefix and performing a Vault lookup for that key at service execution time. Later in this article, I will explain the process the Agent uses to read a secret from Vault.
The Deployment Pipeline in Action
With these improvements to Trebuchet, the interactions between Trebuchet, LaFours and Vault can be drawn out like this.
User Initiates a Deploy
At the start of a deploy operation, the user, shown as the happy developer, makes a request to the Orchestrator using his Lightweight Directory Access Protocol (LDAP) username and password. The Orchestrator follows up with a request to LaFours to verify that the user is an Owner or Maintainer. Since all further Orchestrator operations are coupled tightly with Vault, which has its own authentication with LDAP, the Orchestrator has no need for its own authentication mechanism outside the initial LaFours check.
To read a secret out of Vault, you need two things: a Policy and a Token issued that leverages some Policy. Remember that during service creation in LaFours, a read-only Policy was templated uniquely for that service to allow reading of Vault secrets under that service. In finer detail, LaFours created the read-only Policy and an AppRole for that service, with a few constraints.
- The AppRole can create SecretIDs.
- Those SecretIDs have a short time to live (TTL) and are single-use.
- The SecretID can only be used to issue a Token with the read-only Policy described.
- Created Tokens have a short TTL.
In keeping with the theme of no service having a Token owned by a system user, LDAP users with the read-write Policy assigned by LaFours (Owners and Maintainers) are also granted the ability to issue SecretIDs. Since LaFours was responsible for defining the boundaries of each service’s AppRole, LaFours effectively grants users the ability to issue tightly scoped SecretIDs using their LDAP account for redemption by automated tools. During a deploy operation, the Orchestrator proxies the users LDAP credentials to issue a unique SecretID for each Agent in need and later performs the exchange required to issue a Wrapped Token to each Agent using a combination of AppRoleID and SecretID. This gives us great auditability to know who “authorized” a host to interact with a service’s secrets and to re-issue Tokens outside of the host provisioning process, allowing blank hosts to start with no access to Vault at all.
Token Issuance and Renewal
The final part to Trebuchet’s deploy operations is how it handles issuing Tokens and keeping them valid. As mentioned before, an LDAP user makes a request to the Orchestrator API when initiating a deploy operation. Early in the deployment, the Orchestrator checks if all Agents associated with that particular service deployment have a valid Token. For any hosts with an expired service Token or any new hosts, the Orchestrator uses the LDAP credentials of the user to create a SecretID which is then used to issue a unique Wrapped Token passed down to that Agent for it to unwrap. In this way, neither the user nor the Orchestrator knows the actual Tokens passed along to the Agents allowing host level hardening to protect Token theft. After this process has verified that all service destination hosts have valid Tokens, the rest of the deploy proceeds.
Outside or during a deploy operation, the Trebuchet Agent reacts to being provided with a Wrapped Token via API by unwrapping it and placing it in a location on disk readable only by the Agent. The unique Unwrapped Token is stored for use during further Vault interactions and belongs solely to the service that deployed it. At regular intervals, the Agent renews all at-rest Tokens with monitoring in place to alert if any renewal fails or if Vault seems unavailable. Given the short TTL on Tokens, this means that if for any reason the Agent is unable to interact with Vault, all Tokens on that host are under threat of expiration. If such an event does occur, reissuing the Tokens is as simple as doing another deployment with the currently running service version and all Agents will be brought back to working order.
Using the combination of Vault, LaFours and Trebuchet, Bronto can be flexible in its deployment operations and maintain a high level of auditability through any interaction with production services or data, client or otherwise.