Certificates, Secrets and Keys, Oh My!

The worst time for an essential credential to expire is exactly when you need it. That is, of course, always when it happens. APIs are suddenly unreachable, pipelines fail…it’s not good. But in a project with dozens or even hundreds of such credentials, it can be difficult to keep track of pending expiration dates.

In this post, I want to describe how my team recently addressed this challenge. A brief glimpse of the complexity: our team manages multiple Azure DevOps projects, eight Azure Key Vaults and ten App Registrations, containing hundreds of certificates, secrets, and other credentials, spread across three Azure tenants, for our own team as well as dozens of internal customers. Far too much to monitor manually.

The requirements

In planning our solution, we had the following requirements:

1. We should be alerted proactively about any credential due to expire in a specified time frame.
2. The resulting remediation tasks should be trackable.
3. It should work for the multiple sub-projects run by our team.
4. The whole thing should be automated.

The solution

Azure CLI

The Azure CLI offers great programmatic access to Azure resources and Azure DevOps and is the workhorse of the scripts we wrote for this feature. We filter for all Key Vaults belonging to our subscriptions. The expiration date of each key, certificate, and secret is read out and compared to a time frame, for example fourteen days, provided to the script as a variable. If the expiration date lies within this time frame, it is written to a JSON file, along with the name of the credential and which Key Vault it is in. The client secrets for app registrations governed by our team are handled by a similar script.

The JSON file produced by these scripts provides all the information we need to remediate soon-to-expire credentials and satisfies our first requirement. But it’s unwieldy: nobody actually wants to read a JSON file, and even then there’s no good way to keep track of which credentials we’ve rotated.

To address our second requirement, namely work tracking, another script makes use of the az boards commands to read out the JSON file produced by the previous script and create an Azure DevOps Task for each item in the file. The task contains all the information needed to renew the credential: its name, in which Key Vault or App Registration it can be found, and when the expiration date is.

So far so good, but we need good organization and an easy way to filter for our sub-projects. To organize them all into a single user story, the script generates a new user story at the beginning of each sprint and, in subsequent runs, adds a parent link from each task to that user story (az boards work-item relation add). The fields parameter of the work-item create command allows us to programmatically assign to each task the sub-project System Tags we already use every day. Additionally, using the wiql (Work Item Query Language) parameter of the az boards query command, we run a custom query to determine if there are any tasks already created for that credential, with the status “Active” or “New”. If so, no new task will be created. Here we ignore tasks with status “Closed”, because, in the event the “expiration dates” scripts have found a corresponding credential, it means additional attention is needed: perhaps the task was erroneously closed by a developer before updating the credential. And thus our third requirement is satisfied.

Azure Pipelines

Finally, it all comes together in an Azure yaml pipeline. Using a template, the pipeline iterates through our sub-projects and executes the above scripts for each one. In addition, we implemented a Webhook to send Teams notifications to our common channel, in the event of any alerts. The result is a fully-automated (requirement four!) alerting solution with work tracking via Azure DevOps in a defined User Story for each sprint.

We use the Azure DevOps suite of products start to finish, but a similar solution integrating GitHub or Jenkins with Jira or Trello would also work.

The next steps

The next item on our agenda is incorporating automated certificate rotation to reduce manual effort. In the meantime, we can rest easy knowing our credentials won’t silently expire at an inopportune moment.