The Platform uses AWS SSM Parameter Store to store secrets and other variables that may be necessary for services to function properly. For the purposes of this document, a “secret” can be a password, API key, or other string of information that your application may need to communicate with other applications. This guide provides details and best practices for storing and managing Platform secrets, including:

How secrets are organized

All service secrets are stored in the dsva-vagov/<service-name>/ path.

All shared team secrets (secrets that belong to a team and not to a service) will be in a top-level path leading with the team name, e.g.: cms, devops, frontend-team. The team who creates a path will have full permission to edit any secret in that path.

The Platform Operations team will manage the /devops path and store shared variables that other teams may need to access to ensure we only manage the credential in a single place, e.g.: VA_VSP_BOT_GITHUB_TOKEN, etc.

The Platform Operations team will create and manage service accounts in AWS and will give those service accounts READONLY permissions for the paths they require for the team’s operation. 

Secrets are organized in a tree structure with the below hierarchy:

  • /dsva-vagov

    • /<service-name> e.g.: vets-api, vets-website, etc.

      • /TOP_LEVEL_VARIABLE_1

      • /TOP_LEVEL_SECRET_1

      • /dev

        • /VARIABLE_2

        • /SECRET_1

      • /staging

        • /VARIABLE_1

        • /SECRET_2

      • /prod

        • /VARIABLE_3

        • /SECRET_3

      • /sandbox

        • /VARIABLE_4

        • /SECRET_4

  • /cms

  • /dvp

  • /devops

  • /frontend-team

  • ….

Managing secrets

The following instructions assume that you have the AWS Command Line Interface (AWS CLI) and jq installed.If you don’t have the AWS CLI set up, view our instructions to set up a session token with the AWS CLI before attempting to manage secrets. 

First, retrieve the secrets and store them locally.

aws ssm describe-parameters > ./parameters.json

View a list of parameter names to check if the parameter you want to add already exists.

jq '.Parameters[]?.Name' ./parameters.json | less

Find a secret for a specific service

All the commands have the same result. This example shows a few different use cases for jq:

jq '.Parameters[]? | select(.Name | test("^/dvp/")).Name' ./parameters.json
jq '.Parameters[]? | select( (.Name/"/")[1] == "dvp" ) | .Name' ./parameters.json
jq '.Parameters[]? | select( (.Name|split("/"))[1] == "dvp").Name' ./parameters.json
BASH

Find a secret for a specific environment

jq '.Parameters[]? | select(.Name | test("^/[^/]+/staging")).Name' ./parameters.json
jq '.Parameters[]? | select( (.Name/"/")[2] == "staging" ) | .Name' ./parameters.json
jq '.Parameters[]? | select( (.Name|split("/"))[2] == "staging").Name' ./parameters.json
BASH

All of our secrets currently use the same key for encrypting values, but since that may not always be the case, you can retrieve the key and information (with some info redacted) using the following example:

jq '[.Parameters[]? | select(.Type=="SecureString") | .KeyId] | unique
[
  "alias/aws/ssm"
]
aws kms describe-key --key-id alias/aws/ssm
> {
>     "KeyMetadata": {
>         "AWSAccountId": "",
>         "KeyId": "",
>         "Arn": "",
>         "CreationDate": 1534194438.007,
>         "Enabled": true,
>         "Description": "Default master key that protects my SSM parameters when no other key is defined",
>         "KeyUsage": "ENCRYPT_DECRYPT",
>         "KeyState": "Enabled",
>         "Origin": "AWS_KMS",
>         "KeyManager": "AWS",
>         "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
>         "KeySpec": "SYMMETRIC_DEFAULT",
>         "EncryptionAlgorithms": [
>             "SYMMETRIC_DEFAULT"
>         ],  
>         "MultiRegion": false
>     }   
> }
BASH

How to add, update, and delete secrets

Adding a secret

This command will create an encrypted value. When running the command, it’s given a name, value, type, and tags. All can be done in a single command:

aws ssm put-parameter \
  --name "/test/devops/example" \
  --value 'P@sSwW)rd' \
  --type 'SecureString' \
  --key-id 'alias/aws/ssm' \
  --tags 'Key=Name,Value=example' 'Key=Creator,Value=Ryan'
> {                                    
>     "Version": 1,                              
>     "Tier": "Standard"                                                                                   
> }
BASH

Updating a secret

The following example shows how to update a secret value in a single command:

aws ssm put-parameter \
  --name "/test/devops/example" \
  --description 'An example key to be deleted later' \
  --value 'P@sSwW)rd' \
  --type 'SecureString' \
  --key-id 'alias/aws/ssm' \
  --overwrite
> {                                    
>     "Version": 2,                              
>     "Tier": "Standard"                                                                                   
> }
BASH

Retrieving a secret

Example with decryption

Using the example above, let’s retrieve the full value of the secret. The values that  return will be encrypted unless you specify --with-decryption on the command line:

aws ssm get-parameter --name '/test/devops/example' --with-decryption
CODE

Example with multiple parameters

The following example will allow you to retrieve multiple secrets at once by wrapping the name of the secret in single quotes with a space between each parameter. 

aws ssm get-parameters --names '/test/devops/example1' '/test/devops/example2'
CODE

Example with revision history

The following example allows you to view the revision history for your secret values:

aws ssm get-parameter-history --name '/test/devops/example' --with-decryption
> {
>     "Parameters": [
>         {
>             "Name": "/test/devops/example",
>             "Type": "SecureString",
>             "KeyId": "alias/aws/ssm",
>             "LastModifiedDate": 1636052332.861,
>             "LastModifiedUser": "",
>             "Value": "P@sSwW)rd",
>             "Version": 1,
>             "Labels": [],
>             "Tier": "Standard",
>             "Policies": [],
>             "DataType": "text"
>         },
>         {
>             "Name": "/test/devops/example",
>             "Type": "SecureString",
>             "KeyId": "alias/aws/ssm",
>             "LastModifiedDate": 1636052462.067,
>             "LastModifiedUser": "",
>             "Description": "An example parameter to be deleted",
>             "Value": "P@sSwW)rd",
>             "Version": 2,
>             "Labels": [],
>             "Tier": "Standard",
>             "Policies": [],
>             "DataType": "text"
>         }
>     ]
> }
BASH

Deleting a secret

Once you no longer need a secret, you can delete it by using the following examples.

Single secret

aws ssm delete-parameter --name '/test/devops/example'
CODE

Multiple secrets

aws ssm delete-parameters --names '/test/devops/example1' '/test/devops/example2'
CODE

Additional resources

https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html

In addition to the using AWS CLI, there are 2 alternative ways to interact with AWS Parameter Store:

  1. Use the AWS Web Console. This is the simplest method for which we’ve provided the above examples.

  2. GitHub - smblee/parameter-store-manager. This method uses a cross-platform desktop application that provides an UI to easily view and manage AWS SSM parameters.