Store a secret in Parameter Store

The Platform uses AWS SSM Parameter Store to store secrets and other variables that may be necessary for services to function properly. For the purposes of this document, a “secret” can be a password, API key, or other string of information that your application may need to communicate with other applications. This guide provides details and best practices for storing and managing Platform secrets, including:

How secrets are organized

All service secrets are stored in the dsva-vagov/<service-name>/ path.

All shared team secrets (secrets that belong to a team and not to a service) will be in a top-level path leading with the team name, e.g.: cms, devops, frontend-team. The team who creates a path will have full permission to edit any secret in that path.

The Platform Operations team will manage the /devops path and store shared variables that other teams may need to access to ensure we only manage the credential in a single place, e.g.: VA_VSP_BOT_GITHUB_TOKEN, etc.

The Platform Operations team will create and manage service accounts in AWS and will give those service accounts READONLY permissions for the paths they require for the team’s operation.

Secrets are organized in a tree structure with the below hierarchy:

/dsva-vagov
- /<service-name> e.g.: vets-api, vets-website, etc.
  - /TOP_LEVEL_VARIABLE_1
  - /TOP_LEVEL_SECRET_1
  - /dev
    - /VARIABLE_2
    - /SECRET_1
  - /staging
    - /VARIABLE_1
    - /SECRET_2
  - /prod
    - /VARIABLE_3
    - /SECRET_3
  - /sandbox
    - /VARIABLE_4
    - /SECRET_4
/cms
/dvp
/devops
/frontend-team
….

Managing secrets

The following instructions assume that you have the AWS Command Line Interface (AWS CLI) and jq installed.If you don’t have the AWS CLI set up, view our instructions to set up a session token with the AWS CLI before attempting to manage secrets.

First, retrieve the secrets and store them locally.

aws ssm describe-parameters > ./parameters.json

View a list of parameter names to check if the parameter you want to add already exists.

jq '.Parameters[]?.Name' ./parameters.json | less

Find a secret for a specific service

All the commands have the same result. This example shows a few different use cases for jq:

BASH

jq '.Parameters[]? | select(.Name | test("^/dvp/")).Name' ./parameters.json
jq '.Parameters[]? | select( (.Name/"/")[1] == "dvp" ) | .Name' ./parameters.json
jq '.Parameters[]? | select( (.Name|split("/"))[1] == "dvp").Name' ./parameters.json

Find a secret for a specific environment

BASH

jq '.Parameters[]? | select(.Name | test("^/[^/]+/staging")).Name' ./parameters.json
jq '.Parameters[]? | select( (.Name/"/")[2] == "staging" ) | .Name' ./parameters.json
jq '.Parameters[]? | select( (.Name|split("/"))[2] == "staging").Name' ./parameters.json

All of our secrets currently use the same key for encrypting values, but since that may not always be the case, you can retrieve the key and information (with some info redacted) using the following example:

BASH

jq '[.Parameters[]? | select(.Type=="SecureString") | .KeyId] | unique
[
  "alias/aws/ssm"
]
aws kms describe-key --key-id alias/aws/ssm
> {
>     "KeyMetadata": {
>         "AWSAccountId": "",
>         "KeyId": "",
>         "Arn": "",
>         "CreationDate": 1534194438.007,
>         "Enabled": true,
>         "Description": "Default master key that protects my SSM parameters when no other key is defined",
>         "KeyUsage": "ENCRYPT_DECRYPT",
>         "KeyState": "Enabled",
>         "Origin": "AWS_KMS",
>         "KeyManager": "AWS",
>         "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
>         "KeySpec": "SYMMETRIC_DEFAULT",
>         "EncryptionAlgorithms": [
>             "SYMMETRIC_DEFAULT"
>         ],  
>         "MultiRegion": false
>     }   
> }

How to add, update, and delete secrets

Adding a secret

This command will create an encrypted value. When running the command, it’s given a name, value, type, and tags. All can be done in a single command:

BASH

aws ssm put-parameter \
  --name "/test/devops/example" \
  --value 'P@sSwW)rd' \
  --type 'SecureString' \
  --key-id 'alias/aws/ssm' \
  --tags 'Key=Name,Value=example' 'Key=Creator,Value=Ryan'
> {                                    
>     "Version": 1,                              
>     "Tier": "Standard"                                                                                   
> }

Updating a secret

The following example shows how to update a secret value in a single command:

BASH

aws ssm put-parameter \
  --name "/test/devops/example" \
  --description 'An example key to be deleted later' \
  --value 'P@sSwW)rd' \
  --type 'SecureString' \
  --key-id 'alias/aws/ssm' \
  --overwrite
> {                                    
>     "Version": 2,                              
>     "Tier": "Standard"                                                                                   
> }

Retrieving a secret

Example with decryption

Using the example above, let’s retrieve the full value of the secret. The values that return will be encrypted unless you specify --with-decryption on the command line:

CODE

aws ssm get-parameter --name '/test/devops/example' --with-decryption

Example with multiple parameters

The following example will allow you to retrieve multiple secrets at once by wrapping the name of the secret in single quotes with a space between each parameter.

CODE

aws ssm get-parameters --names '/test/devops/example1' '/test/devops/example2'

Example with revision history

The following example allows you to view the revision history for your secret values:

BASH

aws ssm get-parameter-history --name '/test/devops/example' --with-decryption
> {
>     "Parameters": [
>         {
>             "Name": "/test/devops/example",
>             "Type": "SecureString",
>             "KeyId": "alias/aws/ssm",
>             "LastModifiedDate": 1636052332.861,
>             "LastModifiedUser": "",
>             "Value": "P@sSwW)rd",
>             "Version": 1,
>             "Labels": [],
>             "Tier": "Standard",
>             "Policies": [],
>             "DataType": "text"
>         },
>         {
>             "Name": "/test/devops/example",
>             "Type": "SecureString",
>             "KeyId": "alias/aws/ssm",
>             "LastModifiedDate": 1636052462.067,
>             "LastModifiedUser": "",
>             "Description": "An example parameter to be deleted",
>             "Value": "P@sSwW)rd",
>             "Version": 2,
>             "Labels": [],
>             "Tier": "Standard",
>             "Policies": [],
>             "DataType": "text"
>         }
>     ]
> }

Deleting a secret

DO NOT delete a secret that’s referenced in code. If you delete a parameter that’s still in use (either in the code or in a deployment), a vets-api server will fail because of a missing value. If in doubt, do not delete the parameter. Instead, reach out to #vfs-platform-support for help.

Once you no longer need a secret, you can delete it by using the following examples.

Single secret

CODE

aws ssm delete-parameter --name '/test/devops/example'

Multiple secrets

CODE

aws ssm delete-parameters --names '/test/devops/example1' '/test/devops/example2'

Formatting Caveats

RSA Keys

Note: An improperly-formatted RSA key in AWS Parameter Store can cause the rails server to not boot properly, resulting in an outage.

Ensure that your RSA keys terminate with a newline character, rather than a space. It's critical to thoroughly review your parameters prior to deployment to a live environment to avoid an outage. A correctly formatted RSA key should resemble the following example:

CODE

"-----BEGIN RSA PRIVATE KEY-----\nYOUR_RSA_KEY_HERE\n-----END RSA PRIVATE KEY-----\n"

RSA Key with newline character at the end — RSA Key with newline at the end

Note: Please be mindful of spaces vs newlines. If at anytime you are unsure, please reach out to #vfs-platform-support for additional help.

Additional resources

https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html

In addition to the using AWS CLI, there are 2 alternative ways to interact with AWS Parameter Store:

Use the AWS Web Console. This is the simplest method for which we’ve provided the above examples.
GitHub - smblee/parameter-store-manager. This method uses a cross-platform desktop application that provides an UI to easily view and manage AWS SSM parameters.

Help and feedback

Suggest content changes to this page.
Submit new Platform Website content.
Get help from the Platform Support Team in Slack.
Submit a feature idea to the Platform.