Setting up Guardian of Secrets: Part 1 – Docker Compose Deployment of Vault

Concept of securing communication has existed in human civilisation since time immemorial. The necessity to ensure secure communication has led to the development of encryption. Some form of secret key is an essential component of encryption. The security of your encrypted data is partly reliant on safeguarding the secret key that is required to decrypt the communication/information. This applies exceptionally well to the networked computer systems. This is where our search for guardian of our secret keys began. A guardian that can keep our secrets safe. We need ability to provide controlled access to the secret to authorised entities after verifying their identities. Our quest for the guardian that met our requirements led us to HashiCorp Vault (HCVault).

HCVault provides ways to “Secure, create, store, and tightly control access to tokens, passwords, certificates, and encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API”. I will go over our experience of using HCVault at Kapstan.io.

HCVault supports dev mode for using vault in dev setup Dev Server Mode | Vault | HashiCorp Developer. We strive to maintain minimal disparity between our development and production setups so we use production mode of HCVault for our local development environments also. Let us look at how we deployed vault in our development environment utilising Docker compose. (We perform development activities on MacOS). I’m assuming that the reader has a foundational knowledge of Docker Compose, shell scripts, and TLS certificates. Install docker on your local machine to try out local vault setup.

The setup process can be categorised into four main components:

1. Establishment of the directory for storing critical data (Further details will be provided as we proceed).
2. Configuration file for Vault.
3. Initialisation script for Vault.
4. Docker Compose file for deploying the Vault container and executing the initialisation process.

‍

Vault container setup

Directory Setup

localsetup is the base directory where we will do the vault setup

mkdir localsetup && cd localsetup && mkdir vault && cd vault && \
mkdir config data keys root scripts && cd -

The directory structure will look as below:

localsetup
└── vault
   ├── config Configuration of Vault server
   ├── data This is where vault will store all its data.
   ├── keys * We will store Vault unseal here
   ├── root * We will store Vault root keys here
   └── scripts Script to initialize vault features
* are the directories where sensitive information will be store.

Vault configuration file

We will create vault/config/vault-config.json file to provide configuration parameters to vault. VAULT_LOCAL_CONFIG environment variable can also be used to provide the configuration, but I prefer configuration from file, rather than from environment variable.

Content of vault/config/vault-config.json

{
 "listener": [
   {
     "tcp": {
       "address": "0.0.0.0:8200",
       "tls_disable": 1
      }
    }
 ],
 "storage": {
    "file": {
       "path": "/vault/data"
    }
  },
  "ui": true,
  "api_addr": "http://0.0.0.0:8200"
}

Vault initialisation

If you’re eager to see the vault in action, feel free to jump ahead to the section about creating the contents of ‘vault/config/vault-init.sh.’ You can always return here later to explore the finer details of the vault’s security measures.

Securing sensitive information hinges on the principle of maintaining confidentiality. Think of a vault as a vigilant protector, guarding these classified matters. This naturally leads to an important question: How does the vault ensure the secrecy and security of the entrusted secrets? All the data housed within the vault is shielded by the unyielding ‘Encryption key’—a linchpin for the vault’s functionality. This underscores the critical need to prioritise the security of this key.

Digging deeper, the ‘Encryption key’ itself is safeguarded by the ‘root key.’ This ‘root key’ holds the power to unlock the vault’s encrypted treasures. This naturally prompts the question: How can the security of the ‘root key’ be ensured?

Interestingly, the ‘root key’ is further protected by the ‘unseal key.’ This unseal key acts as the ultimate gatekeeper, granting access to the vault’s kingdom. And here’s the twist: Vault employs Shamir’s Secret Sharing approach. Shamir’s Secret Sharing is a cryptographic technique that divides a secret into multiple “shares” or “fragments” in such a way that a minimum number of these shares are required to reconstruct the original secret. Each share is essentially a piece of the secret, and the secret can only be revealed by combining the required number of shares. Vault offers two key parameters to manage Shamir’s Secret Sharing: -key-shares and -key-threshold. The -key-shares parameter determines the total number of shares created for the unseal key, while the -key-threshold parameter specifies the minimum number of these shares required to reconstruct the unseal key. This approach grants you an unprecedented level of control and security. For more information, you can read about Shamir’s Secret Sharing.”

Source: Architecture | Vault | HashiCorp Developer

The explanation above clarifies that initialising the vault generates crucial information needed for its smooth operation. Additionally, the vault’s security relies on how different keys are managed. In our setup for a development environment, we’ll keep important keys in the filesystem – acceptable for developer setup, but not secure for production environments. Ensuring the security of these keys becomes extremely important when deploying the vault in a production environment. Scripts used for the purpose is well commented, and should provide clarity and reason for any command that gets executed.

Content of vault/scriptsvault-init.sh

#! /bin/sh
set -ex
apk add jq

INIT_FILE=/vault/keys/vault.init
if [[ -f "${INIT_FILE}" ]]; then
 echo "${INIT_FILE} exists. Vault already initialized."
else
 echo "Initializing Vault..."
 sleep 5
  vault operator init -key-shares=3 -key-threshold=2 | tee ${INIT_FILE} > /dev/null ### 3 fragments, 2 are required to unseal
### Store unseal keys to files
  COUNTER=1
 cat ${INIT_FILE} | grep '^Unseal' | awk '{print $4}' | for key in $(cat -); do
  echo "${key}" > /vault/keys/key-${COUNTER}
    COUNTER=$((COUNTER + 1))
  done
 
 ### Store Root Key to file
 cat ${INIT_FILE}| grep '^Initial Root Token' | awk '{print $4}' | tee /vault/root/token > /dev/null
  echo "Vault setup complete."fiif [ ! -s /vault/root/token -o ! -s /vault/keys/key-1 -o ! -s /vault/keys/key-2 ] ; then
 echo "Vault is initialized, but unseal keys or token are mssing"
 return
fi

echo "Unsealing Vault"
export VAULT_TOKEN=$(cat /vault/root/token)
vault operator unseal "$(cat /vault/keys/key-1)"
vault operator unseal "$(cat /vault/keys/key-2)"

vault status

Provide execute permission to the script

chmod 755 vault/scripts/vault-init.sh

docker-compose.yml file
‍

services:
 vault:
   image: "hashicorp/vault:${VAULT_TAG:-latest}"
   container_name: localsetup-vault
   ports:
     - "8200:8200"
   volumes:
     - ./vault/config:/vault/config
     - ./vault/data:/vault/data
   command: ["vault", "server", "-config=/vault/config/vault-config.json"]
   cap_add:
     - IPC_LOCK

 vault-init:
   image: "hashicorp/vault:${VAULT_TAG:-latest}"
   container_name: localsetup-vault-init
   command:
     - "sh"
     - "-c"
     - "/vault/scripts/vault-init.sh"
   environment:
     VAULT_ADDR: http://vault:8200
   volumes:
     - ./vault/scripts/vault-init.sh:/vault/scripts/vault-init.sh
     - ./vault/keys:/vault/keys
     - ./vault/root:/vault/root
   depends_on:
     vault:
       condition: service_started

‍

At the end your directory should look like

localsetup
├── docker-compose.yml              Docker Compose file is provided below
└── vault
   ├── config
   │   └── vault-config.json       Configuration file for Vault is provided below
   ├── data                        This is where vault will store all its data, This data is encrypted at rest.
   ├── keys                        * We will store Vault unseal here  
   ├── root                        * We will store Vault root keys here
   └── scripts
       └── vault-init.sh           Script to help us initialize Vault is provided below

Start vault

docker compose pull     ### Pulls required images
docker compose up -d    ### Brings up containers defined in docker-compose.yml file.

Check unseal key

ls -la vault/keys

We should observe key-1, key-2, key-3 files in this directory.

Debug any issue

docker logs localsetup-vault-init

Setup Service to Access Secrets

At Kapstan | Your In-House DevSecOps Engineer | We prioritise ‘Secret Segregation’ to ensure our services access only essential secrets. We achieve this through a structured process:

1. Dedicated Secret Engine: We create a unique secret engine for each service.
2. Access Control Policy: We establish a policy to govern access to the secret engine.
3. AppRole Integration: An AppRole is configured with the policy to access the secret engine.
4. Service Authentication: Services authenticate with Vault via the AppRole, gaining access to the secret engine.

This rigorous approach reduces the risk of unauthorised access. It guides our choices regarding where secrets are stored and which services can access them, upholding a secure environment.

We are going to take dummy service (kap_backend_app) as example service for this setup

Setting up AppRole in Vault

Create required additional directories

mkdir vault/policies        ### We will be storing the service policy here
mkdir vault/kap_backend_app ### this directory will store role_id and role_secret_id for the service

Create policy for AppRole

Content of the policy file at vault/policies/kap_backend_app.hcl should be as below:

path "kap_backend_app/*" {
 capabilities = ["create", "read", "update", "delete", "list"]
}

AppRole Creation

Changes to vault/scripts/vault-init.sh file to create AppRole

Add below code at end of file in vault/scripts/vault-init.sh

Pay attention to lines 18 and 19—they hold significance. This is where the role_id and role_secret_id are obtained. These details will be utilized by the service to authenticate itself when accessing information from the vault.

KAP_APP_NAME=kap_backend_app
KAP_APP_INIT_FILE=/vault/kap_backend_app/kap_app.init
if [[ -f "${KAP_APP_INIT_FILE}" ]]; then
   echo "${KAP_APP_INIT_FILE} exists. Vault already initialized for ${KAP_APP_NAME}."
else
   echo "Enabling Secrets Engine for ${KAP_APP_NAME}..."
   vault secrets enable -path=${KAP_APP_NAME} kv-v2

   echo "Creating ${KAP_APP_NAME} Policy..."
   vault policy write ${KAP_APP_NAME} /vault/policies/${KAP_APP_NAME}.hcl

   echo "Enabling AppRole Auth Backend..."
   vault auth enable approle

   echo "Creating ${KAP_APP_NAME} Approle Auth Backend..."
   vault write auth/approle/role/${KAP_APP_NAME} token_policies=${KAP_APP_NAME} token_ttl=2h token_max_ttl=6h
   vault read auth/approle/role/${KAP_APP_NAME}
   vault read auth/approle/role/${KAP_APP_NAME}/role-id | grep "role_id" | awk '{print $2}' | tee /vault/${KAP_APP_NAME}/${KAP_APP_NAME}-role-id > /dev/null
   vault write -force auth/approle/role/${KAP_APP_NAME}/secret-id | grep "secret_id" | awk '{print $2}' | head -n 1 | tee /vault/${KAP_APP_NAME}/${KAP_APP_NAME}-role-secret-id > /dev/null

   echo "${KAP_APP_NAME} Approle creation complete."
   touch ${KAP_APP_INIT_FILE}
fi

‍

Update docker-compose.yml file

Change to docker-compose.yml file to mount newly created 2 directories during vault-init (last 2 lines are additions)

volumes:
 - ./vault/scripts/vault-init.sh:/vault/scripts/vault-init.sh
 - ./vault/keys:/vault/keys
 - ./vault/root:/vault/root
 - ./vault/policies:/vault/policies
 - ./vault/kap_backend_app:/vault/kap_backend_app

Deploy Service

We are going to take simple container (alpine:latest) and access vault from that container using AppRole. I will install vault once the container is up. In real development situation this will be part of Dockerfile for the service.

Changes to docker-compose.yml

kap_backend_app:
 image: "alpine:latest"
 container_name: kap_backend_app
 command:
   - "tail"
   - "-f"
   - "/dev/null"
 environment:
   VAULT_ADDR: http://vault:8200
   VAULT_VERSION: 1.14.2

 volumes:
   - ./vault/kap_backend_app:/vault/kap_backend_app
 depends_on:
   vault:
     condition: service_started

‍

Deploy Service container

docker compose pull
docker compose up -d

Access vault from Service container

Let us go to the shell of service container and use AppRole based authentication

docker exec -it kap_backend_app /bin/ash

Most of our services are written in golang, we include vault golang client.

github.com/hashicorp/vault/api
github.com/hashicorp/vault/api/auth/approle
github.com/hashicorp/vault/sdk

‍

To make things simpler, I am demonstrating the access of vault using vault cli.

• Upto line no 6, vault client is installed
• line 9, authenticates to vault using Approle (using the role_id and secret_id)
  • We get client_token from the response and set as VAULT_TOKEN as we are using vault cli.
  • Equivalent of this can be done in golang by invoking SetToken method on vault client.
• Once token is set, we can perform operations allowed by our role, I demonstrate setting of key-value.

### We have used alpine:latest image, i.e. so we will need to install vault inside the container.
apk --no-cache add curl ca-certificates jq && \
   curl -sLo /tmp/vault.zip "https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip" && \
   unzip -d /usr/local/bin /tmp/vault.zip && \
   rm /tmp/vault.zip && \
   chmod +x /usr/local/bin/vault
   
### Let us login to vault using AppRole credentails and obtain token.
export VAULT_TOKEN=$(vault write -format=json auth/approle/login role_id=$(cat /vault/kap_backend_app/kap_backend_app-role-id) secret_id=$(cat /vault/kap_backend_app/kap_backend_app-role-secret-id) | jq -r .auth.client_token)

### Our access to vault is using AppRole authentication
vault kv put kap_backend_app/first_value key1=val1

‍

TLS Certificates for Services

We are a team with a strong emphasis on security. We prioritise the utilisation of TLS for communication between our services, necessitating TLS certificates. To fulfil these requirements, we sought a method to seamlessly generate TLS certificates for our services. Our objective was to establish an automated process for TLS certificate generation that aligns consistently with our production environment. In this regard also we use HCVault.

Do not be surprised when you see intermediate CA being used in our development environments – We want least delta b/w our local development and production environments w.r.t. tooling and developer experience.

We will continue same example further.

Creating TLS certificate can be viewed as 2 step process

1. Configure vault’s PKI engine – required to be done once
2. Generating TLS certificates – required to be done once per service.

Configure Vault’s PKI engine

Create required additional directories

mkdir vault/pki

Configure vault’s PKI engine in vault-init.sh file

Append to vault/scripts/vault-init.sh

KAP_APP_DOMAIN_NAME=$(echo ${KAP_APP_NAME} | tr '_' '-'). ### '_' is not valid domain, and vault produces unclear error

PKI_INIT_FILE=/vault/pki/pki.init
if [[ -f "${PKI_INIT_FILE}" ]]; then
   echo "${PKI_INIT_FILE} exists. Vault pki already initialized."
else
   ## Enable PKI Secrets Engine
   vault secrets enable pki
   ## Tune Max Lease TTL for PKI Secrets Engine
   vault secrets tune -max-lease-ttl=876000h pki
   ## Generate Root Certificate
   vault write pki/root/generate/internal common_name=mydomain.io ttl=876000h
   ## Write Root Certificate to File
   vault write -field=certificate pki/root/generate/internal common_name="mydomain-local" issuer_name="vault-pki" ttl=876000h > /vault/pki/vault_root_ca.crt
   ## Create Certificate Role
   vault write pki/roles/mydomain-local allow_any_name=true
   ## Configure PKI URLs
   vault write pki/config/urls issuing_certificates="http://vault:8200/v1/pki/ca" crl_distribution_points="http://vault:8200/v1/pki/crl"

   ## Enable Intermediate PKI Secrets Engine
   vault secrets enable -path=pki_int pki
   ## Tune Max Lease TTL for Intermediate PKI Secrets Engine
   vault secrets tune -max-lease-ttl=876000h pki_int
   ## Generate Intermediate CSR (Certificate Signing Request)
   vault write -field=csr pki_int/intermediate/generate/internal common_name="MyDomain Local Intermediate Authority" issuer_name="mydomain-local-intermediate" > /vault/pki/pki_intermediate.csr
   ## Display Intermediate CSR
   cat /vault/pki/pki_intermediate.csr
   ## Sign Intermediate CSR with Root Certificate
   vault write -field=certificate pki/root/sign-intermediate issuer_ref="vault-pki" csr=@/vault/pki/pki_intermediate.csr format=pem_bundle ttl="876000h" > /vault/pki/intermediate.cert.pem
   ## Set Signed Intermediate Certificate
   vault write pki_int/intermediate/set-signed certificate=@/vault/pki/intermediate.cert.pem

   ## Create server role
   vault write pki_int/roles/server issuer_ref="$(vault read -field=default pki_int/config/issuers)" allowed_domains=${KAP_APP_DOMAIN_NAME},localhost,127.0.0.1,host.docker.internal allow_subdomains=true allow_bare_domains=true require_cn=false server_flag=true max_ttl=8670h

   ## Create client role
   vault write pki_int/roles/client issuer_ref="$(vault read -field=default pki_int/config/issuers)" require_cn=false client_flag=true allow_any_name=true max_ttl=8670h

   touch ${PKI_INIT_FILE}
fi

‍

Update docker-compose.yml file

Change to docker-compose.yml file to mount newly created pki directory during vault-init (last 1 line is additions)

volumes:
 - ./vault/scripts/vault-init.sh:/vault/scripts/vault-init.sh
 - ./vault/keys:/vault/keys
 - ./vault/root:/vault/root
 - ./vault/policies:/vault/policies
 - ./vault/kap_backend_app:/vault/kap_backend_app
 - ./vault/pki:/vault/pki

‍

Generate TLS certificates for Services:

Append to vault/scripts/vault-init.sh file to generate certificates for service.

KAP_APP_PKI_INIT_FILE=/vault/kap_backend_app/kap_app_pki.init
if [[ -f "${KAP_APP_PKI_INIT_FILE}" ]]; then
   echo "${KAP_APP_PKI_INIT_FILE} exists. Vault pki already initialized."
else
   ## Generating server certificates
   RESULT=$(vault write -format=json pki_int/issue/server common_name=${KAP_APP_DOMAIN_NAME} alt_names="localhost,127.0.0.1,host.docker.internal" ttl="8670h")
   echo $RESULT | jq -r .data.certificate | tee /vault/${KAP_APP_NAME}/server.crt.pem > /dev/null
   echo $RESULT | jq -r .data.private_key| tee /vault/${KAP_APP_NAME}/server.key.pem > /dev/null
   echo $RESULT | jq -r .data.ca_chain[] | tee /vault/${KAP_APP_NAME}/ca-chain.crt.pem > /dev/null
   cat /vault/${KAP_APP_NAME}/ca-chain.crt.pem >> /vault/${KAP_APP_NAME}/server.crt.pem
   echo "Finished Generating Server Certificates for ${KAP_APP_NAME}, Saved In /vault/${KAP_APP_NAME}"

   ## Generating client certificates
   RESULT=$(vault write -format=json pki_int/issue/client ttl="8670h")
   echo $RESULT | jq -r .data.ca_chain[] | tee /vault/${KAP_APP_NAME}/ca-chain.crt.pem > /dev/null
   echo $RESULT | jq -r .data.certificate | tee /vault/${KAP_APP_NAME}/client.crt.pem > /dev/null
   echo $RESULT | jq -r .data.private_key| tee /vault/${KAP_APP_NAME}/client.key.pem > /dev/null
   cat /vault/${KAP_APP_NAME}/ca-chain.crt.pem >> /vault/${KAP_APP_NAME}/client.crt.pem

   touch ${KAP_APP_PKI_INIT_FILE}
fi

‍

Application Services should be configured to use the generated TLS certificates. Configuration would depend on the programming language and framework used in your application. Generally the description/guides to use TLS will be divided in 2 parts, i.e. generating the certificates, and using the certificates for given programming language and framework. You should skip the TLS certificate generation part (we have generated certificates using vault), and follow the part w.r.t. using certificates. I found Securing gRPC connection with SSL/TLS Certificate using Go article to be useful to configure GRPC service with TLS certificates.

Final State of localsetup directory

.
├── docker-compose.yml                          # Docker compose to help setup dev environemnt
└── vault
   ├── config
   │   └── vault-config.json                   # Simple vault configuration
   ├── data
       ...
   ├── kap_backend_app
   │   ├── ca-chain.crt.pem                    # CA cert chain (required to verify any cert generated from our setup)
   │   ├── client.crt.pem                      # Client certificate for our service.
   │   ├── client.key.pem                      # Key for client certificate
   │   ├── kap_app.init                        # File marker indicating that AppRole and sercret engine are initialized for this service
   │   ├── kap_app_pki.init                    # File marker indicating that client and server certificates are generated for this service
   │   ├── kap_backend_app-role-id             # Id of AppRole for this service
   │   ├── kap_backend_app-role-secret-id      # AppRole Secret for this service
   │   ├── server.crt.pem                      # Server certificate for this service
   │   └── server.key.pem                      # Key for Server certificate
   ├── keys
   │   ├── key-1                               # Unseal Key-1
   │   ├── key-2                               # Unseal Key-2
   │   ├── key-3                               # Unseal Key-3
   │   └── vault.init                          # File marker indicating that vault has been initialized
   ├── pki
   │   ├── intermediate.cert.pem
   │   ├── pki.init                            # filter marker indicating that pki engine of vault has been initialized
   │   ├── pki_intermediate.csr
   │   └── vault_root_ca.crt
   ├── policies
   │   └── kap_backend_app.hcl                 # Policy allowing service to access its secrets (but nothing else)
   ├── root
   │   └── token                               # Vault's root token
   └── scripts
       └── vault-init.sh                       # Finally, the script where all this configuration happens.

These details should help you have less delta in your development and production environment w.r.t. handling security.

In the next post of this series, we’ll delve into the details of our production setup. While understanding and experimenting with these concepts firsthand is valuable, applying them across the team while keeping focus on feature velocity and customer deliverable is entirely distinct challenge. Automation is the only viable approach here. We’ll explore this further in our subsequent post and examine how Kapstan can play a role in this aspect.

‍