Effective Secrets Management with HashiCorp Vault on Docker
Written on
Chapter 1: Understanding Vault
HashiCorp Vault is an innovative application designed for the secure storage and management of sensitive digital secrets. These may include passwords, tokens, API keys, and more. Vault addresses the issue of scattered secrets, which often reside in configuration files, scripts, or databases, by centralizing them in one location. This centralization simplifies auditing, key rotation, and revocation processes.
Vault's basic version is free and typically meets the needs of most users, while larger enterprises can opt for the enterprise version, which offers additional features and scalability, including multi-region replication. In this article, I will share insights from a proof of concept (POC) I conducted, which has since been automated within Kubernetes, allowing for the automatic unsealing of the vault.
Section 1.1: Storage Solutions for Vault Data
Vault accommodates various storage providers for its encrypted data, including ZooKeeper, MySQL, Postgres, and Cassandra. However, after evaluating several options, we have chosen Vault's integrated Raft storage as our backend solution.
The benefits of using Raft include:
- Integration directly within Vault
- Support for all configurations within Vault
- Capability for failover and multi-cluster replication (Enterprise version required)
- Reduction in network requests
- Enhanced performance by minimizing disk read/write overhead
- Cost efficiency through fewer virtual machines
- Elimination of an external storage service as a failure point
- Simplified issue diagnosis
Section 1.2: The Importance of Unsealing the Vault
Vault encrypts its data with multiple keys, necessitating these keys for unsealing. Upon reboot, the vault automatically seals itself. During the POC, the unsealing process was performed manually, with different keys potentially held by various individuals within the organization.
Chapter 2: Architecture and Scaling of the Vault Cluster
To set up the POC, we utilized three virtual machines (VMs) to run three Docker nodes. Note that a single node can suffice if high availability isn't required. The Raft storage will be located at /store/vault/raft on the VMs.
Scaling is best achieved with a minimum of three vault nodes for production, as this configuration allows for one node's failure. Vault's design channels all traffic through a single active node for reads and writes, meaning the addition of nodes does not enhance scalability and can actually reduce write throughput. Vault is designed for vertical scaling—enhancing its CPU and RAM—rather than horizontal scaling by adding nodes.
Steps to Deploy the Vault Cluster
While TLS is not mandatory, it is highly recommended to encrypt all communications. Establishing a trusted connection can also facilitate client-server validation via a central authority (CA).
Getting the Vault Docker Container
HashiCorp offers a Vault Docker container. To pull this image, execute the following command:
docker pull vault
Setting Up CA and TLS Certificates
This section assumes DNS entries are configured for vault1, vault2, and vault3, allowing for DNS lookups to resolve IP addresses.
Generate the CA or utilize a pre-existing one:
mkdir CA; cd CA
openssl req -newkey rsa:3072 -keyout vault-server-ca-key.pem -out vault-server-ca.pem -days 3650 -nodes -x509 -subj "/C=US/ST=Colorado/O=Test Domain/OU=Test Org/CN=Vault Server CA"
cd ..
Create certificates for the three vault nodes. If you are only setting up a single host for development, you may skip certain steps.
DNSDOMAIN=`hostname -d`
for i in {1..3}; do
HOST=vault${i}.$DNSDOMAIN
echo "Creating certs for $HOST"
IP=`host $HOST | awk 'NR==1{print $4}'`
mkdir $HOST
cd $HOST
openssl req -newkey rsa:3072 -keyout key.pem -out cert.csr -nodes -subj "/C=US/ST=Colorado/O=Test Domain/OU=Test Org/CN=$HOST"
# Configuration for server cert
...
cp -s ../CA/* .
openssl x509 -days 3600 -req -in cert.csr -CA vault-server-ca.pem -CAkey vault-server-ca-key.pem -CAcreateserial -out cert.pem -sha256 -extfile server_cert.conf
cd ..
done
This creates the CA certificates and the certificates for the three vault nodes.
Creating the Vault Configuration File
On the Vault VM, set up the directory structure and create the configuration file:
mkdir -p /store/vault/{config,raft,logs}
VAULT_NODE_ID=`hostname -s`
VAULT_VIP=10.0.100.120
VAULT_NODE_IP=`hostname -i`
cat > /store/vault/config/config.hcl <<EOF
backend "raft" {
path = "/vault/raft"
}
listener "tcp" {
address = "$VAULT_NODE_IP:8200"
cluster_address = "$VAULT_NODE_IP:8201"
}
EOF
Starting the Vault Docker Container
On the docker host for the vault node, copy the CA certificate to the appropriate location and run the following commands:
docker run -d -v /store/vault/config:/vault/config -v /store/vault/raft:/vault/raft --network host -h hostname -f --cap-add=IPC_LOCK --name vault-server --restart always vault:latest
Repeat this for the second and third nodes.
Initializing Vault
The follower nodes do not require initialization as they will join the primary cluster and replicate the data. To initialize the Vault secret store, use the first vault node:
vault operator init
Vault will output several unseal keys and an initial root token, which must be safeguarded and never shared irresponsibly.
Unsealing the Vault
To unseal the vault, the following steps need to be executed:
export VAULT_TOKEN='your_root_token_here'
vault operator unseal <key1>
vault operator unseal <key2>
vault operator unseal <key3>
If successful, the status will indicate that the vault is unsealed.
Joining Follower Nodes to the Primary Vault Node
The following commands will connect the second and third nodes to the primary vault:
PRIMARY_NODE=vault1.test.domain
vault operator unseal <key1>
vault operator unseal <key2>
vault operator unseal <key3>
To check the status of the raft cluster, use:
vault operator raft list-peers
Web UI Access
Conclusion
This configuration covers the essentials of utilizing Vault. Vault offers additional advanced features such as dynamic credentials and can also function as a central authority (CA) or manage one-time password (OTP) logins for SSH. It is an excellent solution for secret management.
For a deeper understanding of getting started with HashiCorp Vault, watch the following video.
Learn how to securely manage secrets in containers through this informative video.