HashiCorp Vault - Secrets Management Tools | OneNoughtOne

0/319

System Design (HLD)Secrets Management Tools

Secrets Management Tools

LevelAdvanced

Duration90 mins

TopicSecrets Management Tools

1 / 5

HashiCorp Vault

The Foundation of Modern Secrets Management

In the landscape of secrets management, HashiCorp Vault stands as the de facto standard for organizations that demand enterprise-grade security, extensive flexibility, and integration with virtually any infrastructure. Vault isn't merely a secrets storage system—it's a comprehensive secrets lifecycle management platform that fundamentally transforms how organizations handle sensitive data.

Vault emerged from HashiCorp's deep expertise in infrastructure automation, filling a critical gap: how do you manage secrets in dynamic, cloud-native environments where traditional approaches fail? The answer is a system that treats secrets as first-class citizens with their own lifecycle, access controls, and audit capabilities.

What You Will Learn

By the end of this page, you will understand Vault's architecture, core concepts, and operational patterns. You'll be equipped to design Vault deployments for production environments, understand dynamic secret generation, and implement robust access control policies that scale with your organization.

Why HashiCorp Vault

Before diving into Vault's mechanics, it's essential to understand the problems it solves. Traditional secrets management approaches often involve:

Environment variables scattered across deployment configurations
Configuration files with hardcoded secrets checked into version control
Shared credentials used by multiple applications and team members
Static credentials that never rotate and accumulate risk over time
Manual processes for secret distribution and rotation

These approaches create fundamental security risks: secrets sprawl, lack of auditability, impossible-to-track access patterns, and credentials that persist indefinitely even after team members leave or systems are decommissioned.

Problems Vault Solves

•Secret Sprawl — Centralizes all secrets into a single, authoritative source of truth with consistent access patterns and audit logging.
•Static Credentials — Introduces dynamic secrets that are generated on-demand with automatic expiration, eliminating long-lived credentials.
•Access Control Complexity — Provides fine-grained, policy-based access control that integrates with existing identity systems (LDAP, OIDC, cloud IAM).
•Audit Trail Gaps — Records every secret access with detailed context: who accessed what, when, from where, and using which authentication method.
•Key Management Burden — Offers encryption-as-a-service, handling key management, rotation, and cryptographic operations transparently.
•Multi-Cloud Complexity — Provides a unified secrets interface regardless of whether workloads run on AWS, GCP, Azure, Kubernetes, or on-premises.

Vault's Core Value Proposition

Vault's fundamental insight is that secrets should be treated like infrastructure: managed declaratively, versioned, audited, and automatically maintained. Just as Terraform treats infrastructure as code, Vault treats secrets as managed resources with lifecycle, policies, and observability.

Vault Architecture Deep Dive

Understanding Vault's architecture is crucial for designing production deployments that meet security, availability, and performance requirements. Vault follows a client-server model where the Vault server manages all secrets operations while clients interact through a consistent API.

Core Architectural Components:

Vault Server Components
Component	Responsibility	Key Characteristics
Storage Backend	Persistent storage for encrypted data	Pluggable (Consul, Raft, S3, etc.); handles durability, not security
Barrier	Encryption layer protecting storage	All data encrypted before storage; decryption requires unseal keys
Secret Engines	Generate, store, and manage secrets	Pluggable backends (KV, AWS, database, PKI, etc.)
Auth Methods	Verify client identity	Pluggable authentication (LDAP, OIDC, Kubernetes, cloud IAM)
Audit Devices	Record all operations	Multiple backends (file, syslog, socket); mandatory for compliance
System Backend	Internal Vault operations	Policies, leases, tokens, health, seal status management

The Seal/Unseal Mechanism:

Vault implements a critical security feature through its seal mechanism. When a Vault server starts, it begins in a sealed state—all data remains encrypted and inaccessible. To unseal Vault, operators must provide a threshold of unseal keys (typically 3 of 5 shares using Shamir's Secret Sharing algorithm).

This design ensures that:

No single operator can unseal Vault alone
Vault automatically seals if it detects tampering
Even if an attacker gains access to storage, data remains encrypted
Unseal keys can be distributed across different security domains

Master Key and Encryption Hierarchy:

Vault Encryption Hierarchy

Encryption Hierarchy

┌─────────────────────────────────────────────────────────────┐
│                    DATA ENCRYPTION KEY                       │
│     (256-bit AES-GCM, encrypts all stored data)             │
└───────────────────────────┬─────────────────────────────────┘
                            │ Encrypted by
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                      MASTER KEY                              │
│     (256-bit key, reconstructed during unseal)              │
└───────────────────────────┬─────────────────────────────────┘
                            │ Split using Shamir's Secret Sharing
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                     UNSEAL KEYS                              │
│     (N shares, threshold K required to reconstruct)         │
│     Example: 5 shares, 3 required (3-of-5)                  │
└─────────────────────────────────────────────────────────────┘
 
Alternative: Auto-Unseal with Cloud KMS
┌─────────────────────────────────────────────────────────────┐
│                      MASTER KEY                              │
│     (Encrypted by cloud KMS instead of Shamir shares)       │
└───────────────────────────┬─────────────────────────────────┘
                            │ Protected by
                            ▼
┌─────────────────────────────────────────────────────────────┐
│               CLOUD KMS (AWS KMS, GCP CKMS, etc.)           │
│     (Hardware-backed, IAM-controlled key access)            │
└─────────────────────────────────────────────────────────────┘

Auto-Unseal Considerations

While auto-unseal simplifies operations by eliminating manual unseal ceremonies, it shifts trust to your cloud provider's KMS. This is appropriate for most production environments but may not meet certain compliance requirements that mandate operator-controlled unseal processes. Understand your security posture before choosing.

Secret Engines - The Core Abstraction

Secret engines are the heart of Vault's flexibility. Each engine is a component that stores, generates, or transforms data. Engines are mounted at specific paths, and all operations on secrets go through the appropriate engine.

Vault ships with numerous built-in engines, each designed for specific use cases:

Core Secret Engines
Engine	Purpose	Use Cases
KV (Key-Value)	Store arbitrary secrets	API keys, configuration, static credentials
AWS	Generate dynamic AWS credentials	IAM users, STS tokens, assumed roles
Database	Generate dynamic database credentials	MySQL, PostgreSQL, MongoDB, Oracle users
PKI	Certificate authority operations	TLS certificates, client certs, code signing
Transit	Encryption-as-a-service	Data encryption, signing, verification
SSH	Generate SSH credentials	Signed SSH certificates, OTP access
TOTP	Time-based OTP generation	Multi-factor authentication tokens
Identity	Entity and group management	Cross-mount identity correlation

The KV Secrets Engine - Versioned Secrets:

The KV (Key-Value) engine is the most commonly used engine, storing arbitrary key-value pairs. KV v2 introduces versioning, enabling you to:

Retrieve previous versions of a secret
See when each version was created
Set deletion policies for version retention
Perform soft-delete with recovery capability

KV v2 Operations
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# Enable KV v2 at a path
vault secrets enable -path=secret kv-v2
 
# Write a secret (creates version 1)
vault kv put secret/myapp/database \
    username="app_user" \
    password="supersecret123"
 
# Read current version
vault kv get secret/myapp/database
 
# Update secret (creates version 2)
vault kv put secret/myapp/database \
    username="app_user" \
    password="newsecret456"
 
# Read specific version
vault kv get -version=1 secret/myapp/database
 
# View metadata (all versions)
vault kv metadata get secret/myapp/database
 
# Soft delete current version
vault kv delete secret/myapp/database
 
# Undelete (recover soft-deleted version)
vault kv undelete -versions=2 secret/myapp/database
 
# Permanently destroy a version
vault kv destroy -versions=1 secret/myapp/database
 
# Delete all versions and metadata
vault kv metadata delete secret/myapp/database
 
# Configure version retention
vault write secret/metadata/myapp/database \
    max_versions=10 \
    delete_version_after="720h"

Dynamic Secrets - The Game Changer

Dynamic secrets represent Vault's most powerful capability and the feature that fundamentally differentiates it from simple secret storage systems. Rather than storing pre-existing secrets, dynamic secret engines generate secrets on-demand with automatic expiration.

Why Dynamic Secrets Matter:

Consider a traditional database credential workflow:

DBA creates a database user with username/password
Password is stored somewhere (file, env var, secret manager)
Multiple applications use the same credential
Password never changes because rotation is risky and complex
When a developer leaves, the credential remains valid
Breaches are undetectable—who accessed what?

With dynamic secrets:

Application requests a database credential from Vault
Vault creates a new, unique database user instantly
Credential is returned with a TTL (e.g., 1 hour)
Application uses credential during its lease
Vault automatically revokes credential when lease expires
Each access creates an auditable, time-limited credential

Static Credentials

•Created once, used indefinitely
•Shared across applications/users
•No individual accountability
•Manual rotation (rarely done)
•Breach window: forever
•Revocation: change password everywhere

Dynamic Secrets

•Generated on-demand per request
•Unique per application instance
•Full audit trail per credential
•Automatic expiration and rotation
•Breach window: credential TTL only
•Revocation: single API call

Database Secrets Engine Example:

The database secrets engine demonstrates dynamic secrets at their best. Vault connects to your database as a privileged user, then creates/revokes individual credentials on behalf of applications.

Dynamic Database Credentials
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Enable the database secrets engine
vault secrets enable database
 
# Configure the PostgreSQL connection
vault write database/config/production-postgres \
    plugin_name=postgresql-database-plugin \
    allowed_roles="readonly","readwrite","admin" \
    connection_url="postgresql://{{username}}:{{password}}@postgres.prod:5432/appdb?sslmode=require" \
    username="vault_admin" \
    password="vault_admin_password"
 
# Rotate the root credential (Vault now exclusively controls it)
vault write -force database/rotate-root/production-postgres
 
# Create a role for read-only access
vault write database/roles/readonly \
    db_name=production-postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    revocation_statements="REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"
 
# Create a role for read-write access
vault write database/roles/readwrite \
    db_name=production-postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="8h"

Lease Renewal Strategy

For long-running applications, implement lease renewal before expiration. Request a short initial TTL (e.g., 1 hour), then renew every 30 minutes. This provides automatic credential rotation without connection disruption, while still limiting breach exposure if credentials leak.

Authentication Methods

Vault supports numerous authentication methods that allow clients to prove their identity. Each method is suited to different environments and use cases. The authentication process returns a Vault token with attached policies that determine what secrets the client can access.

Token-Based Authentication Model:

All Vault authentication ultimately results in a token. The token carries:

Policies — What paths and operations are allowed
TTL — When the token expires
Renewable — Whether the token can extend its life
Orphan status — Whether it's tied to a parent token
Entity ID — Correlation with Vault's identity system

Common Authentication Methods
Method	Best For	Identity Source
Token	Direct API access, initial setup	Pre-existing Vault token
AppRole	Machines, CI/CD pipelines	Role ID + Secret ID (two-factor for machines)
Kubernetes	Pods in Kubernetes clusters	Kubernetes service account JWT
AWS IAM	EC2 instances, Lambda functions	AWS instance identity or IAM credentials
GCP	GCE instances, Cloud Functions	GCP service account identity
Azure	Azure VMs, AKS	Azure managed identity
LDAP	Enterprise users	LDAP/Active Directory credentials
OIDC	SSO users, web applications	OpenID Connect provider (Okta, Auth0, etc.)
JWT/OIDC	Service-to-service, external JWTs	JWT from trusted issuers
Userpass	Development, simple setups	Username and password

AppRole Authentication - The Machine Identity Standard:

AppRole is Vault's recommended method for authenticating machines and applications. It implements a two-factor authentication model:

Role ID — A relatively static identifier (like a username)
Secret ID — A dynamic, single-use credential (like a password)

This separation enables secure bootstrapping: the Role ID can be embedded in configuration, while the Secret ID is delivered through a separate, secure channel.

AppRole Authentication
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Enable AppRole auth method
vault auth enable approle
 
# Create a role for a specific application
vault write auth/approle/role/payment-service \
    token_policies="payment-service" \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_ttl=10m \
    secret_id_num_uses=1 \
    token_num_uses=0 \
    bind_secret_id=true
 
# Get the Role ID (can be stored in config)
vault read auth/approle/role/payment-service/role-id
# Role ID: abc123-def456-ghi789
 
# Generate a Secret ID (deliver securely to application)
vault write -f auth/approle/role/payment-service/secret-id
# Secret ID: xyz987-uvw654-rst321
 
# Application authenticates with both
vault write auth/approle/login \
    role_id="abc123-def456-ghi789" \
    secret_id="xyz987-uvw654-rst321"
 
# Response contains token
{
  "auth": {
    "client_token": "hvs.CAESIJl...",
    "policies": ["default", "payment-service"],
    "token_type": "service",
    "renewable": true,
    "lease_duration": 3600
  }
}

Kubernetes Auth is Ideal for K8s Workloads

When running on Kubernetes, the Kubernetes auth method provides the cleanest integration. Pods automatically receive service account tokens that Vault can verify against the Kubernetes API. No secret delivery mechanism is needed—identity is derived from the infrastructure itself.

Policies - Fine-Grained Access Control

Vault's policy system controls what secrets and operations each authenticated identity can access. Policies are written in HCL (HashiCorp Configuration Language) and follow a deny-by-default model—everything is forbidden unless explicitly permitted.

Policy Syntax Fundamentals:

Policies define rules at specific paths with allowed capabilities:

Vault Policy Examples

# Allow reading secrets for a specific application
path "secret/data/myapp/*" {
  capabilities = ["read", "list"]
}
 
# Allow full management of database credentials
path "database/creds/myapp-role" {
  capabilities = ["read"]
}
 
# Allow renewing own token
path "auth/token/renew-self" {
  capabilities = ["update"]
}
 
# Allow looking up own token info
path "auth/token/lookup-self" {
  capabilities = ["read"]
}
 
#--------------------------------------------------
# Capability Reference:
#   create  - Create new data (POST to non-existing path)
#   read    - Read data (GET)
#   update  - Modify existing data (POST/PUT)
#   delete  - Delete data (DELETE)
#   list    - List keys at path (LIST)
#   sudo    - Access root-protected paths
#   deny    - Explicitly forbid (overrides other rules)
#--------------------------------------------------

Policy Design Best Practices

Design policies with least privilege: start restrictive and add permissions as needed. Use path templating to reduce policy sprawl. Avoid wildcards at path roots (e.g., 'secret/*'). Regularly audit policies as requirements change. Remember that deny always wins over allow in policy evaluation.

Policy Attachment:

Policies are attached to tokens through authentication methods. When configuring an auth method role, you specify which policies apply:

# AppRole with specific policies
vault write auth/approle/role/payment-service 
    token_policies="payment-secrets,database-readonly,common"

# Kubernetes auth role with policies
vault write auth/kubernetes/role/api-server 
    policies="api-secrets,transit-encrypt"

When a client authenticates, their token inherits the policies defined for their authentication role, creating a clear mapping from identity to permissions.

High Availability and Production Deployment

Production Vault deployments must address availability, disaster recovery, performance, and operational concerns. Vault supports multiple storage backends with different HA characteristics.

Integrated Raft Storage (Recommended):

Vault's integrated Raft storage has become the recommended approach for most deployments. It provides:

Built-in HA without external dependencies
Automatic leader election and failover
Snapshot-based backup and restore
Simpler operational model

Production Configuration

# Production Vault configuration with Raft storage
storage "raft" {
  path = "/vault/data"
  node_id = "vault-1"
  
  # Performance tuning
  performance_multiplier = 1
  
  # Retry configuration for cluster join
  retry_join {
    leader_api_addr = "https://vault-2.example.com:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-3.example.com:8200"
  }
}
 
# Listener configuration
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/vault/certs/vault.crt"
  tls_key_file = "/vault/certs/vault.key"
  tls_min_version = "tls12"
  
  # Enable request logging
  telemetry {
    unauthenticated_metrics_access = false
  }
}
 
# Cluster communication
cluster_addr = "https://vault-1.example.com:8201"
api_addr = "https://vault-1.example.com:8200"
 
# Auto-unseal with AWS KMS
seal "awskms" {
  region     = "us-west-2"
  kms_key_id = "alias/vault-unseal"
}
 
# Telemetry for monitoring
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname = true
}
 
# Audit logging (mandatory for compliance)
audit {
  file "file" {
    path = "/vault/logs/audit.log"
    log_raw = false
    mode = "0600"
  }
}
 
# UI access
ui = true
 
# Default lease settings
default_lease_ttl = "1h"
max_lease_ttl = "24h"
 
# Disable memory locking (required for containers, enable if on dedicated VM)
disable_mlock = true
 
# Log level
log_level = "info"

Production Deployment Checklist

•TLS Everywhere — All Vault communication encrypted; use proper certificates from trusted CA
•Auto-Unseal Configured — Use cloud KMS or HSM to avoid manual unseal ceremonies
•HA Deployment — Minimum 3 nodes for Raft consensus; deploy across availability zones
•Audit Logging Enabled — All secret access logged; alerts on audit failures; logs shipped to SIEM
•Backup Strategy — Regular Raft snapshots; test restore procedures; off-site backup storage
•Monitoring & Alerting — Prometheus metrics collected; alerts on seal status, leader changes, errors
•Network Policies — Vault API only accessible from authorized networks; cluster traffic isolated
•Resource Limits — CPU/memory limits in orchestrated deployments; prevent resource exhaustion

Disaster Recovery Planning

Regularly test disaster recovery procedures. Take Raft snapshots before any upgrades. Store recovery keys and root tokens securely (ideally in separate secure storage like hardware security modules). Document runbooks for common operational scenarios including complete cluster recovery.

Summary: HashiCorp Vault Mastery

HashiCorp Vault represents the gold standard in secrets management, providing a comprehensive platform that addresses the full lifecycle of secrets in modern infrastructure. Let's consolidate the key concepts:

Key Takeaways

•Vault is a secrets lifecycle platform — It manages creation, rotation, revocation, and auditing of secrets, not just storage.
•Dynamic secrets transform security posture — On-demand credential generation with automatic expiration eliminates static credential risks.
•Secret engines provide extensibility — Pluggable backends for KV storage, databases, cloud providers, PKI, encryption, and more.
•Authentication is identity-driven — Multiple auth methods integrate with existing identity systems; tokens carry policies.
•Policies implement least privilege — Fine-grained, path-based access control with deny-by-default semantics.
•Seal mechanism protects at rest — Master key encryption prevents unauthorized access even with storage compromise.
•HA and disaster recovery are essential — Production deployments require multi-node clusters with tested recovery procedures.

What's Next:

With a deep understanding of HashiCorp Vault's architecture and capabilities, the next page explores AWS Secrets Manager—Amazon's managed secrets service. You'll learn how it compares to Vault, when to choose one over the other, and how to leverage AWS-native secrets management for cloud-centric workloads.

Page Complete

You now have a comprehensive understanding of HashiCorp Vault's architecture, secret engines, authentication methods, policies, and production deployment patterns. This knowledge forms the foundation for comparing and evaluating other secrets management solutions in the following pages.