Persistent Volumes - Kubernetes Storage | OneNoughtOne

0/319

System Design (HLD)Kubernetes Storage

Kubernetes Storage

LevelAdvanced

Duration90 mins

TopicKubernetes Storage

1 / 5

Persistent Volumes

The Storage Challenge in Container Orchestration

Containers were designed to be ephemeral—born, executed, and destroyed without leaving traces. This fundamental characteristic that makes containers so powerful for stateless workloads becomes a critical challenge when applications need to persist data beyond the container lifecycle.

The core tension: Kubernetes orchestrates containers across a dynamic, distributed cluster where pods can be scheduled on any node, moved during failures, and scaled across the infrastructure. How do we provide durable, consistent storage to workloads that may run anywhere in the cluster, potentially on different physical machines than where their data resides?

Persistent Volumes (PVs) represent Kubernetes' solution to this fundamental challenge. They abstract the storage infrastructure, decouple storage provisioning from consumption, and enable stateful workloads to operate reliably in a containerized environment.

What You Will Learn

By the end of this page, you will understand the complete Persistent Volume architecture, including PV/PVC binding mechanics, access modes and their implications, reclaim policies for data lifecycle management, volume phases and state transitions, and production best practices for reliable storage in Kubernetes.

The Container Storage Problem

Before diving into Persistent Volumes, we must understand the storage challenges that containers inherently face. This context is essential for appreciating why the PV abstraction exists and how it solves these problems.

Container filesystem fundamentals:

When a container starts, it receives a layered filesystem constructed from its image. The base layers are read-only, shared across all containers using that image. A thin writable layer (the container layer) sits on top, capturing any modifications made during runtime.

This architecture has profound implications:

Container Storage Limitations

•Ephemeral by design — When a container terminates, its writable layer is destroyed. Any data written to the container filesystem is lost permanently.
•Copy-on-write overhead — The writable layer uses copy-on-write (CoW) semantics. Modifying existing files requires copying the entire file, adding latency and consuming additional space.
•Storage driver coupling — The container filesystem performance depends heavily on the storage driver (overlay2, devicemapper, etc.). Write-intensive workloads often suffer significant performance degradation.
•Node-local storage — Container filesystems are local to the node where the container runs. If a pod moves to another node, its data doesn't follow.
•No sharing between containers — Each container has its own writable layer. Sharing data between containers in the same pod requires explicit volume mounts.

The node scheduling challenge:

Consider a database pod running on Node A with data stored locally. If Node A fails, Kubernetes must reschedule the pod to Node B. Without persistent storage abstraction:

The pod starts fresh on Node B with no data
All database state is lost
Applications depending on this database fail
Manual intervention is required to restore from backups

This scenario is unacceptable for production stateful workloads. We need storage that:

Survives container and pod restarts
Is accessible regardless of which node runs the pod
Provides consistent performance characteristics
Can be managed independently of the workloads consuming it

The Stateful Workload Reality

While the "cattle, not pets" philosophy encourages stateless design, production systems inevitably require state: databases, message queues, file storage, session data, and more. Kubernetes must accommodate these workloads with robust storage primitives.

Persistent Volumes Architecture

Persistent Volumes introduce a two-layer abstraction that decouples storage provisioning from consumption. This separation of concerns is fundamental to Kubernetes' storage model.

The PV/PVC model:

Kubernetes uses two distinct API resources for persistent storage:

Persistent Volume (PV): A cluster-level resource representing a piece of physical storage. PVs are provisioned by administrators (or dynamically by Storage Classes) and exist independently of any pod. They represent the actual storage capacity—an NFS share, an AWS EBS volume, a GCE Persistent Disk, or any supported storage backend.

Persistent Volume Claim (PVC): A namespace-scoped request for storage by a user or workload. PVCs specify the desired storage characteristics (size, access mode, storage class) without requiring knowledge of the underlying storage infrastructure. Pods mount PVCs, not PVs directly.

persistent-volume.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# Persistent Volume (PV) - Cluster-level storage resource
# Provisioned by administrators or dynamically via StorageClass
apiVersion: v1
kind: PersistentVolume
metadata:
  name: database-pv-001
  labels:
    type: ssd
    environment: production
    app: postgresql
spec:
  # Storage capacity - must match or exceed PVC requests
  capacity:
    storage: 100Gi
  
  # Volume mode: Filesystem (default) or Block
  volumeMode: Filesystem
  
  # Access modes determine how the volume can be mounted
  # This volume supports ReadWriteOnce (single node read-write)
  accessModes:
    - ReadWriteOnce
  
  # Reclaim policy: what happens when PVC is deleted
  # Retain: keep data, manual cleanup required
  # Delete: automatically delete backing storage
  # Recycle: deprecated, basic scrub (rm -rf)
  persistentVolumeReclaimPolicy: Retain
  
  # Storage class for binding with PVCs
  # Empty string means this PV only binds to PVCs requesting ""
  storageClassName: fast-ssd
  
  # Mount options passed to the mount command
  mountOptions:
    - hard
    - nfsvers=4.1
  
  # NFS storage backend specification
  # Other backends: awsElasticBlockStore, gcePersistentDisk, 
  # azureDisk, csi, hostPath, local, etc.
  nfs:
    path: /exports/database-001
    server: nfs-server.storage.svc.cluster.local
    readOnly: false

Key architectural properties:

PV vs PVC: Separation of Concerns
Aspect	Persistent Volume (PV)	Persistent Volume Claim (PVC)
Scope	Cluster-wide resource	Namespace-scoped resource
Created by	Administrators or dynamically	Developers/workload owners
Represents	Actual storage infrastructure	Request for storage
Contains	Backend-specific configuration	Desired storage characteristics
Lifecycle	Independent of pods/workloads	Bound to application lifecycle
Knowledge required	Storage infrastructure details	Only capacity and access needs

Why this separation matters:

This architecture enables several critical capabilities:

Role separation: Storage administrators manage infrastructure without application knowledge. Developers request storage without infrastructure expertise.
Portability: Applications reference PVCs by name. The same pod specification works across environments where PVCs are satisfied by different underlying storage.
Pre-provisioning: PVs can be created ahead of workloads, ensuring storage is available when applications deploy.
Resource management: PVs can be managed, monitored, and maintained independently of the workloads consuming them.

PVC Specification and Binding

Persistent Volume Claims are the interface through which applications request storage. Understanding PVC specification and the binding process is essential for reliable storage management.

PVC specification anatomy:

persistent-volume-claim.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# Persistent Volume Claim (PVC) - Request for storage
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgresql-data
  namespace: database
  labels:
    app: postgresql
    component: primary
spec:
  # Access modes requested (must be supported by bound PV)
  accessModes:
    - ReadWriteOnce
  
  # Volume mode: Filesystem or Block
  volumeMode: Filesystem
  
  # Resource request (minimum storage capacity)
  resources:
    requests:
      storage: 50Gi
  
  # Storage class to use for binding/provisioning
  # - Specific class name: bind to PV with matching class
  # - Empty string "": bind only to PVs without class
  # - Omitted: use cluster default StorageClass
  storageClassName: fast-ssd
  
  # Optional: Label selector to narrow PV selection
  selector:
    matchLabels:
      type: ssd
      environment: production
    matchExpressions:
      - key: app
        operator: In
        values:
          - postgresql
          - mysql
  
  # Optional: Request binding to specific PV by name
  # This bypasses normal matching logic
  # volumeName: database-pv-001
 
---
# Pod consuming the PVC
apiVersion: v1
kind: Pod
metadata:
  name: postgresql
  namespace: database
spec:
  containers:
    - name: postgresql
      image: postgres:15
      volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
          # Optional: mount a subdirectory of the volume
          subPath: pgdata
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: postgresql-data
        # Optional: mount as read-only
        readOnly: false

The binding algorithm:

When a PVC is created, the Kubernetes PV controller attempts to bind it to an appropriate PV. The binding process follows these rules:

StorageClass matching: PVC storageClassName must match PV storageClassName exactly. If PVC omits the field, cluster default is used.
Access mode compatibility: The PV must support all access modes requested by the PVC.
Capacity satisfaction: The PV capacity must be greater than or equal to the PVC request.
Selector evaluation: If PVC specifies a selector, PV labels must satisfy all conditions.
Volume name: If PVC specifies volumeName, only that specific PV is considered.
Best fit selection: Among qualifying PVs, Kubernetes selects the smallest PV that satisfies the request (to minimize waste).
Binding is exclusive: Each PV binds to exactly one PVC. Once bound, the PV is unavailable to other claims.

Binding Is Permanent

PV-PVC binding is a one-to-one relationship. Once bound, a PV cannot be bound to a different PVC until the current binding is released (PVC deleted). Even if the PVC requests 10Gi and binds to a 100Gi PV, no other claim can use the remaining 90Gi.

Binding scenarios and troubleshooting:

Common PVC Binding Issues
Symptom	Cause	Resolution
PVC stuck in Pending	No PV matches requirements	Create appropriate PV or adjust PVC spec
PVC stuck in Pending	StorageClass has no provisioner	Use a class with dynamic provisioning or pre-create PVs
PVC stuck in Pending	Requested capacity exceeds available PVs	Create larger PV or reduce PVC request
PVC stuck in Pending	Access mode mismatch	Ensure PV supports requested access modes
PVC stuck in Pending	Selector matches no PVs	Verify PV labels match selector criteria
Wrong PV bound	Multiple PVs qualified, smallest picked	Use selectors or volumeName for precise control

Access Modes Deep Dive

Access modes define how a Persistent Volume can be mounted by pods across the cluster. Understanding access modes is critical for designing storage architectures that match workload requirements while respecting infrastructure constraints.

The three access modes:

Kubernetes Access Modes

•ReadWriteOnce (RWO) — The volume can be mounted as read-write by a single node. Multiple pods on the same node can access the volume, but pods on other nodes cannot. This is the most common mode for block storage.
•ReadOnlyMany (ROX) — The volume can be mounted read-only by multiple nodes simultaneously. Useful for sharing static content, configuration data, or trained ML models across many pods.
•ReadWriteMany (RWX) — The volume can be mounted as read-write by multiple nodes simultaneously. Required for shared storage scenarios but supported by fewer storage backends (typically network filesystems like NFS, CephFS, or cloud file services).

Access Mode Enforcement

Access modes are constraints on how storage can be mounted, not guarantees about data consistency. ReadWriteMany doesn't provide distributed locking or prevent write conflicts. Applications must implement their own concurrency control when sharing writable storage.

ReadWriteOncePod (RWOP):

Kubernetes 1.22+ introduced a fourth access mode:

ReadWriteOncePod (RWOP): The volume can be mounted as read-write by a single pod cluster-wide. This is stronger than RWO—even pods on the same node cannot share the mount. Use RWOP when you need to guarantee exclusive access for data integrity.

Storage backend support matrix:

Not all storage backends support all access modes. This table shows common patterns:

Access Mode Support by Storage Type
Storage Backend	RWO	ROX	RWX	RWOP	Notes
AWS EBS	✓	✗	✗	✓	Block storage, single-AZ
GCE Persistent Disk	✓	✓*	✗	✓	*ROX requires disk type configuration
Azure Disk	✓	✗	✗	✓	Block storage, premium SSD available
NFS	✓	✓	✓	✗	Network filesystem, widely compatible
CephFS	✓	✓	✓	✗	Distributed filesystem
AWS EFS	✓	✓	✓	✗	Managed NFS, multi-AZ
Azure Files	✓	✓	✓	✗	SMB/NFS file shares
GCE Filestore	✓	✓	✓	✗	Managed NFS
Local PV	✓	✗	✗	✓	Node-local storage only
HostPath	✓	✗	✗	✗	Single-node, testing only

Choosing access modes:

The access mode decision impacts architecture significantly:

Single stateful pod (database primary): RWO or RWOP is sufficient and preferred for integrity
Multiple read replicas: ROX allows sharing a snapshot or read-only data
Shared application state: RWX required, but consider if a database/cache is more appropriate
Log aggregation: RWX for multi-writer, but streaming to external system may be better
Content distribution: ROX for static assets shared across web server replicas

Reclaim Policies and Volume Lifecycle

The reclaim policy determines what happens to a Persistent Volume's backing storage when its claim is deleted. This is a critical configuration for data safety, cost management, and operational procedures.

The volume lifecycle phases:

Persistent Volumes progress through defined phases:

PV Lifecycle Phases

•Available — The PV is ready for claiming. No PVC is bound to it. For statically provisioned PVs, this is the initial state.
•Bound — The PV is bound to a PVC. The storage is in use by workloads referencing that PVC.
•Released — The PVC has been deleted, but the PV has not yet been reclaimed. The data still exists on the backing storage.
•Failed — Automated reclamation failed. Manual intervention required to recover or clean up the volume.

Reclaim policies:

The persistentVolumeReclaimPolicy field controls transitions from Released state:

PV Reclaim Policies
Policy	Behavior	Use Case	Data Safety
Retain	PV enters Released state, data preserved, manual cleanup required	Production databases, audit data, any critical data	Highest—data preserved until manual action
Delete	PV and backing storage automatically deleted	Ephemeral workloads, development environments, auto-provisioned volumes	None—data destroyed automatically
Recycle (deprecated)	Basic rm -rf on volume, PV made Available again	Legacy compatibility only	Low—simple delete, no secure wipe

reclaim-policy-examples.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# Retain Policy: For production databases
# Data is preserved even after PVC deletion
apiVersion: v1
kind: PersistentVolume
metadata:
  name: production-db-pv
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain  # Data survives PVC deletion
  storageClassName: production-ssd
  awsElasticBlockStore:
    volumeID: vol-0123456789abcdef0
    fsType: ext4
 
---
# Delete Policy: For ephemeral workloads
# Volume and data automatically cleaned up
apiVersion: v1
kind: PersistentVolume
metadata:
  name: dev-workspace-pv
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete  # Auto-cleanup on PVC deletion
  storageClassName: dev-standard
  gcePersistentDisk:
    pdName: dev-workspace-disk
    fsType: ext4

Changing Reclaim Policies

Reclaim policy can be changed on existing PVs: `kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'. This is useful for upgrading development volumes to production protection without recreating storage.

Reclaiming Released volumes:

When using Retain policy, Released PVs require manual steps to make them Available again:

Delete the claimRef: Remove the reference to the old PVC

kubectl patch pv <pv-name> -p '{"spec":{"claimRef": null}}'

Clean up data if needed: Depending on use case, you may need to wipe, backup, or preserve data
Update labels/selectors: Adjust metadata if the PV will serve a different purpose
Verify Available state: kubectl get pv <pv-name> should show status Available

This manual process protects against accidental data loss but requires operational runbooks for volume recycling.

Volume Modes and Block Storage

Kubernetes supports two volume modes that determine how storage is presented to pods. The choice between filesystem and block mode has significant implications for performance, flexibility, and compatibility.

Volume modes explained:

Filesystem Mode (Default)

•Storage formatted with filesystem (ext4, xfs)
•Mounted as directory in container
•Standard file I/O operations
•Suitable for most applications
•Easier to debug (can browse files)
•Some overhead from filesystem layer

Block Mode

•Raw block device (no filesystem)
•Presented as device file (e.g., /dev/xvda)
•Application manages storage directly
•Required for some databases (Oracle, SAP)
•Lower latency, higher throughput
•No filesystem overhead

block-volume-example.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
# Block mode PV - raw device access
apiVersion: v1
kind: PersistentVolume
metadata:
  name: high-performance-block-pv
spec:
  capacity:
    storage: 200Gi
  volumeMode: Block  # Explicitly request block mode
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: high-iops
  local:
    path: /dev/nvme0n1  # Raw NVMe device
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - database-node-01
 
---
# PVC requesting block mode
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-block-storage
spec:
  volumeMode: Block  # Must match PV
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
  storageClassName: high-iops
 
---
# Pod using block volume
apiVersion: v1
kind: Pod
metadata:
  name: database-with-block-storage
spec:
  containers:
    - name: database
      image: oracle-database:19c
      # Block volumes use volumeDevices, not volumeMounts
      volumeDevices:
        - name: data
          devicePath: /dev/xvda  # Device path inside container
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: database-block-storage

Block Mode Requirements

Block volume mode requires: (1) Storage backend support for raw blocks, (2) PV and PVC must both specify volumeMode: Block, (3) Pod must use volumeDevices instead of volumeMounts, (4) Application must handle raw device I/O. Not all CSI drivers support block mode.

When to use block mode:

Performance-critical databases: Oracle, SAP HANA, and some PostgreSQL configurations prefer raw device access
Applications with custom storage engines: Systems that implement their own filesystem or storage layout
Virtualization: When running VMs inside containers that need direct device access
High-frequency trading: Where microseconds matter and filesystem overhead is unacceptable

When to use filesystem mode:

General applications: Web servers, application containers, most microservices
Standard databases: MySQL, MongoDB, and most PostgreSQL deployments
Shared storage: RWX volumes typically require filesystem mode
Debugging and operations: Filesystem volumes are easier to inspect and troubleshoot

Static vs Dynamic Provisioning

Persistent Volumes can be created through two mechanisms: static provisioning (administrator creates PVs manually) or dynamic provisioning (PVs created automatically in response to PVCs). Understanding both approaches and their trade-offs is essential for production operations.

Static provisioning:

With static provisioning, cluster administrators pre-create Persistent Volumes before workloads request them. PVCs then bind to existing PVs that match their requirements.

Static Provisioning Characteristics

•Manual creation: Administrators create PVs using kubectl or infrastructure automation
•Pre-provisioned capacity: Storage capacity planned and allocated ahead of workloads
•Full control: Exact storage configuration, naming, and placement determined by operators
•Backend flexibility: Can use any storage type, including exotic or legacy systems
•Potential waste: Unused PVs consume resources; over-provisioning is common for safety

Dynamic provisioning:

With dynamic provisioning, PVs are created automatically when a PVC is submitted. A Storage Class defines the provisioner and parameters for volume creation.

Dynamic Provisioning Characteristics

•Automatic creation: PVs created on-demand when PVCs appear
•Right-sized volumes: Each PV matches exactly what the PVC requests
•Self-service: Developers request storage without admin intervention
•Cloud-native: Tight integration with cloud provider storage APIs
•Requires provisioner: Only works with storage backends that have CSI drivers or in-tree provisioners

Static vs Dynamic Provisioning Comparison
Aspect	Static Provisioning	Dynamic Provisioning
Administrator involvement	Required for every volume	Only StorageClass setup
Lead time	PVs must exist before workloads	PVs created with workload deployment
Capacity planning	Pre-plan total storage needs	Scale as demands grow
Resource efficiency	Potential over-provisioning	Exact-fit allocation
Legacy storage support	Full support any backend	Requires CSI driver
Naming control	Explicit, predictable names	Auto-generated names
Recovery/migration	Easier to manage known PVs	More automation available

Hybrid Approaches

Production environments often use both: dynamic provisioning for general workloads and self-service, static provisioning for databases and critical systems requiring specific storage placement. Storage Classes can be configured to prefer static PVs when available, falling back to dynamic provisioning.

Production Best Practices

Operating Persistent Volumes in production requires attention to data safety, operational procedures, and monitoring. These best practices represent lessons learned from production incidents across the industry.

Data Safety and Retention

•Default to Retain policy — For any production workload, use persistentVolumeReclaimPolicy: Retain. Delete policy should only be used with explicit intent.
•Implement backup strategies — PVs are not backups. Use volume snapshots, application-level backups, or continuous replication for data protection.
•Use finalizers wisely — Kubernetes finalizers can prevent accidental PV/PVC deletion. Add custom finalizers for critical volumes.
•Label everything — Rich labels (app, environment, team, data-classification) enable automated policies and easier troubleshooting.
•Document recovery procedures — Runbooks for recovering data from Retained volumes, restoring from snapshots, and handling Failed volumes.

Operational Excellence

•Monitor PV capacity — Alert before volumes fill up. 80% utilization should trigger review; 90% is an emergency.
•Track binding delays — PVCs stuck in Pending indicate provisioning problems or capacity exhaustion. Alert on Pending duration.
•Audit access modes — Regular review of RWX usage—shared writable storage often indicates design issues.
•Version control PV definitions — Store static PV manifests in git. Changes should go through code review.
•Capacity planning — Analyze storage growth trends. Provision StorageClass quotas to prevent runaway dynamic provisioning.

Critical Warning: PVC Deletion

Deleting a PVC with Delete reclaim policy immediately triggers backing storage destruction. There is no confirmation, no grace period, no undo. Implement admission webhooks or OPA policies to require additional approval for PVC deletion in production namespaces.

Monitoring and alerting checklist:

# Essential PV/PVC metrics to monitor
metrics:
  - name: kube_persistentvolume_status_phase
    alert_on: [Failed, Released]  # Immediate attention for Failed
    description: Volume phase status
    
  - name: kube_persistentvolumeclaim_status_phase
    alert_on: [Pending > 5m, Lost]
    description: Claim status
    
  - name: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
    alert_on: "> 0.8 for 15m"
    description: Volume utilization percentage
    
  - name: kubelet_volume_stats_inodes_used / kubelet_volume_stats_inodes
    alert_on: "> 0.9"
    description: Inode exhaustion (filesystem only)

Summary: Persistent Volumes Mastery

Persistent Volumes represent Kubernetes' answer to the fundamental challenge of stateful workloads in a containerized, distributed environment. Mastering PVs requires understanding both the API abstractions and the operational realities of production storage.

Key Takeaways

•PV/PVC separation — The two-layer abstraction decouples storage infrastructure from consumption, enabling role separation and portability.
•Binding is exclusive and intentional — Understanding the binding algorithm prevents surprises when PVs don't bind as expected.
•Access modes define sharing — RWO, ROX, RWX, and RWOP control how storage is accessed across the cluster. Choose based on workload needs and backend capabilities.
•Reclaim policies are critical — Retain for production, Delete for ephemeral. There's no undo for Delete policy destruction.
•Volume modes matter for performance — Block mode offers lower latency but requires application support. Filesystem mode is suitable for most workloads.
•Static and dynamic provisioning serve different needs — Self-service dynamic provisioning for developers; static provisioning for critical, controlled deployments.
•Operational vigilance is required — Monitor capacity, track binding issues, implement backup strategies, and document recovery procedures.

What's next:

With a solid understanding of Persistent Volumes, we'll explore Storage Classes—the mechanism that automates PV provisioning, defines storage tiers, and enables self-service storage for Kubernetes users. Storage Classes transform PVs from static infrastructure into dynamic, on-demand resources.

Page Complete

You now understand Persistent Volumes comprehensively—from container storage challenges through PV/PVC architecture, access modes, reclaim policies, volume modes, and production operations. This foundation prepares you for advanced storage patterns with Storage Classes.