Online Backup (Hot Backup) - Backup Implementation Strategies | OneNoughtOne

Database Management SystemsBackup Implementation

Backup Implementation Strategies

LevelIntermediate

Duration60 mins

TopicBackup Implementation

1 / 5

Online Backup (Hot Backup)

The Imperative of Zero-Downtime Data Protection

In the modern digital economy, where databases power everything from global financial systems to real-time e-commerce platforms, the notion of shutting down a database for backup has become increasingly untenable. Online backup, also known as hot backup, represents the gold standard in enterprise data protection—enabling organizations to capture consistent, recoverable database snapshots while the system continues to serve thousands or millions of concurrent users without interruption.

This capability isn't merely a convenience; it's a fundamental business requirement. Consider a global banking system processing 10,000 transactions per second, an e-commerce platform during peak holiday shopping, or a healthcare system where milliseconds can impact patient care. For these systems, even a 30-minute maintenance window for backup represents millions of dollars in lost revenue, severe customer impact, or potentially life-threatening service interruptions.

What You Will Learn

By the end of this page, you will understand the fundamental principles of online backup, master the techniques used to capture consistent snapshots from active databases, recognize the challenges and trade-offs involved, and be able to design online backup strategies for production database systems.

Foundations of Online Backup

Defining Online Backup

An online backup (or hot backup) is a backup operation performed while the database remains fully operational, accepting reads and writes from connected applications. Unlike cold backups that require database shutdown, online backups leverage sophisticated mechanisms to capture a consistent point-in-time snapshot of data without blocking production workloads.

The Fundamental Challenge

To appreciate what online backup accomplishes, consider what happens when you simply copy database files from a running database:

Inconsistent File States: Database files are constantly being modified. Copying file A, then file B moments later means they represent different points in time.
Partial Write Corruption: A page might be half-written during the copy, resulting in a corrupted backup.
Transaction Inconsistency: Active transactions might be partially reflected—some changes captured, others missed.
Referential Integrity Violations: Foreign key relationships might be broken if parent records are captured without their children (or vice versa).

Online backup mechanisms must solve all these problems while minimizing impact on production performance—a technically demanding proposition.

Online Backup vs. Traditional File Copy
Aspect	Simple File Copy (Running DB)	Online Backup
Consistency	Inconsistent—captures mixed states	Point-in-time consistent
Transaction Integrity	Broken—partial transactions captured	Preserved—complete transactions only
Corruption Risk	High—partial page writes	None—complete pages only
Recovery Guarantee	Uncertain—may be unusable	Guaranteed—tested recovery path
Application Availability	Unchanged but backup is worthless	Unchanged and backup is valid

The File Copy Trap

Many junior DBAs have learned the hard way that copying database files from a running system doesn't produce a valid backup. The backup might appear complete, but when recovery is attempted during an actual disaster, the corrupted or inconsistent state renders it unusable. This is why proper online backup mechanisms are essential—not optional.

Core Mechanisms Enabling Online Backup

Online backup relies on several foundational database mechanisms working in concert. Understanding these mechanisms is crucial for both implementing and troubleshooting online backup systems.

Write-Ahead Logging (WAL)

The cornerstone of online backup is the Write-Ahead Log (also called the redo log or transaction log). Before any data modification is written to the actual data files, it is first recorded in the WAL. This principle ensures:

Recoverability: Even if data files are inconsistent, the WAL contains all information needed to reconstruct a consistent state.
Backup Consistency: By capturing WAL records alongside data files, we can replay transactions to reach a consistent point.
Non-blocking Operations: The backup process can read data files even as modifications occur—the WAL captures concurrent changes.

Checkpoint Mechanism

Checkpoints periodically flush modified pages from memory to disk and record a consistency point in the WAL. During online backup:

A forced checkpoint ensures recent changes are written to data files.
The checkpoint's Log Sequence Number (LSN) establishes a known-good starting point.
Recovery can begin from this checkpoint, replaying only subsequent WAL records.

Converting Mermaid diagram...

Buffer Pool Coordination

The buffer pool (or buffer cache) holds frequently accessed data pages in memory. During online backup:

Modified (dirty) pages in the buffer pool must be handled carefully.
The backup might read an older version from disk while a newer version exists only in memory.
The WAL contains all modifications, ensuring nothing is lost.

Page-Level Consistency

Modern databases ensure page-level atomicity—a page is either completely written or not written at all (using techniques like double-write buffers or torn page detection). Online backups leverage this:

Each copied page is guaranteed to be internally consistent.
The page's LSN indicates the latest modification applied to it.
During recovery, pages can be updated to their correct state using WAL replay.

Key Mechanisms Summary

•Write-Ahead Logging (WAL) — Records all changes before they're applied to data files, enabling replay during recovery
•Checkpointing — Creates known-good consistency points and flushes dirty pages to disk
•Buffer Pool Management — Coordinates in-memory modifications with on-disk state
•Page-Level Atomicity — Ensures no partial page writes corrupt the backup
•Log Sequence Numbers (LSN) — Provides ordering and tracking of all database modifications

Online Backup Strategies

Different online backup strategies offer varying trade-offs between complexity, performance impact, and recovery flexibility. Understanding each approach enables selection of the optimal strategy for specific requirements.

Physical Online Backup

Physical backups copy the raw database files (data files, control files, WAL segments) at the file system level. This is the most common enterprise approach:

Advantages:

Fast backup and restore speeds (direct file copy)
Complete capture of all database objects
Platform and version independent recovery
Minimal database overhead during backup

Considerations:

Backup size equals database size (before compression)
Requires file system access
Recovery is all-or-nothing (typically)

Logical Online Backup

Logical backups export database objects as SQL statements or structured data formats (like pg_dump or mysqldump):

Advantages:

Highly portable across versions and platforms
Selective backup of specific tables or schemas
Human-readable format (SQL)
Smaller size for sparse tables

Considerations:

Slower than physical backup
May miss some physical storage optimizations
Consumes more database resources during export
Restore requires SQL execution (slower)

Physical Backup Characteristics

•Copies raw data files and WAL segments
•Typically block-level or file-level copy
•Fast backup: 100-500 MB/sec typical
•Fast restore: Same speed as backup
•Full database recovery (atomic)
•Requires consistent storage layer

Logical Backup Characteristics

•Exports SQL or structured data format
•Object-level granularity available
•Slower: 10-50 MB/sec typical
•Very slow restore: SQL execution
•Selective object recovery possible
•Cross-platform and version portable

Filesystem Snapshot Backup

Leverage storage-level snapshot capabilities (LVM, ZFS, SAN snapshots, cloud volume snapshots) for near-instantaneous backups:

How it works:

Database enters backup mode (holds checkpoint, tracks WAL)
Storage system creates point-in-time snapshot (instant)
Database exits backup mode
Snapshot is copied to backup storage at leisure

Advantages:

Extremely fast backup window (seconds to minutes)
Minimal database I/O impact during snapshot
Often space-efficient (copy-on-write)

Considerations:

Requires snapshot-capable storage
Recovery requires snapshot remount/restore
Database must support 'backup mode' properly

Continuous/Streaming Backup

Modern databases support continuous backup where WAL segments are shipped to backup storage in real-time:

How it works:

Initial full backup establishes baseline
WAL segments continuously archived as generated
Recovery applies full backup + all subsequent WAL
Point-in-time recovery anywhere in WAL history

Advantages:

Minimal data loss (seconds to minutes of WAL)
Point-in-time recovery capability
Lower backup traffic (incremental WAL)

Considerations:

Requires WAL archiving infrastructure
Storage grows with WAL generation rate
Recovery time increases with WAL volume

Database-Specific Implementations

Each major database system implements online backup with its own tooling, terminology, and specific behaviors. Let's examine the approaches used by leading databases.

PostgreSQL: pg_basebackup and Continuous Archiving

PostgreSQL's online backup architecture is elegant and well-integrated:

pg_basebackup: Creates a physical backup via streaming replication protocol
Continuous WAL archiving via archive_command or streaming
pg_start_backup() / pg_stop_backup() for custom backup scripts
Built-in support for backup compression and manifest generation

postgresql_online_backup.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#!/bin/bash
# PostgreSQL Online Backup with pg_basebackup
 
# Environment setup
BACKUP_DIR="/backup/postgresql/$(date +%Y%m%d_%H%M%S)"
PG_HOST="localhost"
PG_PORT="5432"
PG_USER="backup_user"
 
# Create backup directory
mkdir -p "$BACKUP_DIR"
 
# Perform streaming base backup with compression
# -D: Target directory
# -F: Format (plain directory or tar)
# -X: Include WAL (stream method)
# -z: Compress output
# -P: Show progress
# --checkpoint: Checkpoint mode (fast vs spread)
 
pg_basebackup \
    -h "$PG_HOST" \
    -p "$PG_PORT" \
    -U "$PG_USER" \
    -D "$BACKUP_DIR" \
    -F tar \
    -X stream \
    -z \
    -P \
    --checkpoint=fast \
    --label="Daily Backup $(date +%Y-%m-%d)" \
    --manifest-checksums=SHA256
 
# Check backup success
if [ $? -eq 0 ]; then
    echo "Backup completed successfully: $BACKUP_DIR"
    
    # Create backup manifest
    echo "Backup completed: $(date)" > "$BACKUP_DIR/backup_info.txt"
    echo "PostgreSQL version: $(psql --version)" >> "$BACKUP_DIR/backup_info.txt"
    echo "Backup size: $(du -sh $BACKUP_DIR)" >> "$BACKUP_DIR/backup_info.txt"
else
    echo "Backup FAILED!" >&2
    exit 1
fi

MySQL: Enterprise Backup and Percona XtraBackup

MySQL offers multiple online backup approaches:

MySQL Enterprise Backup (commercial): Hot backup with minimal locking
Percona XtraBackup (open source): Industry standard for InnoDB
mysqldump --single-transaction: Logical backup with consistent snapshot

mysql_online_backup.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/bin/bash
# MySQL Online Backup with Percona XtraBackup
 
BACKUP_BASE="/backup/mysql"
FULL_BACKUP_DIR="$BACKUP_BASE/full_$(date +%Y%m%d)"
MYSQL_USER="backup_user"
MYSQL_PASS="secure_password"
 
# Full backup with Percona XtraBackup
# --backup: Perform backup
# --target-dir: Backup destination
# --compress: Enable compression
# --compress-threads: Parallel compression
# --parallel: Parallel file copy threads
 
xtrabackup \
    --backup \
    --user="$MYSQL_USER" \
    --password="$MYSQL_PASS" \
    --target-dir="$FULL_BACKUP_DIR" \
    --compress \
    --compress-threads=4 \
    --parallel=4
 
# Prepare the backup (apply logs for consistency)
# This step is REQUIRED before restore
xtrabackup \
    --prepare \
    --target-dir="$FULL_BACKUP_DIR"
 
echo "MySQL backup complete: $FULL_BACKUP_DIR"
 
# For incremental backup (runs after full backup)
# INCR_BACKUP_DIR="$BACKUP_BASE/incr_$(date +%Y%m%d_%H%M%S)"
# xtrabackup --backup --target-dir="$INCR_BACKUP_DIR" \
#     --incremental-basedir="$FULL_BACKUP_DIR" \
#     --user="$MYSQL_USER" --password="$MYSQL_PASS"

Oracle: RMAN (Recovery Manager)

Oracle's RMAN is the most sophisticated enterprise backup tool:

Integrated with Oracle database for optimal performance
Block-level incremental forever capability
Automatic backup optimization and validation
Comprehensive catalog and reporting

oracle_rman_backup.sql
SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
-- Oracle RMAN Online Backup Script
 
-- Connect to RMAN
-- $ rman target / catalog rman_user/password@catalog_db
 
-- Configure backup settings
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE DEVICE TYPE DISK PARALLELISM 4;
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM';
CONFIGURE ENCRYPTION FOR DATABASE ON;
 
-- Full database backup with compression and parallelism
RUN {
    ALLOCATE CHANNEL c1 DEVICE TYPE DISK;
    ALLOCATE CHANNEL c2 DEVICE TYPE DISK;
    ALLOCATE CHANNEL c3 DEVICE TYPE DISK;
    ALLOCATE CHANNEL c4 DEVICE TYPE DISK;
    
    -- Backup the database
    BACKUP AS COMPRESSED BACKUPSET
        DATABASE
        PLUS ARCHIVELOG
        TAG 'DAILY_FULL_BACKUP';
    
    -- Backup control file and SPFILE
    BACKUP CURRENT CONTROLFILE
        SPFILE
        TAG 'CONTROLFILE_BACKUP';
    
    -- Validate backup integrity
    RESTORE DATABASE VALIDATE;
    
    RELEASE CHANNEL c1;
    RELEASE CHANNEL c2;
    RELEASE CHANNEL c3;
    RELEASE CHANNEL c4;
}
 
-- Crosscheck and delete obsolete backups
CROSSCHECK BACKUP;
DELETE NOPROMPT OBSOLETE;
 
-- Report backup status
LIST BACKUP SUMMARY;

SQL Server: Native Backup with Compression

Microsoft SQL Server provides built-in online backup capabilities:

BACKUP DATABASE ... WITH COPY_ONLY: Non-disruptive backup
Backup compression significantly reduces size and I/O
Striped backups across multiple devices for parallelism
Integrated with Windows VSS for storage snapshots

sqlserver_backup.sql
SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
-- SQL Server Online Full Backup
 
-- Full database backup with compression
BACKUP DATABASE [Production]
TO DISK = 'D:\Backups\Production_Full.bak'
WITH 
    COMPRESSION,                          -- Native compression
    CHECKSUM,                             -- Verify backup integrity
    STATS = 10,                           -- Progress reporting every 10%
    NAME = 'Production-Full Database Backup',
    DESCRIPTION = 'Daily full backup with compression',
    COPY_ONLY;                            -- Don't affect differential baseline
 
-- Backup to multiple files (striped backup for performance)
BACKUP DATABASE [Production]
TO DISK = 'D:\Backups\Production_Stripe1.bak',
   DISK = 'D:\Backups\Production_Stripe2.bak',
   DISK = 'D:\Backups\Production_Stripe3.bak',
   DISK = 'D:\Backups\Production_Stripe4.bak'
WITH 
    COMPRESSION,
    CHECKSUM,
    MAXTRANSFERSIZE = 4194304,            -- 4MB per transfer
    BUFFERCOUNT = 20,                     -- Memory buffers
    STATS = 5;
 
-- Verify backup is readable
RESTORE VERIFYONLY 
FROM DISK = 'D:\Backups\Production_Full.bak'
WITH CHECKSUM;
 
-- Query backup history
SELECT 
    database_name,
    backup_start_date,
    backup_finish_date,
    DATEDIFF(MINUTE, backup_start_date, backup_finish_date) AS duration_minutes,
    backup_size / 1024 / 1024 AS backup_size_mb,
    compressed_backup_size / 1024 / 1024 AS compressed_size_mb,
    CAST(100 - (compressed_backup_size * 100.0 / backup_size) AS DECIMAL(5,2)) AS compression_ratio_pct
FROM msdb.dbo.backupset
WHERE database_name = 'Production'
ORDER BY backup_start_date DESC;

Performance Considerations and Impact

While online backups minimize disruption, they are not zero-impact operations. Understanding and managing performance impact is essential for production deployments.

Resource Consumption During Backup

Online backups consume several system resources:

Disk I/O: Backup reads compete with production I/O. On spinning disks, this can significantly impact query latency.
CPU: Compression and checksum computation consume processor cycles.
Network: Streaming backups to remote storage consume bandwidth.
Memory: Backup processes may affect buffer pool efficiency.

Measuring Backup Impact

Before deploying online backup, measure baseline performance and impact:

Metrics to Monitor During Online Backup
Metric Category	Specific Metrics	Acceptable Increase
Query Performance	Average query latency, P95/P99 latency	10-30% degradation
Throughput	Transactions per second, queries per second	5-20% reduction
Disk I/O	IOPS, throughput MB/s, queue depth	50-100% increase acceptable
CPU	User %, system %, iowait %	20-40% increase if capacity exists
Replication Lag	Seconds behind primary (if replicated)	Minimal increase expected

Mitigation Strategies

1. Schedule During Low-Traffic Periods

Even for 24/7 systems, traffic patterns vary. Schedule full backups during:

Late night/early morning for consumer applications
Weekends for business applications
Post-batch processing windows

2. Throttle Backup I/O

Many backup tools support I/O limiting:

throttled_backup_examples.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# PostgreSQL: pg_basebackup with rate limiting
pg_basebackup -D /backup -r 100M  # Limit to 100 MB/s
 
# XtraBackup: I/O rate limiting
xtrabackup --backup --throttle=40 --target-dir=/backup
# Limits to 40 I/O operations per second
 
# Linux: Use ionice for I/O scheduling priority
ionice -c 3 pg_basebackup -D /backup  # Idle I/O class (lowest priority)
 
# Linux: Use cpulimit to cap CPU usage
cpulimit -l 50 -- pg_basebackup -D /backup  # Limit to 50% CPU
 
# Combination approach
ionice -c 2 -n 7 nice -n 19 pg_basebackup -D /backup -r 50M

3. Use Replica for Backup Source

A powerful pattern is to perform backups from a replica rather than the primary:

Advantages:

Zero impact on primary database performance
Replica can be paused during backup for consistency
Backup I/O isolated from production writes

Considerations:

Replica must be current (monitor replication lag)
Some systems require specific replica configuration
Network transfer if replica is remote

Converting Mermaid diagram...

4. Leverage Storage-Level Features

Modern storage systems offer backup-friendly features:

Snapshot-based backup: Near-instant, minimal database impact
Clone volumes: Create writable copy for backup processing
QoS policies: Prioritize production I/O over backup I/O
Dedicated backup networks: Separate backup traffic from production

5. Optimize Backup Duration

Shorter backups mean shorter impact windows:

Use parallel backup (multiple streams/channels)
Enable compression (reduces data volume to transfer)
Incremental/differential backups (smaller data sets)
Use SSDs or dedicated backup storage

Performance Impact Baseline

Always establish a performance baseline before deploying online backup in production. Run test backups during various load conditions and measure impact on key metrics. This data enables informed decisions about backup scheduling, throttling, and architecture. A well-tuned online backup should cause <15% degradation in production performance during the backup window.

Consistency Guarantees and Recovery Path

The ultimate test of any backup is successful recovery. Online backups must provide clear, guaranteed recovery paths with well-defined consistency semantics.

Crash Consistency vs. Application Consistency

Online backups typically provide crash consistency—the backup represents a state equivalent to abruptly powering off the server at a specific moment:

All committed transactions are preserved
In-flight transactions are rolled back during recovery
No partial transactions exist
Database structural integrity is maintained

Application consistency goes further, ensuring:

Application-specific invariants are met
External system references are valid
Business logic constraints are satisfied

Most online backups provide crash consistency. Application consistency requires additional coordination (quiescing applications, coordinating multi-system backups).

The Recovery Process

Recovering from an online backup involves several steps:

1. Restore Data Files

Copy backup files to target location
Decompress if backup was compressed
Verify file checksums if available

2. Apply Redo Logs (WAL)

Replay all transactions from backup checkpoint forward
Bring database to consistent crash-recovery state
This step is often called 'recovery' or 'prepare'

3. Open Database

Roll back any incomplete transactions
Verify database structural integrity
Database becomes available for connections

4. Validate Recovery

Run consistency checks
Verify application connectivity
Validate sample data integrity

recovery_procedures.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# PostgreSQL: Restore from pg_basebackup
 
# 1. Stop the running PostgreSQL (if any)
sudo systemctl stop postgresql
 
# 2. Clear the data directory (CAUTION: destroys current data)
sudo rm -rf /var/lib/postgresql/14/main/*
 
# 3. Restore the backup
sudo tar -xzf /backup/base.tar.gz -C /var/lib/postgresql/14/main/
sudo tar -xzf /backup/pg_wal.tar.gz -C /var/lib/postgresql/14/main/pg_wal/
 
# 4. Set correct ownership
sudo chown -R postgres:postgres /var/lib/postgresql/14/main
 
# 5. Create recovery signal file (PostgreSQL 12+)
sudo touch /var/lib/postgresql/14/main/recovery.signal
 
# 6. Start PostgreSQL - recovery happens automatically
sudo systemctl start postgresql
 
# 7. Monitor recovery progress
sudo -u postgres psql -c "SELECT pg_is_in_recovery();"
# Returns 'f' (false) when recovery is complete
 
# 8. Validate recovery
sudo -u postgres pg_isready
sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_user_tables;"

Untested Backups Are Not Backups

Many organizations have discovered during actual disasters that their backups were unusable—corrupted, incomplete, or with undocumented dependencies. Regular recovery testing is essential. The only proven backup is one that has been successfully restored. Test your recovery procedures at least monthly, with documented runbooks and timing metrics.

Best Practices and Implementation Checklist

Implementing online backup successfully requires attention to numerous details. The following best practices and checklist ensure robust, production-ready backup systems.

Online Backup Best Practices

•Use database-native tools — pg_basebackup, xtrabackup, RMAN are optimized and well-tested. Avoid generic file copy unless using storage snapshots.
•Always include transaction logs — Data files without WAL/redo logs are often unrecoverable. Configure continuous WAL archiving.
•Compress and encrypt — Reduce storage costs with compression; protect data with encryption at rest and in transit.
•Validate backup integrity — Use checksums, verify backup readability, test restore procedures regularly.
•Monitor backup jobs — Alert on failures, track backup duration trends, verify backup frequency meets RPO.
•Document recovery procedures — Maintain step-by-step runbooks; every team member should be able to perform recovery.
•Store backups offsite — Protect against site-wide disasters with geographic separation of backup storage.
•Retain backup history — Maintain multiple recovery points; don't rely on a single backup.

Online Backup Implementation Checklist
Category	Item	Status
Infrastructure	Dedicated backup storage sized appropriately	☐
Infrastructure	Network bandwidth for backup traffic calculated	☐
Infrastructure	Backup retention storage estimated (30-90 days)	☐
Configuration	Backup user/role with appropriate permissions	☐
Configuration	WAL archiving configured and tested	☐
Configuration	Backup compression enabled	☐
Configuration	Backup encryption configured	☐
Scheduling	Backup schedule aligned with traffic patterns	☐
Scheduling	Backup window sized appropriately for data volume	☐
Monitoring	Backup success/failure alerting configured	☐
Monitoring	Backup duration monitoring in place	☐
Monitoring	Backup size trending tracked	☐
Validation	Automated backup integrity checks enabled	☐
Validation	Regular restore tests scheduled (monthly)	☐
Documentation	Recovery runbook documented	☐
Documentation	Recovery tested by multiple team members	☐

Page Complete

You now understand the principles, mechanisms, and implementation strategies for online (hot) backup. This knowledge enables you to design and deploy backup systems that protect production data without impacting availability. Next, we'll explore offline (cold) backup strategies for scenarios where system shutdown is acceptable or required.

1 / 5

Loading learning content...

Database Management SystemsBackup Implementation

Backup Implementation Strategies

LevelIntermediate

Duration60 mins

TopicBackup Implementation

1 / 5

Online Backup (Hot Backup)

The Imperative of Zero-Downtime Data Protection

What You Will Learn

Foundations of Online Backup

Defining Online Backup

The Fundamental Challenge

To appreciate what online backup accomplishes, consider what happens when you simply copy database files from a running database:

Inconsistent File States: Database files are constantly being modified. Copying file A, then file B moments later means they represent different points in time.
Partial Write Corruption: A page might be half-written during the copy, resulting in a corrupted backup.
Transaction Inconsistency: Active transactions might be partially reflected—some changes captured, others missed.
Referential Integrity Violations: Foreign key relationships might be broken if parent records are captured without their children (or vice versa).

Online backup mechanisms must solve all these problems while minimizing impact on production performance—a technically demanding proposition.

Online Backup vs. Traditional File Copy
Aspect	Simple File Copy (Running DB)	Online Backup
Consistency	Inconsistent—captures mixed states	Point-in-time consistent
Transaction Integrity	Broken—partial transactions captured	Preserved—complete transactions only
Corruption Risk	High—partial page writes	None—complete pages only
Recovery Guarantee	Uncertain—may be unusable	Guaranteed—tested recovery path
Application Availability	Unchanged but backup is worthless	Unchanged and backup is valid

The File Copy Trap

Core Mechanisms Enabling Online Backup

Online backup relies on several foundational database mechanisms working in concert. Understanding these mechanisms is crucial for both implementing and troubleshooting online backup systems.

Write-Ahead Logging (WAL)

Recoverability: Even if data files are inconsistent, the WAL contains all information needed to reconstruct a consistent state.
Backup Consistency: By capturing WAL records alongside data files, we can replay transactions to reach a consistent point.
Non-blocking Operations: The backup process can read data files even as modifications occur—the WAL captures concurrent changes.

Checkpoint Mechanism

Checkpoints periodically flush modified pages from memory to disk and record a consistency point in the WAL. During online backup:

A forced checkpoint ensures recent changes are written to data files.
The checkpoint's Log Sequence Number (LSN) establishes a known-good starting point.
Recovery can begin from this checkpoint, replaying only subsequent WAL records.

Converting Mermaid diagram...

Buffer Pool Coordination

The buffer pool (or buffer cache) holds frequently accessed data pages in memory. During online backup:

Modified (dirty) pages in the buffer pool must be handled carefully.
The backup might read an older version from disk while a newer version exists only in memory.
The WAL contains all modifications, ensuring nothing is lost.

Page-Level Consistency

Each copied page is guaranteed to be internally consistent.
The page's LSN indicates the latest modification applied to it.
During recovery, pages can be updated to their correct state using WAL replay.

Key Mechanisms Summary

•Write-Ahead Logging (WAL) — Records all changes before they're applied to data files, enabling replay during recovery
•Checkpointing — Creates known-good consistency points and flushes dirty pages to disk
•Buffer Pool Management — Coordinates in-memory modifications with on-disk state
•Page-Level Atomicity — Ensures no partial page writes corrupt the backup
•Log Sequence Numbers (LSN) — Provides ordering and tracking of all database modifications

Online Backup Strategies

Physical Online Backup

Physical backups copy the raw database files (data files, control files, WAL segments) at the file system level. This is the most common enterprise approach:

Advantages:

Fast backup and restore speeds (direct file copy)
Complete capture of all database objects
Platform and version independent recovery
Minimal database overhead during backup

Considerations:

Backup size equals database size (before compression)
Requires file system access
Recovery is all-or-nothing (typically)

Logical Online Backup

Logical backups export database objects as SQL statements or structured data formats (like pg_dump or mysqldump):

Advantages:

Highly portable across versions and platforms
Selective backup of specific tables or schemas
Human-readable format (SQL)
Smaller size for sparse tables

Considerations:

Slower than physical backup
May miss some physical storage optimizations
Consumes more database resources during export
Restore requires SQL execution (slower)

Physical Backup Characteristics

•Copies raw data files and WAL segments
•Typically block-level or file-level copy
•Fast backup: 100-500 MB/sec typical
•Fast restore: Same speed as backup
•Full database recovery (atomic)
•Requires consistent storage layer

Logical Backup Characteristics

•Exports SQL or structured data format
•Object-level granularity available
•Slower: 10-50 MB/sec typical
•Very slow restore: SQL execution
•Selective object recovery possible
•Cross-platform and version portable

Filesystem Snapshot Backup

Leverage storage-level snapshot capabilities (LVM, ZFS, SAN snapshots, cloud volume snapshots) for near-instantaneous backups:

How it works:

Database enters backup mode (holds checkpoint, tracks WAL)
Storage system creates point-in-time snapshot (instant)
Database exits backup mode
Snapshot is copied to backup storage at leisure

Advantages:

Extremely fast backup window (seconds to minutes)
Minimal database I/O impact during snapshot
Often space-efficient (copy-on-write)

Considerations:

Requires snapshot-capable storage
Recovery requires snapshot remount/restore
Database must support 'backup mode' properly

Continuous/Streaming Backup

Modern databases support continuous backup where WAL segments are shipped to backup storage in real-time:

How it works:

Initial full backup establishes baseline
WAL segments continuously archived as generated
Recovery applies full backup + all subsequent WAL
Point-in-time recovery anywhere in WAL history

Advantages:

Minimal data loss (seconds to minutes of WAL)
Point-in-time recovery capability
Lower backup traffic (incremental WAL)

Considerations:

Requires WAL archiving infrastructure
Storage grows with WAL generation rate
Recovery time increases with WAL volume

Database-Specific Implementations

Each major database system implements online backup with its own tooling, terminology, and specific behaviors. Let's examine the approaches used by leading databases.

PostgreSQL: pg_basebackup and Continuous Archiving

PostgreSQL's online backup architecture is elegant and well-integrated:

pg_basebackup: Creates a physical backup via streaming replication protocol
Continuous WAL archiving via archive_command or streaming
pg_start_backup() / pg_stop_backup() for custom backup scripts
Built-in support for backup compression and manifest generation

postgresql_online_backup.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#!/bin/bash
# PostgreSQL Online Backup with pg_basebackup
 
# Environment setup
BACKUP_DIR="/backup/postgresql/$(date +%Y%m%d_%H%M%S)"
PG_HOST="localhost"
PG_PORT="5432"
PG_USER="backup_user"
 
# Create backup directory
mkdir -p "$BACKUP_DIR"
 
# Perform streaming base backup with compression
# -D: Target directory
# -F: Format (plain directory or tar)
# -X: Include WAL (stream method)
# -z: Compress output
# -P: Show progress
# --checkpoint: Checkpoint mode (fast vs spread)
 
pg_basebackup \
    -h "$PG_HOST" \
    -p "$PG_PORT" \
    -U "$PG_USER" \
    -D "$BACKUP_DIR" \
    -F tar \
    -X stream \
    -z \
    -P \
    --checkpoint=fast \
    --label="Daily Backup $(date +%Y-%m-%d)" \
    --manifest-checksums=SHA256
 
# Check backup success
if [ $? -eq 0 ]; then
    echo "Backup completed successfully: $BACKUP_DIR"
    
    # Create backup manifest
    echo "Backup completed: $(date)" > "$BACKUP_DIR/backup_info.txt"
    echo "PostgreSQL version: $(psql --version)" >> "$BACKUP_DIR/backup_info.txt"
    echo "Backup size: $(du -sh $BACKUP_DIR)" >> "$BACKUP_DIR/backup_info.txt"
else
    echo "Backup FAILED!" >&2
    exit 1
fi

MySQL: Enterprise Backup and Percona XtraBackup

MySQL offers multiple online backup approaches:

MySQL Enterprise Backup (commercial): Hot backup with minimal locking
Percona XtraBackup (open source): Industry standard for InnoDB
mysqldump --single-transaction: Logical backup with consistent snapshot

mysql_online_backup.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/bin/bash
# MySQL Online Backup with Percona XtraBackup
 
BACKUP_BASE="/backup/mysql"
FULL_BACKUP_DIR="$BACKUP_BASE/full_$(date +%Y%m%d)"
MYSQL_USER="backup_user"
MYSQL_PASS="secure_password"
 
# Full backup with Percona XtraBackup
# --backup: Perform backup
# --target-dir: Backup destination
# --compress: Enable compression
# --compress-threads: Parallel compression
# --parallel: Parallel file copy threads
 
xtrabackup \
    --backup \
    --user="$MYSQL_USER" \
    --password="$MYSQL_PASS" \
    --target-dir="$FULL_BACKUP_DIR" \
    --compress \
    --compress-threads=4 \
    --parallel=4
 
# Prepare the backup (apply logs for consistency)
# This step is REQUIRED before restore
xtrabackup \
    --prepare \
    --target-dir="$FULL_BACKUP_DIR"
 
echo "MySQL backup complete: $FULL_BACKUP_DIR"
 
# For incremental backup (runs after full backup)
# INCR_BACKUP_DIR="$BACKUP_BASE/incr_$(date +%Y%m%d_%H%M%S)"
# xtrabackup --backup --target-dir="$INCR_BACKUP_DIR" \
#     --incremental-basedir="$FULL_BACKUP_DIR" \
#     --user="$MYSQL_USER" --password="$MYSQL_PASS"

Oracle: RMAN (Recovery Manager)

Oracle's RMAN is the most sophisticated enterprise backup tool:

Integrated with Oracle database for optimal performance
Block-level incremental forever capability
Automatic backup optimization and validation
Comprehensive catalog and reporting

oracle_rman_backup.sql
SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
-- Oracle RMAN Online Backup Script
 
-- Connect to RMAN
-- $ rman target / catalog rman_user/password@catalog_db
 
-- Configure backup settings
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE DEVICE TYPE DISK PARALLELISM 4;
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM';
CONFIGURE ENCRYPTION FOR DATABASE ON;
 
-- Full database backup with compression and parallelism
RUN {
    ALLOCATE CHANNEL c1 DEVICE TYPE DISK;
    ALLOCATE CHANNEL c2 DEVICE TYPE DISK;
    ALLOCATE CHANNEL c3 DEVICE TYPE DISK;
    ALLOCATE CHANNEL c4 DEVICE TYPE DISK;
    
    -- Backup the database
    BACKUP AS COMPRESSED BACKUPSET
        DATABASE
        PLUS ARCHIVELOG
        TAG 'DAILY_FULL_BACKUP';
    
    -- Backup control file and SPFILE
    BACKUP CURRENT CONTROLFILE
        SPFILE
        TAG 'CONTROLFILE_BACKUP';
    
    -- Validate backup integrity
    RESTORE DATABASE VALIDATE;
    
    RELEASE CHANNEL c1;
    RELEASE CHANNEL c2;
    RELEASE CHANNEL c3;
    RELEASE CHANNEL c4;
}
 
-- Crosscheck and delete obsolete backups
CROSSCHECK BACKUP;
DELETE NOPROMPT OBSOLETE;
 
-- Report backup status
LIST BACKUP SUMMARY;

SQL Server: Native Backup with Compression

Microsoft SQL Server provides built-in online backup capabilities:

BACKUP DATABASE ... WITH COPY_ONLY: Non-disruptive backup
Backup compression significantly reduces size and I/O
Striped backups across multiple devices for parallelism
Integrated with Windows VSS for storage snapshots

sqlserver_backup.sql
SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
-- SQL Server Online Full Backup
 
-- Full database backup with compression
BACKUP DATABASE [Production]
TO DISK = 'D:\Backups\Production_Full.bak'
WITH 
    COMPRESSION,                          -- Native compression
    CHECKSUM,                             -- Verify backup integrity
    STATS = 10,                           -- Progress reporting every 10%
    NAME = 'Production-Full Database Backup',
    DESCRIPTION = 'Daily full backup with compression',
    COPY_ONLY;                            -- Don't affect differential baseline
 
-- Backup to multiple files (striped backup for performance)
BACKUP DATABASE [Production]
TO DISK = 'D:\Backups\Production_Stripe1.bak',
   DISK = 'D:\Backups\Production_Stripe2.bak',
   DISK = 'D:\Backups\Production_Stripe3.bak',
   DISK = 'D:\Backups\Production_Stripe4.bak'
WITH 
    COMPRESSION,
    CHECKSUM,
    MAXTRANSFERSIZE = 4194304,            -- 4MB per transfer
    BUFFERCOUNT = 20,                     -- Memory buffers
    STATS = 5;
 
-- Verify backup is readable
RESTORE VERIFYONLY 
FROM DISK = 'D:\Backups\Production_Full.bak'
WITH CHECKSUM;
 
-- Query backup history
SELECT 
    database_name,
    backup_start_date,
    backup_finish_date,
    DATEDIFF(MINUTE, backup_start_date, backup_finish_date) AS duration_minutes,
    backup_size / 1024 / 1024 AS backup_size_mb,
    compressed_backup_size / 1024 / 1024 AS compressed_size_mb,
    CAST(100 - (compressed_backup_size * 100.0 / backup_size) AS DECIMAL(5,2)) AS compression_ratio_pct
FROM msdb.dbo.backupset
WHERE database_name = 'Production'
ORDER BY backup_start_date DESC;

Performance Considerations and Impact

While online backups minimize disruption, they are not zero-impact operations. Understanding and managing performance impact is essential for production deployments.

Resource Consumption During Backup

Online backups consume several system resources:

Disk I/O: Backup reads compete with production I/O. On spinning disks, this can significantly impact query latency.
CPU: Compression and checksum computation consume processor cycles.
Network: Streaming backups to remote storage consume bandwidth.
Memory: Backup processes may affect buffer pool efficiency.

Measuring Backup Impact

Before deploying online backup, measure baseline performance and impact:

Metrics to Monitor During Online Backup
Metric Category	Specific Metrics	Acceptable Increase
Query Performance	Average query latency, P95/P99 latency	10-30% degradation
Throughput	Transactions per second, queries per second	5-20% reduction
Disk I/O	IOPS, throughput MB/s, queue depth	50-100% increase acceptable
CPU	User %, system %, iowait %	20-40% increase if capacity exists
Replication Lag	Seconds behind primary (if replicated)	Minimal increase expected

Mitigation Strategies

1. Schedule During Low-Traffic Periods

Even for 24/7 systems, traffic patterns vary. Schedule full backups during:

Late night/early morning for consumer applications
Weekends for business applications
Post-batch processing windows

2. Throttle Backup I/O

Many backup tools support I/O limiting:

throttled_backup_examples.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# PostgreSQL: pg_basebackup with rate limiting
pg_basebackup -D /backup -r 100M  # Limit to 100 MB/s
 
# XtraBackup: I/O rate limiting
xtrabackup --backup --throttle=40 --target-dir=/backup
# Limits to 40 I/O operations per second
 
# Linux: Use ionice for I/O scheduling priority
ionice -c 3 pg_basebackup -D /backup  # Idle I/O class (lowest priority)
 
# Linux: Use cpulimit to cap CPU usage
cpulimit -l 50 -- pg_basebackup -D /backup  # Limit to 50% CPU
 
# Combination approach
ionice -c 2 -n 7 nice -n 19 pg_basebackup -D /backup -r 50M

3. Use Replica for Backup Source

A powerful pattern is to perform backups from a replica rather than the primary:

Advantages:

Zero impact on primary database performance
Replica can be paused during backup for consistency
Backup I/O isolated from production writes

Considerations:

Replica must be current (monitor replication lag)
Some systems require specific replica configuration
Network transfer if replica is remote

Converting Mermaid diagram...

4. Leverage Storage-Level Features

Modern storage systems offer backup-friendly features:

Snapshot-based backup: Near-instant, minimal database impact
Clone volumes: Create writable copy for backup processing
QoS policies: Prioritize production I/O over backup I/O
Dedicated backup networks: Separate backup traffic from production

5. Optimize Backup Duration

Shorter backups mean shorter impact windows:

Use parallel backup (multiple streams/channels)
Enable compression (reduces data volume to transfer)
Incremental/differential backups (smaller data sets)
Use SSDs or dedicated backup storage

Performance Impact Baseline

Consistency Guarantees and Recovery Path

The ultimate test of any backup is successful recovery. Online backups must provide clear, guaranteed recovery paths with well-defined consistency semantics.

Crash Consistency vs. Application Consistency

Online backups typically provide crash consistency—the backup represents a state equivalent to abruptly powering off the server at a specific moment:

All committed transactions are preserved
In-flight transactions are rolled back during recovery
No partial transactions exist
Database structural integrity is maintained

Application consistency goes further, ensuring:

Application-specific invariants are met
External system references are valid
Business logic constraints are satisfied

Most online backups provide crash consistency. Application consistency requires additional coordination (quiescing applications, coordinating multi-system backups).

The Recovery Process

Recovering from an online backup involves several steps:

1. Restore Data Files

Copy backup files to target location
Decompress if backup was compressed
Verify file checksums if available

2. Apply Redo Logs (WAL)

Replay all transactions from backup checkpoint forward
Bring database to consistent crash-recovery state
This step is often called 'recovery' or 'prepare'

3. Open Database

Roll back any incomplete transactions
Verify database structural integrity
Database becomes available for connections

4. Validate Recovery

Run consistency checks
Verify application connectivity
Validate sample data integrity

recovery_procedures.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# PostgreSQL: Restore from pg_basebackup
 
# 1. Stop the running PostgreSQL (if any)
sudo systemctl stop postgresql
 
# 2. Clear the data directory (CAUTION: destroys current data)
sudo rm -rf /var/lib/postgresql/14/main/*
 
# 3. Restore the backup
sudo tar -xzf /backup/base.tar.gz -C /var/lib/postgresql/14/main/
sudo tar -xzf /backup/pg_wal.tar.gz -C /var/lib/postgresql/14/main/pg_wal/
 
# 4. Set correct ownership
sudo chown -R postgres:postgres /var/lib/postgresql/14/main
 
# 5. Create recovery signal file (PostgreSQL 12+)
sudo touch /var/lib/postgresql/14/main/recovery.signal
 
# 6. Start PostgreSQL - recovery happens automatically
sudo systemctl start postgresql
 
# 7. Monitor recovery progress
sudo -u postgres psql -c "SELECT pg_is_in_recovery();"
# Returns 'f' (false) when recovery is complete
 
# 8. Validate recovery
sudo -u postgres pg_isready
sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_user_tables;"

Untested Backups Are Not Backups

Best Practices and Implementation Checklist

Implementing online backup successfully requires attention to numerous details. The following best practices and checklist ensure robust, production-ready backup systems.

Online Backup Best Practices

•Use database-native tools — pg_basebackup, xtrabackup, RMAN are optimized and well-tested. Avoid generic file copy unless using storage snapshots.
•Always include transaction logs — Data files without WAL/redo logs are often unrecoverable. Configure continuous WAL archiving.
•Compress and encrypt — Reduce storage costs with compression; protect data with encryption at rest and in transit.
•Validate backup integrity — Use checksums, verify backup readability, test restore procedures regularly.
•Monitor backup jobs — Alert on failures, track backup duration trends, verify backup frequency meets RPO.
•Document recovery procedures — Maintain step-by-step runbooks; every team member should be able to perform recovery.
•Store backups offsite — Protect against site-wide disasters with geographic separation of backup storage.
•Retain backup history — Maintain multiple recovery points; don't rely on a single backup.

Online Backup Implementation Checklist
Category	Item	Status
Infrastructure	Dedicated backup storage sized appropriately	☐
Infrastructure	Network bandwidth for backup traffic calculated	☐
Infrastructure	Backup retention storage estimated (30-90 days)	☐
Configuration	Backup user/role with appropriate permissions	☐
Configuration	WAL archiving configured and tested	☐
Configuration	Backup compression enabled	☐
Configuration	Backup encryption configured	☐
Scheduling	Backup schedule aligned with traffic patterns	☐
Scheduling	Backup window sized appropriately for data volume	☐
Monitoring	Backup success/failure alerting configured	☐
Monitoring	Backup duration monitoring in place	☐
Monitoring	Backup size trending tracked	☐
Validation	Automated backup integrity checks enabled	☐
Validation	Regular restore tests scheduled (monthly)	☐
Documentation	Recovery runbook documented	☐
Documentation	Recovery tested by multiple team members	☐

Page Complete

1 / 5