TCP Three-Way Handshake

Every reliable connection on the Internet—every web page you load, every file you download, every video stream you watch—begins with a single, carefully crafted packet: the SYN segment. This unassuming 40-byte message initiates a sophisticated dance between two machines, establishing the foundation for reliable, ordered, and error-checked communication.

The SYN (Synchronize) segment represents the client's request to initiate a connection. It carries critical information: a proposed starting sequence number, connection parameters, and optional capabilities. Understanding the SYN segment isn't just academic—it's essential for debugging connection issues, analyzing security attacks, and comprehending why TCP behaves the way it does under various network conditions.

Before TCP can begin reliable data transfer, both endpoints must establish a synchronized understanding of key connection parameters. This synchronization serves multiple critical purposes:

Why Can't TCP Just Start Sending Data?

Unlike UDP, which operates in a fire-and-forget mode, TCP guarantees reliable, ordered delivery. To achieve this, TCP must:

1. Establish bidirectional communication — Both sides must confirm they can send and receive
2. Agree on sequence numbers — Each side must know where the other will begin numbering bytes
3. Negotiate connection parameters — Window sizes, maximum segment sizes, and optional features must be agreed upon
4. Allocate resources — Both endpoints must reserve memory buffers and data structures for the connection

Without synchronization, TCP would have no way to detect lost packets (no reference point for sequence numbers), no ability to reorder out-of-sequence arrivals, and no agreement on how much data can be in flight.

The Synchronization Problem:

Consider what happens if a client simply started sending data without synchronization:

1. The server has no connection state—it doesn't know this client wants to communicate
2. Sequence numbers would be meaningless without an agreed starting point
3. Old packets from previous connections might be mistaken for new ones
4. The server has no receive buffer allocated for this client's data
5. No agreement exists on parameters like maximum segment size

The SYN segment solves all these problems by explicitly establishing a connection context before any data transfer occurs.

The TCP header contains a set of control flags (also called control bits) that indicate the purpose and state of each segment. The SYN flag is one of six primary flags defined in the original TCP specification, each occupying a single bit in the header.

The Six Primary TCP Flags:

The SYN Flag's Special Semantics:

When the SYN flag is set (bit value = 1), the segment has special meaning:

1. Sequence Number Initialization — The sequence number field contains the Initial Sequence Number (ISN) that this endpoint will use
2. No Payload Expected — Pure SYN segments carry no application data (though the specification technically allows it)
3. Connection State Transition — Receipt of a SYN causes the receiver to transition to a specific state in the TCP state machine
4. Resource Allocation Trigger — The receiver must allocate resources (Transmission Control Block) for the new connection

Flag Combinations:

The SYN flag can appear alone (pure SYN) or in combination with the ACK flag (SYN-ACK). However, certain combinations are invalid or suspicious:

The Initial Sequence Number (ISN) carried in the SYN segment is perhaps the most critical value in TCP connection establishment. This 32-bit number serves as the starting reference point for all subsequent byte counting in that direction of communication.

Why Not Just Start at Zero?

A naive implementation might set the ISN to 0 for every connection. This approach fails catastrophically for several reasons:

1. Old Duplicate Problem — Consider a client connecting to a server, sending data with sequences 0-1000, then the connection dies. If a new connection also starts at 0, old packets still wandering the network could be misinterpreted as new.

2. Security Vulnerability — Predictable ISNs allow attackers to inject packets into connections they're not part of. If an attacker knows the expected sequence number, they can forge packets that the receiver will accept.

3. Connection Disambiguation — The same client-server pair might establish multiple connections over time. Unique ISNs help distinguish between these connections.

ISN Generation Strategies:

The evolution of ISN generation reflects advancing understanding of both reliability and security requirements:

Modern ISN Generation (RFC 6528):

Contemporary implementations generate ISNs using a formula similar to:

ISN = M + F(src_ip, src_port, dst_ip, dst_port, secret_key)

Where:

• M is a monotonically increasing value (typically from a counter incremented every 4 microseconds)
• F is a cryptographic hash function (commonly MD5 or a keyed HMAC)
• secret_key is a randomly generated key known only to the host, changed periodically

This approach ensures:

1. Randomness to outsiders — Without the secret key, ISNs appear completely random
2. Consistency for legitimate connections — The same socket pair gets the same ISN offset, preventing old duplicates
3. Forward progress — The counter M ensures ISNs always increase over time globally

A SYN segment is a complete TCP segment with specific field values that mark it as a connection initiation request. Let's examine every field in a typical SYN segment:

Critical Observations:

1. Acknowledgment Number is Meaningless — In a pure SYN (not SYN-ACK), the ACK flag is not set, so the acknowledgment number field has no semantic meaning. Implementations typically set it to 0.

2. Sequence Number Represents a Byte — Although this SYN segment carries no data, the ISN in the sequence field represents a 'virtual byte.' The SYN itself consumes one sequence number, so if ISN = 1000, the first data byte will have sequence number 1001.

3. Window Size is Initial Advertisement — The client declares how much data it's willing to receive in its initial buffer. This may be adjusted by Window Scale option.

4. Options are Critical — Modern TCP heavily relies on options negotiated during the handshake. Without proper options, connections may perform poorly.

The SYN segment is the only opportunity for a TCP endpoint to advertise optional capabilities to its peer. Options that aren't negotiated during the handshake cannot be used for the duration of the connection. This makes option selection in the SYN critically important.

Common Options in SYN Segments:

Deep Dive: Maximum Segment Size (MSS):

The MSS option is arguably the most important SYN option. It declares the largest segment (TCP header + data) the sender is willing to receive. This prevents the peer from sending segments that would require fragmentation.

MSS Calculation:

MSS = MTU - IP_Header_Size - TCP_Header_Size
    = 1500 - 20 - 20
    = 1460 bytes (typical for Ethernet + IPv4)

For IPv6:

MSS = MTU - IPv6_Header_Size - TCP_Header_Size
    = 1500 - 40 - 20
    = 1440 bytes (typical for Ethernet + IPv6)

If MSS option is absent, the default MSS is 536 bytes (576-byte minimum IPv4 MTU minus 40 bytes of headers). This severely limits performance, which is why all modern implementations include MSS.

Option Negotiation Rules:

1. Both sides must agree — If the server's SYN-ACK doesn't include a capability option that the client's SYN included, that capability is not available for the connection

2. Capabilities are per-connection — Each new connection must renegotiate options; they don't persist

3. Options have size limits — Total TCP header including options cannot exceed 60 bytes (15 × 32-bit words), limiting how many options can be included

4. Order doesn't matter — Options can appear in any order, though NOP padding is commonly used for alignment

When a TCP endpoint sends a SYN (client) or receives a SYN (server), it must allocate a data structure to track the connection state. This structure is called the Transmission Control Block (TCB)—the kernel's representation of a TCP connection.

What the TCB Contains:

The TCB stores all information needed to manage the connection:

TCB Creation During Handshake:

Client Side (Active Open):

1. Application calls connect()
2. Kernel creates TCB, sets ISS (Initial Send Sequence)
3. Kernel sends SYN with ISS
4. TCB state transitions to SYN_SENT
5. Kernel stores ISS in TCB, waits for SYN-ACK

Server Side (Passive Open):

1. Application calls listen()
2. SYN arrives from client
3. Kernel creates TCB (or queues SYN in some implementations)
4. Kernel extracts client's ISN, stores as IRS
5. Kernel generates its own ISS
6. Sends SYN-ACK with server's ISS, acknowledges client's ISN
7. TCB state transitions to SYN_RECEIVED

When a SYN segment arrives at a host, the TCP implementation follows a precise processing sequence. Understanding this sequence is essential for diagnosing connection problems and understanding security mechanisms.

Edge Cases in SYN Processing:

Duplicate SYN (Same Connection): If a SYN arrives for an existing connection, behavior depends on the connection state:

• SYN_RECEIVED state — Retransmit the SYN-ACK (client may not have received it)
• ESTABLISHED state — The SYN may indicate the client lost state; respond with ACK of current sequence
• Other states — Complex rules apply; may indicate half-open connection recovery

Simultaneous Open: Rarely, two hosts may each send SYN to each other simultaneously (both acting as clients). TCP handles this by allowing both sides to transition through SYN_RECEIVED to ESTABLISHED without requiring one to be the 'server.'

Understanding theory is important, but examining real SYN packets cements the concepts. Here's an example of a captured SYN segment as seen in a packet analyzer like Wireshark:

Key Observations from this SYN:

1. Port Numbers — Client using ephemeral port 54321, connecting to HTTPS (443)
2. Sequence Number — Raw ISN is 2812345678; Wireshark shows relative 0
3. Header Length — 40 bytes (10 × 4) indicates 20 bytes of options
4. Flags — Only SYN is set (0x002)
5. Window — 65535 bytes initial window, but with scale factor 7, effective max is 65535 × 128 = 8MB
6. Options Present:
   • MSS = 1460 (standard for Ethernet + IPv4)
   • SACK Permitted — Client wants selective acknowledgments
   • Timestamps — Client wants RTT measurement and PAWS
   • Window Scale = 7 — Client supports large windows

The SYN segment, as the initiator of every TCP connection, is a focal point for both attacks and defenses. Understanding these security implications is essential for network architects and security professionals.

Defense Mechanisms:

We've thoroughly examined the SYN segment—the first critical step in TCP connection establishment. Let's consolidate the essential knowledge:

What's Next:

The SYN initiates the conversation, but it's only the first of three messages. In the next page, we'll examine the SYN-ACK segment—the server's response that acknowledges the client's SYN while simultaneously proposing its own sequence number. This bidirectional establishment is what gives TCP its reliability guarantees.