IPSec: AH, ESP, IKEv2 and the two modes

Authentication Header (AH) authenticates parts of the IP header and the payload. It cannot authenticate the whole header, because some fields legitimately change as the packet is forwarded, e.g. Time-to-Live. The fields AH does authenticate are the “normally immutable” ones: source and destination IP, version, length, and similar.

Encapsulating Security Payload (ESP) can provide confidentiality, data origin authentication, connectionless integrity, an anti-replay service, and limited traffic flow confidentiality. It can be used stand-alone or combined with AH. ESP adds another header, positioned after the IP header (and possibly after an AH header), and a trailer after the protected data. That header and trailer carry:

• SPI, identifying the Security Association.
• Sequence Number (32 bits), a monotonically increasing counter giving an anti-replay function.
• An initialization value (IV), or nonce, present if the encryption or authenticated-encryption algorithm needs one.
• Payload/Protected Data (variable), a transport-level segment in transport mode, or a full IP packet in tunnel mode.
• Pad Length (8 bits), the number of pad bytes immediately preceding this field.
• Next Header (8 bits), identifying the type of data in the payload, e.g. an IPv6 extension header or an upper-layer protocol such as TCP.
• Integrity Check Value (variable, an integral number of 32-bit words), computed over the ESP packet minus the authentication data field itself.

IKEv2 provides mutual authentication and key establishment between two parties, negotiating cryptographic parameters such as ciphers and hash functions. It results in a total of four Security Associations, one per party and direction. These SAs are then used by AH and/or ESP, and are stored in the Security Association Database.

Tunnel mode is used when the gateways (routers) at each end are responsible for adding IPSec’s cryptographic protection to packets their hosts send, rather than the hosts doing it themselves. The entire original IP packet, once its AH or ESP fields are added, is treated as the payload of a new, outer IP packet with its own new outer IP header. The whole inner packet travels through a tunnel from one point of the network to another; no router along the way can examine the inner IP header, and since the original packet is fully encapsulated, the new outer packet can carry totally different source and destination addresses. This is why hosts behind a firewall can communicate securely without implementing IPSec themselves: their unprotected packets are tunneled through external networks by tunnel-mode SAs set up at the network boundary.

Transport mode is used when the endpoint of the IP communication is the same principal that adds the protection, i.e. genuine end-to-end host-to-host communication. It protects the payload of an IP packet, e.g. a TCP or UDP segment or an ICMP packet, not the whole packet. ESP in transport mode encrypts and optionally authenticates the IP payload, but not the IP header. AH in transport mode authenticates the IP payload and selected portions of the IP header. Either way, transport mode authenticates and encrypts only the payload: the IP header is only authenticated, never encrypted. This gives confidentiality to any application using it, without needing every application to implement confidentiality itself, at the cost that traffic analysis is still possible on the transmitted packets, since the header remains visible.