241Security protocols and encapsulation modesSecurity protocolsIPsec comes with two security protocols, AH and ESP. They define how to encapsulate IP packets and thesecurity services that they can provide.• AH (protocol 51) defines the encapsulation of the AH header in an IP packet, as shown in Figure 79.AH can provide data origin authentication, data integrity, and anti-replay services to prevent datatampering, but it cannot prevent eavesdropping. Therefore, it is suitable for transmittingnon-confidential data. AH supports authentication algorithms HMAC-MD5 and HMAC-SHA1.• ESP (protocol 50) defines the encapsulation of the ESP header and trailer in an IP packet, as shownin Figure 79. ESP can provide data encryption, data origin authentication, data integrity, andanti-replay services. Unlike AH, ESP can guarantee data confidentiality because it can encrypt thedata before encapsulating the data to IP packets. ESP supports encryption algorithms such as DES,3DES, and AES, and authentication algorithms HMAC-MD5 and HMAC-SHA1.Both AH and ESP provide authentication services, but the authentication service provided by AH isstronger. In practice, you can choose either or both security protocols. When both AH and ESP are used,an IP packet is encapsulated first by ESP and then by AH.Encapsulation modesIPsec supports the following encapsulation modes:• Transport mode—The security protocols protect the upper layer data of an IP packet. Only thetransport layer data is used to calculate the security protocol headers. The calculated securityprotocol headers and the encrypted data (only for ESP encapsulation) are placed after the originalIP header. You can use the transport mode when end-to-end security protection is required (thesecured transmission start and end points are the actual start and end points of the data). Thetransport mode is typically used for protecting host-to-host communications, as shown in Figure 77.• Tunnel mode—The security protocols protect the entire IP packet. The entire IP packet is used tocalculate the security protocol headers. The calculated security protocol headers and the encrypteddata (only for ESP encapsulation) are encapsulated in a new IP packet. In this mode, theencapsulated packet has two IP headers. The inner IP header is the original IP header. The outer IPheader is added by the network device that provides the IPsec service. You must use the tunnelmode when the secured transmission start and end points are not the actual start and end points ofthe data packets (for example, when two gateways provide IPsec but the data start and end pointsare two hosts behind the gateways). The tunnel mode is typically used for protectinggateway-to-gateway communications, as shown in Figure 78.Figure 77 IPsec protection in transport modeHost A Host BData flowIPsec tunnel
