S 5.149 Secure connection of an external network with IPSec

Initiation responsibility: Head of IT, IT Security Officer

Implementation responsibility: Administrator

Internet Protocol Security (IPSec) is a standard defined by the IEEE in a series of RFCs and internet drafts. IPSec consists of a series of protocols for encryption, securing data integrity, authentication, and key administration when communicating using IP. IPSec can also be used to implement secure connections between computer systems that are largely transparent to the users. IPSec is often used, for example, in business and administration to implement VPNs.

Various security mechanisms such as the following are described in IPSec:

The Authentication Header allows for an authentication of the transmitted data and is therefore intended to effectively prevent IP spoofing or session hijacking attacks. The Encapsulating Security Payload (ESP) mechanism also allows for encryption of the transmitted data in addition to authentication. Since ESP can also be used without encryption and therefore for authentication only, the AH mechanism is not widely used.

IPSec offers the following two operating modes to permit as many different types of connections as possible:

In the transport mode, the IP header of the source packets is used for routing purposes. Only the contents of the packets are encrypted in the transport mode, but not the IP header. This mode is only suitable for communication connections where the endpoints of the tunnel are simultaneously the endpoints of communication, i.e. as in direct client/server communication, for example. Since the information needed for transmission is not encrypted, any router between the endpoints may eventually be able to process the information directly.

In the tunnel mode, the entire packet including the IP header is encrypted in order to protect internal address information against unauthorised access. In this case, an attacker can only determine the endpoints of the tunnel and cannot determine the route followed by the connection.

It is necessary to select an appropriate operating mode for the VPN according to the particular area of application. When used to connect the networks of different locations, ESP should be used in combination with the tunnel mode. When used for communication between two computers in the LAN, the transport mode should be selected.

Key administration with IPSec

IPSec uses the Internet Key Exchange protocol (IKE) for key generation and distribution. IKE describes how security parameters are negotiated and shared keys are exchanged. IKE is divided into the following two phases:

Phase 1 is used to negotiate an "ISAKMP Security Association", with "ISAKMP" being "Internet Security Association and Key Management Protocol". A security association (SA) describes an authenticated, encrypted channel and generally consists of a security parameter index, the destination IP address, and a security protocol identifier. The SA can be negotiated in the main mode or in the aggressive mode.

The modes differ in terms of the number of messages needed to be exchanged and the encryption of the data exchanged. The first step for both communication partners in the main mode is to calculate a shared secret key according to the Diffie-Hellman key exchange method. The actual authentication data is then protected by this key during transmission. Authentication can only be performed with the help of a character string (a pre-shared key or PSK) known to both of the communicating parties or using certificates. Six messages are needed in this first phase for negotiation in the main mode.

The aggressive mode, in contrast, only requires three messages to be exchanged, because a separate key for the authentication data does not need to be negotiated. Instead, a checksum is generated from the pre-shared key using a hash function and then transmitted.

A suitable mode must be selected to securely connect an external network using IPSec. The aggressive mode offers advantages in terms of speed in comparison to the main mode, but it should only be used in exceptional cases, because it is insecure. It is possible, for example, to determine the pre-shared key using a dictionary or brute force attack.

Some manufacturers support the XAUTH method to compensate for the inherent weaknesses of the IKE aggressive mode when using pre-shared keys. In this case, the IKE protocol is expanded so that mechanisms such as RADIUS and others can be used.

In Phase 2, the SAs and keys to be used by a security protocol like IPSec or any other protocol requiring cryptographic key material are negotiated.

The following aspects must be taken into consideration for a secure IPSec configuration:

To route protected VPN data communications using untrustworthy networks, the central servers also must be available from the untrustworthy network. It is therefore necessary in this case to minimise the number of possible points of attack to protect against attacks on the LAN. For this reason, the following additional requirements are placed on the VPN servers involved:

The VPN server should not offer any network services other than the network services required for IPSec communication.

If possible, only those connections actually needed from the VPN server to the LAN should be established.

Since IPSec is a highly complex protocol family offering several different services, all unneeded services should be disabled. Only the IKE, ESP, and possibly the AH service should be enabled if this is possible.

The services offered and the authorisations assigned should be documented comprehensibly.

To continuously improve the security of the VPNs, it is also necessary take into consideration the recommendations described in safeguard S 4.321 Secure operation of a VPN.

Review questions:

© Federal Office for Information Security. All rights reserved.