T 4.69 Problems with the IPSec configuration

Internet Protocol Security (IPSec) is a widely used method for securing IP-based communication and is often used for VPNs. IPSec can be understood as a series of protocols for key management, authentication, and encryption. Due to the complexity of the standard, it is possible to configure IPSec incorrectly, which then threatens the security and stability of the communication connections, for example of the VPN channel.

Before actually performing encryption, the communication partners need to agree to the security associations (SAs) to be used. In many IPSec implementations, there is an option for working with fixed SAs and keys. However, their use comes in conjunction with the disadvantages described in T 2.130 Inappropriate selection of VPN authentication methods.

If the aggressive mode is selected for key negotiation (a mode for establishing connections faster), then the identities and signatures (as well as optional certificates) are transmitted as plain text.

Since the IPSec standard was specified in 1998, it does not take some of the methods for network configuration developed later into account. For example, in certain configurations, IPSec has traffic-related problems with the Network Address Translation (NAT) used nowadays in many networks because it changes the IPSec packets. For this reason, the supplemental RFCs 3947 and 3948 were developed with regard to the issue of NAT traversal.

Many manufacturers of VPN software have also implemented one or more versions of solutions to the NAT problem in their products that are mutually incompatible in some cases. When NAT is used, this means that different VPN products may only be able to communicate with each other at a lower security level, with limitations, or they may not even be able to communicate at all.

The most common configuration problems with an IPSec VPN are due to the SA suggestion lists. Problems arise when none of the SA suggestions of the initiator matches a suggestion provided by the other end, i.e. by the responder. Another frequently encountered reason for the inability to establish an IPSec connection is when not enough ports are activated on the security gateways.

Examples:

• An administrator configured the aggressive mode due to its ability to establish connections faster. The transmission of plain text in the aggressive mode has a flaw, though: the gateway sends a hash value derived using the pre-shared keying (PSK) method for authentication over the network. Since this hash value is not encrypted, an attacker could reconstruct the key using a dictionary or brute force attack and gain access to the corporate network.
• During the initial installation of the VPN of a medium-sized company, an inexperienced administrator transmitted the key used in an unencrypted e-mail to the corresponding branch office, and someone eavesdropped on the transmission. The attacker could then dial into the VPN with the help of the information obtained, access confidential documents, and then sell them to a competitor.