S 5.90 Use of IPSec under Windows

Initiation responsibility: IT Security Officer, Head of IT

Implementation responsibility: Administrator

Windows offers IPSec-compliant functionality to protect communications. IPSec is an international standard that permits cryptographic protection of IP-based communication: The decision whether or not to use IPSec to protect communications must be made on a case-by-case basis. When planning the use of Windows, this must already be taken into account and defined by means of a policy.

IPSec consists of the following functions:

• mutual authentication of the communication endpoints
• protection of the integrity of the data transmitted using digital signatures
• protection of the confidentiality of the data transmitted or of the entire IP-data packet using encryption (tunnel mode)

General information on the selection of suitable cryptographic procedures can be found in S 2.164 Selection of a suitable cryptographic procedure. When using IPSec, an algorithm belonging to the SHA-2 family, i.e. SHA-224, SHA-256, SHA-384 or SHA-512, should be used as the hash method.

These are, for example, supported in the IPSec client which is part of the Windows Vista and Windows 7 firewall. The function is available by default since Windows 7 and in Windows Vista since the Service Pack 1. Without Service Pack 1 installed, this IPSec client supports only the weaker MD5 and SHA1 hash methods in Windows Vista. Windows XP without Service Pack 3 installed also supports only the weaker hash methods. They should no longer be used.

In order to ensure that not only the integrity and confidentiality of the data transmitted, but also that the data is actually exchanged between the right communication partners, the communication partners must be authenticated. The Windows implementation allows the following procedures for authentication of the communication end points:

• The Kerberos protocol can be used if both communication partners are within the same Active Directory structure. In this case, the standard Windows authentication procedure is used. This procedure is based on symmetric keys that are then used to encrypt the Kerberos tickets.
• X.509 certificates can be used. In this case, authentication, which is based on asymmetric keys, is granted based on certificate information. In general, a "Challenge Response Procedure" is applied. It is checked if the user to be authenticated possesses the correct private key. The IPSec DirectAccess function uses this option.

The first time an IPSec connection is established, the communication partners first negotiate the algorithms and procedures to be subsequently used for authentication, for protecting integrity, and for maintaining confidentiality, and the results are then stored in the Security Association (SA).

The parameters stored in the SA are then used for all future communication connections until the validity of the SA parameters expires and the procedures are renegotiated. This is generally done fully automatically by the components in the IPSec implementation.

The master key and the session key need to be generated for the actual encryption. In general, the master key, which is used to generate all the other keys, is normally only created once for each connection, but the session keys are recreated several times at regular intervals. It is also possible to recreate the master key at regular intervals, but this requires reauthentication of the communication partners. Reauthentication is usually executed automatically by the components of the IPSec implementation, which has a significant effect on the performance.

IPSec knows two different methods to protect communications: ESP (Encapsulated Security Payload) and AH (Authentication Header). Under Windows Server 2008 and higher, AH is no longer supported, since this method is hardly of any practical importance due to the related disadvantages (no implementation of network addresses per NAT possible).

To control IPSec-based communication, Windows offers IPSec policies that specify which IPSec parameters are to be used for a connection. In Windows Vista and Windows Server 2008, the IPSec policies are also referred to as connection security rules. The following can be achieved using the various policies:

• that IT systems only accept connections protected by IPSec,
• that IT systems require the communication partner to use IPSec-protected connections, but also allow unprotected communication if the partner does not support any IPSec protocols,
• or that IPSec-based communication is not allowed.

Windows 2000 and higher provides three predefined IPSec policies:

• Client (only response): For IT systems that only negotiate IPSec protection upon request by the communication partner and otherwise do not use any communication protection.
• Server (request security): For IT systems that are requested to use IPSec-protected connections by their communication partners but that also accept connections without protection when the communication partner does not support IPSec.
• Server (security required): For IT systems that should only establish IPSec-protected connections and deny requests for unprotected connections.

These predefined rules can be adapted in detail to the local requirements. It is recommended in this case to make a copy of the policy first and make the changes to the copy.

Filter rules are used in connection with an IPSec policy in order to be able to define different IPSec parameters, for example depending on the protocol used. It is possible to specify, for example, that HTTP connections do not have to be encrypted but FTP connections always need to be encrypted.

Windows Vista and Windows 7 allow the configuration of the IPSec policies using group policies in Computer Configuration | Windows Settings | Windows Firewall with Advanced Security | Connection Security Rules; for Windows Server 2008, the configuration editor is in Administrative Tools| Windows Firewall with Advanced Security | Connection Rules. For Vista, Windows 7 and Server 2008, Microsoft does not provide any predefined IPSec policies. However, the Connection Security Rule Wizard can help you configure the IPSec policies. IPSec is enabled using group policies or is enabled locally in the Properties dialogue of a network connection. The enabling in the Properties dialogue is not available in Vista, Windows 7 and Server 2008. Here, IPSec is enabled by creating connection security rules.

Under Windows Server 2008 and higher, the configuration of rules for the local firewall and IPSec in the surface was brought together to simplify administration and to eliminate sources of error resulting from contradictory IPSec and firewall rules.

In general, the following must be taken into account when using IPSec in Windows:

• Before using IPSec, it is necessary to check if the performance losses associated with enabling IPSec can be tolerated. Under certain circumstances, consideration should be given to using network adapters with a TCP/IP Offload Engine (TOE) to handle the complex calculations associated with the TCP/IP protocol stack on the network adapter in order to reduce the load on the CPU.
• The Perfect Forward Secrecy (PFS) option should be enabled to provide stronger protection for the session keys. This ensures that when a session key does become compromised, only the data encrypted with that particular session key can be decrypted. This is accomplished through the following:
• that a session key that has been used to encrypt data cannot be used to generate additional keys and
• that the source key material used to generate a session key is not used again to generate another session key.
  
  This does cause some loss of performance but the actual loss is generally negligible.
• For connections with a high protection requirement, the PFS option can also be enabled for the master key. However, this results in greater performance losses than when using PFS for session keys since the communication partners have to be authenticated every time.
• In each specific case, it is necessary to decide which mechanisms and procedures for authentication and protection of integrity and confidentiality should be available for IPSec negotiations when establishing a connection. It must be taken into account that there must be at least one procedure available that is supported by both of the communication partners.
• If you define your own IPSec policies, then it is always absolutely necessary to define a default response rule. This rule takes effect when no other filter rule applies to the policy. If no default response rule is defined, then it may be impossible to establish a connection between the communication partners. The default response rule is not required for the use of DirectAccess.
• The filter rules of an IPSec policy also allow, among other things, the IPSec protection to be linked to the IP address of the communication partner so that encryption is enabled or disabled depending on the communication partner.
• If the Kerberos mechanism is used for authentication, then authentication is not protected by IPSec since Kerberos does not execute in the context of an IPSec connection. After installing Windows 2000 Service Pack 1, IPSec protection can also be enabled for the Kerberos protocol. To enable it in this case, though, it is necessary to modify the registry (see also Microsoft Knowledge Base Article KB254728 and KB811832 for more information):
  
  In HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services \IPSEC (or from version Windows Vista/7 and higher in HKEY_LOCAL_MACHINE | SYSTEM | CurrentControlSet | Services | PolicyAgent), the NoDefaultExempt key (of type REG_DWORD) must be created and set to the value 1.
• To check if the IPSec connection is being established correctly and that IPSec communication is functioning correctly, Windows 2000 provides the ipsecmon.exe program, Windows XP the MMC snap-in IP Security Monitor, and Windows Vista / 7 / Server 2008 the Windows Firewall with Advanced Security snap-in. The program or snap-in can be used to track down the source of the error if there are problems with IPSec connections. The program is relatively simple and can only be used for an initial determination of the causes of the problem.
• IPSec should be used, among other things, in conjunction with EFS-encrypted files (see also S.147 Secure use of EFS under Windows) when these files are stored on servers and should be protected during transmission to a client over the network. However, it is possible to use any other mechanism instead of IPSec to protect network communications and to protect EFS files stored on a server during transmission.
• If communication with a system that does not have a Windows operating system installed is to be protected using IPSec, then tests must be conducted to check their interoperability and whether they function correctly. The IPSec procedure is standardised, but in some cases, even standardised procedures can cause compatibility problems under some circumstances.

Review questions:

• Is an IPSec policy available?
• Is the performance of the IT systems communicating adequate for IPSec communication?
• Was the Perfect Forward Secrecy (PFS) option enabled?
• Were the IPSec connections checked to make sure they are established correctly?
• Was a default response rule defined?