A VPN connection in commonly involves two phases:
- ISAKMP SA
The two peers needs to agree in the encryption method and authentication algorithms to create an authenticated and secure tunnel by which the network VPN traffic will flow. This is done through automatic keys exchange, but it is done over a (yet) insecure path. Diffie-Hellman (DH) helps at this point by just sending the public keys through the wire.
How DH works? Both ends generate a private/public key pair and exchange with each other the public one. On both sides the same session key (symmetric key) is calculated with the local private and the remote public key (maths magic) without being sent by the channel.
In resume, both peers agree on the DH Group (1, 2 or 5 normally – 768 bit, 1024 bit or 1536 bit prime), encryption (DES, 3DES, AES, …), hashing (MD5, SHA1) and authentication (PSK, certificates DSA/RSA, …), through UDP port 500. This combination is called proposals. The SA has to be uniquely identified (multiple SA’s can coexist at the same device), and it is get done by a combination of a parameter called SPI, IP dst and security protocol (ESP/AH).
This phase has to possible modes (just when using IKE v1):
- Main mode (6 messages): the recommended and more secure mode.
- Agressive mode (3 messages): it would be possible to sniff the keys and/or anyone knowing the PSK could connect to the VPN.
2. IPSEC SA
Now that the secure tunnel is settled, the IPSEC SA (quick mode) is negotiated. Again, a proposal list is sent, composed of: security protocol (ESP/AH), DH Group, encryption algorithm, authentication algorithm, key lifetime, Proxy ID, DH public keys (if PFS is used).
- ESP provides encryption (3DES, DES, AES, …), hashing (MD5, SHA1) and authentication. Payload is encapsulated, so NAT is possible.
- AH does not provide encryption but it is faster than ESP.
- Proxy ID (or IKE ID’s) identifies which SA is used for the VPN. It links multiple subnets to the same VPN.
- PFS (Perfect Forward Secrecy), if enabled, will recalculate the DH keys in phase 2 to make the channel even more secure.
IPSEC tunnels also has to modes:
- Transport mode: create a secure channel between hosts.
- Tunnel mode: create a secure channel between networks (mostly deployed)
So commonly, to complete a VPN configuration you need to agree on the encryption algorithm, hash method, DH group, PSK and IP address of the pairs.