Saturday, 6 January 2018

IPsec VPN for macOS High Sierra and Android Nougat

After switching to OpenWRT from DD-WRT which doesn't support IPsec, this is post describes the settings I use.

Since Android (at least up to Nougat) does not support IKEv2, we will use a certificate-based IKEv1 setup (IPsec Xauth RSA in Android speak).

strongSwan configuration

The configuration is taken from strongSwan's usable examples.

# /etc/ipsec.conf - strongSwan IPsec configuration file

config setup
        #charondebug="asn 2, cfg 2, chd 2, dmn 2, enc 1, esp 2, ike 2, imc 2, imv 2, job 2, knl 2, lib 2, mgr 2, net 2, pts 2, tls 2, tnc 2"

conn %default




        # Android Nougat native client.
        # IKE:AES_CBC_256/HMAC_SHA2_384_192/PRF_HMAC_SHA2_384/MODP_1024
        # macOS High Sierra Cisco IPsec.
        # IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048

        # For some reason, data cannot be transferred using Android's second suggestion.
        # ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ
        # ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ

conn pubkey

conn xauth-pubkey
        # Android's native client requires XAuth in addition to certificates.

conn xauth-hybrid

conn ipsecuritas
        # IPSecuritas apparently needs a specific IP or the CHILD_SA cannot be found.

In order to prevent routing headaches, it's advisable to assign addresses to IPsec clients (rightsourceip) from a different subnet than the LAN ( in this example).

I did not set up IKEv2 connections (for use on macOS, for example) because that would require installing additional modules for EAP, such as eap-md5 or eap-tls.

# /etc/ipsec.secrets - strongSwan IPsec secrets file

: RSA moonKey.pem
client : XAUTH "password"

I have chosen the legacy ipsec configuration in favour of swanctl because the latter is a much bigger packet to install.

Firewall configuration

Confusingly, OpenWRT /etc/config/firewall already has a couple of rules for IPsec (Allow-IPSec-ESP and Allow-ISAKMP), but another important one is missing. Moreover, as of this writing these rules handle IPsec forwarding to the LAN. In order for the router to be able to receive IPsec traffic, the option dest lan must be dropped. Here's the result:

config rule
        option name             Allow-IPSec-ESP
        option src              wan
        #option dest             lan
        option proto            esp
        option target           ACCEPT

config rule
        option name             Allow-ISAKMP
        option src              wan
        #option dest             lan
        option proto            udp
        option dest_port        500
        option target           ACCEPT

config rule
        option name             Allow-NATT
        option src              wan
        option proto            udp
        option dest_port        4500
        option target           ACCEPT

The WAN interface needs to accept packets unwrapped from IPsec tunnels which can be recognised by their »ipsec« policy using the »policy module«. In order for clients connected via IPsec to be reachable from the LAN, packets bound for an IPsec tunnel must not be subjected to NAT. This can be accomplished using the following rule in /etc/firewall.user, for example:

# Accept tunnelled traffic.
iptables \
        -t filter \
        -A input_wan_rule \
        -m policy \
        --dir in \
        --pol ipsec \
        --mode tunnel \
        -j ACCEPT \

# Prevent NATting traffic bound for IPsec tunnel.
iptables \
        -t nat \
        -A postrouting_wan_rule \
        -m policy \
        --dir out \
        --pol ipsec \
        --mode tunnel \
        -j ACCEPT \


Activating the line charondebug in /etc/ipsec.conf (above) or adding the following section to /etc/strongswan.d/charon-logging.conf is helpful when debugging issues:

charon {
    syslog {
        daemon {
            default = 1
# logread -f

The following commands can help with routing and firewall debugging:

# ip -f inet route list table all
# iptables -t [filter|nat] -L -v
# iptables -t filter -I reject --log-prefix 'reject ' -j LOG
# /etc/init.d/firewall stop

Other settings

In order to provide DNS to VPN clients, the DNS server must not be configured as »local only« (localservice=0).

config dnsmasq
        option domainneeded      1
        option rebind_protection 1
        option rebind_localhost  1
        option local             /lan/
        option domain            lan
        option authoritative     1
        option readethers        1
        option leasefile         /tmp/dhcp.leases
        option resolvfile        /tmp/
        option nonwildcard       1
        option localservice      0
        option add_local_fqdn    3

Key and certificate generation

Generating certificates requires strongswan-pki which is a separate install. First, generate the key for your certificate authority.

# ipsec pki --gen --outform pem >caKey.pem

The CA key is particularly sensitive and must never leak. Next, create a certificate for your certificate authority.

# cacert=/etc/ipsec.d/cacerts/caCert.pem
# ipsec pki --self \
        --in caKey.pem \
        --dn 'C=CH, O=rienajouter, CN=rienajouter CA' \
        --ca \
        --outform pem \

The »distinguished name« for every certificate you create must be unique – otherwise the certificates will be considered interchangeable and authentication will fail. Next, create a key and certificate for the VPN server.

# key=/etc/ipsec.d/private/serverKey.pem
# cert=/etc/ipsec.d/certs/serverCert.pem
# ipsec pki --gen --outform pem >"$key"
# ipsec pki --pub --in "$key" \
| ipsec pki --issue \
        --cacert "$cacert" \
        --cakey caKey.pem \
        --dn 'C=CH, O=rienajouter,' \
        --san '' \
        --flag serverAuth \
        --flag ikeIntermediate \
        --outform pem \

For the VPN server certificate to work reliably with macOS, the CN of the distinguished name as well as the subject alt name (--san) must match the DNS name of the VPN gateway and the ikeIntermediate flag must be present. Windows on the other hand requires the serverAuth flag.

Last, for each client create a certificate to be shared with the client.

# key=/etc/ipsec.d/private/clientKey.pem
# cert=/etc/ipsec.d/certs/clientCert.pem
# ipsec pki --gen --outform pem >"$key"
# ipsec pki --pub --in "$key" \
| ipsec pki --issue \
        --cacert "$cacert" \
        --cakey caKey.pem \
        --dn 'C=CH, O=rienajouter, CN=client' \
        --outform pem \

Generating keys and certificates in PEM format helps when exporting them to the clients.

Doing the same with openssl is a nightmare because openssl was created in order to prevent the widespread use of cryptography, and quite successfully so far.

$ openssl req \
        -new \
        -newkey rsa:2048 \
        -subj '/C=CH/O=rienajouter/CN=client' \
        -keyout "$key" \
        -nodes \
| openssl x509 \
        -req \
        -CA "$cacert"  \
        -CAkey caKey.pem \
        -sha256 \
        -days $((3 * 365)) \

In case you forget, here's how to find out what's inside one of the key or certificate files:

$ ipsec pki --print --in "$cert"
$ openssl rsa -in "$key" -text -noout
$ openssl x509 -in "$cert" -text -noout

Certificate distribution

In order to distribute the certificates generated in the previous step, they must be packaged together into PKCS#12 files. The command to do so is:

$ openssl pkcs12 \
        -certfile "$cacert" \
        -inkey "$key" \
        -in "$cert" \
        -export \
        -out client.p12 \

When exporting for macOS, make sure to use a non-empty password when prompted by the above command. After importing the CA and client certificates into the system keychain on macOS, set the CA certificate's »trust« for »IPsec« to »always trust«. You also need to grant access to the private key inside the client certificate to /usr/sbin/racoon. The »access control« tab is only shown for keys, which in turn only appear when selecting »keys« or »certificates« in the left-hand pane of »Keychain Access«.