Building an OpenBSD VPN server with iked and OpenSSL


By Marco W. Soijer

April 2020

iked OpenSSL OpenBSD VPN

Share this post

OpenBSD's IKEv2 daemon iked is great to set up a VPN server that uses X.509 certificates, so mobile devices can connect securely through their respective built-in VPN clients. The OpenBSD iked documentation and VPN FAQ refer to the ikectl utility to create a Certificate Authority (CA) and maintain the X.509 public key infrastructure (PKI), but give little information on how iked actually uses the certificates that ikectl generates.

Unfortunately, ikectl has some limitations. It uses its own, self-signed certificate for the CA, preventing the use of a CA that may readily exist in the network and is known and trusted by your clients. It also does not allow for much automation like batch processing. As such, ikectl is good for for quickly setting up a VPN for a few private devices — but it is less suitable for running a corporate VPN server.

The solution is to use a standard, OpenSSL-generated PKI instead of the one from ikectl. Here's how.

Preparing the Certificate Authority

If you already have a CA on your system that can sign certificates for server authentication and that has been set up for generating a Certificate Revocation List (CRL), you can skip this step and move on to prepare the certificate management for iked. If not, you need to create a self-signed CA certificate.

First, as root, create the RSA private key with a length of 4096 bits:

#openssl genrsa -out /etc/ssl/private/vpn.key 4096

You may want to chmod 600 the resulting key file, just as an additional security measure.

The OpenSSL dialogs can be cut short by putting everything it needs to know in a configuration file and running the tool in batch mode. OpenSSL comes with a large openssl.cnf by default, but this is all you need for a VPN server:

/etc/ssl/vpn.cnf
1# OpenSSL configuration for self-signed CA
2
3[ req ]
4distinguished_name = req_distinguished_name
5x509_extensions = x509v3_ca
6prompt = no
7
8[ req_distinguished_name ]
9commonName = vpn
10
11[ x509v3_ca ]
12subjectKeyIdentifier = hash
13authorityKeyIdentifier = keyid:always, issuer:always
14basicConstraints = CA:true
15keyUsage = digitalSignature, keyCertSign, cRLSign

As the minimum, it specifies which fields to include to identify the CA afterwards, the X.509 fields for a CA, and disables the prompt for the fields' values. You can add more req_distinguished_name fields to uniquely identify your organisation, including an organisation name or locality, but basically only commonName is necessary. Of course, you can also choose a more unique name for your CA than vpn, but this one matches the OpenBSD examples on iked. You can find more information on the structure and contents of the configuration file in the OpenSSL req documentation.

With the configuration file, you can generate the self-signed certificate:

#openssl req -new -x509 -key /etc/ssl/private/vpn.key -config /etc/ssl/vpn.cnf -days 1826 -out /etc/ssl/certs/vpn.crt

You do not want your CA certificate to expire too often. Should the CA certificate get compromised, you need to generate all derived certificates anew anyway and you can easily revoke the CA certificate by removing it from /etc/ssl/certs/. So five years seems a reasonable time.

Preparing certificate management

All you need to do more to have a running Certificate Authority, is to create a file in which the serial number for the new certificates is tracked, plus an empty text file that will serve as the database for storing a list of certificates. I put them together in a new directory called db, but you can store them anywhere you want:

#mkdir /etc/ssl/db
#echo "01" > /etc/ssl/db/vpn.srl
#touch /etc/ssl/db/vpn.txt

A single configuration file can be used to generate certificate signing requests (CSR) for the server and clients, and to inform the CA on what extensions to sign. The basic configuration file looks like this:

/etc/ssl/iked.cnf
1# OpenSSL configuration for iked server and clients
2
3[ req ]
4default_bits = 4096
5distinguished_name = req_distinguished_name
6prompt = no
7
8[ req_distinguished_name ]
9commonName = server1.domain
10
11[ x509v3_server ]
12keyUsage = nonRepudiation, digitalSignature, keyEncipherment, keyAgreement
13nsCertType = server
14subjectAltName = DNS:${req_distinguished_name::commonName}
15extendedKeyUsage = serverAuth
16
17[ x509v3_client ]
18keyUsage = nonRepudiation, digitalSignature, keyEncipherment
19nsCertType = client
20subjectAltName = DNS:${req_distinguished_name::commonName}
21extendedKeyUsage = clientAuth
22
23[ ca ]
24default_ca = CA_default
25
26[ CA_default ]
27certificate = /etc/ssl/certs/vpn.crt
28private_key = /etc/ssl/private/vpn.key
29serial = /etc/ssl/db/vpn.srl
30database = /etc/ssl/db/vpn.txt
31new_certs_dir = /etc/ssl/db
32default_days = 380
33default_crl_days= 380
34default_md = sha1
35policy = CA_policy
36
37[ CA_policy ]
38commonName = supplied

It specifies the length of new keys, key usage for both the server and the clients, and provides certificate, key, and database file locations for the CA, together with the default validity for the certificates and the revocations list, the message digest function, and the policy that the CA will require commonName for identification of the server and clients — implying it will drop all other distinguished name fields.

The configuration file also specifies the host name (in line 9), which is the only thing that needs to be changed for each certificate that is to be created. You can automate setting the variable with a script if you want to; note that OpenBSD does not support passing variables from the environment through $ENV::name, so you will have to come up with some custom way to get specific parameters into the configuration file.

Creating the VPN server certificate

The next step is to create the server certificate. Another private key is needed, but this time, we generate it as part of the CSR creation process:

#openssl req -new -nodes -keyout /etc/ssl/private/server1.domain.key -config /etc/ssl/iked.cnf -out /etc/ssl/certs/server1.domain.csr

Then have the CA sign the request:

#openssl ca -batch -config /etc/ssl/iked.cnf -in /etc/ssl/certs/server1.domain.csr -extensions x509v3_server -extfile /etc/ssl/iked.cnf -out /etc/ssl/certs/server1.domain.crt

and convert the resulting certificate into X.509 PEM format:

#openssl x509 -in /etc/ssl/certs/server1.domain.crt -pubkey -out /etc/ssl/certs/server1.domain.crt

You can rm /etc/ssl/certs/server1.domain.csr afterwards, and have a look at the new server certificate with:

#openssl x509 -in /etc/ssl/certs/server1.domain.crt -noout -text

You should see the serial number, the common name of the CA as the issuer's CN, the validity interval, starting today and ending in 380 days, and the server name as the subject's CN. Within the X.509 extensions, you can find the server name as Subject Alternative Name with a DNS: prefix again, and see that the key can be used for digital signature, non-repudiation, key encipherment and key agreement as set in the config file.

You can find all the details of openssl ca and x509 on the OpenSSL ca and x509 documentation pages.

Firing up iked

The IKEv2 daemon needs the CA certificate, the server certificate, and the server's private key:

#cp /etc/ssl/certs/vpn.crt /etc/iked/ca/
#cp /etc/ssl/certs/server1.domain.crt /etc/iked/certs/
#mv /etc/ssl/private/server1.domain.key /etc/iked/private/local.key

It is good practice to chmod 600 /etc/iked/private/local.key.

You can move the certificates — like the private key — if you no longer need them in /etc/ssl, but you cannot make a symbolic link. iked does not follow those.

The VPN server further needs the CA's certificate revocation list, which can be generated directly into /etc/iked/crls:

#openssl ca -gencrl -config /etc/ssl/iked.cnf -out /etc/iked/crls/vpn.crl

You can always review the revocation list using:

#openssl crl -in /etc/ssl/iked/crls/vpn.crl -noout -text

There should obviously not be any revoked certificates yet.

Now restart the deamon to load the new certificates:

#rcctl restart iked

Creating client certificates (the iked way)

Setting up clients is very similar to generating the server certificate. Replace the server's identifier in line 9 of /etc/ssl/iked.cnf by that of the new client, for example client1.domain. Run

#openssl req -new -nodes -keyout /etc/ssl/private/client1.domain.key -config /etc/ssl/iked.cnf -out /etc/ssl/certs/client1.domain.csr

sign the request with

#openssl ca -batch -config /etc/ssl/iked.cnf -in /etc/ssl/certs/client1.domain.csr -extensions x509v3_client -extfile /etc/ssl/iked.cnf -out /etc/iked/certs/client1.domain.crt

convert the certificate to X.509 PEM format

#openssl x509 -in /etc/iked/certs/client1.domain.crt -pubkey -out /etc/iked/certs/client1.domain.crt

and rm /etc/ssl/certs/client1.domain.csr.

The client needs the X.509 certificate, the corresponding private key, and the CA certificate. You can combine the former two into a PKCS #12 archive with

#openssl pkcs12 -export -inkey /etc/ssl/private/client1.domain.key -in /etc/iked/certs/client1.domain.crt -out client1.domain.pfx -password pass:{password}

in which you replace {password} with a strong keyphrase. Copy the new client1.domain.pfx together with /etc/ssl/certs/vpn.crt to the client machine and install them according to local customs. Theoretically, you can put the CA certificate into the PKCS #12 archive too, but this does not always work on the client side — using a mobile configuration profile for iOS for example requires them to be separate.

You do not want the client's private key to linger around on your server. So at least, finish with

#rm -P /etc/ssl/private/client1.domain.key

or even better: create client certificates the PKI way.

Creating client certificates (the PKI way)

Preventing the client's private key from leaving the client machine in the first place is much better than creating one on the server, trying to transfer it securely to the client machine with password protection (only), and wiping the secret key on the server.

So copy /etc/ssl/iked.cnf to the client — there is nothing secret in the configuration file — run openssl req on the client, transfer the CSR back to the server — again, nothing secret, but you may want to verify integrity of the request, for example with a hash that you compare through a different channel, if your connection is not secure itself and there could be a man-in-the-middle — and sign on the server as usual. Once you return the client and CA certificates to the client, it has all it needs.

Creating clients the PKI way requires you to be able to generate a key and certificate signing request on the client. This is no problem on any macOS or Linux laptop, as OpenSSL is natively available, requires some additional effort on a Windows client, but may be virtually impossible on a tablet or phone. Then you are back to the iked way.

What else needs to be done

For a functioning VPN, you also need to set up /etc/iked.conf for use with X.509 certificates and prepare the packet filter in /etc/pf.conf for handling IPsec traffic. If you run into troubles, iked -dvv starts the server without daemonising it and shows many details on certificate loading, key agreement, and client authentication.