Building an OpenBSD VPN server with iked and OpenSSL

OpenBSD's IKEv2 daemon iked is great to set up a VPN server that uses X.509 certificates, so mobile devices can connect securely through their respective built-in VPN clients. The OpenBSD iked documentation and VPN FAQ refer to the ikectl utility to create a Certificate Authority (CA) and maintain the X.509 public key infrastructure (PKI), but give little information on how iked actually uses the certificates that ikectl generates.

Unfortunately, ikectl has some limitations. It uses its own, self-signed certificate for the CA, preventing the use of a CA that may readily exist in the network and is known and trusted by your clients. It also does not allow for much automation like batch processing. As such, ikectl is good for for quickly setting up a VPN for a few private devices — but it is less suitable for running a corporate VPN server.

The solution is to use a standard, OpenSSL-generated PKI instead of the one from ikectl. Here's how.

Preparing the Certificate Authority

If you already have a CA on your system that can sign certificates for server authentication and that has been set up for generating a Certificate Revocation List (CRL), you can skip this step and move on to prepare the certificate management for iked. If not, you need to create a self-signed CA certificate.

First, as root, create the RSA private key with a length of 4096 bits:

You may want to chmod 600 the resulting key file, just as an additional security measure.

The OpenSSL dialogs can be cut short by putting everything it needs to know in a configuration file and running the tool in batch mode. OpenSSL comes with a large openssl.cnf by default, but this is all you need for a VPN server:

# OpenSSL configuration for self-signed CA

[ req ]

distinguished_name = req_distinguished_name

x509_extensions = x509v3_ca

prompt = no

[ req_distinguished_name ]

commonName = vpn

[ x509v3_ca ]

subjectKeyIdentifier = hash

authorityKeyIdentifier = keyid:always, issuer:always

basicConstraints = CA:true

keyUsage = digitalSignature, keyCertSign, cRLSign

As the minimum, it specifies which fields to include to identify the CA afterwards, the X.509 fields for a CA, and disables the prompt for the fields' values. You can add more req_distinguished_name fields to uniquely identify your organisation, including an organisation name or locality, but basically only commonName is necessary. Of course, you can also choose a more unique name for your CA than vpn, but this one matches the OpenBSD examples on iked. You can find more information on the structure and contents of the configuration file in the OpenSSL req documentation.

With the configuration file, you can generate the self-signed certificate:

You do not want your CA certificate to expire too often. Should the CA certificate get compromised, you need to generate all derived certificates anew anyway and you can easily revoke the CA certificate by removing it from /etc/ssl/certs/. So five years seems a reasonable time.

Preparing certificate management

All you need to do more to have a running Certificate Authority, is to create a file in which the serial number for the new certificates is tracked, plus an empty text file that will serve as the database for storing a list of certificates. I put them together in a new directory called db, but you can store them anywhere you want:

A single configuration file can be used to generate certificate signing requests (CSR) for the server and clients, and to inform the CA on what extensions to sign. The basic configuration file looks like this:

# OpenSSL configuration for iked server and clients

[ req ]

default_bits = 4096

distinguished_name = req_distinguished_name

prompt = no

[ req_distinguished_name ]

commonName = server1.domain

[ x509v3_server ]

keyUsage = nonRepudiation, digitalSignature, keyEncipherment, keyAgreement

nsCertType = server

subjectAltName = DNS:${req_distinguished_name::commonName}

extendedKeyUsage = serverAuth

[ x509v3_client ]

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

nsCertType = client

subjectAltName = DNS:${req_distinguished_name::commonName}

extendedKeyUsage = clientAuth

[ ca ]

default_ca = CA_default

[ CA_default ]

certificate = /etc/ssl/certs/vpn.crt

private_key = /etc/ssl/private/vpn.key

serial = /etc/ssl/db/vpn.srl

database = /etc/ssl/db/vpn.txt

new_certs_dir = /etc/ssl/db

default_days = 380

default_crl_days= 380

default_md = sha1

policy = CA_policy

[ CA_policy ]

commonName = supplied

It specifies the length of new keys, key usage for both the server and the clients, and provides certificate, key, and database file locations for the CA, together with the default validity for the certificates and the revocations list, the message digest function, and the policy that the CA will require commonName for identification of the server and clients — implying it will drop all other distinguished name fields.

The configuration file also specifies the host name (in line 9), which is the only thing that needs to be changed for each certificate that is to be created. You can automate setting the variable with a script if you want to; note that OpenBSD does not support passing variables from the environment through $ENV::name, so you will have to come up with some custom way to get specific parameters into the configuration file.

Creating the VPN server certificate

The next step is to create the server certificate. Another private key is needed, but this time, we generate it as part of the CSR creation process:

Then have the CA sign the request:

and convert the resulting certificate into X.509 PEM format:

You can rm /etc/ssl/certs/server1.domain.csr afterwards, and have a look at the new server certificate with:

You should see the serial number, the common name of the CA as the issuer's CN, the validity interval, starting today and ending in 380 days, and the server name as the subject's CN. Within the X.509 extensions, you can find the server name as Subject Alternative Name with a DNS: prefix again, and see that the key can be used for digital signature, non-repudiation, key encipherment and key agreement as set in the config file.

You can find all the details of openssl ca and x509 on the OpenSSL ca and x509 documentation pages.

Firing up iked

The IKEv2 daemon needs the CA certificate, the server certificate, and the server's private key:

It is good practice to chmod 600 /etc/iked/private/local.key.

You can move the certificates — like the private key — if you no longer need them in /etc/ssl, but you cannot make a symbolic link. iked does not follow those.

The VPN server further needs the CA's certificate revocation list, which can be generated directly into /etc/iked/crls:

You can always review the revocation list using:

There should obviously not be any revoked certificates yet.

Now restart the deamon to load the new certificates:

Creating client certificates (the iked way)

Setting up clients is very similar to generating the server certificate. Replace the server's identifier in line 9 of /etc/ssl/iked.cnf by that of the new client, for example client1.domain. Run

sign the request with

convert the certificate to X.509 PEM format

and rm /etc/ssl/certs/client1.domain.csr.

The client needs the X.509 certificate, the corresponding private key, and the CA certificate. You can combine the former two into a PKCS #12 archive with

in which you replace {password} with a strong keyphrase. Copy the new client1.domain.pfx together with /etc/ssl/certs/vpn.crt to the client machine and install them according to local customs. Theoretically, you can put the CA certificate into the PKCS #12 archive too, but this does not always work on the client side — using a mobile configuration profile for iOS for example requires them to be separate.

You do not want the client's private key to linger around on your server. So at least, finish with

or even better: create client certificates the PKI way.

Creating client certificates (the PKI way)

Preventing the client's private key from leaving the client machine in the first place is much better than creating one on the server, trying to transfer it securely to the client machine with password protection (only), and wiping the secret key on the server.

So copy /etc/ssl/iked.cnf to the client — there is nothing secret in the configuration file — run openssl req on the client, transfer the CSR back to the server — again, nothing secret, but you may want to verify integrity of the request, for example with a hash that you compare through a different channel, if your connection is not secure itself and there could be a man-in-the-middle — and sign on the server as usual. Once you return the client and CA certificates to the client, it has all it needs.

Creating clients the PKI way requires you to be able to generate a key and certificate signing request on the client. This is no problem on any macOS or Linux laptop, as OpenSSL is natively available, requires some additional effort on a Windows client, but may be virtually impossible on a tablet or phone. Then you are back to the iked way.

What else needs to be done

For a functioning VPN, you also need to set up /etc/iked.conf for use with X.509 certificates and prepare the packet filter in /etc/pf.conf for handling IPsec traffic. If you run into troubles, iked -dvv starts the server without daemonising it and shows many details on certificate loading, key agreement, and client authentication.