Der eigene Mailserver I: OpenSMTPD

Running your own mail server is considered to be hard. Setting it up is believed to be complex and there are definitely many things you can do wrong, leaving the doors wide open to hackers and spammers. The main reason for mail's poor reputation is probably the tools chosen. Search for hosting your own mail server and you will most probably find the likes of Postfix and Dovecot. They are good tools — I used to run them too. But they are tremendously complex. So if you need a decent no-bells-no-whistles mail server, forget them. This post is about the easiest, secure SMTP server I have ever seen; there is another post on a matching IMAP server — Running you own mail server II: Cyrus IMAP.

What this post is about — and what not

OpenSMTPD and Cyrus IMAP are the server applications needed for running your own mail server. But they are not enough. You also need various DNS and reverse DNS records, sender policy framework (SFP) records, domain-based message authentication, reporting and conformance (DMARC) records, domain-keys identified mail (DKIM) signatures, and probably spam filtering, for which Rspamd may be a good choice. These are all beyond the scope of this post. Does it mean hosting your own mail server is complicated after all? To the contrary. You can handle each of them one-by-one and there is enough information out there on how to do it. But most importantly: integrating things like spam filtering or DKIM-signatures is much easier with OpenSMTPD than it is with most other server application. So here you go.

Meet OpenSMPTD

Started within the security-focussed OpenBSD-community, OpenSMTPD is an open-source RFC-5321 SMTP daemon that supports all you need to operate a mail server — or better: a mail transfer agent (MTA). OpenSMTPD is distributed under the ISC licence, which is basically a one-clause BSD licence, thus limiting you in no way. In itself, the pedigree of OpenSMPTD and its licence are enough of a reason to make it the mail server of choice. But it gets better.

There are binary distributions for many operating systems, including (Debian) Linux, FreeBSD and obviously OpenBSD. On FreeBSD, installation is as easy as pkg install openstmpd.

After installation, the only thing you need apart from a private key and a certificate for your domain — I use example.com here — is a single configuration file. You can find all the details on the manual page online or locally from man smtpd.conf. When you want to include a spam filter like rspamd you will need a handful more, but for the basic features of a secure mail server, the configuration file has about ten lines. Read that again.

The configuration file

The concept of OpenSMTPD's smtpd.conf centers around three reserved words: listen, action, and match. listen tells OpenSMPTD what network interfaces and ports to listen to, which protocols to support, how to secure connections, and whether and how to authenticate clients. action defines what OpenSMTPD should do with the individual messages. And match glues the two together, by specifying which conditions need to be met for a message arriving at one of the interfaces to be handled by one of the actions.

The catch, if any, is that match rules relate to all interfaces listened to. You will want to use your mail server in two directions, sending out mail that you submit through the mail client you use, and receiving mail from the outside. The interface to receive mail must be open to everyone, or otherwise, you will not be reachable. The interface to send messages should only be open to you, or otherwise the bad guys out there will use it to distribute their spam and phishing mails. However, there is no way to prevent someone from delivering messages to be sent by your server — as you would on your outgoing interface — to the incoming interface. And if there is a matching rule, your server will happily distribute such messages from someone else.

You can and obviously must prevent this with the correct conditions in the match rule. Although listen, action, and match rules can be mixed freely, I believe putting all rules per type together helps to remind us that all match rules apply to all listen interfaces and not only to the one, they may be written next to — even if in practice, with the correct conditions, this will be the case. This is what it boils down to:

table virtuals "/usr/local/etc/mail/virtuals"

table submitters "/usr/local/etc/mail/submitters"

pki example_com cert "/etc/ssl/certs/example_com.crt"

pki example_com key "/etc/ssl/private/example_com.key"

listen on if0 tls pki example_com

listen on if0 port submission tls-require pki example_com auth <submitters>

action receive lmtp "/var/run/lmtp" rcpt-to virtual <virtuals>

action send relay pki example_com

match from any for domain example.com action receive

match auth from any for any action send

Receiving

Starting at line 7, OpenSTMPD is told to listen on interface if0, which is a placeholder for the actual network interface your system has. tls specifies the use of Transport-Layer Security, so that your server will be able to create secure connections to any other mail transfer agent that connects to this interface. You can force the use of TLS by replacing tls with tls-require, to make sure that no mail gets received through an unencrypted channel. By specifying tls-require verify you can go even further and deny any incoming connection from a mail server that does not authenticate itself with a valid certificate. Although most mail agents nowadays do, you will cut yourself of from older or non-compliant servers. As TLS only secures the communication channel and cannot replace end-to-end encryption of any messages, leaving TLS optional is not really a security issue. So the choice is yours.

The pki key word is used to specify the key-certificate pair of files that is to be used for secured connections. example_com is a random name; this one refers to the subject name of the actual certificate, which seems like a logical choice, but take any identifier your want. The actual certificate and key files are specified in lines 4 and 5 with the pki directive for cert and key respectively. They must contain the X.509 certificate for your server and the corresponding private key in PEM format and obviously, use the same identifier as put in line 7.

Line 7 does not specify a port number, so OpenSMTPD listens on port 25, the standard one for SMTP.

Next is the defintion of an action called receive on line 10. The name is arbitrary; receive is not a reserved word. Line 10 tells OpenSMTPD that receive is an action to deliver messages using the Local Mail Transfer Protocol (LMTP) to a socket named /var/run/lmtp. The socket is listened to by the mail delivery agent (MDA) as described in the post on Running you own mail server II: Cyrus IMAP. rcpt-to virtual lets OpenSMTPD deliver the message to a virtual mail box instead of the one for the current system user, so that you can receive messages to different mail boxes. <virtuals> refers to a table — indicated by the angle brackets — where these mail boxes are defined. The name virtuals itself again is arbitrary.

Line 1 defines what the table virtuals actually is: it is the plain-text file referenced. The file itself is nothing more than a list of mail box names, one per line, always followed by the name of the actual user that is responsible for the mail box. That will be our MDA and its user is called cyrus. So the file will look something like this:

Finally, with line 13 of smtpd.conf, we instruct OpenSMTPD when to use the action receive. The match directive applies to messages originating from any (anyone) and meant for the domain example.com. The rule will not apply to any messages that are not for our example domain, but that does not mean anything in itself. Only if there are no other match rules that tell OpenSMTPD what to do with the message, it will inform the sender that the message cannot be delivered.

You now have an MTA for receiving mail and sending the messages to the local virtual mail boxes.

Sending

The steps for configuring OpenSMTPD to send out messages are similar. Line 8 defines another listener. It uses the same interface, but this time explicitly refers to port submission, where submission corresponds to the so-called well-known port number 587 (well-known port numbers are defined in /etc/services). For submitting your own messages, you can be sure to use a TLS-capable mail client, so tls-require is the way to go; pki example_com refers to the PKI as before. The difference with the listener for receiving message is that for sending out mail, you do not want anyone to connect, but only your local users, probably exactly the ones who have a virtual mail box. The listener therefore requires authentication through the reserved word auth and refers to the table called submitters to authenticate against. The table itself is defined by reference to a local file in line 2.

The submitters table is what OpenSMTPD calls a credentials table. It contains user names — typically the mail addresses for the various accounts — followed by a space and an encoded, hashed password, like this:

You encode the password using smtpctl encrypt {password} and copy the result into the submitters file. And make sure you create some strong passwords, preferably random-generated (for example using openssl rand -base64 12).

Line 11 defines the action named send. It instructs OpenSMTPD to relay messages to another — the destination — SMTP server. Without any special instructions, OpenSMPTD will use the destination address to perform an MX-record DNS look-up to find out which is the destination mail server. pki example_com should not be a surprise anymore: it allows OpenSMTPD to authenticate as the legitimate MTA for your domain towards the receiving server (remember tls-require verify?).

The match rule in line 14 basically tells OpenSMTPD to apply the action send to messages from any for any, so that you can send mail to any address in the world. The crucial part is the auth at the beginning. It makes sure that the rule is only applied to authenticated clients and therefore, to clients connecting on port submission and contained in the list of submitters. Anyone connecting on port 25 (line 7) does not even get a chance to authenticate, and therefore, cannot use your mail server to distribute their spam.

Do not forget

There is not much more you need to consider. Set up your firewall to allow connections to ports 25 and 587; you may limit access to the submission port if appropriate. Maintain a valid certificate for your domain. Make sure OpenSMTPD is running and — if you want — starts at boot. Restart the daemon whenever you change the configuration file or one of the table files.

And don't forget to use the time you saved by not using a more complex SMTP daemon on something enjoyable.