Understanding email routing

23 March 2021

Often, it’s easy enough to understand why an email error happened. For instance, the error “incorrect authentication data” means that the email address and/or password were incorrect. It’s also easy to see how to fix the error: the user needs to correct the username and password in their email client.

Other email errors are more difficult to understand. In order to figure out why an error happened it’s useful to understand where the error popped up. This article gives a basic overview of how emails travel from A to B.

Image: a simplified overview of how emails get from A to B.

Mail User Agents

To send and receive emails you need email software, or an email client. This can be a desktop or phone app, such as Outlook or Thunderbird, or a web mail client. The technical term for an email client is Mail User Agent.

Email clients such as Outlook need to be configured. Unfortunately, some email clients use default settings that don’t work. For instance, there are email clients that don’t automatically enable authentication. Some of them also make it difficult to find the setting. Other email clients try to be overly helpful by hiding settings and making an educated guess about what settings to use. Often, that’s worse – if it guesses wrong it can be difficult to change the incorrect setting.

As an aside, this largely explains why web mail has become very popular. It pretty much “just works”. There’s nothing to configure. You simply log in and you’re ready to go.

Mail Transfer Agents

When you sent an email you effectively upload it to your mail server. This is where you might see authentication and SSL/TLS errors. Obviously, the user needs to log in with a valid username and password. And, the user’s MUA also needs to be able to log in securely, using a current TLS protocol.

If the email is accepted the server next delivers it email to the recipient. In technical terms, the email is delivered to a Mail Submission Agent (MSA), which in turn hands the email over to a Mail Transfer Agent (MTA).

If the email is sent to a user that exists on the server then the email is passed to the Local Delivery Agent (more on that shortly). If the email is for an external user then the MTA looks up the MX records of the recipient’s domain and delivers the email to the server.

Of course, the email delivery can fail. It may be that the email address doesn’t exist or that the user’s mailbox is full. Or, it may be that the receiving server refuses the email because it thinks it’s spam. If the error is temporary then the email is kept in the mail queue. This happens for instance when the recipient’s mailbox is full. The MTA will keep the email in the queue for up to a week and try to deliver it at set (and rather complicated) intervals. If the error is permanent then the email is bounced back to the sender.

cPanel servers use Exim as the MTA. Exim’s main log (/var/log/exim_mainlog) should tell you where the email was delivered to and whether or not it was accepted by the receiving mail server. If the delivery failed then it will also tell you why.

Mail Delivery Agent

The server that receives emails from an MTA is a Mail Delivery Agent (MDA). The MDA stores the email so that the recipient’s MUA can retrieve it. As you probably know, there are two protocols for retrieving emails: POP3 or IMAP.

• POP3 (the Post Office Protocol) is a one-way affair. It only retrieves emails from your inbox, and it removes emails from the server after they have been retrieved.
• IMAP (the Internet Message Access Protocol) doesn’t remove emails from the server. Instead, it changes the status of the email. This makes it possible to access all your emails on multiple MUAs, including emails in your Sent folder.

POP3 and IMAP are handled by Dovecot, and all POP3 and IMAP logins are logged in /var/log/maillog. On cPanel servers Dovecot is also the Local Delivery Agent. That’s a posh way saying that Dovecot is responsible for local deliveries. If you sent an email to an address that lives on the same server then Dovecot delivers it. The MTA only gets involved if an email needs to be delivered to an external server.

It’s useful to understand this. People often test their SMTP settings by sending an email to their own email address. That test is somewhat useful, as it tells you if you can connect to the mail server. However, it doesn’t tell you if the MTA can deliver emails to an external mail server. The email never leaves the server. Instead, Dovecot does a local delivery.

The Mail Exchanger

cPanel servers have a Mail Exchanger setting that defines if mail for a domain should be delivered locally or remotely. In most cases, cPanel knows what the setting should be. However, things get complicated when both the DNS and email for a domain are managed elsewhere.

To illustrate the point, let’s imagine you host the website example.com. You use G Suite for your mail@example.com address, and the DNS is managed by Cloudflare. Can you guess what happens with an email sent via your website?

The answer is: it depends. If the website relays emails via a specific SMTP server then the example.com server never sees the email. However, if you use the default cPanel mailbox to relay emails then the server needs to know where to send the email.

This is where the Mail Exchanger comes in. If the Mail Exchanger is set to “remote” then Exim delivers the email. It looks up the MX records for example.com and delivers the email to G Suite. This is what should happen. However, if the Mail Exchanger is set to “local” then Dovecot takes responsibility. It will try to deliver the email locally. As the email address doesn’t exist locally the email will bounce. The bounce email is sent to the default mailbox.

There are a few lessons here. The first is to not rely on the default mailbox to relay form submissions. It’s convenient, but it’s much better to relay email via a specific SMTP server.

Secondly, it’s often useful to manage the DNS on the server that’s hosting your website. It’s not necessary, but it often simplifies things.

And finally, it’s worth checking the default mailbox from time to time. Email in the mailbox often point to a problem, either with your email or services such as cron jobs.