The journey of an email: SMTP, submission and IMAP

Objective

Understand what happens between the click on "Send" and the arrival of the message in a mailbox, know how to speak SMTP to a server by hand, and read the history of a message in its headers.

Key points

• Four roles: the MUA (Thunderbird, Outlook, a phone) writes; the MSA receives authenticated submissions (port 587 or 465); the MTA (Postfix) carries mail from server to server (port 25); the MDA (Dovecot) stores it in the mailbox, which the client reads over IMAP (port 993).
• To find the recipient's server, the MTA asks DNS for the domain's MX record.
• An SMTP session fits in five commands: EHLO, MAIL FROM, RCPT TO, DATA, QUIT.
• The envelope (MAIL FROM, RCPT TO) is used for transport; the headers (From:, To:, Subject:) are what the user sees. They can be different.
• Each server on the way adds a Received: line at the top: so you read the history from bottom to top.

1. The players and the ports

 workstation at the Fes branch                 Internet                       ml-mail-01.masterlinux.lab
┌───────────────┐  587 (STARTTLS)   ┌──────────────┐   25 (SMTP)   ┌─────────────────────────────┐
│ MUA           │ ── + password ───► │ MSA/MTA      │ ─────────────► │ MTA Postfix ─► MDA Dovecot │
│ Thunderbird   │                   │ smtp.gmail…  │  (domain's MX)   │        Maildir mailbox        │
└───────────────┘                   └──────────────┘                  └──────────────┬──────────────┘
                                                                         IMAP 993 (TLS) │
                                                                  support team's phone ◄┘

Port 25 is used between servers. Many ISPs (in Morocco as elsewhere) block outgoing port 25 on home connections: a user who sends mail must go through 587 or 465, authenticated.

2. Finding the recipient's server: the MX

dig MX masterlinux.lab +short
# 10 mail.masterlinux.lab.          ← priority 10 (lower = preferred)
# 20 mail2.masterlinux.lab.         ← backup
dig A mail.masterlinux.lab +short
# 203.0.113.25

Without an MX, the sender tries the A record of the domain itself (fallback rule of RFC 5321). A domain that never receives mail can publish a null MX: @ IN MX 0 ..

3. An SMTP session by hand

On a real server, you test with swaks (package swaks) or by hand with nc or openssl s_client:

swaks --to support@masterlinux.lab --from ops@masterlinux.lab --server 127.0.0.1 --header "Subject: Test"
openssl s_client -connect mail.masterlinux.lab:25 -starttls smtp -crlf -quiet

The dialogue, line by line (C = client, S = server):

S: 220 mail.masterlinux.lab ESMTP Postfix              ← banner
C: EHLO ml-poste-01
S: 250-mail.masterlinux.lab
S: 250-SIZE 10240000                                    ← maximum size accepted
S: 250-STARTTLS                                         ← encryption available
S: 250 8BITMIME
C: MAIL FROM:<ops@masterlinux.lab>                      ← envelope: sender (return address)
S: 250 2.1.0 Ok
C: RCPT TO:<support@masterlinux.lab>                    ← envelope: recipient
S: 250 2.1.5 Ok
C: DATA
S: 354 End data with <CR><LF>.<CR><LF>
C: From: ops@masterlinux.lab                            ← headers (what the user sees)
C: Subject: Premier envoi
C:                                                      ← empty line: end of headers
C: Bonjour.
C: .                                                    ← a dot on its own: end of message
S: 250 2.0.0 Ok: queued as 3F2A61C0D2                   ← ID in the queue
C: QUIT
S: 221 2.0.0 Bye

Read reply codes by their first digit: 2xx success, 3xx continue, 4xx temporary failure (the sender will retry), 5xx permanent failure (returned to the sender). The enhanced code (5.1.1, 4.7.1…) gives the cause: 5.1.1 = unknown mailbox, 5.7.1 = refused by a security rule.

In the labs of this course

The lab machine has no network. It runs ml-smtpd, the lab's training SMTP server, which reads the real /etc/postfix/main.cf file and answers with Postfix messages. It listens on 127.0.0.1:25 and :587, delivers local domains in Maildir format and writes /var/log/mail.log.

# 1. like sendmail -bs: an SMTP session on standard input
printf 'EHLO lab\r\nMAIL FROM:<ops@masterlinux.lab>\r\nRCPT TO:<support@masterlinux.lab>\r\nQUIT\r\n' | ml-smtpd --stdio

# 2. over the local network, with Python (set_debuglevel shows the whole dialogue)
python3 -c 'import smtplib; s=smtplib.SMTP("127.0.0.1", 25); s.set_debuglevel(1); s.noop(); s.quit()'

# 3. by hand with bash (/dev/tcp redirection)
exec 3<>/dev/tcp/127.0.0.1/25; head -1 <&3; printf 'QUIT\r\n' >&3; cat <&3

4. Envelope versus headers

The envelope is like the address written on a postal letter; the headers are the letterhead inside. The postman only reads the envelope.

This is what lets a mailing platform receive bounces (bounces@…) while showing the shop's address. It is also what spoofers exploit: hence SPF, DKIM and DMARC (level 4).

5. Reading the Received lines

Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45])   ← 3. received by OUR server
        by mail.masterlinux.lab (Postfix) with ESMTPS id 4B7F21C0A2 …
Received: by mail-ed1-f45.google.com with SMTP id …                                  ← 2. internal Google transfer
Received: from [192.168.1.23] ([41.250.88.17]) by smtp.gmail.com with ESMTPSA …        ← 1. sent by the client

• ESMTPS = encrypted (TLS), ESMTPA = authenticated, ESMTPSA = both.
• In from X ([IP]), the name X is declared by the client (it can lie); the IP in brackets is seen by the server (reliable if that server is trusted).
• 192.168.1.23 is the private address of the computer behind the home router; 41.250.88.17 is the public address seen by Gmail.

grep -n '^Received:' /root/recu.eml            # the first lines
awk '/^Received:/,/;/' /root/recu.eml          # each full Received (up to the date)

Real-world case

One Monday morning, the support team of the MasterLinux shop gets a complaint: "I never received your reply". Reading the headers of the sent message, the administrator sees that the support workstation was sending with ESMTP (no A) to port 25 of an old server: the message left without authentication and Gmail filed it as spam. Switching the mail client to 587 + STARTTLS + authentication fixed the problem. Reading the Received lines before touching the configuration saves hours.

Common pitfalls

• Mixing up 25 and 587: a mail client set to 25 works at the office, then fails on 4G (port blocked by the mobile operator).
• Trusting the From: header: it is written by the sender, just like the name after from in Received. Only the IPs added by your own servers are reliable.
• Forgetting \r\n in manual SMTP: some servers reject bare line feeds (\n) — this is even the basis of the 2023-2024 "SMTP smuggling" attack, fixed in Postfix with smtpd_forbid_bare_newline.
• Reading Received lines from top to bottom: the first one is the most recent.

Lab

On the lab machine, ml-smtpd plays the role of ml-mail-01 for the masterlinux.lab domain. You will: send a first message over SMTP, send a message whose envelope differs from its header, trigger a 550 5.1.1 rejection, find the originating IP of a real message, and list the mail ports.

Quiz

1. Which DNS record tells you the server that receives mail for a domain?
2. What is the difference between MAIL FROM and the From: header?
3. What does a reply code that starts with 4 mean?
4. Why does a user send on 587 and not on 25?
5. In which order do you read the Received: lines?