import AffiliateLink from ’../../components/AffiliateLink.astro’;

Self-Host Docker Mailserver Without Becoming a Spam Relay

Running your own mail server is still possible. It is also the quickest way to learn that email is less an app than a global reputation system held together with DNS records and suspicion.

I would not self-host email for every project. For a personal domain, a small team, or a lab where privacy and control matter, Docker Mailserver is one of the few routes that stays understandable. For transactional SaaS email, use a specialist provider and keep your weekends.

The boring decision that saves pain

Before touching Docker, get the server and domain basics right. Use a VPS with a static IPv4 address, ask the provider for reverse DNS, and do not put this on a residential connection. Port 25 is frequently blocked, and a bad IP reputation can make an otherwise perfect setup useless.

You need these DNS records before you expect reliable delivery:

A record: mail.example.com points to your VPS IPv4 address.
MX record: example.com sends mail to mail.example.com.
PTR record: your VPS IP resolves back to mail.example.com.
SPF record: authorizes your server to send for the domain.
DKIM record: publishes the key your server signs outgoing messages with.
DMARC record: tells receivers what to do when SPF or DKIM fails.

PTR is the one people skip because it lives at the VPS provider. Receivers notice. Make the hostname, PTR, and SMTP greeting agree or expect mail to land in spam.

A minimal Docker Mailserver stack

Docker Mailserver bundles Postfix, Dovecot, spam filtering, and the glue needed to run them in a single maintained image. That is much saner than assembling five mail components from random Compose examples written three years ago.

Create a working directory:

mkdir -p ~/docker-mailserver/docker-data/dms/config
cd ~/docker-mailserver

Create compose.yaml:

services:
  mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    container_name: mailserver
    hostname: mail
    domainname: example.com
    env_file: mailserver.env
    ports:
      - "25:25"
      - "143:143"
      - "465:465"
      - "587:587"
      - "993:993"
    volumes:
      - ./docker-data/dms/mail-data:/var/mail/
      - ./docker-data/dms/mail-state:/var/mail-state/
      - ./docker-data/dms/mail-logs:/var/log/mail/
      - ./docker-data/dms/config:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    stop_grace_period: 1m
    cap_add:
      - NET_ADMIN

Then create mailserver.env beside it:

OVERRIDE_HOSTNAME=mail.example.com
SSL_TYPE=letsencrypt
PERMIT_DOCKER=none
ENABLE_SPAMASSASSIN=1
ENABLE_CLAMAV=0
ENABLE_FAIL2BAN=1
POSTMASTER_ADDRESS=postmaster@example.com
ONE_DIR=1

Replace example.com everywhere. Start the container with docker compose up -d, then inspect it with docker compose logs -f mailserver. Do not open webmail or IMAP to the world until the server can receive a test message and its certificate is valid.

Create accounts and signing keys

Docker Mailserver ships management commands inside the container. Create a mailbox first:

docker exec -ti mailserver setup email add you@example.com

Generate a DKIM key:

docker exec -ti mailserver setup config dkim keysize 2048 selector mail domain example.com

The command writes the public DNS value into the mounted configuration directory. Publish it as a TXT record named mail._domainkey.example.com. Do not paste a made-up DKIM key from a blog post. It must be the public key generated by your own server.

A reasonable initial SPF record is:

example.com. TXT "v=spf1 mx -all"

Start DMARC in reporting mode while you test:

_dmarc.example.com. TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com; adkim=s; aspf=s"

After you have checked real deliveries for a few weeks, move from p=none to p=quarantine or p=reject. DMARC is not a badge you install once. It is a policy, and an aggressive policy will reject your own mail if your records are wrong.

Test delivery like a skeptic

Send mail to a Gmail, Outlook, and Proton Mail address. Check message headers, not just whether the message appeared. You want SPF, DKIM, and DMARC to pass, and you want the received SMTP hostname to match your DNS.

Also test inbound mail from outside your network. A Compose stack that works on localhost proves almost nothing. Firewalls, provider port blocks, and broken MX records only show up from the public internet.

Useful checks:

dig +short MX example.com
dig +short TXT example.com
dig +short TXT mail._domainkey.example.com
dig +short -x YOUR_SERVER_IP

For a more complete external report, use a mail testing address such as the one provided by Mail-Tester. Treat its score as a debugging signal, not a promise that every recipient will trust a fresh server.

Keep the attack surface small

Mail is a public service, so the default should be boring and restrictive. Allow SMTP submission on 587 with authentication, prefer encrypted IMAP on 993, and avoid exposing the administrative Docker socket to anything.

Never configure an open relay. Docker Mailserver’s PERMIT_DOCKER=none is intentional: containers on a shared Docker network should not automatically gain permission to relay mail through your server.

I also recommend a VPN for server administration. NordVPN is not a replacement for mail authentication, but it is useful when you need to administer a VPS from networks you do not control.

Backups are part of the mail server

Your mail lives in mail-data, while account state and configuration live in the other mounted directories. Back up all of them together, encrypt the backup, and test restoring it somewhere isolated. A backup that has never been restored is a theory.

For a small installation, stop the container briefly and archive the data:

docker compose stop mailserver
tar -czf mailserver-backup-$(date +%F).tar.gz docker-data
docker compose start mailserver

That is deliberately simple. For anything business-critical, use filesystem snapshots or a backup tool such as Restic and test mailbox recovery regularly. Also retain DNS zone exports. Rebuilding a server is manageable; remembering every record from memory is not.

The honest verdict

Self-hosting a mailbox is worth it when you want ownership of a personal domain and are willing to operate it. Docker Mailserver makes the application layer approachable, but it cannot manufacture IP reputation, reliable reverse DNS, or operational discipline.

If your product depends on password resets and invoices reaching strangers, outsource outbound transactional mail. If you want a private mailbox and enjoy knowing exactly where it lives, start with one domain, monitor it closely, and earn complexity one mailbox at a time.

Next, configure an external uptime check for SMTP and IMAPS, then send real test messages every few months. Email fails quietly right until it matters.