SMTP AUTH explained: PLAIN, LOGIN, and what a 535 actually means

Most developers meet SMTP authentication as three environment variables and an error message. MAIL_USERNAME, MAIL_PASSWORD, and — some hours later — 535 5.7.8 Authentication credentials invalid. The framework wraps the protocol so completely that when it fails there is nothing to look at.

This article removes the wrapper. It shows the full exchange between a client and a submission server, explains the two authentication mechanisms you will actually encounter, and reads the reply codes so that the next 535 is a diagnosis rather than a mystery. Everything here can be reproduced by hand against any submission server — including ours, which is what the transcripts were taken from.

The shape of a session

SMTP is a line protocol. The client sends a command, the server replies with a three-digit code and some text. Authentication is one command in a sequence that looks like this on port 587:

S: 220 smtp.mailclutch.com ESMTP
C: EHLO app.example.com
S: 250-smtp.mailclutch.com
S: 250-SIZE 37748736
S: 250-STARTTLS
S: 250-AUTH PLAIN LOGIN
S: 250 8BITMIME
C: STARTTLS
S: 220 2.0.0 Ready to start TLS
   ...TLS handshake...
C: EHLO app.example.com
S: 250-smtp.mailclutch.com
S: 250-AUTH PLAIN LOGIN
S: 250 8BITMIME
C: AUTH PLAIN AG1jX2p...
S: 235 2.7.0 Authentication successful
C: MAIL FROM:<noreply@example.com>
S: 250 2.0.0 OK
C: RCPT TO:<anyone@example.com>
S: 250 2.1.5 OK
C: DATA
S: 354 End data with <CR><LF>.<CR><LF>
C: ...the message...
C: .
S: 250 2.0.0 OK: queued
C: QUIT
S: 221 2.0.0 Bye

Four things to notice, because each one is a place a real client goes wrong.

EHLO is sent twice. The first EHLO discovers what the server offers in clear text. After STARTTLS, the connection is a new one as far as the protocol is concerned, and the client must EHLO again — the server may advertise different extensions inside TLS. A client that skips the second EHLO and jumps to AUTH gets a 503 Bad sequence of commands.

The server tells you which mechanisms it supports. 250-AUTH PLAIN LOGIN means exactly those two. A client insisting on CRAM-MD5 or XOAUTH2 against this server gets 504 5.5.4 Unrecognized authentication type.

SIZE is the maximum message. 37748736 is 36 MiB. A client that ignores it and streams a 50 MB attachment gets a 552 after the upload, not before.

Reply codes carry an enhanced status. 235 2.7.0 — the 2.7.0 is the enhanced code from RFC 3463, and the second number is the subject: 7 is security/policy, which is why every authentication result starts x.7.y.

Why STARTTLS comes before AUTH

Both mechanisms below send the password in a trivially reversible encoding. Base64 is not encryption; it is a way of getting binary safely through a text protocol. Anyone who can read the connection can read the password.

So a submission server has a choice: refuse AUTH until the connection is encrypted, or accept it in the clear and trust the operator's network. Most public servers refuse — you will see 530 5.7.0 Must issue a STARTTLS command first. Some sandbox and internal servers allow it, because their clients are inside a VPC or on a developer laptop and requiring TLS would break every telnet-style test.

Either way: use TLS. Port 465 gives you TLS from the first byte; port 587 gives you it after STARTTLS. The difference between the two is the subject of its own article. What matters here is that a client configured for port 465 that tries to speak clear-text SMTP first will hang or see garbage — the server is waiting for a TLS handshake, and the client is waiting for a 220 that will never arrive.

AUTH PLAIN

PLAIN (RFC 4616) sends everything in one line. The payload is three fields separated by NUL bytes:

[authzid] NUL authcid NUL password

authzid — the identity to act as — is almost always empty. authcid is the username. So for username mc_abcdefghijklmnop and password pw_qrstuvwxyz234567:

printf '\0mc_abcdefghijklmnop\0pw_qrstuvwxyz234567' | base64
# AG1jX2FiY2RlZmdoaWprbG1ub3AAcHdfcXJzdHV2d3h5ejIzNDU2Nw==

And the exchange:

C: AUTH PLAIN AG1jX2FiY2RlZmdoaWprbG1ub3AAcHdfcXJzdHV2d3h5ejIzNDU2Nw==
S: 235 2.7.0 Authentication successful

The client may also send AUTH PLAIN with no argument, wait for a 334 (empty challenge), and then send the payload on its own line. Both forms are valid; the one-line form saves a round trip.

The leading NUL is the empty authzid. Forgetting it is the classic hand-rolled bug: the server parses the username as the authzid, the password as the username, and an empty string as the password — and you get a 535 for a pair that is perfectly correct.

AUTH LOGIN

LOGIN predates PLAIN, was never standardized beyond an expired draft, and is still everywhere — Outlook, older PHP mailers, some MTAs, swaks --auth LOGIN. It sends the username and the password as two separate base64 responses to two prompts:

C: AUTH LOGIN
S: 334 VXNlcm5hbWU6           ← base64("Username:")
C: bWNfYWJjZGVmZ2hpamtsbW5vcA==
S: 334 UGFzc3dvcmQ6           ← base64("Password:")
C: cHdfcXJzdHV2d3h5ejIzNDU2Nw==
S: 235 2.7.0 Authentication successful

The prompts are decorative — the server is not asking a question, it is acknowledging state. A client may also send the username with the initial command (AUTH LOGIN bWNf…) and skip the first prompt; servers that implement LOGIN properly accept both.

There is no security difference between PLAIN and LOGIN. Both are base64 over whatever transport you gave them. The practical difference is that PLAIN is standardized and LOGIN is a compatibility shim, so a server that offers only PLAIN will lose some clients, and a server that offers only LOGIN will confuse some libraries. Offer both — which is what we do.

Reading the reply

535 5.7.8: what the server is actually saying

A 535 means the server received a username and a password, examined them, and decided no. It does not mean the pair in your .env is wrong — it means the pair that arrived is wrong. Between the two are several places for a mangling to happen:

• Whitespace. A trailing space in an environment variable, a \r from a Windows-edited file, a newline captured by $(cat secret). Base64 encodes the whitespace faithfully; the server sees pw_qrstuvwxyz234567 and rejects it.
• Quoting. MAIL_PASSWORD="pw_abc" in a .env file that is later sourced by a shell without stripping quotes yields a password that starts with ".
• Case. Some clients upper-case or lower-case the username. Some servers are case-sensitive. Choose credentials with no case to preserve, if you can — ours are lowercase-only for exactly this reason.
• Wrong mechanism, right pair. A hand-rolled PLAIN missing its leading NUL. A LOGIN implementation sending username and password in one response.
• Rotated or expired. The pair was valid; it is not any more. Sandbox credentials in particular have lifetimes.
• The pool. A stale connection reused from a pool after credentials were rotated.

The efficient diagnosis is to remove the client from the picture and send the exact bytes yourself.

Reproducing by hand

Two tools, both standard.

openssl s_client gives you a raw session. For port 587 with STARTTLS:

openssl s_client -connect smtp.mailclutch.com:587 -starttls smtp -crlf -quiet

Then type:

EHLO test
AUTH PLAIN AG1jX2FiY2RlZmdoaWprbG1ub3AAcHdfcXJzdHV2d3h5ejIzNDU2Nw==

For port 465, drop -starttls smtp — the connection is TLS from the start.

swaks does the whole thing and prints the transcript:

swaks --server smtp.mailclutch.com:587 --tls \
      --auth PLAIN --auth-user mc_abcdefghijklmnop --auth-password pw_qrstuvwxyz234567 \
      --from noreply@example.com --to anyone@example.com \
      --header "Subject: swaks" --body "hello"

Swap --auth PLAIN for --auth LOGIN to see the other mechanism. If swaks gets a 235 and your application gets a 535 with the same pair, the pair is fine and the application is mangling it — go look at whitespace and quoting. If swaks also gets a 535, the pair is wrong or dead, and no amount of application debugging will help.

Where the recipient comes in

One thing that surprises people about a sandbox: the RCPT TO address does not matter. Authentication picks the inbox. Whatever address the message is sent to — a real customer, anyone@example.com, a typo — it lands in the inbox the credentials belong to and goes nowhere else. That is what makes it safe to point a staging environment with a copy of production data at it: the addresses in the database are never contacted, because the credentials, not the addresses, decide where mail goes.

Which is also why authentication is the thing to get right first. Once the 235 is there, everything downstream — the captured message, its headers, its spam score, its authentication results — follows.

In one screen

• EHLO, STARTTLS, EHLO again, AUTH. In that order, on 587. On 465, TLS first, then EHLO, AUTH.
• PLAIN is base64(NUL user NUL pass) in one line. LOGIN is base64(user) then base64(pass) in two.
• Neither is secure without TLS. Both are fine with it.
• 235 is yes. 535 is "the bytes I received are wrong" — check what you sent, not what you meant to send.
• 454 is "not you, not now". 530 is "not yet". 504 is "not like that".
• When in doubt, swaks. It will tell you which side is lying.