cat welcome.txt

 ____  _                      _   _   _    _
/ ___|| |_ _____   _____     / \ | |_| | _(_)_ __  ___
\___ \| __/ _ \ \ / / _ \   / _ \| __| |/ / | '_ \/ __|
 ___) | ||  __/\ V /  __/  / ___ \ |_|   <| | | | \__ \
|____/ \__\___| \_/ \___| /_/   \_\__|_|\_\_|_| |_|___/

 ___  ___ _ __   __| (_)_ __   __ _    ___ _ __ ___   __ _(_) |
/ __|/ _ \ '_ \ / _` | | '_ \ / _` |  / _ \ '_ ` _ \ / _` | | |
\__ \  __/ | | | (_| | | | | | (_| | |  __/ | | | | | (_| | | |
|___/\___|_| |_|\__,_|_|_| |_|\__, |  \___|_| |_| |_|\__,_|_|_|
                              |___/

Transcript and online tools at class.wordtothewise.com

Good afternoon, everyone!

We're going to be sending some email today.

That's really not that complicated, so we should have plenty of time to spare for questions. Interrupt me if you have a question or if something isn't clear.

cat follow.txt

  __       _ _                       _
 / _| ___ | | | _____      __   __ _| | ___  _ __   __ _
| |_ / _ \| | |/ _ \ \ /\ / /  / _` | |/ _ \| '_ \ / _` |
|  _| (_) | | | (_) \ V  V /  | (_| | | (_) | | | | (_| |
|_|  \___/|_|_|\___/ \_/\_/    \__,_|_|\___/|_| |_|\__, |
                                                   |___/

Go to class.wordtothewise.com and click on "transcript me"

If you'd like to see the same stuff that's on screen in your browser, so you can copy and paste things, go to class.wordtothewise.com and click on "transcript me".

Email is old. Really old. Like fifty years old - the first implementations that you'd recognize as kinda like the way we send email today were developed in 1972.

At the very beginning you wouldn't have a mail client. You'd connect to the remote machine where the recipient was and type the commands needed to send a message by hand.

Lots of extra things have been layered on top of this - attachments, rich text, authentication, encryption, bounce management - but the core protocol is still simple enough to run by hand.

So let's do that!

If you'd like to run the commands yourself - and I know all y'all do - then you can go to shell.wordtothewise.com and get a shell with the commands we're using installed.

cat shell.txt

     _          _ _
 ___| |__   ___| | |
/ __| '_ \ / _ \ | |
\__ \ | | |  __/ | |
|___/_| |_|\___|_|_|


Go to shell.wordtothewise.com in your browser to get a shell
with dig, nc and swaks installed.

I'm going to send mail to one of my servers. If you're following along you might want to do the same, rather than sending to your own email address - I'll explain why later when we talk about SWAKS.

cat recipient.txt

 _            _                          _       _            _
| |_   _  ___| | ___   _   _ __ ___  ___(_)_ __ (_) ___ _ __ | |_
| | | | |/ __| |/ / | | | | '__/ _ \/ __| | '_ \| |/ _ \ '_ \| __|
| | |_| | (__|   <| |_| | | | |  __/ (__| | |_) | |  __/ | | | |_
|_|\__,_|\___|_|\_\\__, | |_|  \___|\___|_| .__/|_|\___|_| |_|\__|
                   |___/                  |_|

is discard@reject.wordtothewise.com


Our lucky recipient is discard@reject.wordtothewise.com.

First we need to work out which server will accept mail for reject.wordtothewise.com. The DNS knows everything about hostnames, so we're going to use dig to ask.

figlet dig

     _ _
  __| (_) __ _
 / _` | |/ _` |
| (_| | | (_| |
 \__,_|_|\__, |
         |___/

Dig is a tool for querying DNS. Useful for looking up hostnames, finding mailservers, checking SPF records and all sorts of other things. There are some nice web-based DNS tools you can use that'll give prettier results, but dig is quick and easy.

Other tools you might have for dns are nslookup or host. nslookup is janky, avoid it if you can. host is great, but I prefer dig.

dig +short reject.wordtothewise.com mx

The plus-short asks dig to just give us the result, not all the other information about the DNS query. The other stuff is really useful when you're diagnosing DNS issues but rather scary if you're not used to it. If you were in the DNSSEC session this morning you saw a lot of the scary stuff.

MX means we're asking for the mailserver for that domain. It stands for Mail eXchanger, so I've no idea why it's MX and not ME.

return

10 stuntmx.wordtothewise.com.

(We're going to ignore that "10" for now. Remember it for later.)

So the server stuntmx.wordtothewise.com wants our mail.

We use the Internet for a lot of different things.

cat cats.txt


                      /^--^\     /^--^\     /^--^\
                      \____/     \____/     \____/
                     /      \   /      \   /      \
                    |        | |        | |        |
                     \__  __/   \__  __/   \__  __/
|^|^|^|^|^|^|^|^|^|^|^|^\ \^|^|^|^/ /^|^|^|^|^\ \^|^|^|^|^|^|^|^|^|^|^|^|
| | | | | | | | | | | | |\ \| | |/ /| | | | | | \ \ | | | | | | | | | | |
| | | | | | | | | | | | / / | | |\ \| | | | | |/ /| | | | | | | | | | | |
| | | | | | | | | | | | \/| | | | \/| | | | | |\/ | | | | | | | | | | | |
#########################################################################
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

The Internet is for Cat Pictures. Not that other thing.

                        ,d               
                        88               
 ,adPPYba, ,adPPYYba, MM88MMM ,adPPYba,  
a8"     "" ""     `Y8   88    I8[    ""  
8b         ,adPPPPP88   88     `"Y8ba,   
"8a,   ,aa 88,    ,88   88,   aa    ]8I  
 `"Ybbd8"' `"8bbdP"Y8   "Y888 `"YbbdP"'  

A single server can do a bunch of things at the same time. It might be both a webserver and a mailserver. We need to tell it what we want it to do. We tell it which protocol we're using by which port we use to talk to it. A port is just a number - webservers serve on port 443, dns servers on port 53, mailservers on port 25.

figlet netcat

            _            _
 _ __   ___| |_ ___ __ _| |_
| '_ \ / _ \ __/ __/ _` | __|
| | | |  __/ || (_| (_| | |_
|_| |_|\___|\__\___\__,_|\__|

Netcat is tool for connecting to a remote server on a port and talking to it.

It's the quintessential unix commandline tool in some ways. There are at least three or four different versions of it that do much the same thing, but are subtly or not so subtly incompatible with each other. Also it's called "netcat", but usually installed as "nc".

figlet nc

 _ __   ___
| '_ \ / __|
| | | | (__
|_| |_|\___|

We'll use netcat to connect to our server on port 25.

nc -C stuntmx.wordtothewise.com 25

... that's an upper case C in the -c, it's telling netcat to use the right sort of line-ending. Some netcat's use a lower case C for the same thing, so if it gives weird errors, try the other one ...

return

220 stuntmx.wordtothewise.com ESMTP wttw email wizzard

This is the mailserver telling us that we've connected OK, and that it's name is stuntmx.wordtothewise.com.

The response starts with a three digit number - the first digit is a 2, which means everything is OK.

We should say "Hello" and tell it who we are.

EHLO blighty.com

Hello is spelled E H L O. It used to be spelled H E L O, as that's a four letter way of spelling hello, and they wanted to keep the commands short, maybe? Then they needed to modernize it to add some extensions so they spelled it E H L O. For extended hello, I guess.

return

250-stuntmx.wordtothewise.com
250-PIPELINING
250-SIZE 10240000
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

Whoa. That's a multi-line response.

Again, it starts with a three digit code, it begins with a 2 so everything is fine.

Each line of the response starts with the same code. The server tells us it's name again, and then tells us about all the extensions it offers. We're going to ignore those.

We keep going like this - we send a command, the server responds with a message startng with a three digit code.

MAIL FROM:<steve@wordtothewise.com>

We're sending some mail and it's from me. The email address here isn't the one you'd see in a From: header, it's the bounce address or envelope sender or return path or the 821-from. It has lots of names, but it's the email address bounces will be sent to (and then address that's used for SPF).

For mail we're sending manually, or most 1:1 mail this address will often be the same as the address in the From: header, but for any sort of mail where we want to handle bounces automatically it'll be the email address of our bounce handler.

Yes, you need the angle brackets around the address.

250 2.1.0 OK

The response starts with a 2, so it's all OK.

And who do we want to send mail to?

RCPT TO:<discard@reject.wordtothewise.com>

250 2.1.0 Ok

It starts with a 2, so everything is fine. It likes that recipient.

Now lets send the content of the mail.

DATA

354 Go ahead

This is different. The response code starts with a 3, which means the server is waiting for something more. It wants me to enter the payload here - the headers and body of the mail I'm sending.

To: steve@blighty.com

So I'm using a different email address in the To: header as I did in the RCPT TO command, just to show that I can. They'll usually be the same for 1:1 mail or for personalized bulk mail but will often be different for mail that's CC-ed or Bcc-ed or forwarded. Mail delivery really doesn't care what's in this field - the mail will be delivered to the address we gave in the RCPT TO: command.

From: steve@blighty.com

You can see the headers start with the name of the header, and a colon, and then the value of the header. It's one per line for short headers, for longer headers there's a way of wrapping the lines.

Subject: some test mail

...and I should add a Date header, but I can never remember the date format, so we'll just skip that. We can add any more headers we want here, one on each line.

``

Then a blank line means the end of the headers, and the start of the body. The body is just plain text, so you can

type anything you like here

Love, Steve

``

And we're done.

quirkafleeg :wq

The server is just copying everything I type into the body of the message, so how do I tell it I've finished?

We finish with a full stop, on a line on it's own.

.

250 2.0.0 Ok: queued

And now the server has seen the whole headers and body of the messages and it's happy with what it sees, so we get a response code starting with 2.

Now the server has taken responsibility for the message, and is promising to do it's best to deliver it. If it fails later we won't see anything here, but we mquitight get a bounce message in an email sent to the email address we gave it in the MAIL FROM command.

We could keep going, starting with MAIL FROM again to send another email, but I'm done with this for now.

quit

221 2.0.0 Bye

figlet SUCCESS

 ____  _   _  ____ ____ _____ ____ ____
/ ___|| | | |/ ___/ ___| ____/ ___/ ___|
\___ \| | | | |  | |   |  _| \___ \___ \
 ___) | |_| | |__| |___| |___ ___) |__) |
|____/ \___/ \____\____|_____|____/____/

That's it, we've sent an email successfully.

That's all there is. We send a command, we get a response starting with numbers. Responses begining with 2 or 3 are good. If they begin with 4 or 5 that's bad.

** 12m

I'm going to go back to something I glossed over earlier. We used dig to lookup what hostname we needed to send mail to and we got a nice, simple answer. In the real world it can be more complex, though.

Let's take a look at gmail

Same thing again - dig. short. domain name. mx.

dig +short gmail.com mx

40 alt4.gmail-smtp-in.l.google.com.
20 alt2.gmail-smtp-in.l.google.com.
10 alt1.gmail-smtp-in.l.google.com.
5 gmail-smtp-in.l.google.com.
30 alt3.gmail-smtp-in.l.google.com.

Why do we get five answers?

Everyone really wants mail delivery to be reliable. That's been something we've wanted since MX records were invented in the mid 80s. We want mail to keep working even if a server falls over.

In 2022 we do that with load balancers, high availibility clusters, diverse routing and all sorts of other network layer magic.

In 1987 ... not so much.

So Gmail are advertising five mailservers, all of which will accept mail for our @gmail.com recipient. If one isn't available we can try another one.

Rather than picking one at random, Google would like us to use their main mailserver first - that's the one with the lowest preference, in this case "5". So we should try to deliver to gmail-smtp-in.l.google.com.

nc gmail-smtp-in.l.google.com. 25

220 mx.google.com ESMTP q6-20020adffec6000000b00214578640a6si23172461wrs.337 - gsmtp

Hello, gmail!

We've seen this before, so I'll stop here.

quit

If we couldn't connect to that one, we should try the next highest preference, alt1.gmail-smtp-in-l.google.com. If that fails, the next one. We keep trying them in order of preference until we've connected successfully, or we run out.

On the modern internet they don't fall over much, so we're pretty much always sending mail to the mailserver with the lowest preference.

How about Yahoo?

dig yahoo.com mx

If there are multiple responses with the same preference we should just pick one at random.

It's neat to feel like a hacker and be doing all that fast typing, but it's too much like work. We're tool-using monkeys, so let's use a tool.

cat monkeys.txt

                        .="=.
                      _/.-.-.\_     _
                     ( ( o o ) )    ))
                      |/  "  \|    //
      .-------.        \'---'/    //
     _|~~ ~~  |_       /`"""`\\  ((
   =(_|_______|_)=    / /_,_\ \\  \\
     |:::::::::|      \_\\_'__/ \  ))
     |:::::::[]|       /`  /`~\  |//
     |o=======.|      /   /    \  /
jgs  `"""""""""`  ,--`,--'\/\    /
                   '-- "--'  '--'

We are tool-using monkeys.

There's a tool called SWAKS.

figlet SWAKS

 ______        ___    _  ______
/ ___\ \      / / \  | |/ / ___|
\___ \\ \ /\ / / _ \ | ' /\___ \
 ___) |\ V  V / ___ \| . \ ___) |
|____/  \_/\_/_/   \_\_|\_\____/

It stands for Swiss Army Knife for SMTP. And they're not kidding. It has a ridiculous variety of options, mostly designed for testing bits of your inbound mail system.

(It also stands for Sealed With a Kiss, apparently)

We're going to use a tiny subset of what it can do to send some mail.

swaks --to discard@reject.wordtothewise.com

... and wait for it ...

=== Trying stuntmx.wordtothewise.com:25...
=== Connected to stuntmx.wordtothewise.com.
<-  220 stuntmx.wordtothewise.com ESMTP wttw email wizzard
 -> EHLO kiki.wordtothewise.com
<-  250-stuntmx.wordtothewise.com
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250 DSN
 -> MAIL FROM:<steve@kiki.wordtothewise.com>
<-  250 2.1.0 Ok
 -> RCPT TO:<discard@reject.wordtothewise.com>
<-  250 2.1.0 Ok
 -> DATA
<-  354 Go ahead
 -> Date: Tue, 14 Jun 2022 09:36:41 +0000
 -> To: discard@reject.wordtothewise.com
 -> From: steve@kiki.wordtothewise.com
 -> Subject: test Tue, 14 Jun 2022 09:36:41 +0000
 -> Message-Id: <20220614093641.1778637@kiki.wordtothewise.com>
 -> X-Mailer: swaks v20201014.0 jetmore.org/john/code/swaks/
 ->
 -> This is a test mailing
 ->
 ->
 -> .
<-  250 2.0.0 Ok: queued
 -> QUIT
<-  221 2.0.0 Bye
=== Connection closed with remote host.

You can see that it does all the things we just did by hand.

Most of the lines start with a little arrow. For responses we receive from the server it points left, for commands we sent it points right.

We didn't tell it what server to connect to - it did the DNS lookups to work that out from the recipient's address. We didn't tell it who to send the mail "from" - it made up an email address based on my username and the host I'm running SWAKS on. And it just sent a canned test message.

You can override any of those things, and all sorts of others, but the defaults are often good enough for basic testing.

As well as making things simpler, swaks is also fast. If you're typing SMTP by hand you'll sometimes find a mailserver that times out and disconnects you before you're finished typing. I've not found any that you can't talk to by hand, but I've found a few where you have to type fast and not pause to think. SWAKS avoids that. That's why I warned you off sending email to your own email account when we started.

** 18m

I think I promised to talk about how this helps to diagnose some sorts of delivery issues, though. So lets talk about what happens when the mailserver doesn't like us, and doesn't respond with a happy 2xx response to everything.

There are two broad class of delivery failures - rejections and deferrals.

swaks --to reject@reject.wordtothewise.com

=== Trying stuntmx.wordtothewise.com:25...
=== Connected to stuntmx.wordtothewise.com.
<-  220 stuntmx.wordtothewise.com ESMTP wttw email wizzard
 -> EHLO kiki.wordtothewise.com
<-  250-stuntmx.wordtothewise.com
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250 DSN
 -> MAIL FROM:<steve@kiki.wordtothewise.com>
<-  250 2.1.0 Ok
 -> RCPT TO:<reject@reject.wordtothewise.com>
<** 554 rejecting
 -> QUIT
<-  221 2.0.0 Bye
=== Connection closed with remote host.

You see the server responded to the RCPT TO command with a 554 response code.

figlet 5xx = reject

 ____                                _           _
| ___|_  ____  __  _____   _ __ ___ (_) ___  ___| |_
|___ \ \/ /\ \/ / |_____| | '__/ _ \| |/ _ \/ __| __|
 ___) >  <  >  <  |_____| | | |  __/| |  __/ (__| |_
|____/_/\_\/_/\_\         |_|  \___|/ |\___|\___|\__|
                                  |__/

Any response code starting with a 5 is a rejection.

That means that this mail was rejected. You shouldn't try and deliver this particular mail to this particular recipient again (at least not without some human intervention).

For personal mail this means that you'll get a bounce message in your mailbox. For bulk mail it will record that this message delivery failed in your database.

And deferrals.

swaks --to defer@reject.wordtothewise.com

=== Trying stuntmx.wordtothewise.com:25...
=== Connected to stuntmx.wordtothewise.com.
<-  220 stuntmx.wordtothewise.com ESMTP wttw email wizzard
 -> EHLO kiki.wordtothewise.com
<-  250-stuntmx.wordtothewise.com
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250 DSN
 -> MAIL FROM:<steve@kiki.wordtothewise.com>
<-  250 2.1.0 Ok
 -> RCPT TO:<defer@reject.wordtothewise.com>
<** 451 deferring
 -> QUIT
<-  221 2.0.0 Bye

You can see that the server responded to the RCPT TO command with a 451 response code.

figlet 4xx = defer

 _  _                           _       __
| || |__  ____  __  _____    __| | ___ / _| ___ _ __
| || |\ \/ /\ \/ / |_____|  / _` |/ _ \ |_ / _ \ '__|
|__   _>  <  >  <  |_____| | (_| |  __/  _|  __/ |
   |_|/_/\_\/_/\_\          \__,_|\___|_|  \___|_|

Any response that starts with a 4 is a deferral. It means that something went wrong and the mail wasn't delivered successfully.

As the mail sender that mail is still your responsibility, and you can try and send the exact same mail again. That will be handled automatically by your mail system. It will store the mail that was deferred and try it again in a few minutes. It will keep retrying it for some length of time - hours? days? - and if it keeps being deferred it will finally give up.

When it gives up it will escalate the repeated deferrals to a rejection. Again, for personal mail you'll get a bounce message, for bulk mail it'll be marked as bouncing.

figlet just this mail

   _           _     _   _     _                            _ _
  (_)_   _ ___| |_  | |_| |__ (_)___    ___ _ __ ___   __ _(_) |
  | | | | / __| __| | __| '_ \| / __|  / _ \ '_ ` _ \ / _` | | |
  | | |_| \__ \ |_  | |_| | | | \__ \ |  __/ | | | | | (_| | | |
 _/ |\__,_|___/\__|  \__|_| |_|_|___/  \___|_| |_| |_|\__,_|_|_|
|__/

All of this only applies to delivery of this specific mail to this specific recipient. It doesn't mean anything about delivery of mail to other recipients, or delivery of different emails to this recipient in the future.

Neither SMTP nor any of the other email standards says anything about whether you should try and send future emails after one is rejected or not.

So should we remove the recipient from a mailing list, or not?

That's bounce management, and there's no standard for it. There's no common definition of soft bounces or hard bounces. This is all stuff ESPs have made up independently. So everyone does it slightly differently.

The only time SMTP really talks about bounces, as opposed to rejections, at all is "asynchronous bounces" - that's when the recipient mailserver accepts a mail for delivery, promises to take responsibility for it but then later discovers that it can't actually deliver it to the final recipient. Then it will send a bounce message to the email address we gave it in the MAIL FROM command.

Asynchronous bounces sent in response to legitimate email are fairly rare on the modern Internet, so most of the time we talk about bounce management we're making decisions about whether to send future mail in response to their history of rejecting delivery.

** 22m

In the olden days delivery failures were mostly due to server or network problems, or the recipient not existing.

Now, though, the rejections and deferrals are due to spam filtering as often as not.

What extra information can we intuit about why something was blocked?

The best thing is to read the rejection message.

swaks --to helpful@reject.wordtothewise.com

=== Trying stuntmx.wordtothewise.com:25...
=== Connected to stuntmx.wordtothewise.com.
<-  220 stuntmx.wordtothewise.com ESMTP wttw email wizzard
 -> EHLO kiki.wordtothewise.com
<-  250-stuntmx.wordtothewise.com
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250 DSN
 -> MAIL FROM:<steve@kiki.wordtothewise.com>
<-  250 2.1.0 Ok
 -> RCPT TO:<helpful@reject.wordtothewise.com>
<** 557 Mail rejected see https://postmaster.example.com/rejections/557
 -> QUIT
<-  221 2.0.0 Bye
=== Connection closed with remote host.

gesture

While SMTP itself only really looks at the three digit code there's usually something useful in the human readable bit of the response. Sometimes it's an explanation of why the delivery failed. Sometimes, like this, it's a link you can follow to a longer explanation. Sometimes it'll include a recipient ISP-specific code that tells you how the recipient is classifying the mail. e.g. PH01 (phishing?) or TSS04 (recipient complaints about content).

figlet read the rejection message

                    _   _   _
 _ __ ___  __ _  __| | | |_| |__   ___
| '__/ _ \/ _` |/ _` | | __| '_ \ / _ \
| | |  __/ (_| | (_| | | |_| | | |  __/
|_|  \___|\__,_|\__,_|  \__|_| |_|\___|

           _           _   _
 _ __ ___ (_) ___  ___| |_(_) ___  _ __
| '__/ _ \| |/ _ \/ __| __| |/ _ \| '_ \
| | |  __/| |  __/ (__| |_| | (_) | | | |
|_|  \___|/ |\___|\___|\__|_|\___/|_| |_|
        |__/

 _ __ ___   ___  ___ ___  __ _  __ _  ___
| '_ ` _ \ / _ \/ __/ __|/ _` |/ _` |/ _ \
| | | | | |  __/\__ \__ \ (_| | (_| |  __/
|_| |_| |_|\___||___/___/\__,_|\__, |\___|
                               |___/

If you only remember one thing about delivery troubleshooting from this it's that the actual rejection message sent by the recipients mailserver can be incredibly useful. If your ESP's reporting hides that from you, ask them how you can see the rejection messages.

We've often dug down into why a client is having issues and found that the ESPs reporting either hides a useful detail about why something is being rejected, or even that the reporting is plain wrong - e.g. some ESPs I won't name will list some mail as being delivered successfully even when it's being rejected with a 5xx response.

So you should definitely work out how to see the rejection messages (and, ideally, deferral messages) so you know what they normally look like, and you can recognize what's normal and what's not when you're diagnosing a problem.

It's worth knowing about deferral messages as sometimes they're an early warning. If your mail is being deferred by Yahoo with a "TSS04" cookie in the deferral message it means your mail is causing complaints from recipients. If you don't fix that your delivery is likely to get worse at Yahoo and everywhere else.

**25m


So far we've seen deferrals and rejections after the RCPT TO command, but that's not the only place they can happen.

They can happen after any command, or even when you first connect to the mailserver. If we're diagnosing a rejection due to spam filters or other reputation based filtering it's useful to think about what the recipient mailserver knows when it rejects our mail.

cat blackhole.txt

=== Trying stuntmx.wordtothewise.com:25...
*** Error connecting to 216.189.157.158:25:
***     IO::Socket::INET6: connect: Connection refused

This is about the worst case. It means that the mailserver isn't accepting connections from you at all. It's incredibly rare that a legitimate source of email will be blocked at the network level - it's far more likely that the domain you're sending to doesn't exist any more, or the recipient mailserver is dead, or there's a serious network problem somewhere.

cat banner.txt

=== Trying stuntmx.wordtothewise.com:25...
=== Connected to stuntmx.wordtothewise.com.
<** 554 No email here
 -> QUIT
<-  221 2.0.0 Bye

Here the mailserver will let you connect, but then immediately rejects or defers your connection.

All the recipient knows at this point is your IP address. If you're rejected here it either means that the recipient server has a serious problem, or that they don't want to see any email from your IP address.

If your mail is deferred then it could just mean that the recipient mailserver is overloaded, and maybe that your IP reputation isn't good enough to get "red carpet" delivery.

cat mailfrom.txt

=== Trying stuntmx.wordtothewise.com:25...
=== Connected to stuntmx.wordtothewise.com.
<-  220 stuntmx.wordtothewise.com ESMTP wttw email wizzard
 -> EHLO kiki.wordtothewise.com
<-  250-stuntmx.wordtothewise.com
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250 DSN
 -> MAIL FROM:<steve@kiki.wordtothewise.com>
<** 554 I don't like your return path or your IP address
 -> QUIT
<-  221 2.0.0 Bye

Here the mailserver rejects your mail after the MAIL FROM. It knows your sending IP address, and who the mail is "from". If you're an ESP that probably means that they know which of your customers this mail is from.

They still don't know who you were sending mail to, though, so this is a broad brush block against, at the smallest, an entire customer.

All of these so far a pretty rare. Let's look at a couple of more common cases.

swaks --to dorothy@reject.wordtothewise.com

=== Trying stuntmx.wordtothewise.com:25...
=== Connected to stuntmx.wordtothewise.com.
<-  220 stuntmx.wordtothewise.com ESMTP wttw email wizzard
 -> EHLO kiki.wordtothewise.com
<-  250-stuntmx.wordtothewise.com
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250 DSN
 -> MAIL FROM:<steve@kiki.wordtothewise.com>
<-  250 2.1.0 Ok
 -> RCPT TO:<dorothy@reject.wordtothewise.com>
<** 554 Rejected due to your IP address or MAIL FROM or RCPT TO
 -> QUIT
<-  221 2.0.0 Bye
=== Connection closed with remote host.

gesture

Here we're rejected after the RCPT TO. The mailserver knows who the recipient is, as well as who the mail is from and the sending IP address.

For historical reasons this is the earliest in the transaction you're likely to see much mail rejected, even if it's rejected due to an IP address based block.

But the recipient hasn't seen the body of the message at this point, so you know the rejection is due to the reputation of the mail stream - previous emails sent - rather than the content of this particular message.

Staying with the Wizard of Oz theme.

swaks --to lion@reject.wordtothewise.com

=== Trying stuntmx.wordtothewise.com:25...
=== Connected to stuntmx.wordtothewise.com.
<-  220 stuntmx.wordtothewise.com ESMTP wttw email wizzard
 -> EHLO kiki.wordtothewise.com
<-  250-stuntmx.wordtothewise.com
<-  250-PIPELINING
<-  250-SIZE 10240000
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250 DSN
 -> MAIL FROM:<steve@kiki.wordtothewise.com>
<-  250 2.1.0 Ok
 -> RCPT TO:<lion@reject.wordtothewise.com>
<-  250 2.1.0 Ok
 -> DATA
<-  354 Go ahead
 -> Date: Sun, 12 Jun 2022 09:52:06 +0000
 -> To: lion@reject.wordtothewise.com
 -> From: steve@kiki.wordtothewise.com
 -> Subject: test Sun, 12 Jun 2022 09:52:06 +0000
 -> Message-Id: <20220612095206.1686619@kiki.wordtothewise.com>
 -> X-Mailer: swaks v20201014.0 jetmore.org/john/code/swaks/
 ->
 -> This is a test mailing
 ->
 ->
 -> .
<** 554 Rejected due to your IP or MAIL FROM or RCPT TO or body
 -> QUIT
<-  221 2.0.0 Bye
=== Connection closed with remote host.

gesture This time it was rejected after the end of the DATA section.

This is an increasingly common way of rejecting email. It means the recipient ISP has the entire content of the message before making a decision.

That leads to a lot of things.

One is that the mail might be rejected because of something in the body of the email - a URL, say. Or an image. Or a bitcoin address.

Or it might be rejected because the body of the message looks like mail that's caused complaints before - even if it was sent from a completely different source. That's not unusual in affiliate mail, where different senders of different levels of quality are sending much the same content.

Or it might have been rejected at this point even though the ISP made - or could have made - the decision earlier, due to the recipient or the IP address. Perhaps they want to capture the body for future filtering decisions even though they're not going to deliver this mail.

Or perhaps they just want to reject at a consistent point in the transaction, either for technical reasons or so as not to leak information about why they rejected it. One instance of that is to mitigate listwashing attempts.

What's listwashing? It's removing undeliverable email addresses from a list before mailing them, to hide from an ESP that the list is of poor quality and to attempt to mitigate some sorts of delivery problem.

A common way to wash invalid addresses out of a list is to start sending mail and see what response you get to a RCPT TO. If it's a 250 then it's a valid email address, otherwise it isn't. Then you QUIT out of the transaction and repeat for the next email address. If you only reject the mail after DATA even if the recipient email address doesn't exist then a listwasher can't do that.

With a rejection or deferral after DATA the recipient has the content of the message even though it wasn't delivered. Sometimes a receiver may load images or follow links in the body to look at related content before they respond at the end of DATA. Other times they may defer the message after DATA, then check out the images and links later so they'll have more information about whether to accept the message when it's retried.

That may mean that you see "opens" and "clicks" before a mail was delivered, or even for email that was rejected and never delivered.

figlet 'and so?'

                 _            ___
  __ _ _ __   __| |  ___  ___|__ \
 / _` | '_ \ / _` | / __|/ _ \ / /
| (_| | | | | (_| | \__ \ (_) |_|
 \__,_|_| |_|\__,_| |___/\___/(_)

One reason to know about how email is physically sent is to make it easier to recognise the myths and fairy tales many people tell about email delivery, even when those have been baked into bounce classifiers and open tracking and delivery reporting and list optimization.

Another concrete takeaway from this is that it's useful to know both what the rejection message was, and when in the transaction it happened. Knowing how to extract that information from your ESP gives you another tool to diagnose delivery issues. And if you can get regular reporting on that level of delivery data it can let you see delivery trends you might not spot otherwise.

And if something weird happens, you can try and replicate it with swaks and see exactly what's going on.

** 33m

cat contact.txt

 ____  _                      _   _   _    _
/ ___|| |_ _____   _____     / \ | |_| | _(_)_ __  ___
\___ \| __/ _ \ \ / / _ \   / _ \| __| |/ / | '_ \/ __|
 ___) | ||  __/\ V /  __/  / ___ \ |_|   <| | | | \__ \
|____/ \__\___| \_/ \___| /_/   \_\__|_|\_\_|_| |_|___/

Blog:    wordtothewise.com

Email:   steve@wordtothewise.com

Twitter: @wise_steve