CRLF, PHP, Postfix, and SPAM
Recently we switched our production machines from running sendmail to postfix. In this process our emails started getting caught as SPAM and it took a bit of digging to figure out what exactly was happening. The error we got was:
BAD_HEADER: Improper use of control character (char 0D hex)
If you look up 0D hex character in your favorite ASCI character table you will find that this is a carriage return (CR). As specified in email standards document RFC 822, all of our email headers were separated by CRLF (a carriage return and an line feed - \r\n). At this point I began scratching my head, postfix won’t relay the message because our email conforms to the specs? After a bit more digging and looking at the actual message postfix was trying to send I noticed that instead of CRLF (\r\n) our email had CRCRLF (\r\r\n).
What is actually happening is postfix assumes that any line feed separated header must be missing its carriage return so it takes the liberty to replace any line feed character (LF) with a carriage return and line feed (CRLF). Because we were already formatting our messages correctly this was causing our email headers to be separated with CRCRLF. The spam detection software sees this and assumes we are trying to perform a carraige return injection attack.
The fix is to send email headers that are only line feed separated (\n) because postfix will fix the headers along the way. What we did was make this a configuration parameter in our software since our software runs machines with both sendmail and postfix (depending on the client).
October 23rd, 2007 at 6:50 pm
Something doesn’t add up. Postfix does nothing of the sort to incoming emails.
Are you passing the email to postfix directly via the command-line?
What does PHP have to do with this?
Ciao!
October 24th, 2007 at 8:24 pm
I just realized that I never mentioned that this only happens when I send emails through the php built-in function mail. Below is copied from a posting on a php mailing list (also found here: http://us3.php.net/function.mail).
—————–
Note that there is a big difference between the behavior of this function on Windows systems vs. UNIX systems. On Windows it delivers directly to an SMTP server, while on a UNIX system it uses a local command to hand off to the system’s own MTA.
The upshot of all this is that on a Windows system your message and headers must use the standard line endings \r\n as prescribed by the email specs. On a UNIX system the MTA’s “sendmail” interface assumes that recieved data will use UNIX line endings and will turn any \n to \r\n, so you must supply only \n to mail() on a UNIX system to avoid the MTA hypercorrecting to \r\r\n.
If you use plain old \n on a Windows system, some MTAs will get a little upset. qmail in particular will refuse outright to accept any message that has a lonely \n without an accompanying \r.
October 26th, 2007 at 8:08 pm
Okay, now that makes sense.
Your mistake was … using mail in PHP.
It’s not a good thing, it’s not portable, it won’t let you efficiently
send emails to multiple people, etc. etc.
One of the first things I had to do at RackSpace was write a
replacement mail library in PHP. I tried to make it work using the
mail command, but it offers no protection, no guarantees, etc. I
wrote a semi-lame one that worked for a couple years.
In the end, I wrote it in two parts:
1) It queued the email in a table.
2) It used port 25 to send it to a known good SMTP server in-house and
would only then delete the email from the table.
A future version was going to keep the emails for a while and to match
up bounced emails for tagging later instead of having humans do
that…
I think now there are some good mail libraries, like PHPMailer that can take a
lot of the pain out of mailing in PHP.
Ciao!