Composr Tutorial: Understanding and configuring e-mail

E-mail can be a complex thing. There are a number of e-mail protocols with standards that seem cobbled together, and there are a wide array of different tools that work with e-mail. Making everything compatible can be quite a challenge.

E-mails in Composr

Composr constructs its e-mails using language strings: each different e-mail is built from a different language string (or sometimes a text template). These strings/templates are written in Comcode. Composr sends out e-mails in dual format – both HTML and plain text, so that people can disable HTML in their e-mail software if they wish to. Plain text versions are made by automatic tidying-up of the Comcode (i.e. making it a little more human-readable), and HTML versions are made by parsing the Comcode to HTML and then putting that HTML inside the MAIL.tpl template.

Composr goes to great lengths to reduce the chance of e-mails being marked as spam, and (by default) embeds all CSS and images inside the e-mail instead of linking (so that the user does not need images enabled to see them – most users do not as it can aid spammer tracking).

Just taking one example, the 'MAIL_NEW_PASSWORD' language string from the cns language file, which is:

You can see it is fed with various parameters, and is written in Comcode.

Configuration options

Configuration options warranting particular discussion are:

• "Link to images in e-mails rather than embed" – by default this option is disabled, and when disabled makes Composr embed all images directly within e-mails rather than remotely linking to them. This means that users will see images immediately without having to grant permission for their e-mail client to download them.
• "E-mail 'from' address method" – by default this option is set to separate From and Reply-To addresses, to reduce chance of e-mails being considered fraudulent.

Mail server overview

First, I will start with a brief overview of how e-mail works. This section may be far more than you ever need to know, so don't worry if you don't follow it all.

Consider that an e-mail address is composed of two parts: an account name, and a domain name (with an '@' symbol to separate them).

This is a simple thing to understand but let's look at some more detail. The first question is 'where does it get delivered to?', and the answer is 'the server identified by the MX record of the domain name that it is being sent to'. To deliver an e-mail to someone@example.com we would look up the MX record (a type of DNS record) for the example.com domain, and thus find the IP address of the destination server.

This actual delivery process is performed by the 'SMTP' server, otherwise known as an 'outgoing e-mail server'. When you send an e-mail from a mail client (be that a desktop client, a webmail client, or a webapp like Composr), it is sent to the outgoing SMTP server to be dispatched. That server will put the message in a queue, and then it will (in the SMTP server's own time) send it on to SMTP server on the IP address of the MX record for the domain name ('destination e-mail server'). If it cannot be delivered it is kept in the queue while a few retries are attempted over a few days. The destination server will then then deliver the e-mail to the account specified in the e-mail address, and give a bounce e-mail if no such account exists (assuming it hasn't been set up to forward the e-mail to another account or address, or just silently drop it).


What I have just described is the primary mechanism for e-mail. However, there is a secondary mechanism – actually being able to read e-mails from an inbox (SMTP will populate an inbox but provides no way to actually read it). This are three common ways to read inboxes:

1. Using the IMAP protocol (which is designed to permanently store e-mail on the server)
2. Using the POP3 protocol (which is designed to transfer e-mail from the server to the user's e-mail client)
3. Accessing the mail box directly (webmail often does this) as do UNIX command-line utilities that run directly on the server

It is important to understand that IMAP/POP3/webmail are entirely separate from SMTP itself, except for two areas:

1. They access the same mailbox that SMTP writes to
2. SMTP often whitelists the IP addresses of users who have recently logged into POP3 or IMAP to say that relaying should be allowed from those IP addresses (this is one mechanism for relaying to be allowed, another is authenticated SMTP, and another is from-address whitelisting)

SMTP configuration in Composr

There are two separate issues for us to consider when it comes to Composr:

1. Whether we will want (i) Composr's SMTP-connection code to run, or (ii) PHP's SMTP-connection/local-agent code.
2. Which SMTP server PHP or Composr is connecting to. Neither Composr nor PHP include an actual SMTP server, so you're always going to be configuring one of them to connect to an actual SMTP server. The issue is whether that is your server's own SMTP server (assuming you have one) or whether it is another one (usually your hosting provider's). If you're on a Linux/UNIX server you have no choice but to use your server's own SMTP server if using PHP's SMTP-connection code because the local-agent is used rather than a configured SMTP server.

It is usually best to rely on PHP's SMTP-connection code, so it can be managed on a server level. However there are a few situations where this is not workable:

1. PHP doesn't support SMTP authentication, so if the only e-mail server available requires this, and you're on Windows (which uses SMTP not a local-agent to connect to the SMTP server), you'll need to use Composr's SMTP-connection code (which does support authentication).
2. If you're on Linux and the server doesn't have an SMTP server on it.
3. If the PHP mail configuration is misconfigured or faulty and you can't repair it (see below).

Composr's SMTP-connection code is configured from the Configuration module (Admin Zone > Setup > Configuration > Site options). If the SMTP server hostname is left blank (the default), Composr relies on PHP's SMTP-connection code.

Avoid getting spam-blocked

When a website sends out e-mail there is always a risk that it could get blocked by spam filters. Whether this happens largely depends on the configuration of spam filters at ISPs and on user's own computers, but there also some general causes.

Specific issues can be:

1. Your server is on a spamlist. Also check to see if any bounce messages come back that talk about your server being blocked as a spammer.
   Possible reasons for getting onto a spamlist:
   1. If you are on a shared server it is possible other sites on the server may have gotten the server banned. There are tools to see what other sites share your server's IP address.
   2. You have a stale newsletter producing too many bounces. You should clean out addresses that are bouncing so that network infrastructure doesn't flag you for spammy bulk delivery. If you have some way of tracking which e-mails are read you can delete newsletter subscribers that are not reading your e-mails (Composr does not currently support a mechanism for this directly).
   3. You have bought mailing lists. Rarely if ever buy mailing lists, users will mark you as a spammer.
   4. You have annoyed your legitimate readers. If you send low-quality e-mails, such as ones that have low quality content, use too much data, use dodgy looking link redirects, or are hard to unsubscribe from, you have a higher chance of being reported as a spammer.
2. Your message content appears spammy, which sometimes can happen inadvertently. Very carefully check your spam folder. Spam filters typically run a complex set of calculations to detect if something is 'spam'. It could well by a domain SPF setting is wrong, and combined with Composr e-mails being more complex than some other software, that knocking it over a spam threshold. That is just one of many possibilities that should be looked into if it is indeed a spam-filtering issue.
   Here are some scanner tools you can use:
   1. GlockApps
   2. mailtester
   3. ISnotSPAM (fairly primitive compared to the above solutions)
   These tools will scan for various other issues beyond content problems, they're very useful.
   To send a test e-mail to a few services at once using Commandr:
   

   Code

   :$emails = array('scanner_a@example.com', 'scanner_b@example.com', 'scanner_c@example.com', 'you@example.com'); require_code('mail'); mail_wrap('Verifying delivery', 'This is a message being sent to verify that e-mails are being correctly delivered. Thank you.', $emails, null, '', '', 3, null, false, null, false, false, false, 'MAIL', true);

   Some specific things to look out for when writing your content:
   1. Don't have an unreasonable image to text ratio.
   2. Don't use a broken reply-to address that nobody can reply on such as a noreply.
3. Something subtle is triggering spam analysers:
   1. The PHP mail.add_x_header option is on (it flags up on SpamAssassin, a common spam analysers that companies use). You can check this from Admin Zone > Tools > PHP info.
   2. Inconsistent text and HTML versions. If you've customised the MAIL.tpl template but you've not also customised the MAIL.txt template to have the same text (it flags up on SpamAssassin if the words are inconsistent).
4. Your "Website e-mail address" is for an e-mail address hosted on another server and an SPF record exists for the domain does not grant your web server permission to use the address for sending out mail. Common e-mail services like gmail often have this problem. If this might be the case, you either need to get the SPF record amended to cover your server (impossible for a common service), or use a different "Website e-mail address". Note that Composr uses the "Website e-mail address" as the "From" address in all outgoing e-mails, but by default the reply addresses depend on context (often they are the "Staff address", but they could also be the address of the member who caused the e-mail to be sent.
5. You have a subtle SMTP configuration problem:
   1. You don't have reverse DNS available on your server's IP address. This is the address outbound SMTP connections are made from. Checker tools may complain if the reverse DNS points to a domain name not a hostname (e.g. domain.com not mail.domain.com) – but you can ignore this in practice as it won't cause any problems.
   2. Your server is giving an invalid HELO DNS address when it makes outbound SMTP connections, e.g. something generic like localhost. Preferably it will use your actual domain name, or a subdomain there-of, but this is not mandatory. You can check this using Telnet, telnet 192.252.146.10 25 on Linux or Mac, or using online tools.
   3. The hostname in the above two checks does not match
   4. The Return-Path in e-mails is pointing to an incorrect or invalid e-mail account, so that bounce/receipt e-mails themselves bounce. You may need to enable the "Pass website e-mail address to 'sendmail'" option. You can check your e-mail by manually viewing the headers of an e-mail sent to you.
6. You have your own server and using it for SMTP but your SMTP port is blocked by the webhost or ISP. Some companies do this to reduce the chance of their network being blocked for spamming. If this is the case no e-mails will be sent at all and you need to discuss the correct configuration with your webhost/ISP.
7. You are missing an unsubscribe link on your newsletter template. It's there by default, and it is a good idea to leave it.
8. It's unlikely, but it could be some host/software-specific bug. You can open a bug report if you're willing to give the developers access to run tests on your server, after you've checked it's not a spam folder, and only if you're not on a low-quality free webhost.

Positive advice:

1. Generally it is advisable to set up SPF, as it provides a positive signal that your server is not a spammer. Set both SPF and TXT record types for maximum compatibility (set them to the same values).
2. Ask your visitors to add your website e-mail address to their contacts list. Spam checkers usually will not block mail sent from someone on their contacts list. If their e-mail provider has a "Safe senders" list, that's even better – Microsoft's e-mail services have this. Microsoft's e-mail services do over-block unknown e-mail servers or if users aren't reading your e-mails for long. If you're blocked, your message may not even go into the user's spam folder.
3. Ask users to mark e-mails non-spam if they go into their spam boxes, or move them from their spam/promotion folders to their inbox, or even sometimes reply to them. That should have an effect on learning algorithms.
4. Configure DKIM for your server and domain name (see next section).
5. Configure DMARC for your domain name. Here are some good tool choices to help you:
   1. DMARC Generator
   2. DMARC Wizard and DMARC Inspector
6. Include a physical mailing address in your newsletter footer. To be compliant with the US CAN-SPAM act you need to do this for commercial mailings (read as "advertising e-mail"). ISPs can't know if you are sending a commercial e-mail so are not likely to penalise you much if you don't, but you never know.
7. Enable a List-Unsubscribe header. You can do this with a Commandr command like :set_value('list_unsubscribe_target', 'mailto:you@example.com?subject=Unsubscribe&body=Please%20unsubscribe%20me');.
8. If you find X-Authentication-Warning headers are being included in outgoing e-mails you may need to edit your /etc/mail/trusted-users file. Exact advice depends on your SMTP server.
9. Set up Google's Postmaster Tools

More extreme advice:

1. If you're really stuck with your e-mail server being blocked, you could use a third-party e-mailing service like Mandrill. Mandrill may require the "Enable Blind CC (BCC)" option to be turned off, as we have had a report of it not working on Mandrill, but that they provide an account setting to make CC behave like BCC.
2. You can send out newsletters using a totally different e-mail address and third-party newsletter tool such as MailChimp. This is extreme unless you want to benefit from the features of such a tool, but it will mean that if your newsletter activities get marked as spammy it won't affect your important day-to-day website e-mailing functionality.
3. If you have a lot of cash then you can get your server onto a whitelist.

Implementing DKIM (advanced)

DKIM is an e-mail digital signature system which reduces the chance of people faking e-mails from you, and therefore your e-mails being flagged as spam.

There are 2 ways you can implement DKIM:

1. At the server-level. This is recommended, but may not be available on some shared webhosts. It is recommended because this way all e-mails from your domain may be signed, not just ones sent from Composr.
2. Within Composr (there's a config option to enter the DKIM private key). [version 10.0.5+]

There are a few caveats with Composr's implementation:

1. The PHP openssl extension is required.
2. It only works with the PHP mailer, not the SMTP mailer (for overhead control the SMTP mailer will send e-mails going to multiple users within a single packet, which prevents use of DKIM).
3. It is not recommended on Windows due to severe performance problems with openssl on Windows.
4. Composr will only apply DKIM on your mail e-mail domain (as defined in your _config.php file). It is incompatible with the "Use true 'from' address" option (which is off by default).

Your DKIM public key is to be stored in DNS as per standard DKIM set up.

You can use an online tool to generate your keys, including the specific DNS records you need to add.

• DKIM Wizard
• DKIM Generator

Tools for testing DKIM:

• Check a TXT record (1)
• Check a TXT record (2)
• E-mail scanner and tips

Technical references for debugging:

• Using opendkim to test
• DomainKeys technical summary
• DKIM technical summary
• Full specification

Queues and debugging

Composr has config options for:

1. turning on a mail queue (for efficient delivery).
2. keeping stuff in the queue until you let it out (when testing a site, the "E-mail debug mode" option).

The queue can be viewed from:
Admin Zone > Audit > E-mail queue/log

All (†) e-mails are logged for 14 days (by default) in the same system, regardless of whether the queue is on or not. This is very useful for analysing what your site has been sending out.

† Except support ticket e-mails if IMAP integration is enabled, newsletter e-mails, password reset e-mails, and some other very high security or high volume e-mails.

Testing a server's mail support

If you want to test a server can directly send e-mail using PHP use this simple test script:
(of course substitute in your own e-mail address)

Requesting help

If you are not able to get e-mail working then it is almost certainly a result of your system configuration. Someone helping would likely need to have admin access to your site, or perhaps even to the whole SMTP server you are using. Never-the-less, if you are requesting help then at least make sure you answer the following questions so they have some insight into your configuration:

1. Are you self-hosted or are you on shared hosting?
2. Is your server Windows or is it Linux?
3. If you are on Windows: Have you properly configured the SMTP settings in the server's php.ini file?
4. Is Composr configured to use its own SMTP settings, or they are blank and PHP's mailer is used?
5. If you have not configured Composr to use its own SMTP settings: Are you on a local hosted SMTP server (same server as web server), or are you or a remote relay?
6. Have you checked your spam folder?
7. Have you have looked at the SMTP server logs?
8. Have you installed the better_mail addon?
9. If you have configured Composr to use its own SMTP settings: Is your SMTP configuration secured via some kind of SSL?
10. If you have configured Composr to use its own SMTP settings, and if it is SSL: What exact SSL configuration do you need to use?
11. If you have configured Composr to use its own SMTP settings: Do you need to connect to SMTP with any special port settings?
12. Are you running any exotic SMTP software?

See also

• How the Internet works
• Introduction to Cookies and sessions (and JavaScript)
• How domain names work
• Notifications

Feedback

Have a suggestion? Report an issue on the tracker.