

Building Mails
==============

One of TurboMail's main goals is to make build emails easy. Therefore we create
a Message class which hides all the tedious work like encoding the message 
header lines and building a MIME compliant body.

.. class:: Message([author [, to [, subject [, **kwargs]]]])
    
    All arguments are optional. For the semantics behind these parameters, please
    read the 'Properties' section below. Through the use of kwargs you can set
    all properties documented below directly in the constructor (a TypeError 
    will be raised if kwargs contains a key which is not a valid property).

.. _message_properties:

**Properties:**

* author -- Email address or addresses of the author(s)
* bcc -- Blind carbon-copy address or addresses.
* cc -- Carbon-copy address or addresses.
* date -- Date if message creation either a date string as per RFC 2822, a 
  datetime.datetime instance or the number of seconds since 1970 as float 
  (optional, uses the current local time by default)
* disposition -- Request disposition notification be sent ("email was read") to this address.
* encoding -- Content encoding specific to this message.
* headers -- A list of additional messages headers (either as dictionary or as list of tupels).
* nr_retries - How often should TurboMail retry to deliver this message if the delivery failed? (default: 3)
* organization -- The descriptive Organization header.
* plain -- Plain text content. Will be automatically generated if a rich text part was specified.
* priority -- The X-Priority header, in the range of 1-5.
* reply_to -- Email address to which replies should be sent (Reply-To header)
* rich -- The rich-text ("HTML") part
* sender -- email address of the agent which sends the mail (required if you specified more than one author)
* smtp_from -- The SMTP envelope address, if different from the sender (this option is only useful for the SMTPTransport).
* subject -- A textual description or summary of the content of the message.
* to -- Email address or addresses which should receive the message (To header)

All email addresses can be given either as a string (TurboMail will convert
unicode domain names with idna encoding if possible) or as a tuple 
('Full Name', 'name@example.com').

The plain- and rich-text parts of the message may be defined as any callable 
which returns plain- or rich-text. The callable is executed at the time the 
message is built - usually just prior to the first delivery attempt.

The encoding can be overridden on a per-message basis (you have to pay attention
that all characters in your plain or rich contents may be encoded with the 
chosen encoding). If you use 'UTF-8-QP' as encoding (which is a fake encoding 
provided by TurboMail), the message body will be encoded with UTF-8 Quoted 
Printable (Python's email module uses base64 by default for UTF-8 content).

In order to simplify the use of the Message class even further, you can set 
default values for every property in your configuration. These configuration
values are used in Message's __init__ (constructor) when you don't set a value 
explicitely. The configuration key is the name of the property, prefixed with
'mail.message' e.g. 'mail.message.bcc'. Using that mechanism you can send all
messages in copy to a certain address (assuming you don't overwrite the
bcc property after the message instantiation).

**Methods:**

    .. method:: attach(file, [name])
       
       Attach a given file (either on-disk by passing a path, or in-memory by passing a File descendant) to the message. The name argument is required if passing an in-memory File descendant.
    
    .. method:: embed(file, name)
       
       As per attach, but limited to image files for embedding in the rich-text (HTML) part of the message. Name is the desired CID of the embedded image.
    
    .. method:: send()
       
       Send the message via the currently configured manager. In order to send
       a message, you need to define at least an author, a recipient (either
       in 'to', 'cc' or 'bcc') and the message content (via 'plain' or 'rich').


Plain Text Messages
-------------------

Easy things first: You want to send an email with just some information in it
(e.g. unhandled exception in your application). This kind of mail doesn't have
to look fancy, so we'll just use plain text::

  from turbomail import Message
  message = Message("from@example.com", "to@example.com", "Hello World")
  message.plain = "TurboMail is really easy to use."
  message.send()

You see, building a plain text email is very easy: Just put your message content
in the 'plain' attribute and send the mail. Done. In real world scenarios you 
probably would use some kind of template engine to generate the text body but
that's your choice.


Messages with Attachments
-------------------------

Sometimes you need send some files along your message content (e.g. a PDF 
document or some zip file). There two alternative ways of doing it with 
TurboMail. This is convenient because sometimes you generate a file on-the-fly
(like a PDF report with reportlab/z3c.rml) and don't want to write it to the
hard disk (slow, you have to handle disk full conditions etc)::

  from turbomail import Message
  
  message = Message("from@example.com", "to@example.com", "Sales are skyrocketing!")
  message.plain = "Have a look at the sales statistics attached."
  
  filename = '/home/fs/latest_sales.pdf'
  # The file will be named 'latest_sales.pdf' in the message
  message.attach(filename)
  
  # Alternative way to attach a file (please note that you have to provide a
  # name for the file to attach):
  fp = file(filename)
  # The file will be named 'sales_report.pdf' in the message
  message.attach(fp, 'sales_report.pdf')
  
  message.send()



HTML Messages
-------------

Originally emails contained only plain text which does not provide much formatting
possibilities especially compared to all those fancy looking websites that came
up in the late 90ties. Since the advent of the `MIME standard <http://en.wikipedia.org/wiki/MIME>`_
you can use HTML in your messages to produce a so called 
`HTML e-mail <http://en.wikipedia.org/wiki/HTML_e-mail>`_.

To ::

  html_part = """<html><header/>
  <body>
    <h1>Look,</h1>
    
    this is an HTML message. Really <b>w00t</b>.
  </body>
  </html>
  """
  
  from turbomail import Message
  msg = Message("foo@example.com", "bar@example.com", "Writing a HTML message")
  msg.rich = html_part
  msg.plain = """This is the boring alternative part for people who don't see 
  the HTML part."""
  msg.send()

You can build an HTML email just by assigning a string which contains the HTML 
markup to *msg.rich*. You should always provide *msg.plain* too just in case 
someone does not want to read your HTML. If you don't provide a plain text
TurboMail will derive a plain text part from your HTML by using a very simple 
algorithm [#]_.


In Mozilla Thunderbird 2 the resulting message looks like this:

.. image:: thunderbird_html_message.png


The HTML string above is static. In most applications you probably use a 
template engine like Genshi to produce the final HTML.

Be warned: *Keep your HTML simple.* E-mail clients generally don't support the
full subset of HTML/CSS. If you are frustrated because Internet Explorer does
not like your web page wait until you saw your HTML message rendered by Outlook
express before you start cursing...
Another difficulty is that there are more email clients with significant market
share than browsers (you should count different web mail providers as different 
mail clients because they all cut different parts of your message to protect 
their webmail users).


.. [#] If you need more information about TurboMail's built-in HTML to text
       converter please see :ref:`Clever plain text generation for your rich text messages<html2text>`.


HTML Messages with embedded images
----------------------------------

If you want to have really good looking emails you can add images. Unfortunately
it is somewhat complicated because you can't just attach an image and reference
it from your HTML like in a web page. Instead you have to use the embed 
function which will tell the recipient's mail client that the image is meant to
be shown inside the HTML and is not an ordinary attachment.

The other alternative which keeps the mails also very small is to reference
your images with absolute urls to your server. So the recipient has to fetch
the images every time he wants to read your message. But this method may not be
reliable as some mail clients won't load images from remote servers (or require
special user interaction to do so like in Mozilla Thunderbird) [#]_.


.. [#] Spammers and other evil people used remote images (called `web bugs <http://en.wikipedia.org/wiki/Web_bug>`_)
       to track people so they knew who read their emails. Therefore quite few
       (especially the computer-savvy) people don't like linked images and
       many popular mail clients refuse to load files from remote URLs, 
       displaying your mail without any images.



Pre-built Messages
------------------

Sometimes you don't want to create a Message with TurboMail - maybe the message
is already on your hard drive or it will be generated by some other mechanism
(be it legacy code or just a highly specialized mail-generation library). 
However you want to benefit from the facilities TurboMail provides (e.g. 
testability of mail sending, threaded mail sending, ...). This is when the 
WrappedMessage comes in::

  from turbomail import WrappedMessage
  
  # We're using a hard-coded string here - of course you can get this string 
  # from another library, this is just to make this example as easy as possible.
  generated_message = '...'
  
  message = WrappedMessage('from@example.com', 'to@example.com',
                           message=generated_message)
  message.send()

The WrappedMessage just wraps a message and adds the minimum information which 
is necessary for delivery. The contents of your message will not be modified in
any way.

Please note that you have to specify the 'sender' and the recipient(s) 
explicitly when building a WrappedMessage - although the message you're about
to pass may contain this information already. However, when it comes to delivery
mechanisms like SMTP this information can not be extracted from the message
content without guessing. You can find an in-depth explanation of the underlying
mechanism in your chapter :ref:`details`.



.. _i18n:

How TurboMail handles Internationalization
------------------------------------------

When RFC 822 was written (1982), the world was still a disc: emails were allowed
to contain ASCII characters only. International characters like German umlauts
("ü", "ß") were not allowed. As time went by the original format was extended
in a backward compatible way so you could write plain text messages which
contain Non-ASCII characters.

The default encoding for TurboMail is still ASCII (the same holds true for 
almost all mail servers). If you want to send mails which contain Non-ASCII 
characters, you have to set the configuration option 'mail.message.encoding' 
appropriately (see :ref:`TurboMail Configuration <configuration>`).
The broadest character set available is UTF-8. Unfortunately Python will encode 
the body part using base64 if you choose "UTF-8". This means that a human can 
not read the message by looking at the plain text which may help debugging 
problems. Furthermore it will increase the message's size by approximately 1/3.

To get around that, TurboMail registers a new codec 'UTF-8-QP' which is 
mostly an alias for UTF-8. Python's email module will use UTF-8 Quoted Printable
for messages with 'UTF-8-QP' which is much more human readable than base64 and 
won't increase the message size as much as base64.

Since several years you can use even `internationalized domain names <http://en.wikipedia.org/wiki/Internationalized_domain_name>`_
which means that umlauts may appear left from the @ character as in foo@zääz.de.
In the future we will probably see internationalized top-level domains.
Most email servers will require that you convert internationalized domain names
to punycode so that it only contains ASCII characters. With TurboMail you don't
have to care about the conversion as TurboMail will do it for you. However if 
you choose to use WrappedMessage, you must convert IDNs in your email
header (not the SMTP envelope) yourself [#]_.


.. [#] Technically you *may* not need to do this but often this is necessary so 
       that the readers mail client will display the address correctly.


How to get your Messages through Spam Filters
---------------------------------------------

No, this section is not about how to become a `"Spam King" <http://en.wikipedia.org/wiki/Spam_King>`_. 
Due to the amount of spam everyone receives on a daily basis, most of your 
recipients will use spam filters to filter spam (or rely on their providers 
like Google or Yahoo to do so). Assuming that your application only sends out 
legitimate mail, how do you avoid that your mail is accidentally marked as spam?

"Spamminess" is nothing which can defined just by one score value. Every spam
filter will use its own rules and thresholds when to classify a message as spam.
Most things that influence the spam score are beyond TurboMails' control: 
message content, having enough innocent text, sending mails from a static IP
address with DNS correctly set up etc. It is impossible to assemble a message 
that will pass all your recipients spam filters!

However, TurboMail users reported that Google Mail classified their messages as
spam if they used UTF-8 characters in their message body. The problem was solved
by using the UTF-8 Quoted Printable "encoding" [#]_. The previous section
:ref:`How TurboMail handles Internationalization <i18n>` told you already how to 
do this.

Furthermore it may help if you add the full name of the recipient as sender and 
not just the email address (assuming you know the name)::

  # instead of:
  msg = Message("foo@example.com", "bar@example.com", ...)
  # better write:
  msg = Message(("Foo", "foo@example.com"), ("Bar", "bar@example.com"), ...)

If you build HTML messages with TurboMail's Message class, you should care about
the plain text part, too. Message tries to derive a plain text part from your
rich-text ("HTML") part automatically if you don't set any plain text explicitly.
But the algorithm for generating the plain text part is not very clever so some 
parts of your markup may be included in the plain text part. Some users reported
that spam filters classified their message as spam due to these left-overs. We
try to improve the text filters if we become aware of these issues but you may
think about providing a plain text part explicitly or just disabling the plain
text generation. Another alternative is to :ref:`plug in a better html2text 
converter <html2text>`.


.. [#] If you use Non-ASCII characters in a message, Python's email module may
       resort to base64 as encoding for your body. In the early days of mass spam 
       spammers used this trick to get their advertising to the recipients by 
       encoding their message bodies with base64. Most scanners at that time did
       not preprocess messages and just looked for some keywords but base64 will
       make an unreadable character stream out of your plain text. Something like
       'TurboMail' becomes 'VHVyYm9NYWls\\n' when encoded with base64. The users
       mail client was able to decode base64 without any problems so that may be
       the reason why most spam filters still give base64 a bad rating.



Sending Mails
==============

When you instantiated a message and added all the information (e.g. message 
body), you have to tell TurboMail to send the message explicitly. The easiest 
way of sending a message is calling the send() method of a Message::

    from turbomail import Message
    msg = Message('foo@example.com', 'bar@example.com', 'Subject')
    msg.plain = "Foo Bar"
    msg.send()   # Message will be sent through the configured manager/transport.

When the send method is called, an email.Message will be assembled and handed 
over to the manager you configured. Whether the message will have to wait in a
queue until there are free resources to deliver it to the intended recipient or
if the message is given to the configured transport immediately, depends on your
configuration.

Please note that every time you call the send method, an email is sent. 

In TurboMail 2 you had to call turbomail.enqueue(msg) to send a message. This
method is still available for backwards compatibility but its usage is 
deprecated.

If an error occurs while sending the mail, TurboMail will retry several times to
deliver the message. If that is not successful, the message will be dropped. If
you use the ImmediateManager, you'll get an exception (smtplib.SMTPError or
socket.error). If you use the asynchronous DemandManager, the failure will be
logged. We're not satisfied with that behavior but we were not able to fix this
in time for TurboMail 3.0. For 3.1 we'll build a more robust and flexible system
with hooks so you can do your own error handling routines.


.. _configuration:

TurboMail Configuration
=======================

TurboMail's behavior (e.g. what default encoding should be used) is highly 
configurable. If you use TurboMail in a supported framework which has a 
TurboMail adapter TurboMail will use the general configuration system of your
framework. For TurboGears 1 this means you can put configurations for TurboMail 
in app.cfg and/or {dev, prod}.cfg and TurboMail will pick them up automatically.
The configuration is set when TurboMail starts up and should not be changed 
afterwards.

Configuration is based on key-value pairs where the key is always a string. What
type is required for the value depends on the key. Unless otherwise noted, the 
values should be specified as strings, too.

Later in this manual we use the `ConfigObj <http://www.voidspace.org.uk/python/configobj.html>`_
syntax when we refer to specific settings. For example, "mail.on = True" means 
that TurboMail's configuration contains a key "mail.on" with the boolean value 
True. Which configuration files you have to edit to configure (and if you have 
to use a special syntax for some values) depends on your framework. We have 
more information about the supported frameworks in :ref:`TurboMail Adapters<adapters>`.

Throughout this documentation we use the ConfigObj syntax::

  mail.on             = True        # boolean value True for the option 'mail.on'
  mail.manager        = 'immediate' # string value 'immediate'
  mail.demand.threads = 5           # int value 5

This section deals with general configuration options which are not
specific to a single manager or transport. You find specific options in chapter
:ref:`Detailed TurboMail <details>`.


Configuration options
---------------------

The single most important option is **mail.on**. If try to use TurboMail without
setting this option as True, you will get an exception like this immediately:
"MailNotEnabledError: An attempt was made to use a facility of the TurboMail 
framework but outbound mail hasn't been enabled in the config file [via mail.on].
This was done on purpose because the default transport does not send out mails
(so you don't spam real people with mails accidentally generated in your
unit tests [#]_) but just not sending the mails could be frustrating experience 
for new users when their mails go to /dev/null. Furthermore this saves some bytes
of RAM because neither a manager nor a transport will be loaded when TurboMail is 
not enabled.


* **mail.on** (boolean, default: False) Enable TurboMail.
* mail.manager (string, default: 'immediate') Specify the name of the manager to use
* **mail.transport** (string, default: 'debug') Specify the name of the transport to use

* mail.message.<property name> (string or tuple) -- default values for properties of the Message class (see :ref:`Message properties <message_properties>`)


The individual :ref:`managers` and :ref:`transports` can have their own 
configuration options. These are explained in the section where the specific 
manager/transport is explained in more detail. We just make an exception to 
explain a mandatory option for the SMTPTransport (which is probably by far the 
most widely used one):

* **mail.smtp.server** (string, mandatory for SMTP transport) host name or IP address of the SMTP server which will take care of mails sent by TurboMail (only if SMTP transport was enabled). [#]_


.. [#] You do use foo@example.com as recipient for all your test emails, right?
       The example.com domain is reserved for documentation so you will never
       bother someone if you sent a mail to this domain. Usage of existing domains
       for test purposes is really a pain and kills kittens. And yes, 
       `donotreply.com <http://www.donotreply.com>`_ *does* exist.
.. [#] We deliberately did not set 'localhost' as default value for that because
       on Un*x machines there is often an SMTP server running on the local 
       machine which may decide to deliver messages to real mail servers. After
       you mailed a badly written test email to a few thousand customers 
       accidentally you'll understand our paranoia here :-)


How transports, managers and extensions are found by TurboMail
--------------------------------------------------------------

TurboMail heavily utilizes `setuptools <http://peak.telecommunity.com/DevCenter/setuptools>`_
to build a modular, loosely coupled and extensible system. When you specify 
"mail.transport = 'smtp'" in your configuration, TurboMail tries to load the
class associated to the entry point 'turbomail.transports.smtp'. If no class could
be loaded, an error will be logged and TurboMail will be disabled.

If you try to use TurboMail after the an error, you will get the same 
MailNotEnabledError as if you forgot to specify "mail.on = True".

The same holds true for managers (entry point group 'turbomail.managers') and 
extensions (entry point group 'turbomail.extensions'). The only difference
related to extensions is that TurboMail won't be disabled if an extension could
not be loaded.


Add new transports without using setuptools
-------------------------------------------

Sometimes you don't want to rely on setuptools for manager/transport discovery.
For example if you run the unit test suite for TurboMail itself, we don't want 
to force you to install the TurboMail egg in your environment. Therefore we added
a mechanism so that you can use transports and managers in TurboMail without the 
requirement that setuptools must be able to find them via its normal entry point 
mechanism::

    config = {'mail.on': True, 'mail.manager': 'foo'}
    # assuming you built a manager class called MyOwnManagerClass
    interface.start(config, extra_classes={'foo': MyOwnManagerClass})

If extra classes where provided, TurboMail will look in this dict *before* the
lookup with setuptools. If you use this mechanism, you can be sure that the 
specified manager class is used regardless of the environment configuration 
which is nice especially for unit tests.


Logging
-------

TurboMail supports logging through Python's standard logging module. All loggers
utilized by TurboMail start with 'turbomail.' so you can easily disable all 
logging (don't forget to stop propagation if you want to silence TurboMail). 
However we recommend setting the log level for TurboMail to WARNING so you will
be notified of potential problems.

Furthermore there is a special delivery log where all delivery attempts are 
noted. To handle mail delivery log messages differently from other TurboMail
log messages (e.g. for later auditing), you can define other handler/log level
settings on 'turbomail.delivery' (mail deliveries are logged with log level 
'INFO').


Unit Tests with TurboMail
=========================

Testability was advertised as a feature in the beginning. So how to use 
TurboMail in your (unit) tests?

First of all, you should use the ``ImmediateManager`` and the ``DebugTransport``
for testing so you get sent messages as fast as possible (ImmediateManager) and
you can collect your messages easily without any interaction with your 
environment (DebugTransport). Fortunately, this is just the default configuration
for TurboMail if you don't configure any managers or transports explicitly.

A simple test case looks like this::

    import unittest
    
    from turbomail.control import interface
    from turbomail.message import Message
    
    class TestYourAppWithTurboMail(unittest.TestCase):
        def setUp(self):
            config = {'mail.on': True}
            interface.start(config)
        
        def tearDown(self):
            interface.stop(force=True)
            interface.config = {'mail.on': False}
        
        def test_fetch_sent_messages(self):
            # call your app - do some interesting things
            # ...
            # no you can retrieve the sent mail:
            send_mails = interface.manager.transport.get_sent_mails()
            self.assertEqual(1, len(send_mails))
            sent_mail = send_mails[0]
            # check that the mail contains the expected information

This test assumes that TurboMail is installed in your environment with all the
necessary egg information so that it can load the ImmediateManager and the 
DebugTransport. If you want to be independent of setuptools, please have a look
a TurboMail's test_debug_transport.py test case does not rely on setuptools too.

If you want to test a TurboGears 1.0 application and you use a test case which
calls turbogears.start() you don't have to issue interface.start and 
interface.stop because TurboGears will handle that for you. Additionally you 
should not configure TurboMails' interface but just use turbogears.config to set
the appropriate values for TurboMail.

If you want to see some real test cases with TurboMail, you may want to look in
TurboMail's own test suite for examples.



Compatibility with TurboMail 2.x
================================

For TurboMail 3 we reworked all internals. Nevertheless we tried to maintain
compatibility for the most important interfaces. The idea is that 90% of all
applications using TurboMail 2 will "just work" with TurboMail 3 without the
need to change any code (just one line in the configuration). However we
deprecated some interfaces and properties. If you use one of these, a 
DeprecationWarning will be raised. You can disable these warnings in your 
application easily::

  from warnings import filterwarnings
  filterwarnings('ignore', 'warning text specified with a regular expression', 
                 category=DeprecationWarning)
  # Now you can use the deprecated interface and no warning will be given

The most important changes for upgraders are probably that the default manager
is now the ImmediateManager (message delivery is synchronous by default, your
code needs to deal with exceptions raised during the mail delivery) and the 
default transport is the DebugTransport which means that **no mail is delivered 
unless you use the SMTPTransport in the configuration**.


Configuration
-------------

One of the main features of TurboMail 3 are different types of transports. In 
order to send messages with SMTP in your production system you *must* add this 
configuration option::

  mail.transport = "smtp"

If you forget to do this, no messages will be delivered!


In TurboMail 2 all parameter keys used the naming convention 'mail.<name>'. With
TurboMail 3 this often does not make sense because we have different transports
now (not only SMTP) and many options are only useful in a SMTP context. 
Therefore all SMTP related parameters are renamed from 'mail.foo' to 
'mail.smtp.foo'. The options mail.server, mail.username, mail.password, 
mail.debug and mail.tls were renamed. The old parameter names will continue to 
work but a DeprecationWarning is given when TurboMail encounters one of the old 
configuration names.


Message class
-------------

All properties from the old Message class should be still there. Please note 
that we renamed the 'recipient' property to 'to' to be more clear which mail
header is meant. 'smtpfrom' and 'replyto' were renamed in accordance to 
`PEP 8 <http://www.python.org/dev/peps/pep-0008/>`_ (the official style guide for
Python code) to 'smtp_from' and 'reply_to'.

The most important change functionality-wise is the rename of 'sender'. We felt
that 'sender' is ambiguous because it was used for the From header but there is
another header that is called 'sender', too. Therefore the attribute to set the
From header is now called 'author' and there the property 'sender' will now set
the Sender header. If you don't set the 'author' property but use 'sender',
your 'sender' property will be used for the 'author' so we keep the backwards 
compatibility.



Sending Messages
----------------

In TurboMail 2 you submitted messages for sending by doing this::

  from turbomail import Message, enqueu
  msg = Message(...)
  enqueue(msg)

In TurboMail 3 messages can send themselves so the code above should be written
as::

  from turbomail import Message
  msg = Message(...)
  msg.send()
  
  # alternatively you can send messages by doing:
  from turbomail import send
  send(msg)

Furthermore we felt that 'enqueue' is not a good name any more because now we
have different strategies of sending messages and not all managers will queue
your message. Therefore the 'enqueue' method is still there but is marked as
deprecated.


Broken Compatibility
--------------------

We deliberately broke compatibility for some add-on kludges made in TurboMail in
order to get a much nicer interface in TurboMail 3. The test mode of TurboMail 2
is completely unsupported although the same functionality is provided by the
DebugTransport (see unit testing with TurboMail). Any code that accessed 
turbomail._queue directly will be broken in TurboMail 3.

As noted above, we changed the default manager behavior to be synchronous and 
the default transport (DebugTransport) does not deliver any messages to the
outside world.

In TurboMail 2.x using the 'UTF-8-QP' encoding had the side effect that 
Quoted Printable was used to encode all messages with UTF-8. After sending one
message with UTF-8-QP all later messages using just UTF-8 would be encoded with
Quoted Printable. In TurboMail 3 using the 'UTF-8-QP' encoding will not affect 
other messages. If you want to have all UTF-8 encoded messages delivered with
Quoted Printable, you need to enable the :ref:`UTF8qp extension<utf8qp>`.



