Proposals/Email to Journal

The discussion of this feature can be found at http://mahara.org/interaction/forum/topic.php?id=3814

Preamble

In his keynote presentation at Mahara UK 2011, Mark Osborne described how he currently uses Posterous to e-mail blog posts from his mobile phone to Posterous. He then uses a RSS feed, imported into Mahara to display them on his Pages.

It strikes me that this is something that Mahara itself should support. There should be a way for users to e-mail journal entries in to Mahara.

Overview of suggested functionality

The functionality should allow users to submit both journal posts, and drafts. It should also support submission of entries to each of a users different journals.

Message Subject The subject of the e-mail could (and probably should) be used directly for the Journal entry.

Attachments It should also allow users to e-mail multi-media files within the message subject (e.g. pictures, video, audio) which could then be saved to the users file storage space, and included as normal within the journal entry.

As a related feature, users could also e-mail files to Mahara solely for storage in Mahara and not specifically for a journal entry (e.g. e-mail a video from a users mobile phone for later use in other areas of Mahara).

This could also be made to support uploading of files to groups.

External Videos Links to external videos, such as youtube could be replaced with proper embed code. For instance a link like http://www.youtube.com/watch?v=hl64aiheIPI could be replaced with the full <embed> code for that external video repository.

Tags The processor should also attempt to parse the e-mail to extract certain additional information from it. An example would be that of tags. These could be specified by specifying a line at the bottom of the e-mail such as:

 Tags: foo, bar, baz

To indicate that the tags foo, bar, and baz should be applied to the journal entry.

Receiving E-Mail

Following initial discussion, I can see two main options for this:

a specific e-mail address format; and
an e-mail mailbox which is polled using IMAP.

E-mail Address Format

E-mail is currently processed for the VERP feature in Mahara. When e-mail is sent from Mahara, the VERP system changes the Reply-To address of the outgoing e-mail to something that it can parse. This includes an obfuscated copy of the intended e-mail address, userid, and site key. If a sent e-mail is returned by the MTA, this address is parsed by Mahara. As a result of this use-case, the address format is intended to be machine readable, rather than human readable -- for example:

 AAA-BAQAAAA==dd1d93ca86ee049bd72e830f55e03bec@bounces.mahara.local

For an e-mail to journal-like feature to be a useful feature, the recipient address would need to be in a more human-readabla format. For example:

 <username>-journal[-<journaltitle>][-draft]@mahara.local

Where the journaltitle is an optional variable relating to the specific journal that a user wishes to post to. if a user doesn't specify the journaltitle, then a users' default journal is used. The addition of an optional journaltitle could be added to journals for this purpose.

For the case of users wishing to e-mail files for storage then an address format such as

 <username>[email protected]

If possible, it would be best if the static text strings could be localised (e.g. journal, draft, storefiles).

It should only be possible to submit posts went sent from an e-mail address within the users profile -- otherwise e-mail should be given a draft status.

The benefit of this approach is that messages are received straight into Mahara in a timely fashion.

Mailbox Fetch over IMAP

This option allows for greater adoption - for example in a shared hosting environment where it is not possible to modify the MTA configuration to direct e-mail to a php script.

The idea behind this option is to have a single e-mail account (e.g. [email protected]). This account would be checked using the IMAP protocol by a function called from the Mahara Cron.

Each message in the inbox is retrieved one-by-one and processed based on the certain keywords in the subject.

The recipient is matched against the e-mail address in a users profile, and if none are found, the message is rejected.

For a message where the recipient does match a users profile, the subject is then analysed:

If a valid keyword is found, then a message is posted into the journal associated with this keyword If no keyword is found, or the keyword is invalid, then the message is posted into the default journal.

This allows for subjects such as:

Subject: FWD: blah blah blah

to be published into the default journal, providing a keyword of FWD is not already established for that user.

A keyword could also be used for storing attachments within a message. Subject: save

And groups could also be catered for: Subject: group-<keyword>-save

The benefit of this option is that it would allow for wider adoption. This option is provided at the cost of timeliness to publish. Additionally, locking may have to be visited to prevent an e-mail from being published multiple times.

Technical Changes and Thoughts

Incoming e-mail should be handled by an e-mail processing script. Some e-mail is already processed by htdocs/lib/processemail.php for the VERP feature of Mahara. This could be extended to support journal posts, in addition to MTA returns (for VERP). It may be preferable to use a different mail domain for incoming e-mail than for VERP e-mail to help with the logic for this.

It would probably be best to move the processmail function from htdocs/lib/user.php to htdocs/lib/mahara.php. It should then be possible to separate out VERP-related e-mail from incoming e-mail.

Here are some other technical thoughts relating to the feature:

HTML E-mail would have to be processed and passed through HTML Purifier;
any attachments included within the e-mail would have to be stripped out and stored within the users file storage space. They would then have to be included back into the journal entry;
any attachments must still fit within the user quota;
an appropriate mechanism would have to be considered if the user quota was insufficient;

It would also be ideal if the system supported some level of localisation as mentioned above.

Security Concerns

E-mail isn't exactly a secure system. It's far to easy to fake the sender on an e-mail so this concern would need to be seriously considered. An attacker knowing the e-mail address of a user, and their username, would be able to forge e-mail which would immediately be displayed within that user's journal.

I have considered a number of options:

only allow draft status on journal entries;
changing the incoming address to an address similar to that used by the Secret URL feature; and
include a secret metadata field within the e-mail (e.g. Passcode: 12345 as per the Tags suggestion).

Draft Only

Whilst seeming slightly limiting at first, the draft only feature may be one of the best options. Rather than an e-mail being published straight to a (potentially) world-readable journal without any chance for the user to check that entry's content, it would be saved as a draft. The user could at any point log in to mahara to actually publish the entry.

Upon receiving a journal entry draft, Mahara could then send that user a confirmation e-mail informing them that an entry was successfully received. The entry text could be displayed. They could have the option of clicking a link to confirm that the e-mail is valid and allowing them to publish the entry on their Journal. This would not require that the user log on to Mahara - just that they click the link and have a web browser available.

As an extension of this, it could be combined with the existing VERP functionality to allow a user to simply reply to the confirmation e-mail to publish the entry.

In my opinion, this is probably the best option combining ease of use, with a reasonable level of user verification.

Secret posting address

Initially I considered this a reasonable solution to the problem, but am concerned with the usability of the feature if this were imposed.

The idea behind this option would be that, rather than the suggested e-mail address format, users could generate a per-journal e-mail address for e-mailing data to. This would ideally be a short string to make it easier to type into a mobile device. As an example [email protected]

After speaking with several people who work with end users, I feel that this option is less than ideal. It would require users to store the journal's e-mail address within their address book, and this may mean that they are less likely to actually use the functionality. If a user created a new journal for a specific project, then they would have to create a new address and store this somewhere accessible.

Metadata Field within the message subject or body

This is similar to secret posting address option but even less user friendly. Within the body or the subject of the e-mail, a pass phrase, or pass code could be included.

This requires even more work for users, and leaves potential for users to inadvertently expose their passphrase somehow.

Other Issues

Configuration Complexity

In order for this feature to work, a certain amount of configuration would be required:

an MX record would be required for the mahara host; and
the Mail Transfer Agent would need the appropriate configuration.

I suspect that this may be a limiting for many smaller installations, particularly those offered on shared hosting. I suspect that most shared hosting providers would not allow the relevant MTA configuration for this to work. It may be possible to provide a cron job to check a mailbox via IMAP for new posts, though this would still require appropriate configuration.