Developer Area/Import//Export/Meeting 20081105

The following are notes from a developer meeting on IRC, held from 2008/11/05 10:50AM GMT to 12:30PM GMT. The meeting was held in #mahara-dev on freenode.

Attendees

Penny Leach and Nigel McNie

Agenda

• Architecture of the import side

Notes

Similarly to the discussion about exporting, we wanted to think about importing with a second import format in mind but we couldn't come up with a suitable candidate. Later on in the meeting we also discussed handling imports from Moodle over MNET which may be in a straight 'files' format rather than something rich like LEAP and that seemed to provide enough diversity for a broader view of the architecture.

We talked about whether we want to generate completely new accounts from an import, or just merge data into existing accounts, and agreed that administrative users might want to use an import to create accounts. Nigel raised the issue of how to deal with conflicts in the data when merging but then we got into talking about a 'Holding area' for imports which offsets that problem.

During the Manchester meeting, there was a lot of discussion about the Holding area and it was generally agreed that this was the best approach.

We talked for awhile about what this would look like from a user perspective, and both didn't like the idea of a wizard, but instead a page with two columns. On the left would be a list of information about each item in the import, followed by suggested targets and 'actions' on the right. Using the example of a piece of html, the potential targets might be 'blog post' or 'file', and the blog and file plugins would be asked to offer any additional options about where to import; file might give a list of target folders, and the blog plugin would give a list of available blogs.

Additionally, this can be used to handle conflict resolution. If one of the targets is a single-type artefact (eg some profile information and resume fields), Mahara can offer the user the option to overwrite the existing data.

For each item, the user can also opt to 'discard' it.

There can be multiple holding areas at any given time, and we decided that there should be a separate 'import' quota for users, so they can't just upload multiple large portfolio imports. The 'files' quota doesn't get added to until the user actually sends files into the file area.

These holding areas are persistent, but after a period of time, we might want to send a notification to the user that we're going to remove the import data in a week, to clean up stale imports. One all the items within an import are either sent to artefacts or discarded, the import is removed.

We discussed briefly the ability for users to set preferences about how to deal with these questions but didn't really resolve it. This can be added at an additional time without too much hassle.

We both agreed that it was very important to make sure the interface was user friendly and discussed how it should look for large imports. Both thought we should have the ability to filter by 'suggested type' (eg show me all things that could be blogs) and then bulk actions associated with results. We should be paginating, but when performing bulk actions, offer the ability to perform it for all items on the current page as well as all items matched by the filter, similar to gmail.

We then moved on to talking about the technical details.

Import plugins should live under import/ and the holding area should be common for all formats. We went back and forth about this a lot and eventually decided on the following workflow:

• file arrives in $format
• an entry is inserted into a new import table
• it is passed to import/$format/lib.php to be unpacked
• each potential artefact is inserted into import_item, with some common data
• this record can contain an 'externalid' field that links the record back to something in the import - in the example of LEAP this would be the itemid. Alternatively, if there is more data, but no unique identifier in the import file, the format might elect to put more data into ancillary tables.
• for each item, there's also a number of entries in import_item_artefactype.
• this record can contain an 'externalid' field that links the record back to something in the import - in the example of LEAP this would be the itemid. Alternatively, if there is more data, but no unique identifier in the import file, the format might elect to put more data into ancillary tables.
• for each item, there's also a number of entries in import_item_artefactype.
• the holding area gets back control, and displays all the items to the user. Each target plugin is asked to provide a small preview - eg an image icon or a blog post title. Format plugins can override the default display on the left column although this is unlikely.
• when the user makes a decision about where to send an item, the data is reworken (using the externalid and the import file, or the ancillary tables), and sent to the artefact for import.

Things still to discuss

Action items

• Penny to create very quick mockups.

Files

• File:Import-meeting 0.txt