Developer Area/Dev Team Ideas

To be discussed what is still relevant, what we've already done, or what is not necessary anymore

Current wishlist ideas from the Dev Team, these may be implemented at some point as time allows.

Documentation

API docs

• Developers like PHPDoc documentation, we should provide this on mahara.org (they'll be too big to ship with Mahara itself)
• We should probably generate it per major version, suggestion: api.mahara.org/1.0, api.mahara.org/1.1... what about api.mahara.org/stable and api.mahara.org/unstable as symlinks?
• This will require that we go through the files and change the @phpdoc tags until the structure is generated properly, and of course we will have to document all the functions that need it!
• Functions like the pieform callbacks probably shouldn't appear in the docs
• How and when shall they be generated? From git.mahara.org as a nightly cronjob?

Developer documentation

We have none, we want some. API docs are helpful but aren't exactly task based.

Question: how do we separate them out by version (for when APIs change)?

Writing themes

Things to cover:

• Theme inheritance, including how to write an entirely new theme
• Smarty
• Pieform templates

Artefact plugins

The tutorial method might not work here because they're so big.. but maybe it will.

Search plugins

Probably a low priority thing to document, pending search subsystem rewrite.

Database schema docs

• There are tools available to automatically generate a schema from a postgres database, we should investigate using those to provide schema docs
• We might need to add comments to each table and field (xmldb supports this)
• Not sure where they would be hosted? Probably generated in a similar way to the api docs - perhaps they should go on api.mahara.org?

Testing framework

As Mahara gets bigger, it's going to be harder to make changes without breaking behaviour. Automatic testing helps improve confidence that nothing has been broken by a change.

Testing a webapp is tricky, luckily there are many tools to help. We can use selenium tests for testing the frontend, and unit tests for core libraries. But whatever we do, we need to set up a good framework first

Unit tests

We're looking to draw on some Moodle experience here to help set up a framework.  Penny and Nigel had an IRC meeting with Nico Connault at around 1100 GMT on Dec 6 to see what information he could give us.

We decided on the following

1. We need to identify a strategy for creating test data:
   • Moodle currently has a full generator (which can be used to populate the database for demo purposes as well), and each test tells it which data it needs before the test runs, and cleans it up aftewards. It also keeps a full schema copy with a different dbprefix, and when you try and run a unit test, it checks to see if the schema needs updating (very similar to the main install/upgrade strategy).
   • Another strategy is to ship an SQL dump.
2. We need to identify a few basic low hanging fruit areas in Mahara that we can easily write tests for. This shouldn't be a large area of the codebase that would require a lot of refactoring to get it to a point we can write elegant tests, and it needent even be an area that we place highest priority on for having tests for, but a simple, well contained area that lets us get our hands dirty with a test framework to see how it works and can be used in the rest of the system. Some potential cases are:
   • Adding/removing/moving block instances within a view - View and BlockInstance are both objects and this code is quite well abstracted
3. We should expect that in order to be able to write clean tests to cover a lot of Mahara, we will need to refactor a lot of code. A prime example of this is lib/groups.php which we should refactor into a class and smaller, more self contained methods, splitting out sql-generation and data-fetching methods from functional ones, as we write tests for it.

Major reworking

Various subsystems of Mahara may or will need a rewrite over time.

Theme subsystem work

• Reworking theme support to modularise CSS/JS files
• Cut down size of default theme
• Make certain parts easier to override
  • favicon
• Correct caching of static files
• Documentation

Reworking things to remove the javascript dependency

We want Mahara to work without javascript enabled. This is important for accessibility. Currently, javascript is required for some parts of the system:

• The file artefact plugin absolutely requires it
• The blog artefact plugin absolutely requires it
• ...

We should list all parts requiring javascript, and what we are doing/can do to fix them.

Investigate David Mudrak's language translator

We should look at it and see how it works, what it provides, and if it's good we should advertise it/help with maintainership etc.

Maharactl

Maharactl is a hypothetical script that can drive a Mahara website.

The first prototype was a perl script using perl modules to provide an API to the database. We would probably need a solution that utilised existing code if this idea was to work properly.

This depends on sane core APIs, and may need the core to be decoupled from the web based interface before this is even remotely possible.

Web services api

See Maharactl section - need core APIs.

Note: We have an XMLRPC dispatcher in place which is used for MNET.

Making forums into an artefact plugin

The concept of interaction plugins was born before the work in Mahara 1.1 to make shared group artefacts.

Now we have such artefacts, it makes sense to implement forums as an artefact plugin, which can only be used in groups. This is opposed to something like the file artefact plugin, where both users and groups have file sections. (Note - there is no API to say whether a given artefact type can be used in a group or not - it's up to the artefact plugin itself to export the appropriate pages and code to do this).

Under such a model, groups might own posts and discussions but users would get republish rights on their own posts and discussions they had participated in. This would allow users to embed forum discussions and posts into views.

Profile section reworking

From an e-mail discussion:

> >     o Profile section reworking (make profile data NOT artefacts)
>
> why do you want to do that?  i sell that all the time (everything is an
> artefact) and people really like it.

Well I think this comes about after discussions I've had with RichardM.
Basically, back when views were done as view templates, everything as an
artefact was good and made sense because that's how things got into views.
But the blocktype concept does a great job of abstracting that. You can see
blocks we have now that have 0, 1 or more than one artefact in them.

Given that, the concept of an "artefact" begins to fade a little - at least
from a user's point of view. We used to say "an artefact is something that
can be put in a view" - but that statement does not tell the whole story
any more. Many artefacts can only be put in views in certain ways (e.g.
profile fields). Some have a special blocktype for them but many do not.
Plenty of things that are _not_ artefacts can be put in views.

However, "artefact" still makes sense in the mahara codebase. It's a bit
like Drupal's "node". Lots of things in mahara are artefacts, and thus are
provided by artefact plugins. It makes sense, for example, that forum posts
and discussions are artefacts even. I think it would be great if we can
refactor forum to be an artefact plugin so that posts and discussions can
be put in views.

Anyways, that's a really long way of telling the background to why I think
profile data should not be artefacts. At the end of the day, they're really
one tiny bit of metadata that goes with a user's profile - they're not an
"evidence of learning" in the same way blogs, files, forum posts are. They
can only be put into views in a couple of blocktypes right now and arguably
a bunch of them shouldn't be able to be put in views at all. For example,
why have "first name" and "last name" in the profileinfo blocktype? "Name"
makes more sense - but people know their name and can put it in anyway...
similar arguments apply for many of the other profile fields.

Dispatcher

Martyn reckons that Mahara could implement a call dispatcher to go with the new menu API. This would allow features such as '/profile' to go to '/artefact/internal/'.

Implementation details and issues:

• This would be implemented by index.php becoming the dispatcher, and mod_rewrite to send all unknown requests there
• index.php would look through the menu to find which page to dispatch to, and do so via include()
• index.php would then handle all 404 messages
• This means that existing pages will continue to work, and the site will work without mod_rewrite. The dispatcher will only be used for aliased URLs
• However, this means that every place where there is a URL, it has to be wrapped in a function call, so it can be output as the alias name or the full name depending on whether the dispatcher is being used.

Ported from the Catalyst wiki

(mainly pre 2012)

Log Observing

Nigel thinks it would be a good idea if an emaildigest target for the error logging was written. This target would scan the logs for errors that had occured over the past day and e-mail them to the administrator.

This would be useful because it would give a lot more context than the current postgres error e-mails - for example, full backtraces, times etc.

Other targets may prove useful too - such as an IRCbot target (imagine being joined to a channel that listed errors as they occur, and ask the bot for the backtrace etc).

Use PDO for Database Access

I believe Donal was talking about writing or having written such a layer. This would replace ADODB, which is probably a large culprit in the number of files included per page, and removes a dependency on a library. More discussion probably needed.

Profiling

Some simple profiling with xdebug/apd/other debugger and kcachegrind might reveal some simple performance tweaks that could be made, as well as giving us an idea roughly of what's happening every page load.

Application level logging

This should have been built in from the start so that writing reports would have been just a matter of processing the logs. As it is, we have to go through everything and find all the places that would be important to log and add it (ViewView, ViewUser etc)

Example artefact plugin

Moodle has a NEWMODULE.zip that you can unzip in mod/ and go through and change the name of the stubs and then write them. That would be incredibly useful for Mahara's artefact/ directory. (This could also be done for auth, notification and search).

Cleaning up vserver build scripts on infer

Infer has some scripts in /root/ to rebuild its vservers. They duplicate one another and there's also some scripts/files in there that are 'data' for other scripts (e.g. an apache config that is copied into one of the vservers). It would be better if the whole lot was refactored to prevent duplication.

Open ID support

1) Allow people to log in using open ID 2) Make Mahara an "Open ID provider".

These ideas courtesy of Donal, not sure what they even mean.

Update: Mahara supports BrowserID from Mahara 1.5 on.

Ajaxify Forums

Discuss

Hijaxify some JS only stuff

There's a wikipage for this

Performance

Eventually, we're going to run into a Mahara installation that is big, and might not be performing as well as it could. At this point, an audit of SQL queries will probably find that we need more indexes. Some playing suggests that indexes on tables where deleted=0 will probably help.

Replacing Smarty

Smarty is a rather large third party library we're using for templating. It provides us with a way of writing templates using a special templating language. But it might not be strictly necessary for Mahara.

We could create a new way of templating using next to no code. Richard thinks that Moodle's method of simply including another PHP file at the end of most PHP scripts leads to problems because too many variables are in scope. We could go one better by using a wrapper that takes a file name and hash of variables, and only puts those variables in scope of the template file.

Smarty differs from this in that it's more limited in the syntax it allows. This is potentially a good thing, because it enforces much of the logic to be done in the controller. But you could also argue that it's too limiting - currently we have to deal with things like dates by preprocessing them in the controller, rather than calling a function in the template having been passed a simple unix timestamp.

The end result of this change would be that we could ditch the smarty library - a 500K saving - and have a smaller, faster templating system. We'd have to be careful that we kept the appropriate controller/view logic in the right files, but that is a manageable issue with coding standards.

To do this change, we'd have to do the following:

• Write the templating functions that would eventually replace smarty() and smarty_core(). They'd have to do the same assigning that the smarty functions do.
• Port header.tpl, footer.tpl and the column templates to the new framework.
• Begin converting pages from the old method to the new. As we go, we'd have to see what things were being done in PHP because smarty was too limited, and move them out to functions called by the template files - and potentially move smarty template logic back to PHP
• Also convert places where smarty_core is used
• Once all the templates are translated, remove the smarty libraries and functions

This change should be begun soon after a major release. Hopefully not too close to it however. There is a timing issue, whereby we will probably have to do a reasonable amount of work on the stable branch which might involve template changes, and we wouldn't want to make it too difficult to port those changes to master. However, we'd probably want to make the change before branching from master for any major new features. There's leeway either side of this mythical date where one of the scenarios is harder and one easier, we'd have to pick a good time for it.

There is an upgrade cost too - this will quite literally bust any custom themes. Maybe this is an acceptable cost for a major version upgrade. We can provide instructions for porting themes and plenty of advance warning - and of course, versions of themes can be compatible with versions of Mahara.

Summary

Benefits of a switch

• Can use PHP in template files, which is more expressive than Smarty, potentially allowing us to move display logic out of controllers
• Don't have to include a third party library with Mahara/track it from upstream etc.
• Less code to do the same job. There is a small speed increase, as a) less PHP has to be parsed every page load, and b) smarty has to compile the templates once (NOTE: Nigel established 2008/09/21 that Smarty + templates are accounting for about 20 included files per pageload, out of the 60-100 that happen on a typical load).

Disadvantages of a switch

• Syntax of smarty encourages processing of data in controllers
• Cost to switching (development time, porting templates) is higher than the benefits. (Related - We probably won't get money for this)
• When the switch happens, existing themes will be broken

Undecided advantages

• Smarty is understood by more people vs. PHP is understood by more people

View revisions

Based on the view template and shared artefact work, view revisions are a lot easier to implement now. We can extend the 'copy' mechanism for views/artefacts to do deep copying, and can store these copies in the view/artefact tables, with NULL owner. Adding a new column for 'author' will allow us to keep the relationship between users and revisions intact, then a 'revision' column on the view table is the only other thing required.

We can implement 'roll back' simply by setting owner = author.

The 'author' concept allows us to implement copying of blogs also, which is currently not done because we'd have to change the ownership which seems like lying.

Hashing files on disk

Richard implemented a new field for file artefacts, whereby more than one can be linked to the same file on disk.

We could potentially do one better, and add in a waffer thin layer that hashes files on disk. Though perhaps that's a job for the filesystem, ZFS does this.

Interactions as artefacts

The idea of forum posts as artefacts has been floated. This wasn't thought about when we made the forums. But it makes sense that people might wish to use forum posts as evidence.

While we may have been able to do this before, the concept of shared artefacts raises the idea that maybe group plugins should really actually be artefact plugins that work in groups - similar to how files work in 1.1. Some plugins - such as the hypothetical 'wiki' or 'poll' - won't make sense for users but that wouldn't stop them being implemented as plugins that don't export user menu items/have pages that work for users.

Blockinstances as artefacts (and profile information as artefacts)

The Artefact/View model used to be quite strict - artefacts controlled how they were rendered into views.

Now, with blocktypes, this isn't the case. Some blocktypes have many artefacts in them, some have none at all. While it wouldn't make too much sense, there's no technical reason why certain artefacts might not even be able to be put into views nowadays.

This raises the question of why profile information needs to be artefacts. Nigel thinks it doesn't need to be - it made sense before, but not so much now. We could easily provide blocktypes to render certain profile information that makes sense...

Potentially, blockinstances could be artefacts now. This thinking came about after thinking about how we would implement 'view tabs'/'view pages' (more than one 'page' for a given view). There may well be cases where you'd want the same blockinstance in two tabs, something that's not really possible in the current model.

Subpages

• Usability Sprint '09