Developer Area/Webservices

This page describes Mahara's webservices system. The webservices system provides a standardized way for a Mahara site to communicate with other applications, via HTTPS requests.

It allows Mahara to act as a "service provider", accepting incoming HTTP requests that run functions in Mahara and/or return data about Mahara. Mahara plugins can expose "service groups", which are sets of functionality to be accessed via HTTP. The site admin can then configure access to these via the Webservices administration page.

It also allows Mahara to act as a "service requester", with internal Mahara PHP functions triggering HTTP requests that access webservices on outside applications. Mahara plugins can use the "Connection Manager" interface to describe incoming webservices they would like to be able to consume; site & institution admins can then enable/disable these connections, and configure the specific service providers they point to, and their authentication credentials.

Concepts

Here's a brief explanation of some of the terms used on the Web services admin page.

Web service requester master switch =

This controls whether or not Mahara allows outgoing webservice requests. That is, your Mahara server sending out HTTP requests to an external server. Specifically, it controls whether or not you can use webservices that have been configured using the Mahara connection manager. You can disable that sitewide, for all plugins and institutions, with this switch.

You can check whether it's enabled or not by checking $CFG->webservice_requester_enabled

Note that this does not prevent all code in Mahara from accessing outside services. There is still plenty of code that connects to other servers using Curl to make HTTP requests, and this control has no impact on those. All it controls is the connections configured via the connection manager.

Web service provider master switch

This controls whether or not Mahara will accept incoming webservice requests. That is, your Mahara server accepting requests form other servers, which cause stuff to happen on your server. You can disable that sitewide, for all plugins and services and institutions, with this switch.

You can check this with $CFG->webservice_provider_enabled.

Note that this does not 100% guarantee that no "webservices" will connect to your application. There are some scripts in Mahara that are designed to be machine-readable, which are not controlled by this setting. (For instance, generated RSS feeds.) There may also be programs that "screenscrape" Mahara, using the same HTTP methods as a normal user's browser. All this setting controls, is webservices that are configured via the Mahara webservices API.

Web service protocols

Controls which data formats we accept incoming requests in. REST is the most popular. See the Moodle webservices documents for specifics about these protocols.

These can be checked by calling webservice_protocol_enabled($protocol) where protocol is rest, oauth, xmlrpc, or soap.

These settings are dependent on the "Web service provider master switch". If that switch is disabled, all of these services are disabled.

Functions

These are, as the name implies, individual functions that can be accessed via webservices. They're declared by plugins (or by the /webservice core code), and they map to underlying PHP functions which are executed when the webservice is accessed.

Implementing

Functions are defined in files that sit in the services/functions directory of a plugin, e.g. htdocs/module/mobileapi/webservice/functions. The functions are grouped into classes, with each class in a file that shares its name. For each function that's meant to be exposed as a webservice, the class must also define a function called "*_parameters" and one called "*_returns", which return objects describing the parameters and return values of the webservice function.

This part is essentially unchanged from Moodle's webservices, so their documentation is quite helpful on this: https://docs.moodle.org/dev/Adding_a_web_service_to_a_plugin#Write_the_external_function_descriptions

Once the underlying PHP function has been implemented, along with its associated description functions, you also need to describe it in your plugin's webservice/services.php file. The Moodle documentation describes the format: https://docs.moodle.org/dev/Adding_a_web_service_to_a_plugin#Declare_the_web_service_function (Mahara doesn't support the new "services" field; instead, services have to list which fields they accept)

All functions share the same namespace for their published names, so the published names should be prefaced with their plugin type and name, e.g. module_mobileapi_get_blogs instead of just get_blogs. (However, the code doesn't enforce this.)

You can also look at the "mobileapi" module for an example in Mahara.

Note: for backwards-compatibility, I recommend making every webservice function return an associative array (an external_single_structure), with all return values stored as fields of that associative array. See notes about backward-compatibility design below.

Global variables

Some things to be aware of when running code in a webservices function.

$USER will be set to the user whose username/password was used for authentication, or to the user who owns the token that was used for authentication. This is handled by the webservices library itself (server.php)

$SESSION should not be relied on. Mahara's sessions rely on cookies, and most webservices clients don't accept and send cookies the way a normal web browser does. Additionally, the user is re-authenticated on each webservice request, which will clear most session data anyhow. You should try to write webservice functions so that each one can act alone, using only the data provided in its parameters.

Services / Service groups

A service group (called a "service" in Moodle, and sometimes in Mahara), is a collection of functions, grouped together for access control purposes. You can't grant access to individual functions; instead you have to put one or more functions into a service group, and then grant access to that.

Service groups, like functions, are declared by plugins. (Mahara 15.04 through 16.04 also declared some "demo" service groups, but those have been removed in Mahara 16.10). They can also be created manually via the web UI, by selecting which functions should be included.

Service groups provided by plugins have a component, which indicates which plugin provides them. (Due to coding legacy issues, the component always ends with "/webservice", e.g. "module/mobileapi/webservice".) Core service groups provided by "htdocs/webservice" would belong to the component "webservice". Manually created service groups have no component value (this is what indicates they were manually created). The list of functions in a plugin-created service group cannot be modified by admins via the web interface; manually created service groups can be modified.

Service groups can also have a shortname, to make it easier for automated systems to refer to them. Normally this is unnecessary, because webservice calls usually only need to specify the function's name. However, it is needed when trying to use htdocs/module/mobileapi/json/token.php to auto-generate an authentication token for a service.

Implementing

Services are declared in their plugin's webservice/services.php file (e.g. htdocs/module/mobileapi/webservice/services.php). Core services would go in htdocs/webservice/services.php.

The format for declaring services is unchanged from Moodle: https://docs.moodle.org/dev/Adding_a_web_service_to_a_plugin#Declare_the_service

One addition Mahara has made, is that each service can also have an apiversion value. This is an optional integer to help webservice clients deal with different servers that may be running different versions of the Mahara software. It's recommended to increment this number each time you change the definition or behavior of any of the functions in the service group.

(Currently Mahara core only exposes this number via htdocs/module/mobileapi/json/info.php, and then only for the "maharamobile" service group.)

Tokens

All webservices require are non-public and require authentication. One of the ways to authenticate is via tokens. Tokens are generated-per user, and allows access to one service group. (Only service groups that have "tokenusers" enabled can be accessed by token.)

Admins can generate tokens for users. For the "maharamobile" service group, users can also self-generate tokens through the web interface, or through the Mobile api doing it on their behalf.

Service users

Service users are "bot" user accounts in Mahara, which have been set to have "webservice" as their authentication method. They can authenticate to webservices using their Mahara username and password, if the service group has been set to allow "Restricted users", and the user is on the list for the service. If "Restricted users" and "Token users" are both enabled, users on the list can also authenticate via tokens.

OAuth

Mahara webservices also supports OAuth for authentication. Each application that connects to the server via OAuth will need its own Consumer Key (API Key), which a site admin can set up through the Webservices configuration page. This is rather like setting up an API key for Twitter or Google.

Connection manager

The connection manager is a way for plugins to indicate that they can consume webservices, and for site and institution admins to configure connections to webservice providers. For instance, a Mahara plugin might be written to consume webservices from a Mahara site. With the connection manager, an institution admin can then enable this connection for users in their institution, through Mahara's Administration pages. And different institutions can be configured to connect to different Moodle sites.

This feature is not ported from Moodle. (Although it was written with the idea in mind of connecting to Moodle webservices). Nor are there any connections declared by core plugins yet. So it still has some rough edges For an example of how to use it, see the connection manager examples topic in Gerrit.

Any plugin type can declare "connections", by adding a "define_webservice_connections" method to the plugin's class. For example: https://reviews.mahara.org/#/c/6666/7/htdocs/notification/redirect/lib.php

<?php
// Useless block that shows you the tags you have in another Mahara site.
class PluginBlocktypeRemoteTags extends PluginBlocktype {

    //... the usual blocktype stuff should be here...

    // Define a connection
    public static function define_webservice_connections() {
        return array(
            array(
                  'connection' => 'blocktype_remotetags_tagsource',
                  'name' => 'A connection to a webservice provider that can give me a list of tags',
                  'notes' => 'The service provider should expose the "get_tags" function.',
                  'version' => '1',
                  'type' => WEBSERVICE_TYPE_REST,
                  'isfatal' => false),
            );
    }
}

You'll notice this description is not very specific. There is a human-readable "Notes" section that describes the characteristic the service provider should have, but the connection declaration doesn't describe have a rigorous description of the function signatures, like the Service Groups and Functions did. All it really specifies is an API version number (for this connection), which protocol it prefers to use, and whether an error response from the service should be considered a fatal error or not.

This is because the connection manager is meant to be flexible and lightweight. All it really indicates is a name, on which we can use the web interface to hang some authentication details (the URL of the service provider, maybe a webservice token, maybe an OAuth token, maybe a username and password...). It's up to the admins to make sure they point it at the correct type of service provider. And it's up to our plugin to gracefully handle whatever response it gets from the service provider.

Also notice that I Franken-named the connection field: "blocktype_remotetags_tagsource". All connections share the same namespace, so it's a good idea to Franken-name them so you don't conflict with others.

Once declared, these connections then show up on the institution configuration screen, as well as the webservices connection screen. A site or institution admin can then go in and fill in the service provider's URL and other configuration details for the connection.

To then make use of this connection in your code, you call Plugin::get_webservice_connections() to retrieve any connection available to the current logged in user. This returns a $connection object, and you can call functions on it, which will be automatically translated into webservice requests, and return to you the response from the remote service.

A couple of quirks on this. First, because connections are defined per-institution, we have to indicate which user we're getting connections for. (Again, if we're doing a Mahara->Moodle plugin, we want to make sure we connect to the right Moodle for this user.) Second, because a user may belong to multiple institutions, Plugin::get_webservice_connections() returns an array of connections. It's up to the connection consumer to decide how they want to deal with those.

Going back to that same remote tags block, here's what the code looks like that tries to actually use the connection.

<?php
// Useless block that shows you the tags you have in another Mahara site.
class PluginBlocktypeRemoteTags extends PluginBlocktype {

    //... the usual blocktype stuff should be here...

    //... and then my define_webservice_connections() was here...

    //... and now here's where I use the connection...
    public static function render_instance($blockinstance) {
        global $USER;

        // We retrieve the connections this user can access
        $conns = static::get_webservice_connections($USER, 'blocktype_remotetags_tagsource');
        if (!$conns) {
           return "You have no remote tag sources.";
        }

        $output = '';
        foreach ($conns as $conn) {
           $limit = 50;
           $offset = 0;

           // $conn->call($functionname, $functionparameters)
           $results = $conn->call('get_tags', array($limit, $offset));

           // $conn->connection has all the metadata the admin entered
           // when configuring this connection.
           $output .= "Tags on $conn->connection->name:\n";

           if ($results['errorcode']) {
               $output .= "  Errored out!\n";
           }
           if (count($results['tags'])) {
               foreach ($results['tags'] as $tag) {
                   $output .= "  $tag\n";
               }
               if ($results['total'] > $limit) {
                   $remaining = $results['total'] - $limit;
                   $output .= "  ... and $remaining more!\n";
               }
           }
           else {
               $output .= "  (none!)\n";
           }
           echo "\n";
        }
        return $output;
    }
}

Best practices

Designing for backward-compatibility

In a perfect world, everyone would have the latest, most advanced and secure version of every software program. But in the real world, old software lingers. Especially old Mahara versions, which school IT departments sometimes don't have the resources to keep up to date.

So when writing webservice functions, you should keep in mind that down the road there may be webservice clients talking to really old versions of the server software, and vice versa. This can be a challenge, because the Mahara webservice interface is strictly typed (it has to be, to support things like SOAP). Here are some suggestions about how to write webservices so that backwards (and forwards) compatibility is easier.

Function parameters

If you have to add an additional parameter to an existing function, add it to the end of the function list. And give it a default value (which makes it optional). That way, older clients will still be able to call the function using the old set of parameters.

If you need to deprecate a parameter because it's no longer needed, leave that parameter in place and just ignore it. Note that it is deprecated or ignored in the function's documentation. It makes for messier code, but again, it allows older clients to still call the function with the old set of parameters.

Function return values

Return an associative array (that is, a external_single_structure) as the return value of the function, and make all the data be fields in that array. Then, if you need to output more information from the function in the future, you can simply add additional fields to the associative array. Older clients that don't expect those fields will simply ignore them.

On the other hand, if your function returns a scalar (an external_value) or a normal array (an external_multiple_structure), if you need to add additional return fields you'll have to change the output to an associative array, which will make the ouput be a different data type, which may choke and/or crash older clients.

For example, let's say I write a webservice function to retrieve an array of my Mahara tags:

<?php
// BAD: get_tags version 1
function get_tags() {
    return array_of_my_tags();
}

It returns an array. A Javascript client might try to process this return value with the Array.map() function:

var response = call_web_service('get_tags');
var tags = response.map(
    function(tag) {
        // transform strings to uppercase
        return tag.toUpperCase();
    }
);

But then I find some power users have thousands of tags. It's overwhelming webservice clients when they receive a huge response with every tag in it. So, I decide to make tags pageable. I add parameters to let the client say how many tags it wants to receive, and what "page" it's on. I also need to add a return value saying how many tags total there are, so the client knows whether or not they should continue to request more. The most sensible way to do that, is to return an associative array with the total in one field, and the actual list of tags in another.

<?php
// get_tags version 2
// As recommended, I've made the new parameters have default values, so they're optional.
function get_tags($limit = 0, $offset = 0) {
    $tags = array_of_my_tags($limit = 50, $offset = 0);
    $total = count($tags);
    if ($limit) {
        $tags = array_slice($tags, $limit, $offset);
    }
    return [
        'total' => $total,
        'tags' => $tags
    ];
}

But now, my Javascript client has a problem. In PHP, an associative array and a normal array are the same datatype, but in Javascript an associative array is an Object, and a normal array is an Array. The Object class has no "map()" function, so when my client gets to tags.map() it errors out!

Robust webservice clients will be written to type-check the return values before attempting to process them, but even so, the best case is that the client will now have to tell the user "Sorry, I received a response I don't know what to do with." Or a developer will have to update the client to handle the new return format.

By comparison, if I had done this in the beginning...

<?php
// GOOD: extensible version of get_tags
function get_tags() {
    return [
        'tags' => array_of_my_tags();
    ];
}

Then the Javascript client would have looked like this:

var response = call_web_service('get_tags');
var tags = response.tags.map( // Here's the bit that changed
    function(tag) {
        // transform strings to uppercase
        return tag.toUpperCase();
    }
);

It may feel redundant to have to add that extra layer of properties in the client ("response.tags.map" instead of just "response.map"). But, notice that this client won't choke if I later add that "total" return field. It will still just look at "response.tags", and the presence or absence of "response.total" won't cause any problems.

Use the service "apiversion" field (if available)

One of the changes we made from Moodle's standard was to add an "apiversion" field to services. This lets the client know which version of the API it's dealing with, which means the client code can be more easily written to deal with different versions of the webservice.

(Note: currently apiversion is only published for the "maharamobile" service group, exposed via htdocs/module/mobileapi/json/info.php. You'll need to write additional custom code to find the apiversion for other services. One possibility is to add this via the local_webservice_info function.)

Consider again the previous example with get_tags(). If I'm writing a webservice client that uses that function, odds are that out in "the wild" my client will have to access Mahara servers with both version 1 and version 2 of the function. Even if the function was written in an extensible way so that my client doesn't choke, it's still useful to know which version I'm dealing with so that I can page or not page, depending on whether the server supports it.

Without an explicit "apiversion", I'd have to try to deduce the server's version from its return format:

<?php
// Writing a webservice client in PHP (on another server)
$offset = 0;
$limit = 50;
$response = call_webservice('get_tags', array($offset, $limit));
if (array_key_exists('total', $response)) {
    $tags = $response['tags'];
    $hasmore = $total < ($offset + $limit);
}
else {
    $tags = $response;
    $hasmore = false;
}

if ($hasmore) {
   // ... maybe do some paging to get the rest of the tags.
}

Now, that'll work okay. But a future dev coming along is going to have no idea what the presence of absence of the 'tags' key indicates. That could be improved by adding some coding comments, but this kind of "versioning by deduction" has a bad tendency to lead to unreadable and unmaintainable code.

On the other hand, if I have an apiversion, I can be much more explicit in my code. This makes it easier to understand, and easier to test. If I decide in the future that I want to drop support for an older api version, that's easier as well.

<?php
$offset = 0;
$limit = 50;
$sort = SORT_ALPHABETICAL;
$api = fetch_api_version('tags_service');
if ($api < 2) {
    // We only want to call the server if it
    // supports paging.
    throw new Exception('Server too old!');
}
elseif ($api === 2 ) {
    // version 2 added paging
    $tags = call_webservice('get_tags', array($offset, $limit));
}
elseif ($api >= 3 ) {
    // API version 3 added sort order
    $tags = call_webservice('get_tags', array($offset, $limit, $sort));
}
$hasmore = $tags['total'] < ($offset + $limit);

Note that "apiversion" is a service attribute, not a function attribute. So you should update the apiversion for a service, any time you update any of the functions in the service.

It'd also be nice to provide a changelog detailing the precise changes. (But that's what Git is for. ;) )

Where the code lives

The webservices plugin is based on a port of Moodle's webservices functionality. So the Moodle webservices documentation may be useful: https://docs.moodle.org/dev/Web_services

Most of the library code is under htdocs/webservice. This is a kind of "pseudo-plugin", because the original version of Mahara webservices was written in 2011 before we had the general-purpose "module" plugin type.

Plugin-related functionality for web services, is provided under htdocs/auth/webservice. This acts partly as a placeholder for things like language streams and theme assets. It also provides some actual functionality. When you set a user to have "webservice" as their auth instance, it means that user can authenticate to webservices by username and password; and they can no longer log in to Mahara as a normal user. This can be useful for "bot" users that act as a placeholder to connect to a remote service.

module/mobileapi is a module added to support the Mahara Mobile application. It also adds some scripts that allow users to self-generate Webservice auth tokens via a JSON script; this makes SSO from the Mahara Mobile application easier. Some of this functionality should probably be generalized out to be available to other plugins; but currently it's hard-coded to this module.

History

Legacy APIs

The htdocs/api directory contains code for previous webservice functionality in Mahara.

MNet is a custom protocol implemented for Moodle and Mahara, which allows for single-sign-on between Moodle and/or Mahara sites, and for remote procedure calls between them. Webservices should, eventually, replace MNet.

api/mobile was implemented to support the MaharaDroid native Android application (and some third-party applications have used it as well). It is activated via the "Allow mobile uploads" site config setting. When activated, it makes the scripts under api/mobile live, and it adds a field to the user's account settings page, where the user can manually created "mobile access tokens".

The idea is that the user manually creates an easy-to-type mobile access token. Then they launch their mobile app, and paste the token in there. The mobile app can then authenticate itself to the api/mobile scripts by sending the access token along with the user's username. The mobile access token is "rotated", replaced by a new random GUID, on each page request, and the new value is sent back to the mobile app with the response. This was meant to provide some scant additional security, back when most Mahara sites went out over HTTP rather than HTTPS. However, in practice, it meant that the application's connection would die if a communication error prevented it from getting the new code, requiring an inconvenient new connection.

The 3rd-party webservices plugin

In 2011, a Mahara Dev ported Moodle's (then-experimental) webservices plugin to Mahara. This is described here: https://wiki.mahara.org/wiki/Plugins/Auth/WebServices

The version of the plugin in Mahara 15.04+ is based on this optional plugin, and it attempts to allow a site to migrate from the optional plugin to the new core systems.

Mahara 15.04: Webservices in core

Mahara 15.04 moved the webservices plugin into core. This was primarily an update of the old plugin, as well as cleaning up its user interface. This version of the plugin shipped with some standard "Service groups" which were not designed for any specific application, but were meant to be a kind of demo of the technology.

Mahara 16.10: Connection manager

Mahara 16.04 added the Connection Manager system, which is meant to streamline the ability for Mahara plugins to make outgoing webservices requests. It provides an API by which a plugin may describe the kind of connection it is able to handle (for instance, a plugin to communicate with Moodle might describe which Moodle webservices component and function it wants to call, and which authentication data it needs). Site and Institution admins can then configure "instances" of these connections, with different values for different institutions (to allow, for instance, different institutions to connect to different Moodle sites).

Mahara 16.10: module/mobileapi

Work on Mahara Mobile required cleaning up the webservices code some more, and creating an actual webservice to be used by a real app. The specific webservices needed by the Mahara Mobile app were placed under a dedicated module, mobileapi, which also includes some JSON-based scripts that aren't using the standard Mahara webservices engine.

Changes to the core webservices functionality outside of that module, include:

• Adding a "shortname" component to each service group, to make them easier for connecting applications to identify
• Clarifying how "restrictedusers" and "tokenusers" interact
• Allowing a webservice function parameter's default value to not match the type definition for that parameter. (The idea being that the type-enforcement is for incoming user data; whereas the default is a trustable hard-coded value, and being a different data type is often a useful signal to indicate the lack of a user-supplied value)
• Allowing individual users to generate and delete their own webservice access tokens
  • Although, lacking a general permissions system like Mahara, this functionality is currently limited to the mobileapi module, by hard-coding.