Proposals/MNet replacement/Connection manager

Web Services Client Connection Manager

Mahara tracker item to discuss specifications

Summary

This specification describes the new functionality for managing client connections (Mahara is the client) for web service interactions, and the utility functions/classes that abstract away the complexities of making web services calls for Plugin developers.

Target Release

Mahara 15.04

Owner

TBD

References

Inspiration for the implementation pattern was source from Developer_Area/Core_Subsystems/Artefact_Plugins_//_Artefact_Types, and the Web Services plugin .

Motivation

The idea behind this requirement is to abstract away the management of connection details from individual components of Mahara (eg. a plugin), so that each component does not need to maintain a set of configuration, and the management of that configuration is consistent in behaviour, look and feel.

Once there is a consistent management of Web Service client connections, it is also possible to abstract away the code in helper classes and functions for executing Web Service calls, which should streamline and component development, and ensure that components use a common code base for Web Service interaction.

Use Cases (Stories)

Plugin Developer

Description

As a plugin developer, I want to be able to have a logical reference to a list of connections that refer to 1 or more web service end points. With these connection objects I want to:

• pass attributes to for web service call execution
• use introspection to determine versioning and protocol differences for each end point

Preconditions

• Developing a plugin is dependent on inheriting from class 'Plugin' . The Plugin class needs to encapsulate abstractions for discovery of client connections, and delivery of the connections to the end user (plugin developer).

Steps

1. With each plugin developed, specify a list of connection 'handles' that represent client connections that my plugin will interact with.
2. When I wish to execute a remote web service call, I can call a function that returns a list of prefabricated client connections. These client connections are an abstraction from the complexity of dealing with different web services protocols, authentication, and encoding schemas, so that I can concentrate on formulating the data packet, and interpreting the results.

Postconditions

Metadata for the Plugin related client connections has been defined.

Mahara administrator

Description

As a Mahara administrator I want an interface that enables me to configure a logical reference to a list of web service connections.

Preconditions

• Plugin developers have defined the client connection related metadata.

Steps

1. Login as an Administrator, and go to Webservice Client Configuration Manager.
2. Add, amend, delete or reorder priority of client connection instances.
3. Drill into a client connection and specify name, version. protocol, authentication, and end point configuration.
4. assign one or more institutions to a client connection instance, or alternatively indicate that this instance is system global
5. enable/disable client connection instance.
6. enable/disable all client connections.

Postconditions

• A list of client connection instances are maintained for each Institution.

• The web service client connection instances must adequately describe the supported protocols of SOAP, XML-RPC, and REST using the payload encodings of SOAP, XML, URL, and JSON where appropriate. The connections must also support the configuration of authentication options of token, user+password, OAuth1.x and WSSE where appropriate.

Specification

Client connection objects are inherently tied to a plugin. This enables existing functionality for automatic lookup and discovery of plugin features to be used and also restricts the context of web service client execution to:

1. a single plugin
2. the contextual institutions of the logged-in user, $user passed in, or system global if explicitly specified
3. the access control rights of the logged in user by virtue of the pervasive checks throughout the Mahara code base

In order to support this the changes required fall into two broad categories:

• interactive configuration components for maintaining the connection metadata - the client connection manager
• Plugin framework class modifications to support web service client abstractions

Client Connection Manager

The purpose of the client connection manager is to provide a uniform way of managing web service client connection details, so that plugin developers do not have to boiler plate code (DRY). This will reduce development time, enable Mahara administrators to go to one place only for web service client management, give a common look and feel to extensions, and reduce the potential for errors through repetition.

A plugin or component that needs a Web Service client connection:

• requires an interface that enables the declaration of a reference 'name' put in the plugin 'lib.php' class definition file. This name is a common handle that both connection manager and plugin can communicate with - refer to Mahara 'events' get_event_subscriptions() as an example
• a way of declaring characteristics (via metadata) required of the client connection, by the plugin (most likely by the intended 3rd party integration service)
• a common API interface for execution of the necessary web service calls over the configured client connections

The connection metadata has been designed with this flexibility in mind, covering the common protocols and authentication mechanisms surrounding SOAP, XML-RPC and REST.

Metadata

Client Connection Instance
Mandatory (M) / Optional (O)	Element	Type	Default
M	name/handle (from settings - class level attr)	String	None
M	enable/disable individual instance	Boolean/Checkbox	Disabled
M	priority of instance	Integer/Ordinal	1
O	instance name - not necessarily unique but could be used by plugin to determine a different payload requirement	String	None
O	instance interface version - remote version supported for this instance	String	None
M	is system global - is this instance system global	Boolean/Checkbox	False
O	Institutions - a list of one or more institutions assigned to this connection instance. Disabled if "is system global"==True	String	None
M	is fatal - if error, what do we do ?	Boolean/Checkbox	False
M	type (SOAP, XML-RPC, REST)	String	REST
M	payload encoding (SOAP-SOAP, XML-RPC - XML, REST-URL,XML,JSON)	String	URL Encoded
M	auth (SOAP- token, user+pass, wsse; XML-RPC- cert, token, user+pass; REST - token, user+pass, OAuth1.x)	String	None
M	Web Service endpoint	String	None
O	user/consumer key	String	None
O	password/consumer secret	String	None
O	token	String	None
O	X509 Certificate for XML-RPC	String	None
O	Additional parameter/values to pass	String	None
O	Additional parameters POST body/GET	String	GET

Note:

• one of user/consumer key + password/consumer secret, or token, or X509 certificate must be specified.
• if additional parameters are specified, then POST/GET must be specified

Plugin Abstraction Layer

Return a list of connection classes defined for this plugin. These will be used to determine what connection classes are available for configuration in the connection manager. Empty if none.

class Plugin implments IPlugin {
...
function define_webservice_connections() {
    return array();
}

function get_web_service_connections($user=$_USER) {
  // query connection manager tables for list of connections
  // build and return a list of connection objects based on each client connection instance specification
  // context is set by user object - $_USER is default
...
}

...

class AuthPlugin extends Plugin {
...


class MyAuthPlugin extends PluginAuth {
...
function define_webservice_connections() {
    return array(array('id' => 'my_connection', 'name' => 'My Plugin WebService Connection',
                       'version' => 'X.XX', type => WEBSERVICE_CLIENT_TYPE_SOAP,
                       'payload' => WEBSERVICE_CLIENT_PAYLOAD_SOAP', 'auth' => WEBSERVICE_CLIENT_AUTH_TOKEN,
                       // Further default connection options 
                       ), ...);
}
...

define_webservice_connections() is used by the plugin developer to inform the connection manager what client connection classes it needs. Each client connection class must have an id and name defined, after this default values can be passed for all other metadata elements as described above.

In the plugin developers code, get_webservice_connections() is called and returns a list of client connection objects, that can be called to execute the relevant web services call.

 foreach ($plugin->get_webservice_connections( // optional $user as context
                            ) as $connection) {
    try {
        $results = $connection->call('core_user_get_users_by_field', array('field' => 'username', 'values' => array($dbuser->username)));
    } catch (Exception $e) {
         ...
    }   
 }

The connection manager uses either a passed in $user object of the default global context user - $_USER, to determine what client connections should be returned. The user object provides a list of institutions, which is applied as a lens in conjunction with the classes own define_webservices_connections() metadata to determine which client connections should be handed back.

This will be a combination of the appropriate institution specific and system global connections.

The connection objects returned by get_webservice_connections() provide introspection capabilities so that the Plugin developer can know:

• what client class this is - this should equate to the logical definition of an interface
• is this institution specific or system global
• version
• protocol
• encoding
• authentication details
• parameters
• end point
• error handling (is fatal)

get_webservice_connections() only returns the enabled connections, so if all are disabled including with the master switch, then none are returned.

The connection object can be derived from the existing auth/webservices plugin:

• webservice_rest_client()
• webservice_xmlrpc_client()
• webservice_soap_client()

See: https://github.com/piersharding/mahara-contrib-auth-webservice/blob/master/webservice/testclient.php for example code

Assumptions

• Plugins that use client connections must deal with no connection objects passed by get_webservice_connections(), gracefully.
• it is the Plugin developers responsibility to deal with errors thrown by the connection object. The 'is fatal' flag is essentially a hint that is passed in to help the Plugin developer decide whether or not they wish to intervene on an error being thrown.

Process Flow

Wireframes, & Mockups

Connection Manager

The connection manager is an extension of the exiting Institution configuration page - navigate Administration -> Webservice Client Connections.

'Webservice Client Connections' is a new first level Administration function, separate to 'Web Services'.

Functions

Administer Webservice Client Connections page

• Master switch - the master switch is for disabling/enabling all client connections. This switch only disables client connections, it does not alter configuration defined client connection records.
• Add another web service connection - add another client connection class. a class can be added only once if the class is set as system global (see Connection class name page). Connections can be reordered for priority (up/down arrows), and deleted.
• Click on the (green) connection link to drill into the details of a client connection.

Connection class name

• Specify the client connection instance details - name, enabled, version, fatal, type, authentication method, webservice endpoint, user/password, token, certificate, and parameters.
• an instance can only be marked global if it is the only instance of a class.
• a list of institutions associated with this connection class can be added. An institution association can only exist in one instance within a class.
• SOAP may have token/user/WSSE for authentication
• XML-RPC may have token/user/cert for authentication
• REST may have token/user/OAuth for authentication
• only display and accept value for certificate if XML-RPC is selected for type
• User/Password is relabelled Key/Secret if OAuth is selected for authentication

Classes, Utilities

Modifications as describes above to class Plugin

Database Tables

New table required keyed by institution, connection ID, and client connection instance ID for the connection instance details (see table above).

Connection ID should be prefixed with '<Plugin Name>_' to ensure uniqueness.

Security

Client connection manager requires administration access for the given institution.

The global enable/disable switch for all connections requires site administrator access.

Modifications to Other code

See Client Connection Master switch above.

Testing

As a testing strategy, it should be possible to create a local plugin that can perform web services calls to either another Mahara instance, or a Moodle 2.7 (or above) instance.

Dependencies

This functionality depends on the final implementation of the Web Services plugin into core. The dependency is for:

• Mahara to Mahara testing
• master switch panel
• for the client connection object framework that this plugin provides.

Documentation Impact

Documentation is required for:

• the general client connection configuration process
• how plugin developers use the client connection features

Milestones

1. Client connection master switch
2. DML changes - new table
3. class Plugin client connection abstraction support
4. client connection manager configuration screens
5. reference implementation and test plugin

Future Functionality

1. It would be useful for the Plugin developer to be able to give hints about which client connection attributes are relevant eg: it could fix the protcol to REST, and the payload to JSON, and declare the 'is fatal' flag irrelevant.
2. The Plugin developer could provide a helper routine that calls the API via the connection manager screen so that the Mahara Administrator could know that the connection was correctly configured. Not all web service calls would support this (possibly most would not), and it would rely on cooperative development in the target system.
3. exernal API interface for remote and automatic provisioning of client connections (as a basis for wizards for pain-free setup of integration between systems)