Actions

Difference between revisions of "Proposals/MNet replacement/Connection manager"

From Mahara Wiki

< Proposals‎ | MNet replacement
 
(22 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
= Web Services Client Connection Manager =
 
= Web Services Client Connection Manager =
 +
 +
'''[https://bugs.launchpad.net/mahara/+bug/1393536 Mahara tracker item to discuss specifications]'''
  
 
= Summary =
 
= Summary =
Line 32: Line 34:
 
=== Description ===
 
=== Description ===
  
As a plugin developer, I want to be able to have a logical reference to a list of connections that refer to 1 or more web service end points that I can pass attributes to for web service call execution.
+
As a plugin developer, I want to be able to have a logical reference to a list of connections that refer to 1 or more web service end points
 +
With these connection objects I want to:
 +
* pass attributes to for web service call execution
 +
* use introspection to determine versioning and protocol differences for each end point
  
 
=== Preconditions ===
 
=== Preconditions ===
  
* Developing a plugin is dependent on inheriting from class 'Plugin' .
+
* Developing a plugin is dependent on inheriting from class 'Plugin' .  The Plugin class needs to encapsulate abstractions for discovery of client connections, and delivery of the connections to the end user (plugin developer).
  
 
=== Steps ===
 
=== Steps ===
Line 52: Line 57:
 
=== Description ===
 
=== Description ===
  
As a Mahara administrator I want an interface that enables me to configure a logical reference to a a list of web service connections.
+
As a Mahara administrator I want an interface that enables me to configure a logical reference to a list of web service connections.
  
 
=== Preconditions ===
 
=== Preconditions ===
Line 60: Line 65:
 
=== Steps ===
 
=== Steps ===
  
# Login as an Administrator, and go to Institution specific configuration.
+
# Login as an Administrator, and go to Webservice Client Configuration Manager.
 
# Add, amend, delete or reorder priority of client connection instances.
 
# Add, amend, delete or reorder priority of client connection instances.
 
# Drill into a client connection and specify name, version. protocol, authentication, and end point configuration.
 
# Drill into a client connection and specify name, version. protocol, authentication, and end point configuration.
 +
# assign one or more institutions to a client connection instance, or alternatively indicate that this instance is system global
 
# enable/disable client connection instance.
 
# enable/disable client connection instance.
 
# enable/disable all client connections.
 
# enable/disable all client connections.
Line 74: Line 80:
 
= Specification =
 
= Specification =
  
Client connection objects will be inherently tied to a plugin.  This enables existing functionality for automatic lookup and discovery of plugin features to be used and also restricts the context of web service client execution to:
+
Client connection objects are inherently tied to a plugin.  This enables existing functionality for automatic lookup and discovery of plugin features to be used and also restricts the context of web service client execution to:
 
# a single plugin
 
# a single plugin
# the contextual institution of the logged-in user
+
# the contextual institutions of the logged-in user, $user passed in, or system global if explicitly specified
# the access control rights of the logged in user
+
# the access control rights of the logged in user by virtue of the pervasive checks throughout the Mahara code base
  
 
In order to support this the changes required fall into two broad categories:
 
In order to support this the changes required fall into two broad categories:
* interactive configuration components for maintaining the connection metadata
+
* interactive configuration components for maintaining the connection metadata - the client connection manager
 
* Plugin framework class modifications to support web service client abstractions
 
* Plugin framework class modifications to support web service client abstractions
  
 
=== Client Connection Manager ===
 
=== Client Connection Manager ===
 +
 +
The purpose of the client connection manager is to provide a uniform way of managing web service client connection details, so that plugin developers do not have to boiler plate code (DRY).  This will reduce development time, enable Mahara administrators to go to one place only for web service client management, give a common look and feel to extensions, and reduce the potential for errors through repetition.
  
 
A plugin or component that needs a Web Service client connection:
 
A plugin or component that needs a Web Service client connection:
* requires a reference 'name' put in plugin 'lib.php' class definition file - refer to Mahara  'events' get_event_subscriptions() as an example
+
* requires an interface that enables the declaration of a reference 'name' put in the plugin 'lib.php' class definition file.  This name is a common handle that both connection manager and plugin can communicate with - refer to Mahara  'events' get_event_subscriptions() as an example
* multiple connections to various targets - this could be multiple remote web service function calls, or multiple external systems to be integrated with simultaneously for the same API
+
* a way of declaring characteristics (via metadata) required  of the client connection, by the plugin (most likely by the intended 3rd party integration service)
 +
* a common API interface for execution of the necessary web service calls over the configured client connections
 +
 
 +
The connection metadata has been designed with this flexibility in mind, covering the common protocols and authentication mechanisms surrounding SOAP, XML-RPC and REST.
  
 
==== Metadata ====
 
==== Metadata ====
  
{| class="wikitable" style="text-align: left;"
+
{| class="wikitable" cellspacing="0" border="1" style="text-align: left;"  
 
!colspan="4"  style="text-align: center;"|Client Connection Instance
 
!colspan="4"  style="text-align: center;"|Client Connection Instance
 
|-
 
|-
!Mandatory
+
!Mandatory (M) / Optional (O)
 
!Element
 
!Element
 
!Type
 
!Type
Line 121: Line 132:
 
|O
 
|O
 
|instance interface version - remote version supported for this instance
 
|instance interface version - remote version supported for this instance
 +
|String
 +
|None
 +
|-
 +
|M
 +
|is system global - is this instance system global
 +
|Boolean/Checkbox
 +
|False
 +
|-
 +
|O
 +
|Institutions - a list of one or more institutions assigned to this connection instance.  Disabled if "is system global"==True
 
|String
 
|String
 
|None
 
|None
Line 179: Line 200:
 
|GET
 
|GET
 
|}
 
|}
 +
  
 
Note:  
 
Note:  
 
* one of user/consumer key + password/consumer secret, or token, or X509 certificate must be specified.
 
* one of user/consumer key + password/consumer secret, or token, or X509 certificate must be specified.
 
* if additional parameters are specified, then POST/GET must be specified
 
* if additional parameters are specified, then POST/GET must be specified
 
  
 
=== Plugin Abstraction Layer ===
 
=== Plugin Abstraction Layer ===
  
 
Return a list of connection classes defined for this plugin.  These will be used to determine what connection classes are available for configuration in the connection manager. Empty if none.
 
Return a list of connection classes defined for this plugin.  These will be used to determine what connection classes are available for configuration in the connection manager. Empty if none.
 +
  
 
<source lang="php" enclose="div">
 
<source lang="php" enclose="div">
Line 197: Line 219:
 
}
 
}
  
function get_web_service_connections() {
+
function get_web_service_connections($user=$_USER) {
   # query connection manager tables for list of connections
+
   // query connection manager tables for list of connections
   # build and return a list of connection objects based on each client connection instance specification
+
   // build and return a list of connection objects based on each client connection instance specification
 +
  // context is set by user object - $_USER is default
 
...
 
...
 
}
 
}
Line 212: Line 235:
 
...
 
...
 
function define_webservice_connections() {
 
function define_webservice_connections() {
     return array(array('id' => 'my_connection', 'name' => 'My Plugin WebService Connection'), ...);
+
     return array(array('id' => 'my_connection', 'name' => 'My Plugin WebService Connection',
 +
                      'version' => 'X.XX', type => WEBSERVICE_CLIENT_TYPE_SOAP,
 +
                      'payload' => WEBSERVICE_CLIENT_PAYLOAD_SOAP', 'auth' => WEBSERVICE_CLIENT_AUTH_TOKEN,
 +
                      // Further default connection options
 +
                      ), ...);
 
}
 
}
 
...
 
...
 
</source>
 
</source>
 +
 +
define_webservice_connections() is used by the plugin developer to inform the connection manager what client connection classes it needs. Each client connection class must have an id and name defined, after this default values can be passed for all other metadata elements as described above.
  
  
Line 221: Line 250:
  
 
<source lang="php" enclose="div">
 
<source lang="php" enclose="div">
  foreach ($plugin->get_webservice_connections() as $connection) {
+
  foreach ($plugin->get_webservice_connections( // optional $user as context
 +
                            ) as $connection) {
 
     try {
 
     try {
 
         $results = $connection->call('core_user_get_users_by_field', array('field' => 'username', 'values' => array($dbuser->username)));
 
         $results = $connection->call('core_user_get_users_by_field', array('field' => 'username', 'values' => array($dbuser->username)));
Line 229: Line 259:
 
  }
 
  }
 
</source>
 
</source>
 +
 +
The connection manager uses either a passed in $user object of the default global context user - $_USER, to determine what client connections should be returned.  The user object provides a list of institutions, which is applied as a  lens in conjunction with the classes own define_webservices_connections() metadata to determine which client connections should be handed back.
 +
 +
This will be a combination of the appropriate institution specific and system global connections.
 +
  
 
The connection objects returned by get_webservice_connections() provide introspection capabilities so that the Plugin developer can know:
 
The connection objects returned by get_webservice_connections() provide introspection capabilities so that the Plugin developer can know:
* what client ID this is - this should equate to the logical definition of an interface
+
* what client class this is - this should equate to the logical definition of an interface
 +
* is this institution specific or system global
 
* version
 
* version
 
* protocol
 
* protocol
Line 241: Line 277:
  
 
get_webservice_connections() only returns the enabled connections, so if all are disabled including with the master switch, then none are returned.
 
get_webservice_connections() only returns the enabled connections, so if all are disabled including with the master switch, then none are returned.
 +
 +
The connection object can be derived from the existing auth/webservices plugin:
 +
* webservice_rest_client()
 +
* webservice_xmlrpc_client()
 +
* webservice_soap_client()
 +
 +
See: https://github.com/piersharding/mahara-contrib-auth-webservice/blob/master/webservice/testclient.php for example code
  
 
== Assumptions ==
 
== Assumptions ==
Line 258: Line 301:
 
==== Connection Manager ====
 
==== Connection Manager ====
  
The connection manager is an extension of the exiting Institution configuration page - navigate Adminstration -> Institutions -> 'Edit' Institution.
+
The connection manager is an extension of the exiting Institution configuration page - navigate Administration -> Webservice Client Connections.
 +
 
 +
'Webservice Client Connections' is a new first level Administration function, separate to 'Web Services'.
 +
 
 +
==== Functions ====
  
[[File:Mahara-Connection-Manager.png]]
+
===== Administer Webservice Client Connections page =====
 +
 
 +
 
 +
* Master switch - the master switch is for disabling/enabling all client connections.  This switch only disables client connections, it does not alter configuration defined client connection records.
 +
* Add another web service connection - add another client connection class.  a class can be added only once if the class is set as system global (see Connection class name page).  Connections can be reordered for priority (up/down arrows), and deleted.
 +
* Click on the (green) connection link to drill into the details of a client connection.
 +
 
 +
===== Connection class name =====
  
==== Connection Manager Master Switch ====
+
* Specify the client connection instance details - name, enabled, version, fatal, type, authentication method, webservice endpoint, user/password, token, certificate, and parameters.
 +
* an instance can only be marked global if it is the only instance of a class.
 +
* a list of institutions associated with this connection class can be added.  An institution association can only exist in one instance within a class.
 +
* SOAP may have token/user/WSSE for authentication
 +
* XML-RPC may have token/user/cert for authentication
 +
* REST may have token/user/OAuth for authentication
 +
* only display and accept value for certificate if XML-RPC is selected for type
 +
* User/Password is relabelled Key/Secret if OAuth is selected for authentication
  
The master switch for disabling/enabling all client connections is located on the main Web Services configuration page.
+
[[File:Mahara-Connection-Manager.png]]
  
[[File:Mahara-Connection-Manager-Master-Switch.png]]
 
  
 
== Classes, Utilities ==
 
== Classes, Utilities ==
Line 285: Line 345:
 
Client connection manager requires administration access for the given institution.
 
Client connection manager requires administration access for the given institution.
  
The global enable/disable switch for all connections requires global administrator access.
+
The global enable/disable switch for all connections requires site administrator access.
  
 
== Modifications to Other code ==
 
== Modifications to Other code ==
Line 317: Line 377:
 
# client connection manager configuration screens
 
# client connection manager configuration screens
 
# reference implementation and test plugin
 
# reference implementation and test plugin
 +
 +
 +
== Future Functionality ==
 +
 +
# It would be useful for the Plugin developer to be able to give hints about which client connection attributes are relevant eg: it could fix the protcol to REST, and the payload to JSON, and declare the 'is fatal' flag irrelevant.
 +
# The Plugin developer could provide a helper routine that calls the API via the connection manager screen so that the Mahara Administrator could know that the connection was correctly configured.  Not all web service calls would support this (possibly most would not), and it would rely on cooperative development in the target system.
 +
# exernal API interface for remote and automatic provisioning of client connections (as a basis for wizards for pain-free setup of integration between systems)

Latest revision as of 17:43, 11 July 2020

Web Services Client Connection Manager

Mahara tracker item to discuss specifications

Summary

This specification describes the new functionality for managing client connections (Mahara is the client) for web service interactions, and the utility functions/classes that abstract away the complexities of making web services calls for Plugin developers.

Target Release

Mahara 15.04

Owner

TBD


References

Inspiration for the implementation pattern was source from Developer_Area/Core_Subsystems/Artefact_Plugins_//_Artefact_Types, and the Web Services plugin .


Motivation

The idea behind this requirement is to abstract away the management of connection details from individual components of Mahara (eg. a plugin), so that each component does not need to maintain a set of configuration, and the management of that configuration is consistent in behaviour, look and feel.

Once there is a consistent management of Web Service client connections, it is also possible to abstract away the code in helper classes and functions for executing Web Service calls, which should streamline and component development, and ensure that components use a common code base for Web Service interaction.


Use Cases (Stories)

Plugin Developer

Description

As a plugin developer, I want to be able to have a logical reference to a list of connections that refer to 1 or more web service end points. With these connection objects I want to:

  • pass attributes to for web service call execution
  • use introspection to determine versioning and protocol differences for each end point

Preconditions

  • Developing a plugin is dependent on inheriting from class 'Plugin' . The Plugin class needs to encapsulate abstractions for discovery of client connections, and delivery of the connections to the end user (plugin developer).

Steps

  1. With each plugin developed, specify a list of connection 'handles' that represent client connections that my plugin will interact with.
  2. When I wish to execute a remote web service call, I can call a function that returns a list of prefabricated client connections. These client connections are an abstraction from the complexity of dealing with different web services protocols, authentication, and encoding schemas, so that I can concentrate on formulating the data packet, and interpreting the results.

Postconditions

Metadata for the Plugin related client connections has been defined.


Mahara administrator

Description

As a Mahara administrator I want an interface that enables me to configure a logical reference to a list of web service connections.

Preconditions

  • Plugin developers have defined the client connection related metadata.

Steps

  1. Login as an Administrator, and go to Webservice Client Configuration Manager.
  2. Add, amend, delete or reorder priority of client connection instances.
  3. Drill into a client connection and specify name, version. protocol, authentication, and end point configuration.
  4. assign one or more institutions to a client connection instance, or alternatively indicate that this instance is system global
  5. enable/disable client connection instance.
  6. enable/disable all client connections.

Postconditions

  • A list of client connection instances are maintained for each Institution.
  • The web service client connection instances must adequately describe the supported protocols of SOAP, XML-RPC, and REST using the payload encodings of SOAP, XML, URL, and JSON where appropriate. The connections must also support the configuration of authentication options of token, user+password, OAuth1.x and WSSE where appropriate.

Specification

Client connection objects are inherently tied to a plugin. This enables existing functionality for automatic lookup and discovery of plugin features to be used and also restricts the context of web service client execution to:

  1. a single plugin
  2. the contextual institutions of the logged-in user, $user passed in, or system global if explicitly specified
  3. the access control rights of the logged in user by virtue of the pervasive checks throughout the Mahara code base

In order to support this the changes required fall into two broad categories:

  • interactive configuration components for maintaining the connection metadata - the client connection manager
  • Plugin framework class modifications to support web service client abstractions

Client Connection Manager

The purpose of the client connection manager is to provide a uniform way of managing web service client connection details, so that plugin developers do not have to boiler plate code (DRY). This will reduce development time, enable Mahara administrators to go to one place only for web service client management, give a common look and feel to extensions, and reduce the potential for errors through repetition.

A plugin or component that needs a Web Service client connection:

  • requires an interface that enables the declaration of a reference 'name' put in the plugin 'lib.php' class definition file. This name is a common handle that both connection manager and plugin can communicate with - refer to Mahara 'events' get_event_subscriptions() as an example
  • a way of declaring characteristics (via metadata) required of the client connection, by the plugin (most likely by the intended 3rd party integration service)
  • a common API interface for execution of the necessary web service calls over the configured client connections

The connection metadata has been designed with this flexibility in mind, covering the common protocols and authentication mechanisms surrounding SOAP, XML-RPC and REST.

Metadata

Client Connection Instance
Mandatory (M) / Optional (O) Element Type Default
M name/handle (from settings - class level attr) String None
M enable/disable individual instance Boolean/Checkbox Disabled
M priority of instance Integer/Ordinal 1
O instance name - not necessarily unique but could be used by plugin to determine a different payload requirement String None
O instance interface version - remote version supported for this instance String None
M is system global - is this instance system global Boolean/Checkbox False
O Institutions - a list of one or more institutions assigned to this connection instance. Disabled if "is system global"==True String None
M is fatal - if error, what do we do ? Boolean/Checkbox False
M type (SOAP, XML-RPC, REST) String REST
M payload encoding (SOAP-SOAP, XML-RPC - XML, REST-URL,XML,JSON) String URL Encoded
M auth (SOAP- token, user+pass, wsse; XML-RPC- cert, token, user+pass; REST - token, user+pass, OAuth1.x) String None
M Web Service endpoint String None
O user/consumer key String None
O password/consumer secret String None
O token String None
O X509 Certificate for XML-RPC String None
O Additional parameter/values to pass String None
O Additional parameters POST body/GET String GET


Note:

  • one of user/consumer key + password/consumer secret, or token, or X509 certificate must be specified.
  • if additional parameters are specified, then POST/GET must be specified

Plugin Abstraction Layer

Return a list of connection classes defined for this plugin. These will be used to determine what connection classes are available for configuration in the connection manager. Empty if none.


class Plugin implments IPlugin {
...
function define_webservice_connections() {
    return array();
}

function get_web_service_connections($user=$_USER) {
  // query connection manager tables for list of connections
  // build and return a list of connection objects based on each client connection instance specification
  // context is set by user object - $_USER is default
...
}

...

class AuthPlugin extends Plugin {
...


class MyAuthPlugin extends PluginAuth {
...
function define_webservice_connections() {
    return array(array('id' => 'my_connection', 'name' => 'My Plugin WebService Connection',
                       'version' => 'X.XX', type => WEBSERVICE_CLIENT_TYPE_SOAP,
                       'payload' => WEBSERVICE_CLIENT_PAYLOAD_SOAP', 'auth' => WEBSERVICE_CLIENT_AUTH_TOKEN,
                       // Further default connection options 
                       ), ...);
}
...

define_webservice_connections() is used by the plugin developer to inform the connection manager what client connection classes it needs. Each client connection class must have an id and name defined, after this default values can be passed for all other metadata elements as described above.


In the plugin developers code, get_webservice_connections() is called and returns a list of client connection objects, that can be called to execute the relevant web services call.

 foreach ($plugin->get_webservice_connections( // optional $user as context
                            ) as $connection) {
    try {
        $results = $connection->call('core_user_get_users_by_field', array('field' => 'username', 'values' => array($dbuser->username)));
    } catch (Exception $e) {
         ...
    }   
 }

The connection manager uses either a passed in $user object of the default global context user - $_USER, to determine what client connections should be returned. The user object provides a list of institutions, which is applied as a lens in conjunction with the classes own define_webservices_connections() metadata to determine which client connections should be handed back.

This will be a combination of the appropriate institution specific and system global connections.


The connection objects returned by get_webservice_connections() provide introspection capabilities so that the Plugin developer can know:

  • what client class this is - this should equate to the logical definition of an interface
  • is this institution specific or system global
  • version
  • protocol
  • encoding
  • authentication details
  • parameters
  • end point
  • error handling (is fatal)

get_webservice_connections() only returns the enabled connections, so if all are disabled including with the master switch, then none are returned.

The connection object can be derived from the existing auth/webservices plugin:

  • webservice_rest_client()
  • webservice_xmlrpc_client()
  • webservice_soap_client()

See: https://github.com/piersharding/mahara-contrib-auth-webservice/blob/master/webservice/testclient.php for example code

Assumptions

  • Plugins that use client connections must deal with no connection objects passed by get_webservice_connections(), gracefully.
  • it is the Plugin developers responsibility to deal with errors thrown by the connection object. The 'is fatal' flag is essentially a hint that is passed in to help the Plugin developer decide whether or not they wish to intervene on an error being thrown.

Issues

Process Flow

Wireframes, & Mockups

Connection Manager

The connection manager is an extension of the exiting Institution configuration page - navigate Administration -> Webservice Client Connections.

'Webservice Client Connections' is a new first level Administration function, separate to 'Web Services'.

Functions

Administer Webservice Client Connections page
  • Master switch - the master switch is for disabling/enabling all client connections. This switch only disables client connections, it does not alter configuration defined client connection records.
  • Add another web service connection - add another client connection class. a class can be added only once if the class is set as system global (see Connection class name page). Connections can be reordered for priority (up/down arrows), and deleted.
  • Click on the (green) connection link to drill into the details of a client connection.
Connection class name
  • Specify the client connection instance details - name, enabled, version, fatal, type, authentication method, webservice endpoint, user/password, token, certificate, and parameters.
  • an instance can only be marked global if it is the only instance of a class.
  • a list of institutions associated with this connection class can be added. An institution association can only exist in one instance within a class.
  • SOAP may have token/user/WSSE for authentication
  • XML-RPC may have token/user/cert for authentication
  • REST may have token/user/OAuth for authentication
  • only display and accept value for certificate if XML-RPC is selected for type
  • User/Password is relabelled Key/Secret if OAuth is selected for authentication

Mahara-Connection-Manager.png


Classes, Utilities

Modifications as describes above to class Plugin


Database Tables

New table required keyed by institution, connection ID, and client connection instance ID for the connection instance details (see table above).

Connection ID should be prefixed with '<Plugin Name>_' to ensure uniqueness.


Security

Client connection manager requires administration access for the given institution.

The global enable/disable switch for all connections requires site administrator access.

Modifications to Other code

See Client Connection Master switch above.


Testing

As a testing strategy, it should be possible to create a local plugin that can perform web services calls to either another Mahara instance, or a Moodle 2.7 (or above) instance.

Dependencies

This functionality depends on the final implementation of the Web Services plugin into core. The dependency is for:

  • Mahara to Mahara testing
  • master switch panel
  • for the client connection object framework that this plugin provides.


Documentation Impact

Documentation is required for:

  • the general client connection configuration process
  • how plugin developers use the client connection features

Milestones

  1. Client connection master switch
  2. DML changes - new table
  3. class Plugin client connection abstraction support
  4. client connection manager configuration screens
  5. reference implementation and test plugin


Future Functionality

  1. It would be useful for the Plugin developer to be able to give hints about which client connection attributes are relevant eg: it could fix the protcol to REST, and the payload to JSON, and declare the 'is fatal' flag irrelevant.
  2. The Plugin developer could provide a helper routine that calls the API via the connection manager screen so that the Mahara Administrator could know that the connection was correctly configured. Not all web service calls would support this (possibly most would not), and it would rely on cooperative development in the target system.
  3. exernal API interface for remote and automatic provisioning of client connections (as a basis for wizards for pain-free setup of integration between systems)