Proposals/SmartEvidence/SmartEvidence: Technical considerations for the MVP

• Use the Open Badges alliance framework for storing competency framework data because it would be extensible with badges (we wouldn't have to use them right away or all of them, but at least we'd have a framework in place that supports it)
• We are looking at a first implementation of SmartEvidence and thus know that some things are created manually / hard-coded on the backend.

Setting up a competency framework

• Initially, competency frameworks with up to 2 levels (standard and sub-standard) are set up directly in the code and made available to the institutions for choosing in collections so that users can be assigned sharing and copying permissions as per default Mahara.
• Administrator interface functionality would come at a later stage.
• Institution admins have a “Yes/No” toggle (similarly to skins) where they can decide whether to use SmartEvidence or not. If they don't, they won't get the toggle for associating a competency framework with a collection when viewing the collection settings page.
• Each standard / sub-standard gets
  • a short name which is used to identify the standard / sub-standard on the overview page; no longer than 10 or 15 characters;
  • a title which along with the short name is used to identify it in the annotation block; up to 255 characters;
  • an optional description which contains the key indicators or other narratives describing the standard / sub-standard; no character limit
• For each SmartEvidence collection the admin selects whether the portfolio author or an assessor (with the role of staff or admin) can “rate” evidence.
• It's also determined what the “rating” should be. The initial one would be: Meets, Partially meets, Doesn't meet. These strings are hard-coded at the moment for each framework. Thus, if someone wanted to have the same SmartEvidence collection in a different language, they could copy the code and change the language strins. As there is no global agree-on terminology for the “ratings”, language strings in core Mahara would be too hard, but we could look into offering a couple default ones.
• NOTE: A SmartEvidence collection would need to be set up in each language though in which it is to be used as we do not yet have the functionality to translate content on the fly / offer two versions.
• ? Provide a couple of optional scales that could be provided via Mahara to get people started?
• The table of the sub-standard has a “parent” column so it can easily be associated with a standard and then also exapnded to a third level for example.
• Only the top standard would get the grey background / be shown across, but not any other subsequent levels.
• When a framework only has one level / no further categorization, then the grey level is dropped.
• Make the overview table accessible: http://webaim.org/techniques/tables/data
• We are not looking into a smartphone interface at the moment, but want to get something to work for 10” plus screens as the matrix is already complicated enough.

Associating a collection with a competency framework

• SmartEvidence collections can only be set up on the institution or site level.
• In contrast to the initial idea of using categories to distinguish SmartEvidence collections, we are implementing a “Yes/No” selector on the “Title/description” settings page for the collection which is visible only on institution and site collections.
• Then the admin selects the competency framework that is available for the institution in which the collection is created from a select2 drop-down menu where the available frameworks are shown but the admin can also start typing to get the one they want.
• Once the framework has been chosen, the SmartEvidence overview page is added automatically as first page in the collection. At a later stage, users / admin would be able to decide whether to place it at the end, but right now the beginning would do.
• If it were easier and the SmartEvidence overview page should be treated more like a regular page (esp. seeing that our collections are still accessible via page URLs), then it should not be possible to remove it from the collection (it should only be able to be moved in the order of pages) and a user should not be able to delete that overview page as long as the entire collection still exists. Since the user can't edit the page itself, it should not get an edit button.
• At a later stage we could look into offering to change the title of that overview page, but for right now I would say an admin should be the one to decide what to call it as part of the hard-coded setup.
• The administrator can add pages to the SmartEvidence collection to indicate how many different pieces of evidence he expects. The creating and adding of pages functions in the standard Mahara way as is the deletion.
• Now, administrators can share the collection as usual.
• Aaron had mentioned to put some “Ajaxy” stuff on the SmartEvidence overview page, but I don't remember whether that was for the page navigation of for the imagery on the page.
• A user can add more pages to the SmartEvidence collection than are provided by the admin the same way they add pages to a collection right now.
• When a page is about to be deleted from a SmartEvidence collection, the learner will need to get a special notification to be alerted to the imact, e.g. losing annotations, not seeing the page in the competency framework anymore... The annotation blocks and all their data will be deleted.
• Student working with SmartEvidence overview page
• When a student clicks on one of the standards and wants to associate that with a page, a modal is opened in which the student can provide an annotation to explain the reasons for associating it with this standard / substandard. Since the user clicked on a particular standard from the overview page, the block would already have selected the appropriate standard in the drop-down menu.
• The student cannot change the annotation (except for 10 minutes right after finishing editing and before any feedback has been left).
• The annotation block is put in the bottom row of the evidence page and the block is retracted automatically.
• When a student is on a page in the collection that she wants to associate with a standard, she can put the annotation block into the page anywhere and again, “Automatically retract” is selected. The title of the block is automatically set according to the selection of the competency standard and the short name is added, e.g. “Annotation for 3.1” or “Annotation for Innovative Design”.
• As soon as a student put the annotation block onto a page and added her annotation, the overview page shows the icon for the standard having been associated with that page. Currently, that is the ring icon.
• Students can associate more than one standard with a page. For each standard, an annotation block is created automatically if the student selected the standard from the overview page or added manually when done from the page itself. That way, when a student decides to remove a page from a standard because it doesn't fit anymore, all other annotations would not be affected.
• Once an annotation block has been deleted, all its content (annotation and feedback) is gone and cannot be retrieved anymore. Students will receive a warning message when they are about to delete the block to know of the consequences.
• At a later stage, we could look into changing the annotations from the a block to something else, but for the time being, this solution might be the easiest.
• When a student puts an annotation block onto the page herself, she will need to select the appropriate standard. Since the page is in a particular SmartEvidence collection, a select2 drop-down menu in the block only displays the standards available in this framework. The student cannot change the selection of the standard once the block has been saved (except for up to 10 minutes and before any feedback has been left).
• LATER: We also discussed the idea to have a section on the page in which all annotations are displayed with tabs to easily move between them (might be tricky on mobile as there could potentially be yet another hamburger unless we don't use one). Or to have an overlay so that the annotations can be placed on the side. Again, mobile might be tricky here and it requires more investigation.

Assessor working with SmartEvidence overview page

• When an assessor has been given permission to view the collection (either by regular permissions or by portfolio submission), he can view the overview page as well.
• An assessor can only leave feedback for a particular competency standard when the student has already placed an annotation for that standard. Otherwise, the assessor can use the regular commenting functionality on pages and artefacts, but cannot yet assess the evidence using the framework.
• Once a standard has an annotation (the ring icon), he can click that standard, view the annotation by the student and leave feedback. The student then receives a notification about the feedback and the assessor is automatically added to the list of people who get notified when more feedback is left on that annotation (this should be very similar to how notifications work on comments). The assessor can also edit his feedback for up to 10 minutes or until the first additional feedback (this should be similar to comments; I can't remember if you could edit until another person left feedback or if it was for a maximum of 10 minutes).
• The assessor needs the staff or admin role in order to “rate” evidence.
• In addition, the assessor can “rate” the evidence and select from a drop-down menu which attribute it should get: Meets, partially meets, doesn't meet. These language strings are hard-coded into the framework in the beginning. A corresponding symbol indicating this status is then displayed on the overview page.
• The assessor can change the “rating” at any time and doesn't necessarily have to leave feedback.

Others viewing a SmartEvidence collection

The regular sharing permissions for collections apply to a SmartEvidence collection. As soon as someone has access to view the collection, they can also see the overview page and add feedback to an annotation. Regular users, i.e. those without staff or admin permissions, are not able to “rate” evidence. They can only leave feedback.

SmartEvidence overview page for the MVP

1. Pages contained within a SE collection are listed horizontally. For the MVP, we are not looking into adding a page thumbnail displaying an image or a thumbnail of the page, but just the title of the page (which could sit within the space). The row of pages is always visible so it's easy to review the competency standards.
2. An arrow to the right indicates that there are more pages that can be scrolled to. On mobile devices this could be a swipe in the future.
3. If a competency framework consists of 2 levels, the first level is displayed in a kind of heading to separate the individual sub-standards from one another more easily. That heading consists of shortname and title of the standard. A description is not displayed.
4. A sub-standard is displayed only with its shortname.
5. We gather the following information for each sub-standard:
   • short name for the overview table (in the mock-up that is 1.6)
   • title: the actual title of the element
   • description: The long description, list of indicators etc. that can be displayed when hovering over a sub-standard or clicking on it.
6. This column contains the number of items of evidence already collected for this sub-standard.
7. The two columns with the short name and the number of pieces of evidence for the sub-standard do not scroll when more pages are displayed.
8. Icon for when evidence has not (yet) been associated with a sub-standard.
9. Icon for when evidence has been associated with a sub-standard.
10. Icon for when the evidence was “rated” “Partially meets” the standard's criteria. This could also be another icon seeing that Font Awesome doesn't have a half-circle. We could employ the traffic light colors and have the circle with the tick for the “Meets” category.
11. Icon for when evidence was “rated” “Meets” the standard's criteria. This could also be a green circle with a tick in it.
12. There is no icon yet for “Doesn't meet the criteria”. It would be good to have a third possibility to make it easier for an assessor to see which evidence they have already reviewed. This could be a circle in red.

Most of the annotation block functionality already exists (except for the “rating” and some newer commenting features).

1. The annotation by the student is always shown at the top as it's the point of reference.
2. An assessor (someone with staff / admin permissions) sees the drop-down menu to “rate” the association with the standard / sub-standard.
3. Anybody who has access to the page, incl. author, can leave feedback. It is optional though and follows the same principles as comments (it can be made private).
4. The feedback is displayed in the same order as comments. They can also be replied to directly like comments.

1. Author of the SmartEvidence collection can write an annotation explaining why they associate the selected evidence page with the sub-standard.
2. The author can select the standard from a select2 drop-down menu in which the short name and the title are listed. Once the standard has been selected, the description is displayed below.
3. The rest of the block stays as already implemented.

Work flows for the MVP

Setting up a competency framework

• This is done in the code itself.
• Framework is associated with an institution.
• If a framework is associated with “No institution”, it can be used by any other institution as well.

Admin: Connect a competency framework with a collection

1. Site or institution admin sets up a collection on the site or institution level respectively.
2. On the “Title and description” page, there are two new items:
   • SmartEvidence: Yes/No
   • Competency framework: Drop-down menu (Select2 with options listed, but text can also be entered)
   • The “Competency framework” option is only visible (or not greyed out) once “SmartEvidence” is set to “Yes”. When SmartEvidence is selected, the “Competency framework” option is required.
3. When the collection is saved with the framework, an additional “page” is available in the collection. It is the SmartEvidence overview page. It is called “SmartEvidence”. It cannot be deleted from the collection, but like any other page, it can be moved. That way, users can have the overview page anywhere in their collection and not only necessarily at the beginning.
4. The “SmartEvidence” page does not appear in the list of Pages under Portfolio -> Pages because it cannot be edited or modified itself.
5. When the admin sets “SmartEvidence” to “No” on the “Title and description” page, a pop-up asks for confirmation as the collection will lose its SmartEvidence functionality and becomes a regular collection. The collections of the users who have already copied it are not affected though.
6. SmartEvidence collections follow the same permission rules as other collections: They can be shared and copied.

Student copies and views a SmartEvidence collection

1. A student can go to the URL of the collection that the admin has set up and click the "Copy" button as usual for any collection that they can copy.
2. . Once the copying is completed, the collection is in the student's portfolio an can be viewed there.
3. . By clicking the collection title, the student opens the collection. Typically, the SmartEvidence map would be the first or last page in the collection, but it can be moved to a different position.
4. . All pages that are part of the collection are displayed on the SmartEvidence overview page.
5. . A student can add or remove pages from the collection, but cannot remove the SmartEvidence overview page.

Sample matrix files

You can take a look at the collection of framework files and download those that you want to use or adapt.

Instructions on how to set up a file are available in the user manual.