Actions

Difference between revisions of "Proposals/Audio and video recordings"

From Mahara Wiki

< Proposals
(2 intermediate revisions by the same user not shown)
Line 20: Line 20:
 
=Implementation idea=
 
=Implementation idea=
  
−
So, with all that info, and keeping in mind that we still need to clarify some requirements, the rough implementation plan would be as follows:
+
==General==
 +
So, with all that info, and keeping in mind that we still need to clarify some requirements, the rough implementation plan by AaaronW would be as follows:
  
−
* We enhance the existing "filebrowser" Pieform element to have additional controls for the recording of audio & video using getUserMedia(). (Ideally we'll use some existing wrapper library around getUserMedia() for better UI, cross-browser compatibility, and to make it easier to adapt when browser standards change)
+
# To the filepicker and the "File -> Contents" screen, we add "Record video" and "Record audio" buttons near the current "Upload" button/drag-drop.
−
* We use browser sniffing to detect whether getUserMedia() will be present, and if it's not, we fall back to an <input type="file" accept=...> element.
+
# We use OS-sniffing to customize these buttons depending on the user's browser. On Mobile they'll say "Record", and on Desktop they'll say "Upload", to reflect what will probably happen on the user's computer when they click them.
−
* Because iOS doesn't support uploading audio files, we add server-side file conversion for this use-case. Uploaded files that need to be converted will be put into a queue, and a cron job will convert them, using ffmpeg. While implementing this, it may also be a good idea to add automatic format conversion for all uploaded audio/video files into formats that have optimal cross-browser compatibility.  
+
# These buttons will use <input type="file" accept="audio/*"> and "video/*" tags. Browsers will respond to this by either launching an audio/video capture application, or opening a file upload dialog that tries to filter to just audio/video files. (Demo: https://jsfiddle.net/dipish/WWqR5/ )
−
* We create a new TinyMCE plugin (or 2 separate ones) for embedding audio & video files into rich text areas. This will be based on Mahara's existing image embed TinyMCE plugin, which uses the pieform filebrowser (which we'll have already extended to support recording audio/video). It will place a widget into the TinyMCE display area, which can use the same HTML5 player as our embedded media block, to display the audio & video in a playable manner. This will additionally require adding backend code to handle permissions on embedded audio/video, which will be based on the existing systems for permissions for embedded images.
+
# iOS doesn't support <input type="file" accept="audio/*"> so it needs special treatment, which I'll describe further down.
 +
# Once the file is uploaded, we put it into a conversion queue. A cron job looks through that queue and makes a "display copy" of the file suitable for display in a web browser via HTML5. For audio, this will be MP3 format, and for video it'll be H.264 MP4. The conversion will be done via the command-line utilities "avconv" or "ffmpeg". The path to these commands will be configurable by a config setting (similar to the old pathtozip config). If it's not present, conversion won't be done. This code can be written in an extensible manner so that new conversion programs for different operating systems can be added later, and the admin can choose which one to use.
 +
# We treat the "display copy" much like we treat image thumbnails. Its only purpose is as a workaround for web browser limitations, not an end in itself. So we don't ask the user for permission to do it, we don't count it against their file quota, and we delete or create them as needed.
 +
# To limit potential disk over-usage due to a/v display copies, we can make admin settings to set a maximum size of file that will be converted. We can also use high-compression settings on the converted files to limit the amount of space they use. We might also consider only doing the conversion once a file is placed into a block... but I prefer doing the conversion right away, so that we can display the file on its artefact detail page.
 +
# Because the "display copy" is only a sort of thumbnail, we retain the original file as well. The "Download" link for the file gives the user the original file, not the display copy.
 +
 
 +
==Audio in iOS==
 +
 
 +
In addition to the above:
 +
 
 +
# If we sniff that the user is on iOS, for the "Record audio" button we'll actually serve an "accept=video/*" button.
 +
# When an iOS user clicks this "Record audio" button, it will launch their video recorder/selector.
 +
# Once they record and upload a video, on the server side we will note that they were trying to record audio, and so we will convert the file to an audio file, change its name in the file area to end with the correct file suffix, and discard the original video file.
 +
# In testing on my local machine, I found that it takes about 1 second to convert each minute of video. So for small files (5 minutes or less) it would provide the best user interface if we just converted them immediately, as part of the upload script.
 +
# For larger files, we'll need to put them into the conversion cron queue. This means there may be several minutes during which the file has been uploaded but is not yet "usable" as an audio file. I propose that during this period we list the file in the user's content area (so they don't get confused and try to upload it again), but makes its name indicate "Pending conversion" or something similar, so that they know it's not yet usable. To avoid needing special display code to gray out Pending files, we could simply say that if they're downloaded, it serves an empty, zero-length file. Once the conversion queue converts the file, the "Pending" automatically leaves the filename, and download attempts correctly get the converted file.
 +
 
 +
==Recording audio/video in-browser, not yet==
 +
 
 +
There is an HTML5 API for accessing the camera and microphone from a web browser, the "navigate.getUserMedia()" API. But this technology does not yet seem to be mature. Tellingly, none of the major audio/video sharing websites use it (such as YouTube). And crucially, it doesn't work in iOS Safari.
 +
 
 +
The API is also very low-level, and would require additional extensive coding to add features needed for a good user experience, like time codes, indications of file size used and available remaining memory, the ability to trim the file after recording, etc.
 +
 
 +
The state of the audio API seems to be more mature than the video API at this point. There is an excellent demo available at https://webaudiodemos.appspot.com/AudioRecorder/index.html of an audio recorder that shows a sound wave as the audio is recorded. However, as mentioned, it doesn't work in iOS.
 +
 
 +
So for now, our users will get a better user experience from relying on recording programs outside the browser.
  
 
=Mahara Mobile=
 
=Mahara Mobile=
Line 38: Line 63:
 
=Further links=
 
=Further links=
  
−
[https://mahara.org/interaction/forum/topic.php?id=5473 Discussion on mahara.org]
+
* [https://mahara.org/interaction/forum/topic.php?id=5473 Discussion around audio and video recording in general on mahara.org]
 +
* [https://mahara.org/interaction/forum/topic.php?id=7498 Discussion around PoodLL on mahara.org]

Revision as of 15:44, 12 May 2016

Idea

It would be nice to be able to record audio and video directly into Mahara.

Browser support

The standard that the web seems to be triangulating on is the HTML 5 "getUserMedia/Stream" API: [1]. I've successfully used the demo in Chromium, Firefox, and Chrome for Android. Additionally, the web says that Edge (Microsoft) supports it. You can try some demos of it here: https://github.com/webrtc/samples#getusermedia

Unfortunately, there is currently *no* way to record audio and video directly in the browser on iOS. This is a rapidly-evolving area of standardization, so it may become available in the next year or two, but it's not there now.

What does PoodLL do?

There is a Moodle plugin called PoodLL, which allows the recording of audio and video in the browser, for Moodle. It uses two systems:

  1. If Flash is available, it uses a flash object to access the camera and microphone.
  2. If Flash is not available, it presents a standard HTML <input type="file"> file uploader object, with the additional flag "accept=audio/*" or "accept=video/*", which prompts a mobile browser to open its applications that can share or create audio & video files.
  3. In iOS, "accept=audio/*" is not supported, so PoodLL gives you an "accept=video/*" instead and converts the video file into an audio file on the server-side after it is uploaded. It can either use a file converter webservice provided by PoodLL, or it can convert them using ffmpeg on the web server.
  4. PoodLL uses a CSS hack to superimpose a nicer-looking button on top of the standard HTML file upload button.

Implementation idea

General

So, with all that info, and keeping in mind that we still need to clarify some requirements, the rough implementation plan by AaaronW would be as follows:

  1. To the filepicker and the "File -> Contents" screen, we add "Record video" and "Record audio" buttons near the current "Upload" button/drag-drop.
  2. We use OS-sniffing to customize these buttons depending on the user's browser. On Mobile they'll say "Record", and on Desktop they'll say "Upload", to reflect what will probably happen on the user's computer when they click them.
  3. These buttons will use <input type="file" accept="audio/*"> and "video/*" tags. Browsers will respond to this by either launching an audio/video capture application, or opening a file upload dialog that tries to filter to just audio/video files. (Demo: https://jsfiddle.net/dipish/WWqR5/ )
  4. iOS doesn't support <input type="file" accept="audio/*"> so it needs special treatment, which I'll describe further down.
  5. Once the file is uploaded, we put it into a conversion queue. A cron job looks through that queue and makes a "display copy" of the file suitable for display in a web browser via HTML5. For audio, this will be MP3 format, and for video it'll be H.264 MP4. The conversion will be done via the command-line utilities "avconv" or "ffmpeg". The path to these commands will be configurable by a config setting (similar to the old pathtozip config). If it's not present, conversion won't be done. This code can be written in an extensible manner so that new conversion programs for different operating systems can be added later, and the admin can choose which one to use.
  6. We treat the "display copy" much like we treat image thumbnails. Its only purpose is as a workaround for web browser limitations, not an end in itself. So we don't ask the user for permission to do it, we don't count it against their file quota, and we delete or create them as needed.
  7. To limit potential disk over-usage due to a/v display copies, we can make admin settings to set a maximum size of file that will be converted. We can also use high-compression settings on the converted files to limit the amount of space they use. We might also consider only doing the conversion once a file is placed into a block... but I prefer doing the conversion right away, so that we can display the file on its artefact detail page.
  8. Because the "display copy" is only a sort of thumbnail, we retain the original file as well. The "Download" link for the file gives the user the original file, not the display copy.

Audio in iOS

In addition to the above:

  1. If we sniff that the user is on iOS, for the "Record audio" button we'll actually serve an "accept=video/*" button.
  2. When an iOS user clicks this "Record audio" button, it will launch their video recorder/selector.
  3. Once they record and upload a video, on the server side we will note that they were trying to record audio, and so we will convert the file to an audio file, change its name in the file area to end with the correct file suffix, and discard the original video file.
  4. In testing on my local machine, I found that it takes about 1 second to convert each minute of video. So for small files (5 minutes or less) it would provide the best user interface if we just converted them immediately, as part of the upload script.
  5. For larger files, we'll need to put them into the conversion cron queue. This means there may be several minutes during which the file has been uploaded but is not yet "usable" as an audio file. I propose that during this period we list the file in the user's content area (so they don't get confused and try to upload it again), but makes its name indicate "Pending conversion" or something similar, so that they know it's not yet usable. To avoid needing special display code to gray out Pending files, we could simply say that if they're downloaded, it serves an empty, zero-length file. Once the conversion queue converts the file, the "Pending" automatically leaves the filename, and download attempts correctly get the converted file.

Recording audio/video in-browser, not yet

There is an HTML5 API for accessing the camera and microphone from a web browser, the "navigate.getUserMedia()" API. But this technology does not yet seem to be mature. Tellingly, none of the major audio/video sharing websites use it (such as YouTube). And crucially, it doesn't work in iOS Safari.

The API is also very low-level, and would require additional extensive coding to add features needed for a good user experience, like time codes, indications of file size used and available remaining memory, the ability to trim the file after recording, etc.

The state of the audio API seems to be more mature than the video API at this point. There is an excellent demo available at https://webaudiodemos.appspot.com/AudioRecorder/index.html of an audio recorder that shows a sound wave as the audio is recorded. However, as mentioned, it doesn't work in iOS.

So for now, our users will get a better user experience from relying on recording programs outside the browser.

Mahara Mobile

Additional possibilities open up if we consider the Mahara native app. We're currently in the process of migrating this from a native Android app (MaharaDroid) into a cross-platform Apache Cordova app (working name "Mahara Mobile"). If we do that, then we could:

  • Record audio & video directly in the Mahara Mobile app. There is a Cordova plugin for this, and evidence of people successfully using it in iOS and Android.
  • Expand Mahara Mobile to support importing audio & video from other applications (like it currently does with images)
    • Android: This is easy to do in Android. You just fire off an "Intent" that asks for audio or video, and the Android OS will open up all the applications that have registered as being able to provide that. It may also be possible to access the device's media storage area where music & ringtone files are stored.
    • iOS: This is more complicated in iOS. Well, for video it's easy: you can ask the system to access the camera and gallery apps. But there is no similar system for audio. One possibility is to make it a general filesystem access call, in which case we could access files recorded by other apps if they make them available. Another possibility is to use one of the third-party audio-sharing SDKs like AudioCopy or AudioBus. However, those SDKs are meant more for music composition & mixing applications, and may be too heavy-duty for Mahara Mobile.

Further links

Retrieved from ‘https://wiki.mahara.org/index.php?title=Proposals/Audio_and_video_recordings&oldid=7567’