Mirrors/coursera-dl
1
0
Fork 0
mirror of https://github.com/coursera-dl/coursera-dl.git synced 2026-07-18 00:45:26 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #607 from dzhuang/new_dl_features

Browse source 
New downloading features along with adaptations to the new APIs

- Enable multiple subtitles and transcripts with alternatives.
Sometimes we need to download subititles in different languages, for example, 'en' and 'zh-CN', which is not supported currently. The PR enabled downloading multiple subtitles & transcripts, along with fallback for each language, by using the following format:
```
--subtitle-language <lang1,lang2|fallback1_for_lang2|fallback2_for_lang2>
```
This can will also be an alternative fix for #596 .
- Enable downloading resources blocks.
Some materials are now in a new blocks named ``Resources``, which corresponed to the new API ``onDemandReferences.v1``. Fixing #597 

- Enable file to store parameters.  
There are cases where there are too many parameters (besides username and password) to be set each time we download a course. I use ``configargparse`` to read configurations stored in a local file named ``coursera-dl.conf``. Fixing #97 

- Enable downloading new typename ``programming``
That is different from original ``gradedProgramming`` and ``ungradedProgramming`` so it is not recognized by coursera-dl currently. An example can be find at the end of [this page].
The new API is ``onDemandProgrammingImmediateInstructions.v1``.

- Allow alternative ``MathJax.js`` cdn url when downloading webpages, as the default [``MathJax.js`` url](https://cdn.mathjax.org/mathjax/latest/MathJax.js) is not accessible in some countries (this is the case at least for me).
This commit is contained in:
Yuri Bochkarev 2017-05-23 11:25:39 +03:00 • committed by GitHub
commit e02fc5177a
20 changed files with 730 additions and 46 deletions
Download patch file Download diff file Expand all files Collapse all files

29
README.md
View file

@ -70,6 +70,7 @@ I've downloaded many other good videos such as those from Khan Academy.
certain resources.
* File format extension filter to grab resource types you want.
* Login credentials accepted on command-line or from `.netrc` file.
* Default arguments loaded from `coursera-dl.conf` file.
* Core functionality tested on Linux, Mac and Windows.
# Disclaimer
@ -278,6 +279,23 @@ instead. This is especially convenient, as typing usernames (email
addresses) and passwords directly on the command line can get tiresome (even
more if you happened to choose a "strong" password).
Alternatively, if you want to store your preferred parameters (which might
also include your username and password), create a file named `coursera-dl.conf`
where the script is supposed to be executed, with the following format:
--username <user>
--password <pass>
--subtitle-language en,zh-CN|zh-TW
--download-quizzes True
#--mathjax-cdn https://cdn.bootcss.com/mathjax/2.7.1/MathJax.js
# more other parameters
Parameter which is stored in the file will be overriden if it is again specifed
in your commandline script
**Note:** In `coursera-dl.conf`, all the parameters should not be wrapped
with quotes.
## Resuming downloads
In default mode when you interrupt the download process by pressing
@ -341,7 +359,7 @@ one of the following actions solve your problem:
* If results show 0 sections, you most likely have provided invalid
credentials (username and/or password in the command line or in your
`.netrc` file).
`.netrc` file or in your `coursera-dl.conf` file).
* For courses that have not started yet, but have had a previous iteration
sometimes a preview is available, containing all the classes from the last
@ -456,6 +474,15 @@ If you still have the problem, please read the following issues for more ideas o
This is also worth reading:
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning
## Use an alternative cdn url for `MathJax.js`
When saving a course page, we enabled `MathJax` rendering for math equations, by
injecting `MathJax.js` in the header. The script is using a cdn service provided
by [mathjax.org](https://cdn.mathjax.org/mathjax/latest/MathJax.js). However, that
url is not accessible in some countries/regions, you can provide a
`--mathjax-cdn <MATHJAX_CDN>` parameter to specify the `MathJax.js` file that is
accessible in your region.
# Reporting issues
Before reporting any issue please follow the steps below:

227
coursera/api.py
View file

@ -25,12 +25,18 @@ from .define import (OPENCOURSE_SUPPLEMENT_URL,
OPENCOURSE_ONDEMAND_COURSE_MATERIALS,
OPENCOURSE_VIDEO_URL,
OPENCOURSE_MEMBERSHIPS,
OPENCOURSE_REFERENCES_POLL_URL,
OPENCOURSE_REFERENCE_ITEM_URL,
OPENCOURSE_PROGRAMMING_IMMEDIATE_INSTRUCTIOINS_URL,
POST_OPENCOURSE_API_QUIZ_SESSION,
POST_OPENCOURSE_API_QUIZ_SESSION_GET_STATE,
POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS,
POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS_GET_STATE,
INSTRUCTIONS_HTML_INJECTION,
INSTRUCTIONS_HTML_INJECTION_PRE,
INSTRUCTIONS_HTML_MATHJAX_URL,
INSTRUCTIONS_HTML_INJECTION_AFTER,
IN_MEMORY_EXTENSION,
IN_MEMORY_MARKER)
@ -135,9 +141,12 @@ class QuizExamToMarkupConverter(object):
class MarkupToHTMLConverter(object):
def __init__(self, session):
def __init__(self, session, mathjax_cdn_url=None):
self._session = session
self._asset_retriever = AssetRetriever(session)
if not mathjax_cdn_url:
mathjax_cdn_url = INSTRUCTIONS_HTML_MATHJAX_URL
self._mathjax_cdn_url = mathjax_cdn_url
def __call__(self, markup):
"""
@ -170,7 +179,11 @@ class MarkupToHTMLConverter(object):
soup.insert(0, meta)
# 1. Inject basic CSS style
css_soup = BeautifulSoup(INSTRUCTIONS_HTML_INJECTION)
css = "".join([
INSTRUCTIONS_HTML_INJECTION_PRE,
self._mathjax_cdn_url,
INSTRUCTIONS_HTML_INJECTION_AFTER])
css_soup = BeautifulSoup(css)
soup.append(css_soup)
# 2. Replace <text> with <p>
@ -386,7 +399,8 @@ class CourseraOnDemand(object):
"""
def __init__(self, session, course_id, course_name,
unrestricted_filenames=False):
unrestricted_filenames=False,
mathjax_cdn_url=None):
"""
Initialize Coursera OnDemand API.
@ -409,7 +423,7 @@ class CourseraOnDemand(object):
self._user_id = None
self._quiz_to_markup = QuizExamToMarkupConverter(session)
self._markup_to_html = MarkupToHTMLConverter(session)
self._markup_to_html = MarkupToHTMLConverter(session, mathjax_cdn_url=mathjax_cdn_url)
self._asset_retriever = AssetRetriever(session)
def obtain_user_id(self):
@ -721,29 +735,11 @@ class CourseraOnDemand(object):
video_url = sources[0]['formatSources']['video/mp4']
video_content['mp4'] = video_url
# subtitles and transcripts
subtitle_nodes = [
('subtitles', 'srt', 'subtitle'),
('subtitlesTxt', 'txt', 'transcript'),
]
for (subtitle_node, subtitle_extension, subtitle_description) in subtitle_nodes:
logging.debug('Gathering %s URLs for video_id <%s>.', subtitle_description, video_id)
subtitles = dom.get(subtitle_node)
if subtitles is not None:
if subtitle_language == 'all':
for current_subtitle_language in subtitles:
video_content[current_subtitle_language + '.' + subtitle_extension] = make_coursera_absolute_url(subtitles.get(current_subtitle_language))
else:
if subtitle_language != 'en' and subtitle_language not in subtitles:
logging.warning("%s unavailable in '%s' language for video "
"with video id: [%s], falling back to 'en' "
"%s", subtitle_description.capitalize(), subtitle_language, video_id, subtitle_description)
subtitle_language = 'en'
subtitle_link = self._extract_subtitles_from_video_dom(
dom, subtitle_language, video_id)
subtitle_url = subtitles.get(subtitle_language)
if subtitle_url is not None:
# some subtitle urls are relative!
video_content[subtitle_language + '.' + subtitle_extension] = make_coursera_absolute_url(subtitle_url)
for key, value in iteritems(subtitle_link):
video_content[key] = value
lecture_video_content = {}
for key, value in iteritems(video_content):
@ -751,6 +747,102 @@ class CourseraOnDemand(object):
return lecture_video_content
def _extract_subtitles_from_video_dom(self, video_dom,
subtitle_language, video_id):
# subtitles and transcripts
subtitle_nodes = [
('subtitles', 'srt', 'subtitle'),
('subtitlesTxt', 'txt', 'transcript'),
]
subtitle_set_download = set()
subtitle_set_nonexist = set()
subtitle_links = {}
for (subtitle_node, subtitle_extension, subtitle_description) \
in subtitle_nodes:
logging.debug('Gathering %s URLs for video_id <%s>.',
subtitle_description, video_id)
subtitles = video_dom.get(subtitle_node)
download_all_subtitle = False
if subtitles is not None:
subtitles_set = set(subtitles)
requested_subtitle_list = [s.strip() for s in
subtitle_language.split(",")]
for language_with_alts in requested_subtitle_list:
if download_all_subtitle:
break
grouped_language_list = [l.strip() for l in
language_with_alts.split("|")]
for language in grouped_language_list:
if language == "all":
download_all_subtitle = True
break
elif language in subtitles_set:
subtitle_set_download.update([language])
break
else:
subtitle_set_nonexist.update([language])
if download_all_subtitle and subtitles is not None:
subtitle_set_download = set(subtitles)
if not download_all_subtitle and subtitle_set_nonexist:
logging.warning("%s unavailable in '%s' language for video "
"with video id: [%s],"
"%s", subtitle_description.capitalize(),
", ".join(subtitle_set_nonexist), video_id,
subtitle_description)
if not subtitle_set_download:
logging.warning("%s all requested subtitles are unavaliable,"
"with video id: [%s], falling back to 'en' "
"%s", subtitle_description.capitalize(),
video_id,
subtitle_description)
subtitle_set_download = set(['en'])
for current_subtitle_language in subtitle_set_download:
subtitle_url = subtitles.get(current_subtitle_language)
if subtitle_url is not None:
# some subtitle urls are relative!
subtitle_links[
"%s.%s" % (current_subtitle_language, subtitle_extension)
] = make_coursera_absolute_url(subtitle_url)
return subtitle_links
def extract_links_from_programming_immediate_instructions(self, element_id):
"""
Return a dictionary with links to supplement files (pdf, csv, zip,
ipynb, html and so on) extracted from graded programming assignment.
@param element_id: Element ID to extract files from.
@type element_id: str
@return: @see CourseraOnDemand._extract_links_from_text
"""
logging.debug('Extracting links from programming immediate '
'instructions for element_id <%s>.', element_id)
try:
# Assignment text (instructions) contains asset tags which describe
# supplementary files.
text = ''.join(
self._extract_programming_immediate_instructions_text(element_id))
if not text:
return {}
supplement_links = self._extract_links_from_text(text)
instructions = (IN_MEMORY_MARKER + self._markup_to_html(text),
'instructions')
extend_supplement_links(
supplement_links, {IN_MEMORY_EXTENSION: [instructions]})
return supplement_links
except requests.exceptions.HTTPError as exception:
logging.error('Could not download programming assignment %s: %s',
element_id, exception)
if is_debug_run():
logging.exception('Could not download programming assignment %s: %s',
element_id, exception)
return None
def extract_links_from_programming(self, element_id):
"""
Return a dictionary with links to supplement files (pdf, csv, zip,
@ -876,6 +968,87 @@ class CourseraOnDemand(object):
'url': element['url'].strip()}
for element in dom['elements']]
def extract_references_poll(self):
try:
dom = get_page(self._session,
OPENCOURSE_REFERENCES_POLL_URL.format(
course_id=self._course_id),
json=True
)
logging.info('Downloaded resource poll (%d bytes)', len(dom))
return dom['elements']
except requests.exceptions.HTTPError as exception:
logging.error('Could not download resource section: %s',
exception)
if is_debug_run():
logging.exception('Could not download resource section: %s',
exception)
return None
def extract_links_from_reference(self, short_id):
"""
Return a dictionary with supplement files (pdf, csv, zip, ipynb, html
and so on) extracted from supplement page.
@return: @see CourseraOnDemand._extract_links_from_text
"""
logging.debug('Gathering resource URLs for short_id <%s>.', short_id)
try:
dom = get_page(self._session, OPENCOURSE_REFERENCE_ITEM_URL,
json=True,
course_id=self._course_id,
short_id=short_id)
resource_content = {}
# Supplement content has structure as follows:
# 'linked' {
# 'openCourseAssets.v1' [ {
# 'definition' {
# 'value'
for asset in dom['linked']['openCourseAssets.v1']:
value = asset['definition']['value']
# Supplement lecture types are known to contain both <asset> tags
# and <a href> tags (depending on the course), so we extract
# both of them.
extend_supplement_links(
resource_content, self._extract_links_from_text(value))
instructions = (IN_MEMORY_MARKER + self._markup_to_html(value),
'resources')
extend_supplement_links(
resource_content, {IN_MEMORY_EXTENSION: [instructions]})
return resource_content
except requests.exceptions.HTTPError as exception:
logging.error('Could not download supplement %s: %s',
short_id, exception)
if is_debug_run():
logging.exception('Could not download supplement %s: %s',
short_id, exception)
return None
def _extract_programming_immediate_instructions_text(self, element_id):
"""
Extract assignment text (instructions).
@param element_id: Element id to extract assignment instructions from.
@type element_id: str
@return: List of assignment text (instructions).
@rtype: [str]
"""
dom = get_page(self._session, OPENCOURSE_PROGRAMMING_IMMEDIATE_INSTRUCTIOINS_URL,
json=True,
course_id=self._course_id,
element_id=element_id)
return [element['assignmentInstructions']['definition']['value']
for element in dom['elements']]
def _extract_assignment_text(self, element_id):
"""
Extract assignment text (instructions).

30
coursera/commandline.py
View file

@ -6,13 +6,15 @@ handling. The primary candidate is argument parser.
import os
import sys
import logging
import argparse
import configargparse as argparse
from coursera import __version__
from .credentials import get_credentials, CredentialsError, keyring
from .utils import decode_input
LOCAL_CONF_FILE_NAME = 'coursera-dl.conf'
def class_name_arg_required(args):
"""
@ -33,8 +35,14 @@ def parse_args(args=None):
Parse the arguments/options passed to the program on the command line.
"""
parser = argparse.ArgumentParser(
description='Download Coursera.org lecture material and resources.')
parse_kwargs = {
"description": 'Download Coursera.org lecture material and resources.'
}
conf_file_path = os.path.join(os.getcwd(), LOCAL_CONF_FILE_NAME)
if os.path.isfile(conf_file_path):
parse_kwargs["default_config_files"] = [conf_file_path]
parser = argparse.ArgParser(**parse_kwargs)
# Basic options
group_basic = parser.add_argument_group('Basic options')
@ -93,7 +101,15 @@ def parse_args(args=None):
action='store',
default='all',
help='Choose language to download subtitles and transcripts. (Default: all)'
'Use special value "all" to download all available.')
'Use special value "all" to download all available.'
'To download subtitles and transcripts of multiple languages,'
'use comma(s) (without spaces) to seperate the names of the languages, i.e., "en,zh-CN".'
'To download subtitles and transcripts of alternative language(s) '
'if only the current language is not available,'
'put an "|<lang>" for each of the alternative languages after '
'the current language, i.e., "en|fr,zh-CN|zh-TW|de", and make sure the parameter are wrapped with '
'quotes when "|" presents.'
)
# Selection of material to download
group_material = parser.add_argument_group('Selection of material to download')
@ -316,6 +332,12 @@ def parse_args(args=None):
default=False,
help='generate M3U playlists for course weeks')
group_adv_misc.add_argument('--mathjax-cdn',
dest='mathjax_cdn_url',
default='https://cdn.mathjax.org/mathjax/latest/MathJax.js',
help='the cdn address of MathJax.js'
)
# Debug options
group_debug = parser.add_argument_group('Debugging options')

4
coursera/coursera_dl.py
View file

@ -135,7 +135,9 @@ def download_on_demand_class(args, class_name):
args.unrestricted_filenames,
args.subtitle_language,
args.video_resolution,
args.download_quizzes)
args.download_quizzes,
args.mathjax_cdn_url
)
if is_debug_run or args.cache_syllabus():
with open(cached_syllabus_filename, 'w') as file_object:

12
coursera/define.py
View file

@ -67,6 +67,12 @@ OPENCOURSE_SUPPLEMENT_URL = 'https://www.coursera.org/api/onDemandSupplements.v1
'{course_id}~{element_id}?includes=asset&fields=openCourseAssets.v1%28typeName%29,openCourseAssets.v1%28definition%29'
OPENCOURSE_PROGRAMMING_ASSIGNMENTS_URL = \
'https://www.coursera.org/api/onDemandProgrammingLearnerAssignments.v1/{course_id}~{element_id}?fields=submissionLearnerSchema'
OPENCOURSE_PROGRAMMING_IMMEDIATE_INSTRUCTIOINS_URL = \
'https://www.coursera.org/api/onDemandProgrammingImmediateInstructions.v1/{course_id}~{element_id}'
OPENCOURSE_REFERENCES_POLL_URL = \
"https://www.coursera.org/api/onDemandReferences.v1/?courseId={course_id}&q=courseListed&fields=name%2CshortId%2Cslug%2Ccontent&includes=assets"
OPENCOURSE_REFERENCE_ITEM_URL = \
"https://www.coursera.org/api/onDemandReferences.v1/?courseId={course_id}&q=shortId&shortId={short_id}&fields=name%2CshortId%2Cslug%2Ccontent&includes=assets"
# These are ids that are present in <asset> tag in assignment text:
#
@ -772,7 +778,7 @@ FORMAT_MAX_LENGTH = 20
TITLE_MAX_LENGTH = 200
#: CSS that is usen to prettify instructions
INSTRUCTIONS_HTML_INJECTION = '''
INSTRUCTIONS_HTML_INJECTION_PRE = '''
<style>
body {
padding: 50px 85px 50px 85px;
@ -812,7 +818,9 @@ pre {
</style>
<script type="text/javascript" async
src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
src="'''
INSTRUCTIONS_HTML_MATHJAX_URL = 'https://cdn.mathjax.org/mathjax/latest/MathJax.js'
INSTRUCTIONS_HTML_INJECTION_AFTER ='''?config=TeX-AMS-MML_HTMLorMML">
</script>
<script type="text/x-mathjax-config">

39
coursera/extractors.py
View file

@ -47,13 +47,13 @@ class CourseraExtractor(PlatformExtractor):
def get_modules(self, class_name,
reverse=False, unrestricted_filenames=False,
subtitle_language='en', video_resolution=None,
download_quizzes=False):
download_quizzes=False, mathjax_cdn_url=None):
page = self._get_on_demand_syllabus(class_name)
error_occured, modules = self._parse_on_demand_syllabus(
page, reverse, unrestricted_filenames,
subtitle_language, video_resolution,
download_quizzes)
download_quizzes, mathjax_cdn_url)
return error_occured, modules
def _get_on_demand_syllabus(self, class_name):
@ -71,7 +71,9 @@ class CourseraExtractor(PlatformExtractor):
unrestricted_filenames=False,
subtitle_language='en',
video_resolution=None,
download_quizzes=False):
download_quizzes=False,
mathjax_cdn_url=None
):
"""
Parse a Coursera on-demand course listing/syllabus page.
@ -90,7 +92,9 @@ class CourseraExtractor(PlatformExtractor):
json_modules = dom['courseMaterial']['elements']
course = CourseraOnDemand(session=self._session, course_id=dom['id'],
course_name=course_name,
unrestricted_filenames=unrestricted_filenames)
unrestricted_filenames=unrestricted_filenames,
mathjax_cdn_url=mathjax_cdn_url
)
course.obtain_user_id()
ondemand_material_items = OnDemandCourseMaterialItems.create(
session=self._session, course_name=course_name)
@ -155,6 +159,10 @@ class CourseraExtractor(PlatformExtractor):
if download_quizzes:
links = course.extract_links_from_exam(lecture['id'])
elif typename == 'programming':
if download_quizzes:
links = course.extract_links_from_programming_immediate_instructions(lecture['id'])
else:
logging.info('Unsupported typename "%s" in lecture "%s"',
typename, lecture_slug)
@ -174,4 +182,27 @@ class CourseraExtractor(PlatformExtractor):
if modules and reverse:
modules.reverse()
# Processing resources section
json_references= course.extract_references_poll()
references = []
if json_references:
logging.info('Processing resources')
for json_reference in json_references:
reference = []
reference_slug = json_reference['slug']
logging.info('Processing resource %s',
reference_slug)
links = course.extract_links_from_reference(json_reference['shortId'])
if links is None:
error_occured = True
elif links:
reference.append(('', links))
if reference:
references.append((reference_slug, reference))
if references:
modules.append(("Resources", references))
return error_occured, modules

24
coursera/test/fixtures/json/references-poll-output.json vendored Normal file
View file

@ -0,0 +1,24 @@
[
{
"id": "Tk_5NiCREeeSVwJCrBEADA",
"slug": "tutorials",
"content": {
"org.coursera.ondemand.reference.AssetReferenceContent": {
"assetId": "4e66aa537abf1bdec8ecc508324891ac"
}
},
"shortId": "zVvo7",
"name": "Tutorials"
},
{
"id": "Tk_5MSCREeeSVwJCrBEADA",
"slug": "test-cases",
"content": {
"org.coursera.ondemand.reference.AssetReferenceContent": {
"assetId": "7c84e5c5249eb551d95444c172592274"
}
},
"shortId": "a4I28",
"name": "Test Cases"
}
]

47
coursera/test/fixtures/json/references-poll-reply.json vendored Normal file
View file

@ -0,0 +1,47 @@
{
"paging": {},
"linked": {
"openCourseAssets.v1": [
{
"typeName": "cml",
"definition": {
"dtdId": "supplement/1",
"value": "<co-content><text>supplement1</text></co-content>"
},
"id": "4e66aa537abf1bdec8ecc508324891ac"
},
{
"typeName": "cml",
"definition": {
"dtdId": "supplement/1",
"value": "<co-content><text>supplement2</text></co-content>"
},
"id": "7c84e5c5249eb551d95444c172592274"
}
]
},
"elements": [
{
"name": "Tutorials",
"id": "Tk_5NiCREeeSVwJCrBEADA",
"slug": "tutorials",
"content": {
"org.coursera.ondemand.reference.AssetReferenceContent": {
"assetId": "4e66aa537abf1bdec8ecc508324891ac"
}
},
"shortId": "zVvo7"
},
{
"name": "Test Cases",
"id": "Tk_5MSCREeeSVwJCrBEADA",
"slug": "test-cases",
"content": {
"org.coursera.ondemand.reference.AssetReferenceContent": {
"assetId": "7c84e5c5249eb551d95444c172592274"
}
},
"shortId": "a4I28"
}
]
}

18
coursera/test/fixtures/json/supplement-programming-immediate-instructions-empty-instructions.json vendored Normal file
View file

@ -0,0 +1,18 @@
{
"elements": [
{
"id": "Gtv4Xb1-EeS-ViIACwYKVQ~8f3qT",
"itemId": "8f3qT",
"courseId": "Gtv4Xb1-EeS-ViIACwYKVQ",
"assignmentInstructions": {
"definition": {
"dtdId": "",
"value": "<co-content><text/><text/><text/><text/><text/><text/></co-content>"
},
"typeName": "cml"
}
}
],
"linked": {},
"paging": {}
}

6
coursera/test/fixtures/json/supplement-programming-immediate-instructions-no-instructions.json vendored Normal file
View file

@ -0,0 +1,6 @@
{
"elements": [
],
"paging": null,
"linked": null
}

18
coursera/test/fixtures/json/supplement-programming-immediate-instructions-one-asset.json vendored Normal file
View file

@ -0,0 +1,18 @@
{
"elements": [
{
"id": "Gtv4Xb1-EeS-ViIACwYKVQ~e4hZk",
"itemId": "e4hZk",
"courseId": "Gtv4Xb1-EeS-ViIACwYKVQ",
"assignmentInstructions": {
"definition": {
"dtdId": "",
"value": "<co-content><text><asset id=\"yeJ7Q8VAEeWPRQ4YsSEORQ\" name=\"statement-pca\" extension=\"pdf\" assetType=\"generic\"/>. </text></co-content>"
},
"typeName": "cml"
}
}
],
"linked": {},
"paging": {}
}

16
coursera/test/fixtures/json/video-output-1-all.json vendored Normal file
View file

@ -0,0 +1,16 @@
{
"zh-CN.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=eNyKwEu_aMQtn7bg0mUj6uIyVZvjahFSE5x2CrbOXOU&fileExtension=txt",
"en.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=afqFhv9FWfxxEeSka8PCA4ihiyX3g2Z6K4jWJPFlcdo&fileExtension=srt",
"zh-CN.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=nmGzGoF4oNLv28ZDLUtX5dF4xPXUABgym76XMs4UzDE&fileExtension=srt",
"en.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=2Z37WW5Rc7GoT0eft1vdK0HX5imBqoZTKULMTiZ2EjM&fileExtension=txt",
"hi.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=qk--Ptsc4w3u6c-5BFPO9vhjyczMHzlSqUOQskjbfZ0&fileExtension=srt",
"es.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=Ts5QKzu0jwhUafwsaHk7RKoQJK26d4_bzrX2M6iuRaQ&fileExtension=srt",
"pl.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=mcaMPGeK3J7Fn9RRwnuVFnHkyr1COFnLXYKVkUbyfSg&fileExtension=srt",
"ja.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=huh5qtCJVj4rEJnsJ6D7MJdCcqN-s9cMd-M6xlSicLc&fileExtension=srt",
"pt-BR.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=XYyDJ71d9gl3HOqNplyJeEr7Wd2UhU3DhT-9w_Yudzs&fileExtension=srt",
"hi.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=earWLk_RUi3K5UpZfEVOlBgOcpSE9efXz2njRKu31rQ&fileExtension=txt",
"es.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=sd6_C14J-qEkvvbqNTgI8W5eUCvOKwW6RzHcz8yF2Jk&fileExtension=txt",
"pl.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=sFwO_BWNlhZEDHsXYkFlnOEtHBIX8lSsVGIOLIHeZZ0&fileExtension=txt",
"ja.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=WMhDBDbF6SiBuvRwg_QEkglLSK36bj8_5y6kZ9z94YY&fileExtension=txt",
"pt-BR.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=uQaL2V2AJ_Wp5dlCZH1HeyTU_AQo9VdJ2cphUhG8yxk&fileExtension=txt"
}

4
coursera/test/fixtures/json/video-output-1-en.json vendored Normal file
View file

@ -0,0 +1,4 @@
{
"en.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=afqFhv9FWfxxEeSka8PCA4ihiyX3g2Z6K4jWJPFlcdo&fileExtension=srt",
"en.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=2Z37WW5Rc7GoT0eft1vdK0HX5imBqoZTKULMTiZ2EjM&fileExtension=txt"
}

6
coursera/test/fixtures/json/video-output-1.json vendored Normal file
View file

@ -0,0 +1,6 @@
{
"zh-CN.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=eNyKwEu_aMQtn7bg0mUj6uIyVZvjahFSE5x2CrbOXOU&fileExtension=txt",
"en.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=afqFhv9FWfxxEeSka8PCA4ihiyX3g2Z6K4jWJPFlcdo&fileExtension=srt",
"zh-CN.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=nmGzGoF4oNLv28ZDLUtX5dF4xPXUABgym76XMs4UzDE&fileExtension=srt",
"en.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=2Z37WW5Rc7GoT0eft1vdK0HX5imBqoZTKULMTiZ2EjM&fileExtension=txt"
}

6
coursera/test/fixtures/json/video-output-2.json vendored Normal file
View file

@ -0,0 +1,6 @@
{
"zh-TW.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=-sOeJbk_bICP9OMfbtkjLuwUAIZZcjGasIMk8JO6n0Q&fileExtension=srt",
"en.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=xhMK0SSslbfwxl-vzjAXy-bd_iQQTY9iAIrNP4QHxq4&fileExtension=txt",
"en.srt": "https://www.coursera.org/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=nO6NGCExQ5FO0aFFnr_YVXtd_lVW4JQaT34WS9tJi6c&fileExtension=srt",
"zh-TW.txt": "https://www.coursera.org/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=O9DKhZW6bOsI7ncNZIZPBMXmsreSrgulhGf3eyTCULo&fileExtension=txt"
}

47
coursera/test/fixtures/json/video-reply-1.json vendored Normal file
View file

@ -0,0 +1,47 @@
{
"sources": [
{
"resolution": "540p",
"formatSources": {
"video/webm": "https://d3c33hcgiwev3.cloudfront.net/20.1-Conclusion-SummaryAndThankYou.63149a70b22b11e4aca907c8d9623f2b/full/540p/index.webm?Expires=1495238400&Signature=Oj2j7hCpfrpp1ugtZHjRITM9D4MjaOJm2x34ecUPGH2nm~BIvt6RY25XpKgCFZ0qbIK01eloymAUolfupBwzJYIjwANxibwpIJ3bX43dtxnoy1dRh2F1YZoZ6lbPCVOSCxODJFod7bZPCuqRTfXvK6X6F0o-IzkbXy8myk6G5Js_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A",
"video/mp4": "https://d3c33hcgiwev3.cloudfront.net/20.1-Conclusion-SummaryAndThankYou.63149a70b22b11e4aca907c8d9623f2b/full/540p/index.mp4?Expires=1495238400&Signature=TW8nTCrNKfrBGLCloVNFvNB~7qsNXaRI8T~gBUCxyxxPzumARwdw8W9ONVd-8j2TrI8Zvm~j4ysS4UedtLTKynDewxOzjrbsVc3HRBLTqrcQNjjLkG9vzbrGz2wUMMRUwX6qlwT8xFTVuNjh7-W72gq83bzk4eyaALHO~YKXvxk_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A"
}
}
],
"subtitles": {
"hi": "/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=qk--Ptsc4w3u6c-5BFPO9vhjyczMHzlSqUOQskjbfZ0&fileExtension=srt",
"en": "/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=afqFhv9FWfxxEeSka8PCA4ihiyX3g2Z6K4jWJPFlcdo&fileExtension=srt",
"zh-CN": "/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=nmGzGoF4oNLv28ZDLUtX5dF4xPXUABgym76XMs4UzDE&fileExtension=srt",
"es": "/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=Ts5QKzu0jwhUafwsaHk7RKoQJK26d4_bzrX2M6iuRaQ&fileExtension=srt",
"pl": "/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=mcaMPGeK3J7Fn9RRwnuVFnHkyr1COFnLXYKVkUbyfSg&fileExtension=srt",
"ja": "/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=huh5qtCJVj4rEJnsJ6D7MJdCcqN-s9cMd-M6xlSicLc&fileExtension=srt",
"pt-BR": "/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=XYyDJ71d9gl3HOqNplyJeEr7Wd2UhU3DhT-9w_Yudzs&fileExtension=srt"
},
"playlists": {
"hls": "https://d3c33hcgiwev3.cloudfront.net/assetMasterHlsPlaylists.v1/DB-HfUh-EeWWUA71mMib3w?expiry=1495238400000&hmac=kaIpHBiD8US9yrHVsABKCkRPRgIzsBA7tWzB-PHEqhY&mediaCdn=cloudfront"
},
"subtitlesVtt": {
"hi": "/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=4S0NgqfShX81v0QJckTU0IbeAJJnD9m_iZcTYJltbNc&fileExtension=vtt",
"en": "/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=lMlV1hdtArRLJvePvHHqFJhekv1Gs-P6WzPQz_TEnzE&fileExtension=vtt",
"zh-CN": "/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=H7jCFYLCxt9yS5y5YiLkUGwd3-0fWiioGGVVwiYFfjo&fileExtension=vtt",
"es": "/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=nTa427_IEx68vJJYeNQvNQhWQaNOSwJkWUqfAYW2tYI&fileExtension=vtt",
"pl": "/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=jPndFFisYDio-2FgSWp-vMVOg9Ybx-Zh4tODJsHvUWY&fileExtension=vtt",
"ja": "/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=m9-S0joHnqjaCJ72qKLtL199dsngS9zoP1jmvI7_AA4&fileExtension=vtt",
"pt-BR": "/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=ePd5ZewhjQBA5J6QybTLY-U8yEz2FlIqZuvCE7gdYxQ&fileExtension=vtt"
},
"subtitlesTxt": {
"hi": "/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=earWLk_RUi3K5UpZfEVOlBgOcpSE9efXz2njRKu31rQ&fileExtension=txt",
"en": "/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=2Z37WW5Rc7GoT0eft1vdK0HX5imBqoZTKULMTiZ2EjM&fileExtension=txt",
"zh-CN": "/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=eNyKwEu_aMQtn7bg0mUj6uIyVZvjahFSE5x2CrbOXOU&fileExtension=txt",
"es": "/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=sd6_C14J-qEkvvbqNTgI8W5eUCvOKwW6RzHcz8yF2Jk&fileExtension=txt",
"pl": "/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=sFwO_BWNlhZEDHsXYkFlnOEtHBIX8lSsVGIOLIHeZZ0&fileExtension=txt",
"ja": "/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=WMhDBDbF6SiBuvRwg_QEkglLSK36bj8_5y6kZ9z94YY&fileExtension=txt",
"pt-BR": "/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=uQaL2V2AJ_Wp5dlCZH1HeyTU_AQo9VdJ2cphUhG8yxk&fileExtension=txt"
},
"posters": [
{
"url": "https://d3c33hcgiwev3.cloudfront.net/imageAssetProxy.v1/20.1-Conclusion-SummaryAndThankYou.63149a70b22b11e4aca907c8d9623f2b/thumbnails/540p/0.jpg?expiry=1495238400000&hmac=LuHD_gGzBtwVeNIQqABnGU69sWteFl8xdMozFAsGPco",
"resolution": "540p"
}
]
}

77
coursera/test/fixtures/json/video-reply-2.json vendored Normal file
View file

@ -0,0 +1,77 @@
{
"posters": [
{
"url": "https://d3c33hcgiwev3.cloudfront.net/imageAssetProxy.v1/qHTMkA4fEeW2rSIAC2yC6g.processed/thumbnails/540p/0.jpg?expiry=1495238400000&hmac=ES_Ho42kS5yI6VWTPDEY2LNWyvBMOoKwoGXJHb6LqnU",
"resolution": "540p"
}
],
"subtitles": {
"en": "/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=nO6NGCExQ5FO0aFFnr_YVXtd_lVW4JQaT34WS9tJi6c&fileExtension=srt",
"fr": "/api/subtitleAssetProxy.v1/wAkpEeN1SE6JKRHjdWhOTw?expiry=1495238400000&hmac=DeXzNpOf_7RhvGBaqMWqud5J96PoIh6At1eSDWF1RUM&fileExtension=srt",
"lt": "/api/subtitleAssetProxy.v1/qDmNTUOsQoG5jU1DrEKBtw?expiry=1495238400000&hmac=ifyDt77JEmZu5SgJ0nubLGsy6JV9d2IAzNvZdVevBcE&fileExtension=srt",
"ko": "/api/subtitleAssetProxy.v1/BatzP4LfQA6rcz-C35AOXQ?expiry=1495238400000&hmac=U70Yc3wumD-G_XE3AYEISRbqtjtl9WSOqQMjlHI2OGM&fileExtension=srt",
"hi": "/api/subtitleAssetProxy.v1/-v2-jzNFSxq9vo8zRVsa5Q?expiry=1495238400000&hmac=teLyzOnnfT0P8sZVvD4O8fFDgd0RojSYCxB3n6RvTk0&fileExtension=srt",
"fa": "/api/subtitleAssetProxy.v1/MVu-cT46QyqbvnE-OvMq0g?expiry=1495238400000&hmac=gQgvsV9WFOC-cbOuUjLEIepCtka_W8fup6mltVcnJq8&fileExtension=srt",
"es": "/api/subtitleAssetProxy.v1/_mNV07xnT_ejVdO8Z2_3Ig?expiry=1495238400000&hmac=fC2H9Lajgsm_WkxnsuNaL6TIfgEzpfhdY2dIJhNasMI&fileExtension=srt",
"he": "/api/subtitleAssetProxy.v1/keFd5T42SI6hXeU-NkiO1w?expiry=1495238400000&hmac=DiJ-h_TfbyqgajhWVb-1302MXPoqD5x0JEachin0c3Q&fileExtension=srt",
"ar": "/api/subtitleAssetProxy.v1/vHGgsHVsQ2CxoLB1bKNg9w?expiry=1495238400000&hmac=pmjuKhL8-SNhzXi8FAETaJcakRt5S1yqay-G--r4C0I&fileExtension=srt",
"bn": "/api/subtitleAssetProxy.v1/hi47OjiORsmuOzo4jqbJBQ?expiry=1495238400000&hmac=tcHBX4hMne23haoOlu1olxiKa7M1n_CSOX4eOmH0U14&fileExtension=srt",
"pl": "/api/subtitleAssetProxy.v1/lBH5IsEBRySR-SLBAQcknw?expiry=1495238400000&hmac=QHDKKoNSt2A9Bd4PfRiR82OJwSAsCQmbxZEZoFhLnIg&fileExtension=srt",
"tr": "/api/subtitleAssetProxy.v1/jdBjiWbSSlWQY4lm0rpVzw?expiry=1495238400000&hmac=5WoVi4jm3274ClP3U4JZAIhJt-V5ulNHkNtUACqgR6w&fileExtension=srt",
"zh-TW": "/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=-sOeJbk_bICP9OMfbtkjLuwUAIZZcjGasIMk8JO6n0Q&fileExtension=srt",
"hr": "/api/subtitleAssetProxy.v1/k_PHTdx0Rw-zx03cdOcPyw?expiry=1495238400000&hmac=EeciDHPlRVHxIgsgzSJ0uAOjnaatGmqt4bw1hAyGh9A&fileExtension=srt",
"id": "/api/subtitleAssetProxy.v1/xIop0OAYTKaKKdDgGKym3g?expiry=1495238400000&hmac=BT9i5NUXcz5RTOUzjMTK8NOQciU5o-gGI6rTxjNWhbM&fileExtension=srt"
},
"sources": [
{
"formatSources": {
"video/webm": "https://d3c33hcgiwev3.cloudfront.net/qHTMkA4fEeW2rSIAC2yC6g.processed/full/540p/index.webm?Expires=1495238400&Signature=iAKFovwfSpnnEflg4blQajE9Tle2peDHKv0ScvXSK5rMVBQnX9SWq8M2CvsawxPwKdA2Xo02lOZBi~-5I5F6fCag9mzzlV-8Q-dxVoS1yWZLu7HtGROStDSwOiJYvGoLGSgS2dT0dGYG4rNJ9hxQmElQzBCOYkeQsP6lIsh0Ejg_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A",
"video/mp4": "https://d3c33hcgiwev3.cloudfront.net/qHTMkA4fEeW2rSIAC2yC6g.processed/full/540p/index.mp4?Expires=1495238400&Signature=SkYBCgVvj~W2eeSAKjtX4igYZ2WLKq8crzqLbFqgDbZxhH48jW3nXZNWB5~H6ev0EIHkSAbMSQWP4xUGKhzGhcciL8B9jB8LjI180wsSv1jfNknCP9S5p9vd1mdCidheNmJftBtIjfr54m8CgFYUqe1WAo2aLUzRimiS~nf8kxk_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A"
},
"resolution": "540p"
},
{
"formatSources": {
"video/webm": "https://d3c33hcgiwev3.cloudfront.net/qHTMkA4fEeW2rSIAC2yC6g.processed/full/360p/index.webm?Expires=1495238400&Signature=CBaZCNZGbCJ9UpJdWyfGA~KwtUx4UxqnwM8v6GS7T2fsynUVexCfjBcE7IMowDzWa~GZ-jdOI43~s5e4kVc4hbiNOaQoZ-p-te1AffsRJaMhlhI529vxfWUQJGhO3bUGbn9Az9ueSehoe8WLojtHHb5q-IIr53MX-rftl~d3srI_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A",
"video/mp4": "https://d3c33hcgiwev3.cloudfront.net/qHTMkA4fEeW2rSIAC2yC6g.processed/full/360p/index.mp4?Expires=1495238400&Signature=dt9-1ARw8I5U1IrIKJGbQzy5MkCqjySGXutT~KFNx8~~UD0v3T0cFwUG3ggLhL3lkyMAztA-dTpARKfi2igKgq6Q8qfcnTfs8~iu5Ayt1vRZVHDfgomlasB3aElmOHB7WaWkQbZJCChXYlVgg2fDKLGxMcfEGf797AzzDrhEFBY_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A"
}
}
],
"subtitlesTxt": {
"en": "/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=xhMK0SSslbfwxl-vzjAXy-bd_iQQTY9iAIrNP4QHxq4&fileExtension=txt",
"fr": "/api/subtitleAssetProxy.v1/wAkpEeN1SE6JKRHjdWhOTw?expiry=1495238400000&hmac=Y9ysEdDIlObbFPyLBWorn1XEOeJ57TEYPxWsnzE3x5Q&fileExtension=txt",
"lt": "/api/subtitleAssetProxy.v1/qDmNTUOsQoG5jU1DrEKBtw?expiry=1495238400000&hmac=92ArkEZV3O5nkxp5f7pxyUClVWbywOWjJ8DFE86foKA&fileExtension=txt",
"ko": "/api/subtitleAssetProxy.v1/BatzP4LfQA6rcz-C35AOXQ?expiry=1495238400000&hmac=aVqHA-CIZDni9UrmUGB6aR2VjDstYzdquDrOvzusKbc&fileExtension=txt",
"hi": "/api/subtitleAssetProxy.v1/-v2-jzNFSxq9vo8zRVsa5Q?expiry=1495238400000&hmac=fN1qhDyCL5aMzYRW2NbkfNyikEypjzB57LJtO9QXb2Q&fileExtension=txt",
"fa": "/api/subtitleAssetProxy.v1/MVu-cT46QyqbvnE-OvMq0g?expiry=1495238400000&hmac=JLJVoBvCluUXGXqzDti9uaW0gDjCsRWQIBOqxARAM9w&fileExtension=txt",
"es": "/api/subtitleAssetProxy.v1/_mNV07xnT_ejVdO8Z2_3Ig?expiry=1495238400000&hmac=FzDR-l0J3CTljW9aywGiBn56TTWdmzh1TEYzsfmdceo&fileExtension=txt",
"he": "/api/subtitleAssetProxy.v1/keFd5T42SI6hXeU-NkiO1w?expiry=1495238400000&hmac=xwD0e-3s7FwTbDuQn-nNVIi9eoAueviOZl4Ezofd_rY&fileExtension=txt",
"ar": "/api/subtitleAssetProxy.v1/vHGgsHVsQ2CxoLB1bKNg9w?expiry=1495238400000&hmac=it_tXtSCiX5oX9MV9VDZQuz1hj5IFZOohCAyYOn8pR4&fileExtension=txt",
"bn": "/api/subtitleAssetProxy.v1/hi47OjiORsmuOzo4jqbJBQ?expiry=1495238400000&hmac=fZebyPIojlQlJL3HuHkEOgoQHlwPsJJ5YEC4Pd_a4Sg&fileExtension=txt",
"pl": "/api/subtitleAssetProxy.v1/lBH5IsEBRySR-SLBAQcknw?expiry=1495238400000&hmac=jmygEnmsUvBv4-sDUbYZK0MsJht9Mg24AAyaI5iMhmg&fileExtension=txt",
"tr": "/api/subtitleAssetProxy.v1/jdBjiWbSSlWQY4lm0rpVzw?expiry=1495238400000&hmac=kgkFFJSswv5HLtPMkjV7rsgLQZzoSBpWHbIffW1FUKc&fileExtension=txt",
"zh-TW": "/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=O9DKhZW6bOsI7ncNZIZPBMXmsreSrgulhGf3eyTCULo&fileExtension=txt",
"hr": "/api/subtitleAssetProxy.v1/k_PHTdx0Rw-zx03cdOcPyw?expiry=1495238400000&hmac=DO3oN6U9JBwZcScxzOsIAI8Nn2CTaGnlWGi4pAxjDEE&fileExtension=txt",
"id": "/api/subtitleAssetProxy.v1/xIop0OAYTKaKKdDgGKym3g?expiry=1495238400000&hmac=HIQ_jyC6_xWBpz4dCF6hxiG5ay1tVSJwQJF8LSCo0gk&fileExtension=txt"
},
"subtitlesVtt": {
"en": "/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=WtAfot596syGVoSQ-UJ-QpyWQWqhSQ4auwDRijJy7IM&fileExtension=vtt",
"fr": "/api/subtitleAssetProxy.v1/wAkpEeN1SE6JKRHjdWhOTw?expiry=1495238400000&hmac=KDSprAsOxTLlNvgUVon4RRUAAN7BhOd6rRuTm8KWEU0&fileExtension=vtt",
"lt": "/api/subtitleAssetProxy.v1/qDmNTUOsQoG5jU1DrEKBtw?expiry=1495238400000&hmac=Hwu1AvpVFsSZUrlSh-VxQ2Rvj6dJ3pgu1_VXDflnH-k&fileExtension=vtt",
"ko": "/api/subtitleAssetProxy.v1/BatzP4LfQA6rcz-C35AOXQ?expiry=1495238400000&hmac=I1QSwiy02AWEChUfrmw5C8XKrDReprSmP0mqBv-ipno&fileExtension=vtt",
"hi": "/api/subtitleAssetProxy.v1/-v2-jzNFSxq9vo8zRVsa5Q?expiry=1495238400000&hmac=p6JE3hXLwYbJmlTCQf9vZtIO0Gsg8TVAvPjobSFU0eg&fileExtension=vtt",
"fa": "/api/subtitleAssetProxy.v1/MVu-cT46QyqbvnE-OvMq0g?expiry=1495238400000&hmac=BkasYNlGSg-zm2GSKqcDLzfwzISJu6agsq1qirTV3xU&fileExtension=vtt",
"es": "/api/subtitleAssetProxy.v1/_mNV07xnT_ejVdO8Z2_3Ig?expiry=1495238400000&hmac=zlOw3ifj1lIy4i7GNROw5muCIBidW2MCFnSHRHZ5vhI&fileExtension=vtt",
"he": "/api/subtitleAssetProxy.v1/keFd5T42SI6hXeU-NkiO1w?expiry=1495238400000&hmac=9kbtSLEO8NqxGhzvRMUEJGIZ0rlUQ0IO2F_Un-Bpmj0&fileExtension=vtt",
"ar": "/api/subtitleAssetProxy.v1/vHGgsHVsQ2CxoLB1bKNg9w?expiry=1495238400000&hmac=YnlfoVEGoQ7LAtnEjrMFXfRPFy5F0-tq_C7s_Vlqf8o&fileExtension=vtt",
"bn": "/api/subtitleAssetProxy.v1/hi47OjiORsmuOzo4jqbJBQ?expiry=1495238400000&hmac=J0pV67Lu5PRMeSP_6Mk-pN6CdlguHW024MYQDf1ZOjI&fileExtension=vtt",
"pl": "/api/subtitleAssetProxy.v1/lBH5IsEBRySR-SLBAQcknw?expiry=1495238400000&hmac=knptQ0ipo3LojO72PQawrSjdiy6VqBjlFHX62ECyFPg&fileExtension=vtt",
"tr": "/api/subtitleAssetProxy.v1/jdBjiWbSSlWQY4lm0rpVzw?expiry=1495238400000&hmac=9gYXktWObxjtSPjkiBB3qo__LZhsRzWoTEPc32uRxhA&fileExtension=vtt",
"zh-TW": "/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=coTFw8DgESXYsiMqySKkp1JXRfUkwx2fBY_lniYl-i0&fileExtension=vtt",
"hr": "/api/subtitleAssetProxy.v1/k_PHTdx0Rw-zx03cdOcPyw?expiry=1495238400000&hmac=OUzErWIwafoewJ97evxPxAdJRiHgTUQEkPydMHHJElk&fileExtension=vtt",
"id": "/api/subtitleAssetProxy.v1/xIop0OAYTKaKKdDgGKym3g?expiry=1495238400000&hmac=rt0Afn6mQCoiOAVn258RUI0qiXnInP73f7DM3qldWYY&fileExtension=vtt"
},
"playlists": {
"hls": "https://d3c33hcgiwev3.cloudfront.net/assetMasterHlsPlaylists.v1/qHTMkA4fEeW2rSIAC2yC6g?expiry=1495238400000&hmac=PwXl2HLjLXI3lkJ4YeUu1pNM9Y8XG39j2QzOLFwQ8F4&mediaCdn=cloudfront"
}
}

164
coursera/test/test_api.py vendored
View file

@ -89,6 +89,47 @@ def test_extract_links_from_quiz_http_error(get_page, course):
assert None == course.extract_links_from_quiz('0')
@patch('coursera.api.get_page')
def test_extract_references_poll_http_error(get_page, course):
"""
This test checks that downloader skips locked programming assignments
instead of throwing an error. (Locked == returning 403 error code)
"""
locked_response = Response()
locked_response.status_code = define.HTTP_FORBIDDEN
get_page.side_effect = HTTPError('Mocked HTTP error',
response=locked_response)
assert None == course.extract_references_poll()
@patch('coursera.api.get_page')
def test_extract_links_from_reference_http_error(get_page, course):
"""
This test checks that downloader skips locked resources
instead of throwing an error. (Locked == returning 403 error code)
"""
locked_response = Response()
locked_response.status_code = define.HTTP_FORBIDDEN
get_page.side_effect = HTTPError('Mocked HTTP error',
response=locked_response)
assert None == course.extract_links_from_reference('0')
@patch('coursera.api.get_page')
def test_extract_links_from_programming_immediate_instructions_http_error(
get_page, course):
"""
This test checks that downloader skips locked programming immediate instructions
instead of throwing an error. (Locked == returning 403 error code)
"""
locked_response = Response()
locked_response.status_code = define.HTTP_FORBIDDEN
get_page.side_effect = HTTPError('Mocked HTTP error',
response=locked_response)
assert (
None == course.extract_links_from_programming_immediate_instructions('0'))
@patch('coursera.api.get_page')
def test_ondemand_programming_supplement_no_instructions(get_page, course):
no_instructions = slurp_fixture('json/supplement-programming-no-instructions.json')
@ -98,6 +139,16 @@ def test_ondemand_programming_supplement_no_instructions(get_page, course):
assert {} == output
@patch('coursera.api.get_page')
def test_ondemand_from_programming_immediate_instructions_no_instructions(
get_page, course):
no_instructions = slurp_fixture(
'json/supplement-programming-immediate-instructions-no-instructions.json')
get_page.return_value = json.loads(no_instructions)
output = course.extract_links_from_programming_immediate_instructions('0')
assert {} == output
@patch('coursera.api.get_page')
def test_ondemand_programming_supplement_empty_instructions(get_page, course):
empty_instructions = slurp_fixture('json/supplement-programming-empty-instructions.json')
@ -113,6 +164,23 @@ def test_ondemand_programming_supplement_empty_instructions(get_page, course):
assert {} == output
@patch('coursera.api.get_page')
def test_ondemand_programming_immediate_instructions_empty_instructions(
get_page, course):
empty_instructions = slurp_fixture(
'json/supplement-programming-immediate-instructions-empty-instructions.json')
get_page.return_value = json.loads(empty_instructions)
output = course.extract_links_from_programming_immediate_instructions('0')
# Make sure that SOME html content has been extracted, but remove
# it immeditely because it's a hassle to properly prepare test input
# for it. FIXME later.
assert 'html' in output
del output['html']
assert {} == output
@patch('coursera.api.get_page')
def test_ondemand_programming_supplement_one_asset(get_page, course):
one_asset_tag = slurp_fixture('json/supplement-programming-one-asset.json')
@ -134,6 +202,41 @@ def test_ondemand_programming_supplement_one_asset(get_page, course):
assert expected_output == output
@patch('coursera.api.get_page')
def test_extract_references_poll(get_page, course):
"""
Test extracting course references.
"""
get_page.side_effect = [
json.loads(slurp_fixture('json/references-poll-reply.json'))
]
expected_output = json.loads(
slurp_fixture('json/references-poll-output.json'))
output = course.extract_references_poll()
assert expected_output == output
@patch('coursera.api.get_page')
def test_ondemand_programming_immediate_instructions_one_asset(get_page, course):
one_asset_tag = slurp_fixture('json/supplement-programming-immediate-instructions-one-asset.json')
one_asset_url = slurp_fixture('json/asset-urls-one.json')
asset_json = json.loads(one_asset_url)
get_page.side_effect = [json.loads(one_asset_tag),
json.loads(one_asset_url)]
expected_output = {'pdf': [(asset_json['elements'][0]['url'],
'statement-pca')]}
output = course.extract_links_from_programming_immediate_instructions('0')
# Make sure that SOME html content has been extracted, but remove
# it immeditely because it's a hassle to properly prepare test input
# for it. FIXME later.
assert 'html' in output
del output['html']
assert expected_output == output
@patch('coursera.api.get_page')
def test_ondemand_programming_supplement_three_assets(get_page, course):
three_assets_tag = slurp_fixture('json/supplement-programming-three-assets.json')
@ -190,6 +293,7 @@ def test_extract_links_from_lecture_assets_typname_url_and_asset(get_page, cours
output = json.loads(json.dumps(output))
assert expected_output == output
@patch('coursera.api.get_page')
def test_list_courses(get_page, course):
"""
@ -204,6 +308,32 @@ def test_list_courses(get_page, course):
assert expected_output == output
@pytest.mark.parametrize(
"input_filename,output_filename,subtitle_language,video_id", [
('video-reply-1.json', 'video-output-1.json',
'en,zh-CN|zh-TW', "None"),
('video-reply-1.json', 'video-output-1-en.json',
'zh-TW', "None"),
('video-reply-1.json', 'video-output-1-en.json',
'en', "None"),
('video-reply-1.json', 'video-output-1-all.json',
'all', "None"),
('video-reply-1.json', 'video-output-1-all.json',
'zh-TW,all|zh-CN', "None"),
('video-reply-2.json', 'video-output-2.json',
'en,zh-CN|zh-TW', "None"),
]
)
def test_extract_subtitles_from_video_dom(input_filename,output_filename,subtitle_language, video_id):
video_dom = json.loads(slurp_fixture('json/%s' % input_filename))
expected_output = json.loads(slurp_fixture('json/%s' % output_filename))
course = api.CourseraOnDemand(
session=Mock(cookies={}), course_id='0', course_name='test_course')
actual_output = course._extract_subtitles_from_video_dom(video_dom, subtitle_language, video_id)
actual_output = json.loads(json.dumps(actual_output))
assert actual_output == expected_output
@pytest.mark.parametrize(
"input_filename,output_filename", [
('empty-input.json', 'empty-output.txt'),
@ -238,17 +368,33 @@ class TestMarkupToHTMLConverter:
STYLE = None
def setup_method(self, test_method):
self.STYLE = self._p(define.INSTRUCTIONS_HTML_INJECTION)
self.STYLE = self._p(
"".join([define.INSTRUCTIONS_HTML_INJECTION_PRE,
define.INSTRUCTIONS_HTML_MATHJAX_URL,
define.INSTRUCTIONS_HTML_INJECTION_AFTER])
)
self.markup_to_html = api.MarkupToHTMLConverter(session=None)
ALTERNATIVE_MATHJAX_CDN = "https://alternative/mathjax/cdn.js"
self.STYLE_WITH_ALTER = self._p(
"".join([define.INSTRUCTIONS_HTML_INJECTION_PRE,
ALTERNATIVE_MATHJAX_CDN,
define.INSTRUCTIONS_HTML_INJECTION_AFTER])
)
self.markup_to_html_with_alter_mjcdn = api.MarkupToHTMLConverter(
session=None, mathjax_cdn_url=ALTERNATIVE_MATHJAX_CDN)
def test_empty(self):
output = self.markup_to_html("")
assert self._p("""
output_with_alter_mjcdn = self.markup_to_html_with_alter_mjcdn("")
markup = """
<meta charset="UTF-8"/>
""") + self.STYLE == output
"""
assert self._p(markup) + self.STYLE == output
assert self._p(markup) + self.STYLE_WITH_ALTER == output_with_alter_mjcdn
def test_replace_text_tag(self):
output = self.markup_to_html("""
markup = """
<co-content>
<text>
Test<text>Nested</text>
@ -257,8 +403,8 @@ class TestMarkupToHTMLConverter:
Test2
</text>
</co-content>
""")
assert self._p("""
"""
result = """
<meta charset="UTF-8"/>
<co-content>
<p>
@ -268,7 +414,11 @@ class TestMarkupToHTMLConverter:
Test2
</p>
</co-content>\n
""") + self.STYLE == output
"""
output = self.markup_to_html(markup)
output_with_alter_mjcdn = self.markup_to_html_with_alter_mjcdn(markup)
assert self._p(result) + self.STYLE == output
assert self._p(result) + self.STYLE_WITH_ALTER == output_with_alter_mjcdn
def test_replace_heading(self):
output = self.markup_to_html("""

1
requirements.txt
View file

@ -4,3 +4,4 @@ six>=1.5.0
urllib3>=1.10
pyasn1>=0.1.7
keyring>=4.0
configargparse>=0.12.0

1
tox.ini
View file

@ -14,6 +14,7 @@ deps =
six>=1.5.0
urllib3>=1.10
keyrings.alt>=1.1
configargparse>=0.12.0
commands = py.test -v --junitxml={envlogdir}/result.xml coursera/test
# Original command: install_command = pip install {opts} {packages}