README: Update versions of Python that we support.

According to
<https://en.wikipedia.org/wiki/History_of_Python#Table_of_versions>, the
(current) earliest supported version is 3.6 (with only 3.8 and 3.9 being
supported with binaries, 3.6 and 3.7 supported only with source code).

Some things to keep in mind here, according to the current download page of
Python for Windows (which, I suppose, is where the majority of our users
are):

* Python 3.6, 3.7 and 3.8 support [only Windows Vista and newer][python38].
* Python 3.9 doesn't suport Windows 7; [only 8.1 or newer][python39].

[python38]: https://docs.python.org/3.8/using/windows.html#using-python-on-windows
[python39]: https://docs.python.org/3.9/using/windows.html#using-python-on-windows

I'm still keeping Python 2.7 in the support tests (despite it being dead),
but am considering dropping it in the near term (and removing the
workarounds to support multiple versions).

.editorconfig

@@ -0,0 +1,7 @@
+root = true
+
+[*]
+charset = utf-8
+indent_style = space
+indent_size = 4
+insert_final_newline = true

.gitattributes

@@ -0,0 +1 @@
+coursera/test/* linguist-vendored

.gitignore

@@ -21,3 +21,4 @@ venv3
 .python-version
 .ipynb_checkpoints
 .ropeproject
+.mypy_cache

.travis.yml

@@ -1,10 +1,10 @@
 language: python
 python:
-  - "2.6"
   - "2.7"
-  - "3.3"
-  - "3.4"
-  - "3.5"
+  - "3.6"
+  - "3.7"
+  - "3.8"
+  - "3.9"
   - "pypy"
 matrix:
   allow_failures:

CHANGELOG.md

@@ -1,5 +1,77 @@
 # Change Log
 
+## 0.11.5 (2019-12-16)
+
+Features:
+  - add --cauth argument to specify CAUTH cookie directly from command-line (#724)
+
+## 0.11.4 (2018-06-24)
+
+Features:
+  - Do not expand class names if there is a specialization with the same name,
+    but add --specialization flag to do that explicitly (#673)
+
+## 0.11.3 (2018-06-24)
+
+Bugfixes:
+  - Switch to newer API for syllabus and lecture retrieval (#665, #673, #634)
+
+Features:
+  - You can now download specializations: the child courses will be
+    downloaded automatically
+
+## 0.11.2 (2018-06-03)
+
+Bugfixes:
+  - Use TLS v1.2 instead of v1.0
+  - Switched to api.coursera.org subdomain for subtitles requests (#664)
+
+## 0.11.1 (2018-06-02)
+
+Bugfixes:
+  - Specify utf-8 encoding in setup.py to fix installation on Windows (#662)
+
+## 0.11.0 (2018-06-02)
+
+Features:
+  - Add support for "peer assignment" section (#650)
+
+Bugfixes:
+  - Switched to api.coursera.org subdomain for API requests (#660)
+
+## 0.10.0 (2018-02-19)
+
+Features:
+  - Support Coursera Notebooks (option: `--download-notebooks`)
+  - Add hints in the documentation for users in China
+
+## 0.9.0 (2017-05-25)
+
+Features:
+  - Default arguments are loaded from `coursera-dl.conf` file
+  - Added option `--mathjax-cdn <MATHJAX_CDN>` to specify alternative MathJax CDN
+  - Added support for Resources section
+
+## 0.8.0 (2016-10-04)
+
+Features:
+  - Add `--download-delay` option that adds a specified delay in seconds
+    before downloading next course. This is useful when downloading many
+    courses at once. Default value is 60 seconds.
+  - Add `--only-syllabus` option which is when activated, allows to skip
+    download of the course content. Only syllabus is parsed.
+  - Add support for `reflect` and `mcqReflect` question types in quizzes.
+  - Courses that encountered an error while parsing syllabus will be listed
+    in the end of the program execution, after all courses have been
+    processed (hopefully, downloaded). This helps skip vast output and easily
+    see which courses need user's attention, e.g. enrollment, session
+    switching or just patience until the course start date.
+
+Bugfixes:
+  - Locked programming assignments in syllabus used to crash coursera-dl.
+    Now the script goes on parsing syllabus and skips locked assignments.
+  - Add missing import statement to playlist generation module
+
 ## 0.7.0 (2016-07-28)
 
 Features:

CONTRIBUTING.md

@@ -4,7 +4,7 @@ developers/maintainers feel good when trying to change code that other
 people contributed.
 
 For the record, when this document mentions "I", it mostly means Rogério
-Brito's (@rbrito) is the one to blame.
+Theodoro de Brito's (@rbrito) is the one to blame.
 
 # Write good commit messages
 
@@ -237,7 +237,23 @@ DRAFT
    `git add ... & git ci -m 'Bump version (old_version -> new_version)'`
 4. `git tag new_version`
 5. `git push && git push --tags`
-6. `pandoc --from=markdown --to=rst --output=README.rst README.md`.
-   I think this is required for PyPI description to look nice.
-7. `python setup.py sdist` to build the package
-8. `twine upload dist/coursera-dl-0.6.1.tar.gz` to deploy the package.
+6. `python setup.py sdist bdist_wheel --universal` to build the package
+7. `twine upload dist/coursera-dl-0.6.1.tar.gz` to deploy the package.
+
+## Docker
+
+Build new Docker image from PyPI package:
+
+```
+docker build --tag courseradl/courseradl --build-arg VERSION=0.11.2 .
+```
+
+Run the image:
+```
+docker run --rm -it -v "$(pwd):/courses" -v "$HOME/.netrc:/netrc" courseradl -n /netrc -- google-machine-learning
+```
+
+Publish the image:
+```
+docker push courseradl/courseradl
+```

Dockerfile

@@ -0,0 +1,15 @@
+FROM python:3.6-slim
+
+LABEL maintainer "https://github.com/coursera-dl/"
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends gcc g++ libssl-dev && \
+    rm -rf /var/lib/apt/lists/* && \
+    apt-get purge -y --auto-remove gcc g++ libssl-dev
+
+ARG VERSION
+RUN pip install coursera-dl==$VERSION
+
+WORKDIR /courses
+ENTRYPOINT ["coursera-dl"]
+CMD ["--help"]

MANIFEST.in

@@ -1,3 +1,33 @@
 include requirements*.txt
 include CONTRIBUTING.md
 include LICENSE
+
+exclude .coveragerc
+exclude .ctags
+exclude .gitattributes
+exclude .github/ISSUE_TEMPLATE.md
+exclude .github/PULL_REQUEST_TEMPLATE.md
+exclude .gitignore
+exclude .travis.yml
+exclude AUTHORS.md
+exclude CHANGELOG.md
+exclude README.md
+exclude appveyor.yml
+exclude appveyor/install.ps1
+exclude appveyor/run_with_env.cmd
+exclude assets/hat-logo.svg
+exclude coursera-dl
+exclude coursera-dl.bat
+exclude deploy/.netrc
+exclude deploy/Dockerfile
+exclude deploy/README.md
+exclude deploy/build.sh
+exclude deploy/download.sh
+exclude fabfile.py
+exclude tox.ini
+
+prune appveyor/
+prune assets/
+prune deploy/
+prune coursera/test/
+prune .github/

README.md

@@ -4,9 +4,11 @@
 [![Build status](https://ci.appveyor.com/api/projects/status/3hru0ycv5fbny5k8/branch/master?svg=true)](https://ci.appveyor.com/project/balta2ar/coursera-dl/branch/master)
 [![Coverage Status](https://coveralls.io/repos/coursera-dl/coursera-dl/badge.svg)](https://coveralls.io/r/coursera-dl/coursera-dl)
 [![Latest version on PyPI](https://img.shields.io/pypi/v/coursera-dl.svg)](https://pypi.python.org/pypi/coursera-dl)
-[![Downloads from PyPI](https://img.shields.io/pypi/dm/coursera-dl.svg)](https://pypi.python.org/pypi/coursera-dl)
 [![Code Climate](https://codeclimate.com/github/coursera-dl/coursera-dl/badges/gpa.svg)](https://codeclimate.com/github/coursera-dl/coursera-dl)
 
+<!-- TOC -->
+
+- [Coursera Downloader](#coursera-downloader)
 - [Introduction](#introduction)
 - [Features](#features)
 - [Disclaimer](#disclaimer)
@@ -14,19 +16,28 @@
     - [Recommended installation method for all Operating Systems](#recommended-installation-method-for-all-operating-systems)
     - [Alternative ways of installing missing dependencies](#alternative-ways-of-installing-missing-dependencies)
         - [Alternative installation method for Unix systems](#alternative-installation-method-for-unix-systems)
+        - [ArchLinux](#archlinux)
         - [Installing dependencies on your own](#installing-dependencies-on-your-own)
+    - [Docker](#docker)
+    - [Windows](#windows)
     - [Create an account with Coursera](#create-an-account-with-coursera)
-    - [Running the script](#running-the-script)
+- [Running the script](#running-the-script)
     - [Resuming downloads](#resuming-downloads)
 - [Troubleshooting](#troubleshooting)
+    - [China issues](#china-issues)
     - [Found 0 sections and 0 lectures on this page](#found-0-sections-and-0-lectures-on-this-page)
+    - [Download timeouts](#download-timeouts)
+    - [Windows: proxy support](#windows-proxy-support)
     - [Windows: Failed to create process](#windows-failed-to-create-process)
-    - [SSLError: Errno 1 _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure](#sslerror-errno-1-_sslc504-error14094410ssl-routinesssl3_read_bytessslv3-alert-handshake-failure)
+    - [SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure](#sslerror-errno-1-_sslc504-error14094410ssl-routinesssl3_read_bytessslv3-alert-handshake-failure)
+    - [Alternative CDN for `MathJax.js`](#alternative-cdn-for-mathjaxjs)
 - [Reporting issues](#reporting-issues)
 - [Filing an issue/Reporting a bug](#filing-an-issuereporting-a-bug)
 - [Feedback](#feedback)
 - [Contact](#contact)
 
+<!-- /TOC -->
+
 # Introduction
 
 [Coursera][1] is arguably the leader in *massive open online courses* (MOOC)
@@ -68,6 +79,7 @@ I've downloaded many other good videos such as those from Khan Academy.
     certain resources.
   * File format extension filter to grab resource types you want.
   * Login credentials accepted on command-line or from `.netrc` file.
+  * Default arguments loaded from `coursera-dl.conf` file.
   * Core functionality tested on Linux, Mac and Windows.
 
 # Disclaimer
@@ -89,11 +101,11 @@ relevant excerpt:
 # Installation instructions
 
 `coursera-dl` requires Python 2 or Python 3 and a free Coursera account
-enrolled in the class of interest. (As of February of 2016, we test
-automatically the execution of the program with Python versions 2.6, 2.7,
-Pypy, 3.2, 3.3, 3.4, and 3.5).
+enrolled in the class of interest. (As of February of 2020, we test
+automatically the execution of the program with Python versions 2.7, Pypy,
+3.6, 3.7, 3.8, and 3.9).
 
-**Note:** We *strongly* recommend that you use a Python 3 interpreter (3.4
+**Note:** We *strongly* recommend that you use a Python 3 interpreter (3.9
 or later).
 
 On any operating system, ensure that the Python executable location is added
@@ -108,7 +120,7 @@ particular courses that you want to use with `coursera-dl`.
 
 ## Recommended installation method for all Operating Systems
 
-From a command line (preferrably, from a virtual environment), simply issue
+From a command line (preferably, from a virtual environment), simply issue
 the command:
 
     pip install coursera-dl
@@ -133,7 +145,7 @@ installed in your system (or they can interfere with `coursera-dl`).  Prefer
 to use the option `--user` to `pip install`, if you need can.
 
 **Note 2:** As already mentioned, we *strongly* recommend that you use a new
-Python 3 interpreter (e.g., 3.4 or later), since Python 3 has better support
+Python 3 interpreter (e.g., 3.9 or later), since Python 3 has better support
 for SSL/TLS (for secure connections) than earlier versions.<br/>
 If you must use Python 2, be sure that you have at least Python 2.7.9 (later
 versions are OK).<br/>
@@ -167,7 +179,7 @@ following steps (create/adapt first the directory
     cd my-coursera
     source bin/activate
     git clone https://github.com/coursera-dl/coursera-dl
-    cd coursera
+    cd coursera-dl
     pip install -r requirements.txt
     ./coursera-dl ...
 
@@ -175,7 +187,7 @@ To further download new videos from your classes, simply perform:
 
     cd /directory/where/I/want/my/courses/my-coursera
     source bin/activate
-    cd coursera
+    cd coursera-dl
     ./coursera-dl ...
 
 We are working on streamlining this whole process so that it is as simple as
@@ -211,60 +223,142 @@ your own, please check that the versions of your modules are at least those
 listed in the `requirements.txt` file (and, `requirements-dev.txt` file, if
 applicable).
 
+## Docker
+
+If you prefer you can run this software inside Docker:
+
+```
+docker run --rm -it -v \
+    "$(pwd):/courses" \
+    courseradl/courseradl -u <USER> -p <PASSWORD>
+```
+
+Or using netrc file:
+
+```
+docker run --rm -it \
+    -v "$(pwd):/courses" -v "$HOME/.netrc:/netrc" \
+    courseradl/courseradl -n /netrc
+```
+
+The actual working dir for coursera-dl is /courses, all courses will be
+downloaded there if you don't specify otherwise.
+
+## Windows
+
+`python -m pip install coursera-dl`
+
+Be sure that the Python install path is added to the PATH system environment
+variables. This can be found in Control Panel > System > Advanced System
+Settings > Environment Variables.
+
+```
+Example:
+C:\Python39\Scripts\;C:\Python39\;
+```
+
+Or if you have restricted installation permissions and you've installed Python
+under AppData, add this to your PATH.
+
+```
+Example:
+C:\Users\<user>\AppData\Local\Programs\Python\Python39-32\Scripts;C:\Users\<user>\AppData\Local\Programs\Python\Python39-32;
+```
+
+Coursera-dl can now be run from commandline or powershell.
+
 ## Create an account with Coursera
 
 If you don't already have one, create a [Coursera][1] account and enroll in
 a class. See https://www.coursera.org/courses for the list of classes.
 
-## Running the script
+# Running the script
+
+Refer to `coursera-dl --help` for a complete, up-to-date reference on the runtime options
+supported by this utility.
 
 Run the script to download the materials by providing your Coursera account
 credentials (e.g. email address and password or a `~/.netrc` file), the
 class names, as well as any additional parameters:
-
+```
     General:                     coursera-dl -u <user> -p <pass> modelthinking-004
+
+    With CAUTH parameter:	 coursera-dl -ca 'some-ca-value-from-browser' modelthinking-004
+```
+If you don't want to type your password in command line as plain text, you can use the
+script without `-p` option. In this case you will be prompted for password  once the
+script is run.
+
+Here are some examples of how to invoke `coursera-dl` from the command line:
+```
+    Without -p field:            coursera-dl -u <user> modelthinking-004
     Multiple classes:            coursera-dl -u <user> -p <pass> saas historyofrock1-001 algo-2012-002
     Filter by section name:      coursera-dl -u <user> -p <pass> -sf "Chapter_Four" crypto-004
     Filter by lecture name:      coursera-dl -u <user> -p <pass> -lf "3.1_" ml-2012-002
     Download only ppt files:     coursera-dl -u <user> -p <pass> -f "ppt" qcomp-2012-001
     Use a ~/.netrc file:         coursera-dl -n -- matrix-001
     Get the preview classes:     coursera-dl -n -b ni-001
+	Download videos at 720p:     coursera-dl -n --video-resolution 720p ni-001
     Specify download path:       coursera-dl -n --path=C:\Coursera\Classes\ comnetworks-002
     Display help:                coursera-dl --help
 
     Maintain a list of classes in a dir:
       Initialize:              mkdir -p CURRENT/{class1,class2,..classN}
       Update:                  coursera-dl -n --path CURRENT `\ls CURRENT`
-
+```
 **Note:** If your `ls` command is aliased to display a colorized output, you
 may experience problems.  Be sure to escape the `ls` command (use `\ls`) to
 assure that no special characters get sent to the script.
 
-Note that we *do* support the New Platform ("on-demand") classes.
+Note that we *do* support the New Platform ("on-demand") courses.
+
+By default, videos are downloaded at 540p resolution. For on-demand courses, the
+`--video-resolution` flag accepts 360p, 540p, and 720p values.
+
+To download just the `.txt` and/or `.srt` subtitle files instead of the videos,
+use `-ignore-formats mp4 --subtitle-language en` or whatever format the videos
+are encoded in and desired languages for subtitles.
 
 On \*nix platforms, the use of a `~/.netrc` file is a good alternative to
 specifying both your username (i.e., your email address) and password every
 time on the command line. To use it, simply add a line like the one below to
 a file named `.netrc` in your home directory (or the [equivalent][8], if you
 are using Windows) with contents like:
-
+```
     machine coursera-dl login <user> password <pass>
-
+```
 Create the file if it doesn't exist yet.  From then on, you can switch from
 using `-u` and `-p` to simply call `coursera-dl` with the option `-n`
 instead.  This is especially convenient, as typing usernames (email
 addresses) and passwords directly on the command line can get tiresome (even
 more if you happened to choose a "strong" password).
 
+Alternatively, if you want to store your preferred parameters (which might
+also include your username and password), create a file named `coursera-dl.conf`
+where the script is supposed to be executed, with the following format:
+```
+    --username <user>
+    --password <pass>
+    --subtitle-language en,zh-CN|zh-TW
+    --download-quizzes
+    #--mathjax-cdn https://cdn.bootcss.com/mathjax/2.7.1/MathJax.js
+    # more other parameters
+```
+Parameters which are specified in the file will be overriden if they are 
+provided again on the commandline.
+
+**Note:** In `coursera-dl.conf`, all the parameters should not be wrapped
+with quotes.
+
 ## Resuming downloads
 
 In default mode when you interrupt the download process by pressing
 <kbd>CTRL</kbd>+<kbd>C</kbd>, partially downloaded files will be deleted from your disk and
-you have to start the download process from the begining. If your
+you have to start the download process from the beginning. If your
 download was interrupted by something other than KeyboardInterrupt
 (<kbd>CTRL</kbd>+<kbd>C</kbd>) like sudden system crash, partially downloaded files will
 remain on your disk and the next time you start the process again,
-these files will be discraded from download list!, therefore it's your
+these files will be discarded from download list!, therefore it's your
 job to delete them manually before next start. For this reason we
 added an option called `--resume` which continues your downloads from
 where they stopped:
@@ -295,7 +389,7 @@ one of the following actions solve your problem:
 
 * Make sure the class name you are using corresponds to the resource name
   used in the URL for that class:
-    `https://class.coursera.org/<CLASS_NAME>/class/index`
+    `https://www.coursera.org/learn/<CLASS_NAME>/home/welcome`
 
 * Have you tried to clean the cached cookies/credentials with the
   `--clear-cache` option?
@@ -319,7 +413,7 @@ one of the following actions solve your problem:
 
 * If results show 0 sections, you most likely have provided invalid
   credentials (username and/or password in the command line or in your
-  `.netrc` file).
+  `.netrc` file or in your `coursera-dl.conf` file).
 
 * For courses that have not started yet, but have had a previous iteration
   sometimes a preview is available, containing all the classes from the last
@@ -340,7 +434,7 @@ one of the following actions solve your problem:
     * You get an error when using `-n` to specify that you want to use a
       `.netrc` file and,
     * You want the script to use your default netrc file and,
-    * You get a message saying `coursera-dl: error: too few arguments`  
+    * You get a message saying `coursera-dl: error: too few arguments`
 
       Then you should specify `--` as an argument after `-n`, that is, `-n --`
       or change the order in which you pass the arguments to the script, so that
@@ -359,6 +453,13 @@ one of the following actions solve your problem:
       pip install coursera-dl
   ```
 
+## China issues
+
+If you are from China and you're having problems downloading videos,
+adding "52.84.167.78 d3c33hcgiwev3.cloudfront.net" in the hosts file
+(/etc/hosts) and freshing DNS with "ipconfig/flushdns" may work
+(see https://github.com/googlehosts/hosts for more info).
+
 ## Found 0 sections and 0 lectures on this page
 
 First of all, make sure you are enrolled to the course you want to download.
@@ -373,13 +474,50 @@ file that lists all the course materials. Maybe your friend who is enrolled
 could save that course page for you. In that case use the `--process_local_page`
 option.
 
-Alternatively you may want to try this Chrome extension: https://chrome.google.com/webstore/detail/coursera-materials-downlo/ijkboagofaehocnjacacdhdcbbcpilih
+Alternatively you may want to try this various browser extensions designed for
+this problem.
 
 If none of the above works for you, there is nothing we can do.
 
+## Download timeouts
+
+Coursera-dl supports external downloaders but note that they are only used to
+download materials after the syllabus has been parsed, e.g. videos, PDFs, some
+handouts and additional files (syllabus is always downloaded using the internal
+downloader). If you experience problems with downloading such materials, you may
+want to start using external downloader and configure its timeout values. For
+example, you can use aria2c downloader by passing `--aria` option:
+
+```
+coursera-dl -n --path . --aria2  <course-name>
+```
+
+And put this into aria2c's configuration file `~/.aria2/aria2.conf` to reduce
+timeouts:
+
+```
+connect-timeout=2
+timeout=2
+bt-stop-timeout=1
+```
+
+Timeout configuration for internal downloader is not supported.
+
+## Windows: proxy support
+
+If you're on Windows behind a proxy, set up the environment variables
+before running the script as follows:
+
+```
+set HTTP_PROXY=http://host:port
+set HTTPS_PROXY=http://host:port
+```
+
+Related discussion: [#205](https://github.com/coursera-dl/coursera-dl/issues/205)
+
 ## Windows: Failed to create process
 
-In `C:\Users\<user>\AppData\Local\Programs\Python\Python35-32\Scripts`
+In `C:\Users\<user>\AppData\Local\Programs\Python\Python39-32\Scripts`
 or wherever Python installed (above is default for Windows)
 edit below file in idle: (right click on script name and select 'edit with idle in menu)
 
@@ -390,13 +528,13 @@ coursera-dl-script
 from
 
 ```
-#!c:\users\<user>\appdata\local\programs\python\python35-32\python.exe
+#!c:\users\<user>\appdata\local\programs\python\python39-32\python.exe
 ```
 
 to
 
 ```
-#"!c:\users\<user>\appdata\local\programs\python\python35-32\python.exe"
+#"!c:\users\<user>\appdata\local\programs\python\python39-32\python.exe"
 ```
 
 (add quotes). This is a known pip bug.
@@ -422,6 +560,15 @@ If you still have the problem, please read the following issues for more ideas o
 This is also worth reading:
 https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning
 
+## Alternative CDN for `MathJax.js`
+
+When saving a course page, we enabled `MathJax` rendering for math equations, by
+injecting `MathJax.js` in the header. The script is using a cdn service provided
+by [mathjax.org](https://cdn.mathjax.org/mathjax/latest/MathJax.js). However, that
+url is not accessible in some countries/regions, you can provide a
+`--mathjax-cdn <MATHJAX_CDN>` parameter to specify the `MathJax.js` file that is
+accessible in your region.
+
 # Reporting issues
 
 Before reporting any issue please follow the steps below:
@@ -488,10 +635,9 @@ I enjoy getting feedback. Here are a few of the comments I've received:
 
 # Contact
 
-Please, post bugs and issues on [github][11]. Send other comments to Rogério
-Theodoro de Brito (the current maintainer): rbrito@ime.usp.br (twitter:
-[@rtdbrito][21]) or to John Lehmann (the original author): first last at
-geemail dotcom (twitter: [@jplehmann][12]).
+Please, post bugs and issues on [github][11]. Please, **DON'T** send support
+requests privately to the maintainers! We are quite swamped with day-to-day
+activities. If you have problems, **PLEASE**, file them on the issue tracker.
 
 [1]: https://www.coursera.org
 [2]: https://sourceforge.net/projects/gnuwin32/files/wget/1.11.4-1/wget-1.11.4-1-setup.exe
@@ -504,7 +650,6 @@ geemail dotcom (twitter: [@jplehmann][12]).
 [9]: https://chrome.google.com/webstore/detail/cookietxt-export/lopabhfecdfhgogdbojmaicoicjekelh
 [10]: https://addons.mozilla.org/en-US/firefox/addon/export-cookies/
 [11]: https://github.com/coursera-dl/coursera-dl/issues
-[12]: https://twitter.com/jplehmann
 [13]: http://techcrunch.com/2013/02/20/coursera-adds-29-schools-90-courses-and-4-new-languages-to-its-online-learning-platform/
 [14]: http://www.tunapanda.org
 [15]: https://github.com/html5lib/html5lib-python
@@ -513,11 +658,8 @@ geemail dotcom (twitter: [@jplehmann][12]).
 [18]: http://ww45.python-distribute.org/pip_distribute.png
 [19]: https://pypi.python.org/pypi/six/
 [20]: https://www.coursera.org/about/terms
-[21]: https://twitter.com/rtdbrito
 [22]: https://pypi.python.org/
 [23]: https://pypi.python.org/pypi/coursera-dl
 [issue213]: https://github.com/coursera-dl/coursera-dl/issues/213
 [issue500]: https://github.com/coursera-dl/coursera-dl/issues/500
 [pipinstallerbug]: http://stackoverflow.com/questions/31808180/installing-pyinstaller-via-pip-leads-to-failed-to-create-process
-
-[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/coursera-dl/coursera-dl/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

appveyor.yml

@@ -24,14 +24,6 @@ environment:
     # a later point release.
     # See: http://www.appveyor.com/docs/installed-software#python
 
-    - PYTHON: "C:\\Python26"
-      PYTHON_VERSION: "2.6.x" # currently 2.6.6
-      PYTHON_ARCH: "32"
-
-    - PYTHON: "C:\\Python26-x64"
-      PYTHON_VERSION: "2.6.x" # currently 2.6.6
-      PYTHON_ARCH: "64"
-
     - PYTHON: "C:\\Python27"
       PYTHON_VERSION: "2.7.x" # currently 2.7.11
       PYTHON_ARCH: "32"
@@ -40,28 +32,36 @@ environment:
       PYTHON_VERSION: "2.7.x" # currently 2.7.11
       PYTHON_ARCH: "64"
 
-    - PYTHON: "C:\\Python33"
-      PYTHON_VERSION: "3.3.x" # currently 3.3.5
+    - PYTHON: "C:\\Python36"
+      PYTHON_VERSION: "3.6.x" # currently 3.6.?
       PYTHON_ARCH: "32"
 
-    - PYTHON: "C:\\Python33-x64"
-      PYTHON_VERSION: "3.3.x" # currently 3.3.5
+    - PYTHON: "C:\\Python36-x64"
+      PYTHON_VERSION: "3.6.x" # currently 3.6.?
       PYTHON_ARCH: "64"
 
-    - PYTHON: "C:\\Python34"
-      PYTHON_VERSION: "3.4.x" # currently 3.4.3
+    - PYTHON: "C:\\Python37"
+      PYTHON_VERSION: "3.7.x" # currently 3.7.?
       PYTHON_ARCH: "32"
 
-    - PYTHON: "C:\\Python34-x64"
-      PYTHON_VERSION: "3.4.x" # currently 3.4.3
+    - PYTHON: "C:\\Python37-x64"
+      PYTHON_VERSION: "3.7.x" # currently 3.7.?
       PYTHON_ARCH: "64"
 
-    - PYTHON: "C:\\Python35"
-      PYTHON_VERSION: "3.5.x" # currently 3.5.1
+    - PYTHON: "C:\\Python38"
+      PYTHON_VERSION: "3.8.x" # currently 3.8.?
       PYTHON_ARCH: "32"
 
-    - PYTHON: "C:\\Python35-x64"
-      PYTHON_VERSION: "3.5.x" # currently 3.5.1
+    - PYTHON: "C:\\Python38-x64"
+      PYTHON_VERSION: "3.8.x" # currently 3.8.?
+      PYTHON_ARCH: "64"
+
+    - PYTHON: "C:\\Python39"
+      PYTHON_VERSION: "3.8.x" # currently 3.9.?
+      PYTHON_ARCH: "32"
+
+    - PYTHON: "C:\\Python38-x64"
+      PYTHON_VERSION: "3.8.x" # currently 3.9.?
       PYTHON_ARCH: "64"
 
 init:
@@ -83,7 +83,7 @@ install:
 
   # Upgrade to the latest version of pip to avoid it displaying warnings
   # about it being out of date.
-  - "pip install --disable-pip-version-check --user --upgrade pip"
+  - "python -m pip install --disable-pip-version-check --user --upgrade pip"
 
   # Install requirements
   - "%CMD_IN_ENV% pip install -r requirements.txt"

coursera-dl

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 
 from coursera import coursera_dl

coursera/__init__.py

@@ -1 +1 @@
-__version__ = '0.7.0'
+__version__ = '0.11.5'

coursera/commandline.py

@@ -6,319 +6,447 @@ handling. The primary candidate is argument parser.
 import os
 import sys
 import logging
-import argparse
+import configargparse as argparse
 
 from coursera import __version__
 
 from .credentials import get_credentials, CredentialsError, keyring
 from .utils import decode_input
 
+LOCAL_CONF_FILE_NAME = 'coursera-dl.conf'
+
+
+def class_name_arg_required(args):
+    """
+    Evaluates whether class_name arg is required.
+
+    @param args: Command-line arguments.
+    @type args: namedtuple
+    """
+    no_class_name_flags = ['list_courses', 'version']
+    return not any(
+        getattr(args, flag)
+        for flag in no_class_name_flags
+    )
+
 
 def parse_args(args=None):
     """
     Parse the arguments/options passed to the program on the command line.
     """
 
-    parser = argparse.ArgumentParser(
-        description='Download Coursera.org lecture material and resources.')
+    parse_kwargs = {
+        "description":  'Download Coursera.org lecture material and resources.'
+    }
+
+    conf_file_path = os.path.join(os.getcwd(), LOCAL_CONF_FILE_NAME)
+    if os.path.isfile(conf_file_path):
+        parse_kwargs["default_config_files"] = [conf_file_path]
+    parser = argparse.ArgParser(**parse_kwargs)
 
     # Basic options
     group_basic = parser.add_argument_group('Basic options')
 
-    group_basic.add_argument('class_names',
-                             action='store',
-                             nargs='+',
-                             help='name(s) of the class(es) (e.g. "ml-005")')
+    group_basic.add_argument(
+        'class_names',
+        action='store',
+        nargs='*',
+        help='name(s) of the class(es) (e.g. "ml-005")')
 
-    group_basic.add_argument('-u',
-                             '--username',
-                             dest='username',
-                             action='store',
-                             default=None,
-                             help='coursera username')
+    group_basic.add_argument(
+        '-u',
+        '--username',
+        dest='username',
+        action='store',
+        default=None,
+        help='username (email) that you use to login to Coursera')
 
-    group_basic.add_argument('-p',
-                             '--password',
-                             dest='password',
-                             action='store',
-                             default=None,
-                             help='coursera password')
+    group_basic.add_argument(
+        '-p',
+        '--password',
+        dest='password',
+        action='store',
+        default=None,
+        help='coursera password')
 
-    group_basic.add_argument('--jobs',
-                             dest='jobs',
-                             action='store',
-                             default=1,
-                             type=int,
-                             help='number of parallel jobs to use for '
-                             'downloading resources. (Default: 1)')
+    group_basic.add_argument(
+        '--jobs',
+        dest='jobs',
+        action='store',
+        default=1,
+        type=int,
+        help='number of parallel jobs to use for '
+        'downloading resources. (Default: 1)')
 
-    group_basic.add_argument('-b',  # FIXME: kill this one-letter option
-                             '--preview',
-                             dest='preview',
-                             action='store_true',
-                             default=False,
-                             help='get videos from preview pages. (Default: False)')
+    group_basic.add_argument(
+        '--download-delay',
+        dest='download_delay',
+        action='store',
+        default=60,
+        type=int,
+        help='number of seconds to wait before downloading '
+        'next course. (Default: 60)')
 
-    group_basic.add_argument('--path',
-                             dest='path',
-                             action='store',
-                             default='',
-                             help='path to where to save the file. (Default: current directory)')
+    group_basic.add_argument(
+        '-b',  # FIXME: kill this one-letter option
+        '--preview',
+        dest='preview',
+        action='store_true',
+        default=False,
+        help='get videos from preview pages. (Default: False)')
 
-    group_basic.add_argument('-sl',  # FIXME: deprecate this option
-                             '--subtitle-language',
-                             dest='subtitle_language',
-                             action='store',
-                             default='all',
-                             help='Choose language to download subtitles and transcripts. (Default: all)'
-                             'Use special value "all" to download all available.')
+    group_basic.add_argument(
+        '--path',
+        dest='path',
+        action='store',
+        default='',
+        help='path to where to save the file. (Default: current directory)')
+
+    group_basic.add_argument(
+        '-sl',  # FIXME: deprecate this option
+        '--subtitle-language',
+        dest='subtitle_language',
+        action='store',
+        default='all',
+        help='Choose language to download subtitles and transcripts.'
+        '(Default: all) Use special value "all" to download all available.'
+        'To download subtitles and transcripts of multiple languages,'
+        'use comma(s) (without spaces) to seperate the names of the languages,'
+        ' i.e., "en,zh-CN".'
+        'To download subtitles and transcripts of alternative language(s) '
+        'if only the current language is not available,'
+        'put an "|<lang>" for each of the alternative languages after '
+        'the current language, i.e., "en|fr,zh-CN|zh-TW|de", and make sure '
+        'the parameter are wrapped with quotes when "|" presents.'
+
+    )
 
     # Selection of material to download
-    group_material = parser.add_argument_group('Selection of material to download')
+    group_material = parser.add_argument_group(
+        'Selection of material to download')
 
-    group_material.add_argument('--download-quizzes',
-                                dest='download_quizzes',
-                                action='store_true',
-                                default=False,
-                                help='download quiz and exam questions. (Default: False)')
+    group_material.add_argument(
+        '--specialization',
+        dest='specialization',
+        action='store_true',
+        default=False,
+        help='treat given class names as specialization names and try to '
+        'download its courses, if available. Note that there are name '
+        'clashes, e.g. "machine-learning" is both a course and a '
+        'specialization (Default: False)')
 
-    group_material.add_argument('--about',  # FIXME: should be --about-course
-                                dest='about',
-                                action='store_true',
-                                default=False,
-                                help='download "about" metadata. (Default: False)')
+    group_material.add_argument(
+        '--only-syllabus',
+        dest='only_syllabus',
+        action='store_true',
+        default=False,
+        help='download only syllabus, skip course content. '
+        '(Default: False)')
 
-    group_material.add_argument('-f',
-                                '--formats',
-                                dest='file_formats',
-                                action='store',
-                                default='all',
-                                help='file format extensions to be downloaded in'
-                                ' quotes space separated, e.g. "mp4 pdf" '
-                                '(default: special value "all")')
+    group_material.add_argument(
+        '--download-quizzes',
+        dest='download_quizzes',
+        action='store_true',
+        default=False,
+        help='download quiz and exam questions. (Default: False)')
 
-    group_material.add_argument('--ignore-formats',
-                                dest='ignore_formats',
-                                action='store',
-                                default=None,
-                                help='file format extensions of resources to ignore'
-                                ' (default: None)')
+    group_material.add_argument(
+        '--download-notebooks',
+        dest='download_notebooks',
+        action='store_true',
+        default=False,
+        help='download Python Jupyther Notebooks. (Default: False)')
 
-    group_material.add_argument('-sf',  # FIXME: deprecate this option
-                                '--section_filter',
-                                dest='section_filter',
-                                action='store',
-                                default=None,
-                                help='only download sections which contain this'
-                                ' regex (default: disabled)')
+    group_material.add_argument(
+        '--about',  # FIXME: should be --about-course
+        dest='about',
+        action='store_true',
+        default=False,
+        help='download "about" metadata. (Default: False)')
 
-    group_material.add_argument('-lf',  # FIXME: deprecate this option
-                                '--lecture_filter',
-                                dest='lecture_filter',
-                                action='store',
-                                default=None,
-                                help='only download lectures which contain this regex'
-                                ' (default: disabled)')
+    group_material.add_argument(
+        '-f',
+        '--formats',
+        dest='file_formats',
+        action='store',
+        default='all',
+        help='file format extensions to be downloaded in'
+        ' quotes space separated, e.g. "mp4 pdf" '
+        '(default: special value "all")')
 
-    group_material.add_argument('-rf',  # FIXME: deprecate this option
-                                '--resource_filter',
-                                dest='resource_filter',
-                                action='store',
-                                default=None,
-                                help='only download resources which match this regex'
-                                ' (default: disabled)')
+    group_material.add_argument(
+        '--ignore-formats',
+        dest='ignore_formats',
+        action='store',
+        default=None,
+        help='file format extensions of resources to ignore'
+        ' (default: None)')
 
-    group_material.add_argument('--video-resolution',
-                                dest='video_resolution',
-                                action='store',
-                                default='540p',
-                                help='video resolution to download (default: 540p); '
-                                'only valid for on-demand courses; '
-                                'only values allowed: 360p, 540p, 720p')
+    group_material.add_argument(
+        '-sf',  # FIXME: deprecate this option
+        '--section_filter',
+        dest='section_filter',
+        action='store',
+        default=None,
+        help='only download sections which contain this'
+        ' regex (default: disabled)')
 
-    group_material.add_argument('--disable-url-skipping',
-                                dest='disable_url_skipping',
-                                action='store_true',
-                                default=False,
-                                help='disable URL skipping, all URLs will be '
-                                'downloaded (default: False)')
+    group_material.add_argument(
+        '-lf',  # FIXME: deprecate this option
+        '--lecture_filter',
+        dest='lecture_filter',
+        action='store',
+        default=None,
+        help='only download lectures which contain this regex'
+        ' (default: disabled)')
+
+    group_material.add_argument(
+        '-rf',  # FIXME: deprecate this option
+        '--resource_filter',
+        dest='resource_filter',
+        action='store',
+        default=None,
+        help='only download resources which match this regex'
+        ' (default: disabled)')
+
+    group_material.add_argument(
+        '--video-resolution',
+        dest='video_resolution',
+        action='store',
+        default='540p',
+        help='video resolution to download (default: 540p); '
+        'only valid for on-demand courses; '
+        'only values allowed: 360p, 540p, 720p')
+
+    group_material.add_argument(
+        '--disable-url-skipping',
+        dest='disable_url_skipping',
+        action='store_true',
+        default=False,
+        help='disable URL skipping, all URLs will be '
+        'downloaded (default: False)')
 
     # Parameters related to external downloaders
     group_external_dl = parser.add_argument_group('External downloaders')
 
-    group_external_dl.add_argument('--wget',
-                                   dest='wget',
-                                   action='store',
-                                   nargs='?',
-                                   const='wget',
-                                   default=None,
-                                   help='use wget for downloading,'
-                                   'optionally specify wget bin')
-    group_external_dl.add_argument('--curl',
-                                   dest='curl',
-                                   action='store',
-                                   nargs='?',
-                                   const='curl',
-                                   default=None,
-                                   help='use curl for downloading,'
-                                   ' optionally specify curl bin')
-    group_external_dl.add_argument('--aria2',
-                                   dest='aria2',
-                                   action='store',
-                                   nargs='?',
-                                   const='aria2c',
-                                   default=None,
-                                   help='use aria2 for downloading,'
-                                   ' optionally specify aria2 bin')
-    group_external_dl.add_argument('--axel',
-                                   dest='axel',
-                                   action='store',
-                                   nargs='?',
-                                   const='axel',
-                                   default=None,
-                                   help='use axel for downloading,'
-                                   ' optionally specify axel bin')
-    group_external_dl.add_argument('--downloader-arguments',
-                                   dest='downloader_arguments',
-                                   default='',
-                                   help='additional arguments passed to the'
-                                   ' downloader')
+    group_external_dl.add_argument(
+        '--wget',
+        dest='wget',
+        action='store',
+        nargs='?',
+        const='wget',
+        default=None,
+        help='use wget for downloading,'
+        'optionally specify wget bin')
 
-    parser.add_argument('--list-courses',
-                        dest='list_courses',
-                        action='store_true',
-                        default=False,
-                        help='list course names (slugs) and quit. Listed '
-                        'course names can be put into program arguments')
+    group_external_dl.add_argument(
+        '--curl',
+        dest='curl',
+        action='store',
+        nargs='?',
+        const='curl',
+        default=None,
+        help='use curl for downloading,'
+        ' optionally specify curl bin')
 
-    parser.add_argument('--resume',
-                        dest='resume',
-                        action='store_true',
-                        default=False,
-                        help='resume incomplete downloads (default: False)')
+    group_external_dl.add_argument(
+        '--aria2',
+        dest='aria2',
+        action='store',
+        nargs='?',
+        const='aria2c',
+        default=None,
+        help='use aria2 for downloading,'
+        ' optionally specify aria2 bin')
 
-    parser.add_argument('-o',
-                        '--overwrite',
-                        dest='overwrite',
-                        action='store_true',
-                        default=False,
-                        help='whether existing files should be overwritten'
-                             ' (default: False)')
+    group_external_dl.add_argument(
+        '--axel',
+        dest='axel',
+        action='store',
+        nargs='?',
+        const='axel',
+        default=None,
+        help='use axel for downloading,'
+        ' optionally specify axel bin')
 
-    parser.add_argument('--verbose-dirs',
-                        dest='verbose_dirs',
-                        action='store_true',
-                        default=False,
-                        help='include class name in section directory name')
+    group_external_dl.add_argument(
+        '--downloader-arguments',
+        dest='downloader_arguments',
+        default='',
+        help='additional arguments passed to the'
+        ' downloader')
 
-    parser.add_argument('--quiet',
-                        dest='quiet',
-                        action='store_true',
-                        default=False,
-                        help='omit as many messages as possible'
-                             ' (only printing errors)')
+    parser.add_argument(
+        '--list-courses',
+        dest='list_courses',
+        action='store_true',
+        default=False,
+        help='list course names (slugs) and quit. Listed '
+        'course names can be put into program arguments')
 
-    parser.add_argument('-r',
-                        '--reverse',
-                        dest='reverse',
-                        action='store_true',
-                        default=False,
-                        help='download sections in reverse order')
+    parser.add_argument(
+        '--resume',
+        dest='resume',
+        action='store_true',
+        default=False,
+        help='resume incomplete downloads (default: False)')
 
-    parser.add_argument('--combined-section-lectures-nums',
-                        dest='combined_section_lectures_nums',
-                        action='store_true',
-                        default=False,
-                        help='include lecture and section name in final files')
+    parser.add_argument(
+        '-o',
+        '--overwrite',
+        dest='overwrite',
+        action='store_true',
+        default=False,
+        help='whether existing files should be overwritten'
+        ' (default: False)')
 
-    parser.add_argument('--unrestricted-filenames',
-                        dest='unrestricted_filenames',
-                        action='store_true',
-                        default=False,
-                        help='Do not limit filenames to be ASCII-only')
+    parser.add_argument(
+        '--verbose-dirs',
+        dest='verbose_dirs',
+        action='store_true',
+        default=False,
+        help='include class name in section directory name')
+
+    parser.add_argument(
+        '--quiet',
+        dest='quiet',
+        action='store_true',
+        default=False,
+        help='omit as many messages as possible'
+        ' (only printing errors)')
+
+    parser.add_argument(
+        '-r',
+        '--reverse',
+        dest='reverse',
+        action='store_true',
+        default=False,
+        help='download sections in reverse order')
+
+    parser.add_argument(
+        '--combined-section-lectures-nums',
+        dest='combined_section_lectures_nums',
+        action='store_true',
+        default=False,
+        help='include lecture and section name in final files')
+
+    parser.add_argument(
+        '--unrestricted-filenames',
+        dest='unrestricted_filenames',
+        action='store_true',
+        default=False,
+        help='Do not limit filenames to be ASCII-only')
 
     # Advanced authentication
-    group_adv_auth = parser.add_argument_group('Advanced authentication options')
+    group_adv_auth = parser.add_argument_group(
+        'Advanced authentication options')
 
-    group_adv_auth.add_argument('-c',
-                                '--cookies_file',
-                                dest='cookies_file',
-                                action='store',
-                                default=None,
-                                help='full path to the cookies.txt file')
+    group_adv_auth.add_argument(
+        '-ca',
+        '--cauth',
+        dest='cookies_cauth',
+        action='store',
+        default=None,
+        help='cauth cookie value from browser')
 
-    group_adv_auth.add_argument('-n',
-                                '--netrc',
-                                dest='netrc',
-                                nargs='?',
-                                action='store',
-                                const=True,
-                                default=False,
-                                help='use netrc for reading passwords, uses default'
-                                ' location if no path specified')
+    group_adv_auth.add_argument(
+        '-c',
+        '--cookies_file',
+        dest='cookies_file',
+        action='store',
+        default=None,
+        help='full path to the cookies.txt file')
 
-    group_adv_auth.add_argument('-k',
-                                '--keyring',
-                                dest='use_keyring',
-                                action='store_true',
-                                default=False,
-                                help='use keyring provided by operating system to '
-                                'save and load credentials')
+    group_adv_auth.add_argument(
+        '-n',
+        '--netrc',
+        dest='netrc',
+        nargs='?',
+        action='store',
+        const=True,
+        default=False,
+        help='use netrc for reading passwords, uses default'
+        ' location if no path specified')
 
-    group_adv_auth.add_argument('--clear-cache',
-                                dest='clear_cache',
-                                action='store_true',
-                                default=False,
-                                help='clear cached cookies')
+    group_adv_auth.add_argument(
+        '-k',
+        '--keyring',
+        dest='use_keyring',
+        action='store_true',
+        default=False,
+        help='use keyring provided by operating system to '
+        'save and load credentials')
+
+    group_adv_auth.add_argument(
+        '--clear-cache',
+        dest='clear_cache',
+        action='store_true',
+        default=False,
+        help='clear cached cookies')
 
     # Advanced miscellaneous options
-    group_adv_misc = parser.add_argument_group('Advanced miscellaneous options')
+    group_adv_misc = parser.add_argument_group(
+        'Advanced miscellaneous options')
 
-    group_adv_misc.add_argument('--hook',
-                                dest='hooks',
-                                action='append',
-                                default=[],
-                                help='hooks to run when finished')
+    group_adv_misc.add_argument(
+        '--hook',
+        dest='hooks',
+        action='append',
+        default=[],
+        help='hooks to run when finished')
 
-    group_adv_misc.add_argument('-pl',
-                                '--playlist',
-                                dest='playlist',
-                                action='store_true',
-                                default=False,
-                                help='generate M3U playlists for course weeks')
+    group_adv_misc.add_argument(
+        '-pl',
+        '--playlist',
+        dest='playlist',
+        action='store_true',
+        default=False,
+        help='generate M3U playlists for course weeks')
+
+    group_adv_misc.add_argument(
+        '--mathjax-cdn',
+        dest='mathjax_cdn_url',
+        default='https://cdn.mathjax.org/mathjax/latest/MathJax.js',
+        help='the cdn address of MathJax.js'
+    )
 
     # Debug options
     group_debug = parser.add_argument_group('Debugging options')
 
-    group_debug.add_argument('--skip-download',
-                             dest='skip_download',
-                             action='store_true',
-                             default=False,
-                             help='for debugging: skip actual downloading of files')
+    group_debug.add_argument(
+        '--skip-download',
+        dest='skip_download',
+        action='store_true',
+        default=False,
+        help='for debugging: skip actual downloading of files')
 
-    group_debug.add_argument('--debug',
-                             dest='debug',
-                             action='store_true',
-                             default=False,
-                             help='print lots of debug information')
+    group_debug.add_argument(
+        '--debug',
+        dest='debug',
+        action='store_true',
+        default=False,
+        help='print lots of debug information')
 
-    group_debug.add_argument('--cache-syllabus',
-                             dest='cache_syllabus',
-                             action='store_true',
-                             default=False,
-                             help='cache course syllabus into a file')
+    group_debug.add_argument(
+        '--cache-syllabus',
+        dest='cache_syllabus',
+        action='store_true',
+        default=False,
+        help='cache course syllabus into a file')
 
-    group_debug.add_argument('--version',
-                             dest='version',
-                             action='store_true',
-                             default=False,
-                             help='display version and exit')
+    group_debug.add_argument(
+        '--version',
+        dest='version',
+        action='store_true',
+        default=False,
+        help='display version and exit')
 
-    group_debug.add_argument('-l',  # FIXME: remove short option from rarely used ones
-                             '--process_local_page',
-                             dest='local_page',
-                             help='uses or creates local cached version of syllabus'
-                             ' page')
+    group_debug.add_argument(
+        '-l',  # FIXME: remove short option from rarely used ones
+        '--process_local_page',
+        dest='local_page',
+        help='uses or creates local cached version of syllabus'
+        ' page')
 
     # Final parsing of the options
     args = parser.parse_args(args)
@@ -335,10 +463,16 @@ def parse_args(args=None):
         logging.basicConfig(level=logging.INFO,
                             format='%(message)s')
 
+    if class_name_arg_required(args) and not args.class_names:
+        parser.print_usage()
+        logging.error('You must supply at least one class name')
+        sys.exit(1)
+
     # show version?
     if args.version:
         # we use print (not logging) function because version may be used
-        # by some external script while logging may output excessive information
+        # by some external script while logging may output excessive
+        # information
         print(__version__)
         sys.exit(0)
 
@@ -354,7 +488,8 @@ def parse_args(args=None):
 
     # check arguments
     if args.use_keyring and args.password:
-        logging.warning('--keyring and --password cannot be specified together')
+        logging.warning(
+            '--keyring and --password cannot be specified together')
         args.use_keyring = False
 
     if args.use_keyring and not keyring:
@@ -365,7 +500,7 @@ def parse_args(args=None):
         logging.error('Cookies file not found: %s', args.cookies_file)
         sys.exit(1)
 
-    if not args.cookies_file:
+    if not args.cookies_file and not args.cookies_cauth:
         try:
             args.username, args.password = get_credentials(
                 username=args.username, password=args.password,
@@ -375,5 +510,3 @@ def parse_args(args=None):
             sys.exit(1)
 
     return args
-
-

coursera/cookies.py

@@ -25,7 +25,7 @@ from .utils import mkdir_p, random_string
 # Monkey patch cookielib.Cookie.__init__.
 # Reason: The expires value may be a decimal string,
 # but the Cookie class uses int() ...
-__orginal_init__ = cookielib.Cookie.__init__
+__original_init__ = cookielib.Cookie.__init__
 
 
 def __fixed_init__(self, version, name, value,
@@ -41,7 +41,7 @@ def __fixed_init__(self, version, name, value,
                    rfc2109=False):
     if expires is not None:
         expires = float(expires)
-    __orginal_init__(self, version, name, value,
+    __original_init__(self, version, name, value,
                      port, port_specified,
                      domain, domain_specified, domain_initial_dot,
                      path, path_specified,
@@ -53,6 +53,7 @@ def __fixed_init__(self, version, name, value,
                      rest,
                      rfc2109=False)
 
+
 cookielib.Cookie.__init__ = __fixed_init__
 
 
@@ -68,15 +69,15 @@ class AuthenticationFailed(BaseException):
     """
 
 
-def prepape_auth_headers(session, include_cauth=False):
+def prepare_auth_headers(session, include_cauth=False):
     """
-    This function prepapes headers with CSRF/CAUTH tokens that can
+    This function prepares headers with CSRF/CAUTH tokens that can
     be used in POST requests such as login/get_quiz.
 
     @param session: Requests session.
     @type session: requests.Session
 
-    @param include_cauth: Flag that indicates whethe CAUTH cookies should be
+    @param include_cauth: Flag that indicates whether CAUTH cookies should be
         included as well.
     @type include_cauth: bool
 
@@ -132,7 +133,7 @@ def login(session, username, password, class_name=None):
             logging.error(e)
             raise ClassNotFound(class_name)
 
-    headers = prepape_auth_headers(session, include_cauth=False)
+    headers = prepare_auth_headers(session, include_cauth=False)
 
     data = {
         'email': username,
@@ -150,8 +151,8 @@ def login(session, username, password, class_name=None):
         # for coursera!!!
         v = session.cookies.pop('CAUTH')
         session.cookies.set('CAUTH', v)
-    except requests.exceptions.HTTPError:
-        raise AuthenticationFailed('Cannot login on coursera.org.')
+    except requests.exceptions.HTTPError as e:
+        raise AuthenticationFailed('Cannot login on coursera.org: %s' % e)
 
     logging.info('Logged in on coursera.org.')
 
@@ -169,8 +170,9 @@ def down_the_wabbit_hole(session, class_name):
 
     try:
         r.raise_for_status()
-    except requests.exceptions.HTTPError:
-        raise AuthenticationFailed('Cannot login on class.coursera.org.')
+    except requests.exceptions.HTTPError as e:
+        raise AuthenticationFailed(
+            'Cannot login on class.coursera.org: %s' % e)
 
     logging.debug('Exiting "deep" authentication.')
 
@@ -353,7 +355,7 @@ def get_cookies_for_class(session, class_name,
     Get the cookies for the given class.
 
     We do not validate the cookies if they are loaded from a cookies file
-    because this is intented for debugging purposes or if the coursera
+    because this is intended for debugging purposes or if the coursera
     authentication process has changed.
     """
     if cookies_file:
@@ -375,8 +377,9 @@ class TLSAdapter(HTTPAdapter):
     A customized HTTP Adapter which uses TLS v1.2 for encrypted
     connections.
     """
+
     def init_poolmanager(self, connections, maxsize, block=False):
         self.poolmanager = PoolManager(num_pools=connections,
                                        maxsize=maxsize,
                                        block=block,
-                                       ssl_version=ssl.PROTOCOL_TLSv1)
+                                       ssl_version=ssl.PROTOCOL_TLSv1_2)

coursera/coursera_dl.py

@@ -3,7 +3,7 @@
 
 # Authors and copyright:
 #     © 2012-2013, John Lehmann (first last at geemail dotcom or @jplehmann)
-#     © 2012-2015, Rogério Brito (r lastname at ime usp br)
+#     © 2012-2020, Rogério Theodoro de Brito
 #     © 2013, Jonas De Taeye (first dt at fastmail fm)
 #
 # Contributions are welcome, but please add new unit tests to test your changes
@@ -46,6 +46,7 @@ import json
 import logging
 import os
 import re
+import time
 import shutil
 
 from distutils.version import LooseVersion as V
@@ -59,15 +60,17 @@ import requests
 
 from .cookies import (
     AuthenticationFailed, ClassNotFound,
-    get_cookies_for_class, make_cookie_values, TLSAdapter)
+    get_cookies_for_class, make_cookie_values, TLSAdapter, login)
 from .define import (CLASS_URL, ABOUT_URL, PATH_CACHE)
 from .downloaders import get_downloader
 from .workflow import CourseraDownloader
 from .parallel import ConsecutiveDownloader, ParallelDownloader
 from .utils import (clean_filename, get_anchor_format, mkdir_p, fix_url,
                     print_ssl_error_message,
-                    decode_input, BeautifulSoup, is_debug_run)
+                    decode_input, BeautifulSoup, is_debug_run,
+                    spit_json, slurp_json)
 
+from .api import expand_specializations
 from .network import get_page, get_page_and_url
 from .commandline import parse_args
 from .extractors import CourseraExtractor
@@ -102,38 +105,48 @@ def list_courses(args):
     @type args: namedtuple
     """
     session = get_session()
-    extractor = CourseraExtractor(session, args.username, args.password)
+    login(session, args.username, args.password)
+    extractor = CourseraExtractor(session)
     courses = extractor.list_courses()
     logging.info('Found %d courses', len(courses))
     for course in courses:
         logging.info(course)
 
 
-def download_on_demand_class(args, class_name):
+def download_on_demand_class(session, args, class_name):
     """
-    Download all requested resources from the on-demand class given in class_name.
+    Download all requested resources from the on-demand class given
+    in class_name.
 
-    Returns True if the class appears completed.
+    @return: Tuple of (bool, bool), where the first bool indicates whether
+        errors occurred while parsing syllabus, the second bool indicates
+        whether the course appears to be completed.
+    @rtype: (bool, bool)
     """
 
-    session = get_session()
-    extractor = CourseraExtractor(session, args.username, args.password)
+    error_occurred = False
+    extractor = CourseraExtractor(session)
 
     cached_syllabus_filename = '%s-syllabus-parsed.json' % class_name
     if args.cache_syllabus and os.path.isfile(cached_syllabus_filename):
-        with open(cached_syllabus_filename) as syllabus_file:
-            modules = json.load(syllabus_file)
+        modules = slurp_json(cached_syllabus_filename)
     else:
-        modules = extractor.get_modules(class_name,
-                                        args.reverse,
-                                        args.unrestricted_filenames,
-                                        args.subtitle_language,
-                                        args.video_resolution,
-                                        args.download_quizzes)
+        error_occurred, modules = extractor.get_modules(
+            class_name,
+            args.reverse,
+            args.unrestricted_filenames,
+            args.subtitle_language,
+            args.video_resolution,
+            args.download_quizzes,
+            args.mathjax_cdn_url,
+            args.download_notebooks
+        )
 
     if is_debug_run or args.cache_syllabus():
-        with open(cached_syllabus_filename, 'w') as file_object:
-            json.dump(modules, file_object, indent=4)
+        spit_json(modules, cached_syllabus_filename)
+
+    if args.only_syllabus:
+        return error_occurred, False
 
     downloader = get_downloader(session, class_name, args)
     downloader_wrapper = ParallelDownloader(downloader, args.jobs) \
@@ -165,7 +178,7 @@ def download_on_demand_class(args, class_name):
     if course_downloader.failed_urls:
         print_failed_urls(course_downloader.failed_urls)
 
-    return completed
+    return error_occurred, completed
 
 
 def print_skipped_urls(skipped_urls):
@@ -188,14 +201,17 @@ def print_failed_urls(failed_urls):
     logging.info('-' * 80)
 
 
-def download_class(args, class_name):
+def download_class(session, args, class_name):
     """
     Try to download on-demand class.
 
-    Returns True if the class appears completed.
+    @return: Tuple of (bool, bool), where the first bool indicates whether
+        errors occurred while parsing syllabus, the second bool indicates
+        whether the course appears to be completed.
+    @rtype: (bool, bool)
     """
     logging.debug('Downloading new style (on demand) class %s', class_name)
-    return download_on_demand_class(args, class_name)
+    return download_on_demand_class(session, args, class_name)
 
 
 def main():
@@ -206,6 +222,7 @@ def main():
     args = parse_args()
     logging.info('coursera_dl version %s', __version__)
     completed_classes = []
+    classes_with_errors = []
 
     mkdir_p(PATH_CACHE, 0o700)
     if args.clear_cache:
@@ -215,11 +232,24 @@ def main():
         list_courses(args)
         return
 
-    for class_name in args.class_names:
+    session = get_session()
+    if args.cookies_cauth:
+        session.cookies.set('CAUTH', args.cookies_cauth)
+    else:
+        login(session, args.username, args.password)
+    if args.specialization:
+        args.class_names = expand_specializations(session, args.class_names)
+
+    for class_index, class_name in enumerate(args.class_names):
         try:
-            logging.info('Downloading class: %s', class_name)
-            if download_class(args, class_name):
+            logging.info('Downloading class: %s (%d / %d)',
+                         class_name, class_index + 1, len(args.class_names))
+            error_occurred, completed = download_class(
+                session, args, class_name)
+            if completed:
                 completed_classes.append(class_name)
+            if error_occurred:
+                classes_with_errors.append(class_name)
         except requests.exceptions.HTTPError as e:
             logging.error('HTTPError %s', e)
             if is_debug_run():
@@ -229,15 +259,32 @@ def main():
             print_ssl_error_message(e)
             if is_debug_run():
                 raise
-        except ClassNotFound as cnf:
-            logging.error('Could not find class: %s', cnf)
-        except AuthenticationFailed as af:
-            logging.error('Could not authenticate: %s', af)
+        except ClassNotFound as e:
+            logging.error('Could not find class: %s', e)
+        except AuthenticationFailed as e:
+            logging.error('Could not authenticate: %s', e)
+
+        if class_index + 1 != len(args.class_names):
+            logging.info('Sleeping for %d seconds before downloading next course. '
+                         'You can change this with --download-delay option.',
+                         args.download_delay)
+            time.sleep(args.download_delay)
 
     if completed_classes:
+        logging.info('-' * 80)
         logging.info(
             "Classes which appear completed: " + " ".join(completed_classes))
 
+    if classes_with_errors:
+        logging.info('-' * 80)
+        logging.info('The following classes had errors during the syllabus'
+                     ' parsing stage. You may want to review error messages and'
+                     ' courses (sometimes enrolling to the course or switching'
+                     ' session helps):')
+        for class_name in classes_with_errors:
+            logging.info('%s (https://www.coursera.org/learn/%s)',
+                         class_name, class_name)
+
 
 if __name__ == '__main__':
     main()

coursera/credentials.py

@@ -134,7 +134,8 @@ def authenticate_through_netrc(path=None):
 
     error_messages = '\n'.join(str(e) for e in errors)
     raise CredentialsError(
-        'Did not find valid netrc file:\n' + error_messages)
+        'Did not find valid netrc file:\n' + error_messages +
+        '\nPlease run this command: chmod og-rw ~/.netrc')
 
 
 def get_credentials(username=None, password=None, netrc=None, use_keyring=False):

coursera/define.py

@@ -8,13 +8,16 @@ import os
 import getpass
 import tempfile
 
-COURSERA_URL = 'https://www.coursera.org'
+
+HTTP_FORBIDDEN = 403
+
+COURSERA_URL = 'https://api.coursera.org'
 AUTH_URL = 'https://accounts.coursera.org/api/v1/login'
-AUTH_URL_V3 = 'https://www.coursera.org/api/login/v3'
+AUTH_URL_V3 = 'https://api.coursera.org/api/login/v3'
 CLASS_URL = 'https://class.coursera.org/{class_name}'
 
-# The following link is left just for illustative purposes:
-# https://www.coursera.org/api/courses.v1?fields=display%2CpartnerIds%2CphotoUrl%2CstartDate%2Cpartners.v1(homeLink%2Cname)&includes=partnerIds&q=watchlist&start=0
+# The following link is left just for illustrative purposes:
+# https://api.coursera.org/api/courses.v1?fields=display%2CpartnerIds%2CphotoUrl%2CstartDate%2Cpartners.v1(homeLink%2Cname)&includes=partnerIds&q=watchlist&start=0
 # Reply is as follows:
 # {
 #     "elements": [
@@ -31,10 +34,10 @@ CLASS_URL = 'https://class.coursera.org/{class_name}'
 #     },
 #     "linked": {}
 # }
-OPENCOURSE_LIST_COURSES = 'https://www.coursera.org/api/courses.v1?q=watchlist&start={start}'
+OPENCOURSE_LIST_COURSES = 'https://api.coursera.org/api/courses.v1?q=watchlist&start={start}'
 
-# The following link is left just for illustative purposes:
-# https://www.coursera.org/api/memberships.v1?fields=courseId,enrolledTimestamp,grade,id,lastAccessedTimestamp,onDemandSessionMembershipIds,onDemandSessionMemberships,role,v1SessionId,vc,vcMembershipId,courses.v1(courseStatus,display,partnerIds,photoUrl,specializations,startDate,v1Details,v2Details),partners.v1(homeLink,name),v1Details.v1(sessionIds),v1Sessions.v1(active,certificatesReleased,dbEndDate,durationString,hasSigTrack,startDay,startMonth,startYear),v2Details.v1(onDemandSessions,plannedLaunchDate,sessionsEnabledAt),specializations.v1(logo,name,partnerIds,shortName)&includes=courseId,onDemandSessionMemberships,vcMembershipId,courses.v1(partnerIds,specializations,v1Details,v2Details),v1Details.v1(sessionIds),v2Details.v1(onDemandSessions),specializations.v1(partnerIds)&q=me&showHidden=true&filter=current,preEnrolled
+# The following link is left just for illustrative purposes:
+# https://api.coursera.org/api/memberships.v1?fields=courseId,enrolledTimestamp,grade,id,lastAccessedTimestamp,onDemandSessionMembershipIds,onDemandSessionMemberships,role,v1SessionId,vc,vcMembershipId,courses.v1(courseStatus,display,partnerIds,photoUrl,specializations,startDate,v1Details,v2Details),partners.v1(homeLink,name),v1Details.v1(sessionIds),v1Sessions.v1(active,certificatesReleased,dbEndDate,durationString,hasSigTrack,startDay,startMonth,startYear),v2Details.v1(onDemandSessions,plannedLaunchDate,sessionsEnabledAt),specializations.v1(logo,name,partnerIds,shortName)&includes=courseId,onDemandSessionMemberships,vcMembershipId,courses.v1(partnerIds,specializations,v1Details,v2Details),v1Details.v1(sessionIds),v2Details.v1(onDemandSessions),specializations.v1(partnerIds)&q=me&showHidden=true&filter=current,preEnrolled
 # Sample reply:
 # {
 #     "elements": [
@@ -57,13 +60,21 @@ OPENCOURSE_LIST_COURSES = 'https://www.coursera.org/api/courses.v1?q=watchlist&s
 #         ]
 #     }
 # }
-OPENCOURSE_MEMBERSHIPS = 'https://www.coursera.org/api/memberships.v1?includes=courseId,courses.v1&q=me&showHidden=true&filter=current,preEnrolled'
-OPENCOURSE_CONTENT_URL = 'https://www.coursera.org/api/opencourse.v1/course/{class_name}?showLockedItems=true'
-OPENCOURSE_VIDEO_URL = 'https://www.coursera.org/api/opencourse.v1/video/{video_id}'
-OPENCOURSE_SUPPLEMENT_URL = 'https://www.coursera.org/api/onDemandSupplements.v1/'\
+OPENCOURSE_MEMBERSHIPS = 'https://api.coursera.org/api/memberships.v1?includes=courseId,courses.v1&q=me&showHidden=true&filter=current,preEnrolled'
+OPENCOURSE_ONDEMAND_LECTURE_VIDEOS_URL = \
+    'https://api.coursera.org/api/onDemandLectureVideos.v1/'\
+    '{course_id}~{video_id}?includes=video&'\
+    'fields=onDemandVideos.v1(sources%2Csubtitles%2CsubtitlesVtt%2CsubtitlesTxt)'
+OPENCOURSE_SUPPLEMENT_URL = 'https://api.coursera.org/api/onDemandSupplements.v1/'\
     '{course_id}~{element_id}?includes=asset&fields=openCourseAssets.v1%28typeName%29,openCourseAssets.v1%28definition%29'
 OPENCOURSE_PROGRAMMING_ASSIGNMENTS_URL = \
-    'https://www.coursera.org/api/onDemandProgrammingLearnerAssignments.v1/{course_id}~{element_id}?fields=submissionLearnerSchema'
+    'https://api.coursera.org/api/onDemandProgrammingLearnerAssignments.v1/{course_id}~{element_id}?fields=submissionLearnerSchema'
+OPENCOURSE_PROGRAMMING_IMMEDIATE_INSTRUCTIOINS_URL = \
+    'https://api.coursera.org/api/onDemandProgrammingImmediateInstructions.v1/{course_id}~{element_id}'
+OPENCOURSE_REFERENCES_POLL_URL = \
+    "https://api.coursera.org/api/onDemandReferences.v1/?courseId={course_id}&q=courseListed&fields=name%2CshortId%2Cslug%2Ccontent&includes=assets"
+OPENCOURSE_REFERENCE_ITEM_URL = \
+    "https://api.coursera.org/api/onDemandReferences.v1/?courseId={course_id}&q=shortId&shortId={short_id}&fields=name%2CshortId%2Cslug%2Ccontent&includes=assets"
 
 # These are ids that are present in <asset> tag in assignment text:
 #
@@ -86,7 +97,24 @@ OPENCOURSE_PROGRAMMING_ASSIGNMENTS_URL = \
 #   "linked": null
 # }
 OPENCOURSE_ASSET_URL = \
-    'https://www.coursera.org/api/assetUrls.v1?ids={ids}'
+    'https://api.coursera.org/api/assetUrls.v1?ids={ids}'
+
+# Sample response:
+#  "linked": {
+#    "openCourseAssets.v1": [
+#      {
+#        "typeName": "asset",
+#        "definition": {
+#          "assetId": "fytYX5rYEeedWRLokafKRg",
+#          "name": "Lecture slides"
+#        },
+#        "id": "j6g7VZrYEeeUVgpv-dYMig"
+#      }
+#    ]
+#  }
+OPENCOURSE_ONDEMAND_LECTURE_ASSETS_URL = \
+    'https://api.coursera.org/api/onDemandLectureAssets.v1/'\
+    '{course_id}~{video_id}/?includes=openCourseAssets'
 
 # These ids are provided in lecture json:
 #
@@ -134,7 +162,7 @@ OPENCOURSE_ASSET_URL = \
 #   "linked": null
 # }
 OPENCOURSE_ASSETS_URL = \
-    'https://www.coursera.org/api/openCourseAssets.v1/{id}'
+    'https://api.coursera.org/api/openCourseAssets.v1/{id}'
 
 # These asset ids are ids returned from OPENCOURSE_ASSETS_URL request:
 # See example above.
@@ -157,13 +185,39 @@ OPENCOURSE_ASSETS_URL = \
 #   "linked": null
 # }
 OPENCOURSE_API_ASSETS_V1_URL = \
-    'https://www.coursera.org/api/assets.v1?ids={id}'
+    'https://api.coursera.org/api/assets.v1?ids={id}'
 
 OPENCOURSE_ONDEMAND_COURSE_MATERIALS = \
-    'https://www.coursera.org/api/onDemandCourseMaterials.v1/?'\
-        'q=slug&slug={class_name}&includes=moduleIds%2ClessonIds%2CpassableItemGroups%2CpassableItemGroupChoices%2CpassableLessonElements%2CitemIds%2Ctracks'\
-        '&fields=moduleIds%2ConDemandCourseMaterialModules.v1(name%2Cslug%2Cdescription%2CtimeCommitment%2ClessonIds%2Coptional)%2ConDemandCourseMaterialLessons.v1(name%2Cslug%2CtimeCommitment%2CelementIds%2Coptional%2CtrackId)%2ConDemandCourseMaterialPassableItemGroups.v1(requiredPassedCount%2CpassableItemGroupChoiceIds%2CtrackId)%2ConDemandCourseMaterialPassableItemGroupChoices.v1(name%2Cdescription%2CitemIds)%2ConDemandCourseMaterialPassableLessonElements.v1(gradingWeight)%2ConDemandCourseMaterialItems.v1(name%2Cslug%2CtimeCommitment%2Ccontent%2CisLocked%2ClockableByItem%2CitemLockedReasonCode%2CtrackId)%2ConDemandCourseMaterialTracks.v1(passablesCount)'\
-        '&showLockedItems=true'
+    'https://api.coursera.org/api/onDemandCourseMaterials.v1/?'\
+    'q=slug&slug={class_name}&includes=moduleIds%2ClessonIds%2CpassableItemGroups%2CpassableItemGroupChoices%2CpassableLessonElements%2CitemIds%2Ctracks'\
+    '&fields=moduleIds%2ConDemandCourseMaterialModules.v1(name%2Cslug%2Cdescription%2CtimeCommitment%2ClessonIds%2Coptional)%2ConDemandCourseMaterialLessons.v1(name%2Cslug%2CtimeCommitment%2CelementIds%2Coptional%2CtrackId)%2ConDemandCourseMaterialPassableItemGroups.v1(requiredPassedCount%2CpassableItemGroupChoiceIds%2CtrackId)%2ConDemandCourseMaterialPassableItemGroupChoices.v1(name%2Cdescription%2CitemIds)%2ConDemandCourseMaterialPassableLessonElements.v1(gradingWeight)%2ConDemandCourseMaterialItems.v1(name%2Cslug%2CtimeCommitment%2Ccontent%2CisLocked%2ClockableByItem%2CitemLockedReasonCode%2CtrackId)%2ConDemandCourseMaterialTracks.v1(passablesCount)'\
+    '&showLockedItems=true'
+
+OPENCOURSE_ONDEMAND_COURSE_MATERIALS_V2 = \
+    'https://api.coursera.org/api/onDemandCourseMaterials.v2/?q=slug&slug={class_name}'\
+    '&includes=modules%2Clessons%2CpassableItemGroups%2CpassableItemGroupChoices%2CpassableLessonElements%2Citems%2Ctracks%2CgradePolicy&'\
+    '&fields=moduleIds%2ConDemandCourseMaterialModules.v1(name%2Cslug%2Cdescription%2CtimeCommitment%2ClessonIds%2Coptional%2ClearningObjectives)%2ConDemandCourseMaterialLessons.v1(name%2Cslug%2CtimeCommitment%2CelementIds%2Coptional%2CtrackId)%2ConDemandCourseMaterialPassableItemGroups.v1(requiredPassedCount%2CpassableItemGroupChoiceIds%2CtrackId)%2ConDemandCourseMaterialPassableItemGroupChoices.v1(name%2Cdescription%2CitemIds)%2ConDemandCourseMaterialPassableLessonElements.v1(gradingWeight%2CisRequiredForPassing)%2ConDemandCourseMaterialItems.v2(name%2Cslug%2CtimeCommitment%2CcontentSummary%2CisLocked%2ClockableByItem%2CitemLockedReasonCode%2CtrackId%2ClockedStatus%2CitemLockSummary)%2ConDemandCourseMaterialTracks.v1(passablesCount)'\
+    '&showLockedItems=true'
+
+OPENCOURSE_ONDEMAND_SPECIALIZATIONS_V1 = \
+    'https://api.coursera.org/api/onDemandSpecializations.v1?q=slug'\
+    '&slug={class_name}&fields=courseIds,interchangeableCourseIds,launchedAt,'\
+    'logo,memberships,metadata,partnerIds,premiumExperienceVariant,'\
+    'onDemandSpecializationMemberships.v1(suggestedSessionSchedule),'\
+    'onDemandSpecializationSuggestedSchedule.v1(suggestedSessions),'\
+    'partners.v1(homeLink,name),courses.v1(courseProgress,description,'\
+    'membershipIds,startDate,v2Details,vcMembershipIds),v2Details.v1('\
+    'onDemandSessions,plannedLaunchDate),memberships.v1(grade,'\
+    'vcMembershipId),vcMemberships.v1(certificateCodeWithGrade)'\
+    '&includes=courseIds,memberships,partnerIds,'\
+    'onDemandSpecializationMemberships.v1(suggestedSessionSchedule),'\
+    'courses.v1(courseProgress,membershipIds,v2Details,vcMembershipIds),'\
+    'v2Details.v1(onDemandSessions)'
+
+OPENCOURSE_ONDEMAND_COURSES_V1 = \
+    'https://api.coursera.org/api/onDemandCourses.v1?q=slug&slug={class_name}&'\
+    'includes=instructorIds%2CpartnerIds%2C_links&'\
+    'fields=brandingImage%2CcertificatePurchaseEnabledAt%2Cpartners.v1(squareLogo%2CrectangularLogo)%2Cinstructors.v1(fullName)%2CoverridePartnerLogos%2CsessionsEnabledAt%2CdomainTypes%2CpremiumExperienceVariant%2CisRestrictedMembership'
 
 ABOUT_URL = ('https://api.coursera.org/api/catalog.v1/courses?'
              'fields=largeIcon,photo,previewLink,shortDescription,smallIcon,'
@@ -176,7 +230,111 @@ ABOUT_URL = ('https://api.coursera.org/api/catalog.v1/courses?'
 AUTH_REDIRECT_URL = ('https://class.coursera.org/{class_name}'
                      '/auth/auth_redirector?type=login&subtype=normal')
 
-#POST_OPENCOURSE_API_QUIZ_SESSION = 'https://www.coursera.org/api/opencourse.v1/user/4958/course/text-mining/item/7OQHc/quiz/session'
+# Sample URL:
+#
+# https://api.coursera.org/api/onDemandPeerAssignmentInstructions.v1/?q=latest&userId=4958&courseId=RcnRZHHtEeWxvQr3acyajw&itemId=2yTvX&includes=gradingMetadata%2CreviewSchemas%2CsubmissionSchemas&fields=instructions%2ConDemandPeerAssignmentGradingMetadata.v1(requiredAuthoredReviewCount%2CisMentorGraded%2CassignmentDetails)%2ConDemandPeerReviewSchemas.v1(reviewSchema)%2ConDemandPeerSubmissionSchemas.v1(submissionSchema)
+#
+# Sample response:
+#
+# {
+#   "elements": [
+#     {
+#       "instructions": {
+#         "introduction": {
+#           "typeName": "cml",
+#           "definition": {
+#             "dtdId": "assess/1",
+#             "value": "<co-content><text>Ваше первое задание заключается в установке Python и библиотек..</text></li></list></co-content>"
+#           }
+#         },
+#         "sections": [
+#           {
+#             "typeId": "unknown",
+#             "title": "Review criteria",
+#             "content": {
+#               "typeName": "cml",
+#               "definition": {
+#                 "dtdId": "assess/1",
+#                 "value": "<co-content><text>В результате работы вы установите на компьютер Python и библиотеки, необходимые для дальнейшего прохождения курса..</text></co-content>"
+#               }
+#             }
+#           }
+#         ]
+#       },
+#       "id": "4958~RcnRZHHtEeWxvQr3acyajw~2yTvX~8x7Qhs66EeW2Tw715xhIPQ@13"
+#     }
+#   ],
+#   "paging": {},
+#   "linked": {
+#     "onDemandPeerSubmissionSchemas.v1": [
+#       {
+#         "submissionSchema": {
+#           "parts": [
+#             {
+#               "details": {
+#                 "typeName": "fileUpload",
+#                 "definition": {
+#                   "required": false
+#                 }
+#               },
+#               "id": "_fcfP3bPT5W4pkfkshmUAQ",
+#               "prompt": {
+#                 "typeName": "cml",
+#                 "definition": {
+#                   "dtdId": "assess/1",
+#                   "value": "<co-content><text>Загрузите скриншот №1.</text></co-content>"
+#                 }
+#               }
+#             },
+#             {
+#               "details": {
+#                 "typeName": "fileUpload",
+#                 "definition": {
+#                   "required": false
+#                 }
+#               },
+#               "id": "92ea4b4e-3492-41eb-ee32-2624ee807bd3",
+#               "prompt": {
+#                 "typeName": "cml",
+#                 "definition": {
+#                   "dtdId": "assess/1",
+#                   "value": "<co-content><text>Загрузите скриншот №2.</text></co-content>"
+#                 }
+#               }
+#             }
+#           ]
+#         },
+#         "id": "4958~RcnRZHHtEeWxvQr3acyajw~2yTvX~8x7Qhs66EeW2Tw715xhIPQ@13"
+#       }
+#     ],
+#     "onDemandPeerAssignmentGradingMetadata.v1": [
+#       {
+#         "assignmentDetails": {
+#           "typeName": "phased",
+#           "definition": {
+#             "receivedReviewCutoffs": {
+#               "count": 3
+#             },
+#             "passingFraction": 0.8
+#           }
+#         },
+#         "requiredAuthoredReviewCount": 3,
+#         "isMentorGraded": false,
+#         "id": "4958~RcnRZHHtEeWxvQr3acyajw~2yTvX~8x7Qhs66EeW2Tw715xhIPQ@13"
+#       }
+#     ],
+#     "onDemandPeerReviewSchemas.v1": []
+#   }
+# }
+#
+# This URL is used to retrieve "phasedPeer" typename instructions' contents
+OPENCOURSE_PEER_ASSIGNMENT_INSTRUCTIONS = (
+    'https://api.coursera.org/api/onDemandPeerAssignmentInstructions.v1/?'
+    'q=latest&userId={user_id}&courseId={course_id}&itemId={element_id}&'
+    'includes=gradingMetadata%2CreviewSchemas%2CsubmissionSchemas&'
+    'fields=instructions%2ConDemandPeerAssignmentGradingMetadata.v1(requiredAuthoredReviewCount%2CisMentorGraded%2CassignmentDetails)%2ConDemandPeerReviewSchemas.v1(reviewSchema)%2ConDemandPeerSubmissionSchemas.v1(submissionSchema)')
+
+#POST_OPENCOURSE_API_QUIZ_SESSION = 'https://api.coursera.org/api/opencourse.v1/user/4958/course/text-mining/item/7OQHc/quiz/session'
 # Sample response:
 #
 # {
@@ -192,9 +350,9 @@ AUTH_REDIRECT_URL = ('https://class.coursera.org/{class_name}'
 #     "progressState": "Started"
 #   }
 # }
-POST_OPENCOURSE_API_QUIZ_SESSION = 'https://www.coursera.org/api/opencourse.v1/user/{user_id}/course/{class_name}/item/{quiz_id}/quiz/session'
+POST_OPENCOURSE_API_QUIZ_SESSION = 'https://api.coursera.org/api/opencourse.v1/user/{user_id}/course/{class_name}/item/{quiz_id}/quiz/session'
 
-#POST_OPENCOURSE_API_QUIZ_SESSION_GET_STATE = 'https://www.coursera.org/api/opencourse.v1/user/4958/course/text-mining/item/7OQHc/quiz/session/opencourse~bVgqTevEEeWvGQrWsIkLlw:4958:BiNDdOvPEeWAkwqbKEEh3w@13:1468773901987@1/action/getState?autoEnroll=false'
+#POST_OPENCOURSE_API_QUIZ_SESSION_GET_STATE = 'https://api.coursera.org/api/opencourse.v1/user/4958/course/text-mining/item/7OQHc/quiz/session/opencourse~bVgqTevEEeWvGQrWsIkLlw:4958:BiNDdOvPEeWAkwqbKEEh3w@13:1468773901987@1/action/getState?autoEnroll=false'
 # Sample response:
 #
 # {
@@ -276,9 +434,9 @@ POST_OPENCOURSE_API_QUIZ_SESSION = 'https://www.coursera.org/api/opencourse.v1/u
 #   }
 # }
 #
-POST_OPENCOURSE_API_QUIZ_SESSION_GET_STATE = 'https://www.coursera.org/api/opencourse.v1/user/{user_id}/course/{class_name}/item/{quiz_id}/quiz/session/{session_id}/action/getState?autoEnroll=false'
+POST_OPENCOURSE_API_QUIZ_SESSION_GET_STATE = 'https://api.coursera.org/api/opencourse.v1/user/{user_id}/course/{class_name}/item/{quiz_id}/quiz/session/{session_id}/action/getState?autoEnroll=false'
 
-#POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS = 'https://www.coursera.org/api/onDemandExamSessions.v1/-N44X0IJEeWpogr5ZO8qxQ~YV0W4~10!~1467462079068/actions?includes=gradingAttempts'
+#POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS = 'https://api.coursera.org/api/onDemandExamSessions.v1/-N44X0IJEeWpogr5ZO8qxQ~YV0W4~10!~1467462079068/actions?includes=gradingAttempts'
 # Sample response:
 #
 # {
@@ -419,14 +577,14 @@ POST_OPENCOURSE_API_QUIZ_SESSION_GET_STATE = 'https://www.coursera.org/api/openc
 # Request payload:
 # {"courseId":"-N44X0IJEeWpogr5ZO8qxQ","itemId":"YV0W4"}
 #
-#POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS = 'https://www.coursera.org/api/onDemandExamSessions.v1/-N44X0IJEeWpogr5ZO8qxQ~YV0W4~10!~1467462079068/actions?includes=gradingAttempts'
+#POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS = 'https://api.coursera.org/api/onDemandExamSessions.v1/-N44X0IJEeWpogr5ZO8qxQ~YV0W4~10!~1467462079068/actions?includes=gradingAttempts'
 
 # Response for this request is empty. Result (session_id) should be taken
 # either from Location header or from X-Coursera-Id header.
 #
 # Request payload:
 # {"courseId":"-N44X0IJEeWpogr5ZO8qxQ","itemId":"YV0W4"}
-POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS = 'https://www.coursera.org/api/onDemandExamSessions.v1'
+POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS = 'https://api.coursera.org/api/onDemandExamSessions.v1'
 
 # Sample response:
 # {
@@ -738,7 +896,7 @@ POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS = 'https://www.coursera.org/api/onDemandE
 #
 # Request payload:
 # {"name":"getState","argument":[]}
-POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS_GET_STATE = 'https://www.coursera.org/api/onDemandExamSessions.v1/{session_id}/actions?includes=gradingAttempts'
+POST_OPENCOURSE_ONDEMAND_EXAM_SESSIONS_GET_STATE = 'https://api.coursera.org/api/onDemandExamSessions.v1/{session_id}/actions?includes=gradingAttempts'
 
 ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
 # define a per-user cache folder
@@ -769,7 +927,7 @@ FORMAT_MAX_LENGTH = 20
 TITLE_MAX_LENGTH = 200
 
 #: CSS that is usen to prettify instructions
-INSTRUCTIONS_HTML_INJECTION = '''
+INSTRUCTIONS_HTML_INJECTION_PRE = '''
 <style>
 body {
     padding: 50px 85px 50px 85px;
@@ -809,7 +967,9 @@ pre {
 </style>
 
 <script type="text/javascript" async
-  src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
+  src="'''
+INSTRUCTIONS_HTML_MATHJAX_URL = 'https://cdn.mathjax.org/mathjax/latest/MathJax.js'
+INSTRUCTIONS_HTML_INJECTION_AFTER = '''?config=TeX-AMS-MML_HTMLorMML">
 </script>
 
 <script type="text/x-mathjax-config">
@@ -822,3 +982,9 @@ pre {
   });
 </script>
 '''
+
+# The following url is the root url (tree) for a Coursera Course
+OPENCOURSE_NOTEBOOK_DESCRIPTIONS = "https://hub.coursera-notebooks.org/hub/coursera_login?token={authId}&next=/"
+OPENCOURSE_NOTEBOOK_LAUNCHES = "https://api.coursera.org/api/onDemandNotebookWorkspaceLaunches.v1/?fields=authorizationId%2CcontentPath%2CuseLegacySystem"
+OPENCOURSE_NOTEBOOK_TREE = "https://hub.coursera-notebooks.org/user/{jupId}/api/contents/{path}?type=directory&_={timestamp}"
+OPENCOURSE_NOTEBOOK_DOWNLOAD = "https://hub.coursera-notebooks.org/user/{jupId}/files/{path}?download=1"

coursera/extractors.py

@@ -9,11 +9,11 @@ import abc
 import json
 import logging
 
-from .api import CourseraOnDemand, OnDemandCourseMaterialItems
-from .define import OPENCOURSE_CONTENT_URL
-from .cookies import login
+from .api import (CourseraOnDemand, OnDemandCourseMaterialItemsV1,
+                  ModulesV1, LessonsV1, ItemsV2)
+from .define import OPENCOURSE_ONDEMAND_COURSE_MATERIALS_V2
 from .network import get_page
-from .utils import is_debug_run
+from .utils import is_debug_run, spit_json
 
 
 class PlatformExtractor(object):
@@ -27,9 +27,8 @@ class PlatformExtractor(object):
 
 
 class CourseraExtractor(PlatformExtractor):
-    def __init__(self, session, username, password):
-        login(session, username, password)
-
+    def __init__(self, session):
+        self._notebook_downloaded = False
         self._session = session
 
     def list_courses(self):
@@ -47,115 +46,194 @@ class CourseraExtractor(PlatformExtractor):
     def get_modules(self, class_name,
                     reverse=False, unrestricted_filenames=False,
                     subtitle_language='en', video_resolution=None,
-                    download_quizzes=False):
+                    download_quizzes=False, mathjax_cdn_url=None,
+                    download_notebooks=False):
 
         page = self._get_on_demand_syllabus(class_name)
-        modules = self._parse_on_demand_syllabus(
+        error_occurred, modules = self._parse_on_demand_syllabus(
+            class_name,
             page, reverse, unrestricted_filenames,
             subtitle_language, video_resolution,
-            download_quizzes)
-        return modules
+            download_quizzes, mathjax_cdn_url, download_notebooks)
+
+        return error_occurred, modules
 
     def _get_on_demand_syllabus(self, class_name):
         """
         Get the on-demand course listing webpage.
         """
 
-        url = OPENCOURSE_CONTENT_URL.format(class_name=class_name)
+        url = OPENCOURSE_ONDEMAND_COURSE_MATERIALS_V2.format(
+            class_name=class_name)
         page = get_page(self._session, url)
-        logging.info('Downloaded %s (%d bytes)', url, len(page))
+        logging.debug('Downloaded %s (%d bytes)', url, len(page))
 
         return page
 
-    def _parse_on_demand_syllabus(self, page, reverse=False,
+    def _parse_on_demand_syllabus(self, course_name, page, reverse=False,
                                   unrestricted_filenames=False,
                                   subtitle_language='en',
                                   video_resolution=None,
-                                  download_quizzes=False):
+                                  download_quizzes=False,
+                                  mathjax_cdn_url=None,
+                                  download_notebooks=False
+                                  ):
         """
         Parse a Coursera on-demand course listing/syllabus page.
+
+        @return: Tuple of (bool, list), where bool indicates whether
+            there was at least on error while parsing syllabus, the list
+            is a list of parsed modules.
+        @rtype: (bool, list)
         """
 
         dom = json.loads(page)
-        course_name = dom['slug']
+        class_id = dom['elements'][0]['id']
 
-        logging.info('Parsing syllabus of on-demand course. '
-                     'This may take some time, please be patient ...')
+        logging.info('Parsing syllabus of on-demand course (id=%s). '
+                     'This may take some time, please be patient ...',
+                     class_id)
         modules = []
-        json_modules = dom['courseMaterial']['elements']
-        course = CourseraOnDemand(session=self._session, course_id=dom['id'],
-                                  course_name=course_name,
-                                  unrestricted_filenames=unrestricted_filenames)
+
+        json_modules = dom['linked']['onDemandCourseMaterialItems.v2']
+        course = CourseraOnDemand(
+            session=self._session, course_id=class_id,
+            course_name=course_name,
+            unrestricted_filenames=unrestricted_filenames,
+            mathjax_cdn_url=mathjax_cdn_url)
         course.obtain_user_id()
-        ondemand_material_items = OnDemandCourseMaterialItems.create(
+        ondemand_material_items = OnDemandCourseMaterialItemsV1.create(
             session=self._session, course_name=course_name)
 
         if is_debug_run():
-            with open('%s-syllabus-raw.json' % course_name, 'w') as file_object:
-                json.dump(dom, file_object, indent=4)
-            with open('%s-course-material-items.json' % course_name, 'w') as file_object:
-                json.dump(ondemand_material_items._items, file_object, indent=4)
+            spit_json(dom, '%s-syllabus-raw.json' % course_name)
+            spit_json(json_modules, '%s-material-items-v2.json' % course_name)
+            spit_json(ondemand_material_items._items,
+                      '%s-course-material-items.json' % course_name)
 
-        for module in json_modules:
-            module_slug = module['slug']
-            logging.info('Processing module  %s', module_slug)
-            sections = []
-            json_sections = module['elements']
-            for section in json_sections:
-                section_slug = section['slug']
-                logging.info('Processing section     %s', section_slug)
+        error_occurred = False
+
+        all_modules = ModulesV1.from_json(
+            dom['linked']['onDemandCourseMaterialModules.v1'])
+        all_lessons = LessonsV1.from_json(
+            dom['linked']['onDemandCourseMaterialLessons.v1'])
+        all_items = ItemsV2.from_json(
+            dom['linked']['onDemandCourseMaterialItems.v2'])
+
+        for module in all_modules:
+            logging.info('Processing module  %s', module.slug)
+            lessons = []
+            for section in module.children(all_lessons):
+                logging.info('Processing section     %s', section.slug)
                 lectures = []
-                json_lectures = section['elements']
+                available_lectures = section.children(all_items)
 
                 # Certain modules may be empty-looking programming assignments
-                # e.g. in data-structures, algorithms-on-graphs ondemand courses
-                if not json_lectures:
-                    lesson_id = section['id']
-                    lecture = ondemand_material_items.get(lesson_id)
+                # e.g. in data-structures, algorithms-on-graphs ondemand
+                # courses
+                if not available_lectures:
+                    lecture = ondemand_material_items.get(section.id)
                     if lecture is not None:
-                        json_lectures = [lecture]
+                        available_lectures = [lecture]
 
-                for lecture in json_lectures:
-                    lecture_slug = lecture['slug']
-                    typename = lecture['content']['typeName']
+                for lecture in available_lectures:
+                    typename = lecture.type_name
 
                     logging.info('Processing lecture         %s (%s)',
-                                 lecture_slug, typename)
-                    links = None
+                                 lecture.slug, typename)
+                    # Empty dictionary means there were no data
+                    # None means an error occurred
+                    links = {}
 
                     if typename == 'lecture':
-                        lecture_video_id = lecture['content']['definition']['videoId']
-                        assets = lecture['content']['definition'].get('assets', [])
+                        # lecture_video_id = lecture['content']['definition']['videoId']
+                        # assets = lecture['content']['definition'].get(
+                        #     'assets', [])
+                        lecture_video_id = lecture.id
+                        # assets = []
 
                         links = course.extract_links_from_lecture(
+                            class_id,
                             lecture_video_id, subtitle_language,
-                            video_resolution, assets)
+                            video_resolution)
 
                     elif typename == 'supplement':
                         links = course.extract_links_from_supplement(
-                            lecture['id'])
+                            lecture.id)
+
+                    elif typename == 'phasedPeer':
+                        links = course.extract_links_from_peer_assignment(
+                            lecture.id)
 
                     elif typename in ('gradedProgramming', 'ungradedProgramming'):
-                        links = course.extract_links_from_programming(lecture['id'])
+                        links = course.extract_links_from_programming(
+                            lecture.id)
 
                     elif typename == 'quiz':
                         if download_quizzes:
-                            links = course.extract_links_from_quiz(lecture['id'])
+                            links = course.extract_links_from_quiz(
+                                lecture.id)
 
                     elif typename == 'exam':
                         if download_quizzes:
-                            links = course.extract_links_from_exam(lecture['id'])
+                            links = course.extract_links_from_exam(
+                                lecture.id)
 
-                    if links:
-                        lectures.append((lecture_slug, links))
+                    elif typename == 'programming':
+                        if download_quizzes:
+                            links = course.extract_links_from_programming_immediate_instructions(
+                                lecture.id)
+
+                    elif typename == 'notebook':
+                        if download_notebooks and not self._notebook_downloaded:
+                            logging.warning(
+                                'According to notebooks platform, content will be downloaded first')
+                            links = course.extract_links_from_notebook(
+                                lecture.id)
+                            self._notebook_downloaded = True
+
+                    else:
+                        logging.info(
+                            'Unsupported typename "%s" in lecture "%s" (lecture id "%s")',
+                            typename, lecture.slug, lecture.id)
+                        continue
+
+                    if links is None:
+                        error_occurred = True
+                    elif links:
+                        lectures.append((lecture.slug, links))
 
                 if lectures:
-                    sections.append((section_slug, lectures))
+                    lessons.append((section.slug, lectures))
 
-            if sections:
-                modules.append((module_slug, sections))
+            if lessons:
+                modules.append((module.slug, lessons))
 
         if modules and reverse:
             modules.reverse()
 
-        return modules
+        # Processing resources section
+        json_references = course.extract_references_poll()
+        references = []
+        if json_references:
+            logging.info('Processing resources')
+            for json_reference in json_references:
+                reference = []
+                reference_slug = json_reference['slug']
+                logging.info('Processing resource  %s',
+                             reference_slug)
+
+                links = course.extract_links_from_reference(
+                    json_reference['shortId'])
+                if links is None:
+                    error_occurred = True
+                elif links:
+                    reference.append(('', links))
+
+                if reference:
+                    references.append((reference_slug, reference))
+
+        if references:
+            modules.append(("Resources", references))
+
+        return error_occurred, modules

coursera/filtering.py

@@ -94,15 +94,16 @@ def find_resources_to_get(lecture, file_formats, resource_filter, ignored_format
         logging.info("The following file formats will be ignored: " + ",".join(ignored_formats))
 
     for fmt, resources in iteritems(lecture):
-
         fmt0 = fmt
-        if '.' in fmt:
-            fmt = fmt.split('.')[1]
 
-        if fmt in ignored_formats:
+        short_fmt = None
+        if '.' in fmt:
+            short_fmt = fmt.split('.')[1]
+
+        if fmt in ignored_formats or (short_fmt != None and short_fmt in ignored_formats) :
             continue
 
-        if fmt in file_formats or 'all' in file_formats:
+        if fmt in file_formats or (short_fmt != None and short_fmt in file_formats) or 'all' in file_formats:
             for r in resources:
                 if resource_filter and r[1] and not re.search(resource_filter, r[1]):
                     logging.debug('Skipping b/c of rf: %s %s',

coursera/network.py

@@ -9,7 +9,7 @@ import logging
 import requests
 
 
-def get_reply(session, url, post=False, data=None, headers=None):
+def get_reply(session, url, post=False, data=None, headers=None, quiet=False):
     """
     Download an HTML page using the requests session. Low-level function
     that allows for flexible request configuration.
@@ -29,6 +29,10 @@ def get_reply(session, url, post=False, data=None, headers=None):
     @param headers: Additional headers to send with request.
     @type headers: dict
 
+    @param quiet: Flag that tells whether to print error message when status
+        code != 200.
+    @type quiet: bool
+
     @return: Requests response.
     @rtype: requests.Response
     """
@@ -46,8 +50,9 @@ def get_reply(session, url, post=False, data=None, headers=None):
     try:
         reply.raise_for_status()
     except requests.exceptions.HTTPError as e:
-        logging.error("Error %s getting page %s", e, url)
-        logging.error("The server replied: %s", reply.text)
+        if not quiet:
+            logging.error("Error %s getting page %s", e, url)
+            logging.error("The server replied: %s", reply.text)
         raise
 
     return reply
@@ -59,6 +64,7 @@ def get_page(session,
              post=False,
              data=None,
              headers=None,
+             quiet=False,
              **kwargs):
     """
     Download an HTML page using the requests session.
@@ -82,7 +88,8 @@ def get_page(session,
     @rtype: str
     """
     url = url.format(**kwargs)
-    reply = get_reply(session, url, post=post, data=data, headers=headers)
+    reply = get_reply(session, url, post=post, data=data, headers=headers,
+                      quiet=quiet)
     return reply.json() if json else reply.text
 
 

coursera/playlist.py

@@ -1,4 +1,5 @@
 import os
+import glob
 
 
 def create_m3u_playlist(section_dir):

coursera/test/fixtures/json/peer-assignment-instructions-all.json

@@ -0,0 +1,29 @@
+{
+  "elements": [
+    {
+      "instructions": {
+        "introduction": {
+          "typeName": "cml",
+          "definition": {
+            "dtdId": "assess/1",
+            "value": "<co-content><text>intro</text></li></list></co-content>"
+          }
+        },
+        "sections": [
+          {
+            "typeId": "unknown",
+            "title": "Review criteria",
+            "content": {
+              "typeName": "cml",
+              "definition": {
+                "dtdId": "assess/1",
+                "value": "<co-content><text>section</text></co-content>"
+              }
+            }
+          }
+        ]
+      },
+      "id": "4958~RcnRZHHtEeWxvQr3acyajw~2yTvX~8x7Qhs66EeW2Tw715xhIPQ@13"
+    }
+  ]
+}

coursera/test/fixtures/json/peer-assignment-instructions-no-title.json

@@ -0,0 +1,28 @@
+{
+  "elements": [
+    {
+      "instructions": {
+        "introduction": {
+          "typeName": "cml",
+          "definition": {
+            "dtdId": "assess/1",
+            "value": "<co-content><text>intro</text></li></list></co-content>"
+          }
+        },
+        "sections": [
+          {
+            "typeId": "unknown",
+            "content": {
+              "typeName": "cml",
+              "definition": {
+                "dtdId": "assess/1",
+                "value": "<co-content><text>section</text></co-content>"
+              }
+            }
+          }
+        ]
+      },
+      "id": "4958~RcnRZHHtEeWxvQr3acyajw~2yTvX~8x7Qhs66EeW2Tw715xhIPQ@13"
+    }
+  ]
+}

coursera/test/fixtures/json/peer-assignment-instructions-only-introduction.json

@@ -0,0 +1,16 @@
+{
+  "elements": [
+    {
+      "instructions": {
+        "introduction": {
+          "typeName": "cml",
+          "definition": {
+            "dtdId": "assess/1",
+            "value": "<co-content><text>intro</text></li></list></co-content>"
+          }
+        }
+      },
+      "id": "4958~RcnRZHHtEeWxvQr3acyajw~2yTvX~8x7Qhs66EeW2Tw715xhIPQ@13"
+    }
+  ]
+}

coursera/test/fixtures/json/peer-assignment-instructions-only-sections.json

@@ -0,0 +1,22 @@
+{
+  "elements": [
+    {
+      "instructions": {
+        "sections": [
+          {
+            "typeId": "unknown",
+            "title": "Review criteria",
+            "content": {
+              "typeName": "cml",
+              "definition": {
+                "dtdId": "assess/1",
+                "value": "<co-content><text>section</text></co-content>"
+              }
+            }
+          }
+        ]
+      },
+      "id": "4958~RcnRZHHtEeWxvQr3acyajw~2yTvX~8x7Qhs66EeW2Tw715xhIPQ@13"
+    }
+  ]
+}

coursera/test/fixtures/json/peer-assignment-no-instructions.json

@@ -0,0 +1,4 @@
+{
+  "elements": [
+  ]
+}

coursera/test/fixtures/json/quiz-to-markup/question-type-mcqReflect-input.json

@@ -0,0 +1,44 @@
+{
+  "questions": [
+    {
+      "id": "8uUpMzm_EeaetxLgjw7H8Q@0",
+      "variant": {
+        "detailLevel": "Full",
+        "definition": {
+          "prompt": {
+            "typeName": "cml",
+            "definition": {
+              "dtdId": "assess/1",
+              "value": "<co-content><text>Lorem ipsum</text></co-content>"
+            }
+          },
+          "options": [
+            {
+              "id": "0.9109180361318947",
+              "display": {
+                "typeName": "cml",
+                "definition": {
+                  "dtdId": "assess/1",
+                  "value": "<co-content><text>Answer 1</text></co-content>"
+                }
+              }
+            },
+            {
+              "id": "0.11974743029080992",
+              "display": {
+                "typeName": "cml",
+                "definition": {
+                  "dtdId": "assess/1",
+                  "value": "<co-content><text>Answer 2</text></co-content>"
+                }
+              }
+            }
+          ]
+        }
+      },
+      "question": {
+        "type": "mcqReflect"
+      }
+    }
+  ]
+}

coursera/test/fixtures/json/quiz-to-markup/question-type-mcqReflect-output.txt

@@ -0,0 +1,15 @@
+<h3>Question 1</h3>
+<co-content><text>Lorem ipsum</text></co-content>
+<form>
+<label><input type="radio" name="0"><co-content>
+ <span>
+  Answer 1
+ </span>
+</co-content><br></label>
+<label><input type="radio" name="0"><co-content>
+ <span>
+  Answer 2
+ </span>
+</co-content><br></label>
+</form>
+<hr>

coursera/test/fixtures/json/quiz-to-markup/question-type-reflect-input.json

@@ -0,0 +1,27 @@
+{
+  "evaluation": null,
+  "questions": [
+    {
+      "id": "jeVDBjnNEeaetxLgjw7H8Q@0",
+      "variant": {
+        "detailLevel": "Full",
+        "definition": {
+          "prompt": {
+            "typeName": "cml",
+            "definition": {
+              "dtdId": "assess/1",
+              "value": "<co-content><text>Lorem ipsum</text></co-content>"
+            }
+          }
+        }
+      },
+      "weightedScoring": {
+        "maxScore": 1
+      },
+      "isSubmitAllowed": true,
+      "question": {
+        "type": "reflect"
+      }
+    }
+  ]
+}

coursera/test/fixtures/json/quiz-to-markup/question-type-reflect-output.txt

@@ -0,0 +1,4 @@
+<h3>Question 1</h3>
+<co-content><text>Lorem ipsum</text></co-content>
+<form><label>Enter answer here:<input type="text" name=""><br></label></form>
+<hr>

coursera/test/fixtures/json/references-poll-output.json

@@ -0,0 +1,24 @@
+[
+    {
+        "id": "Tk_5NiCREeeSVwJCrBEADA",
+        "slug": "tutorials",
+        "content": {
+            "org.coursera.ondemand.reference.AssetReferenceContent": {
+                "assetId": "4e66aa537abf1bdec8ecc508324891ac"
+            }
+        },
+        "shortId": "zVvo7",
+        "name": "Tutorials"
+    },
+    {
+        "id": "Tk_5MSCREeeSVwJCrBEADA",
+        "slug": "test-cases",
+        "content": {
+            "org.coursera.ondemand.reference.AssetReferenceContent": {
+                "assetId": "7c84e5c5249eb551d95444c172592274"
+            }
+        },
+        "shortId": "a4I28",
+        "name": "Test Cases"
+    }
+]

coursera/test/fixtures/json/references-poll-reply.json

@@ -0,0 +1,47 @@
+{
+    "paging": {},
+    "linked": {
+        "openCourseAssets.v1": [
+            {
+                "typeName": "cml",
+                "definition": {
+                    "dtdId": "supplement/1",
+                    "value": "<co-content><text>supplement1</text></co-content>"
+                },
+                "id": "4e66aa537abf1bdec8ecc508324891ac"
+            },
+            {
+                "typeName": "cml",
+                "definition": {
+                    "dtdId": "supplement/1",
+                    "value": "<co-content><text>supplement2</text></co-content>"
+                },
+                "id": "7c84e5c5249eb551d95444c172592274"
+            }
+        ]
+    },
+    "elements": [
+        {
+            "name": "Tutorials",
+            "id": "Tk_5NiCREeeSVwJCrBEADA",
+            "slug": "tutorials",
+            "content": {
+                "org.coursera.ondemand.reference.AssetReferenceContent": {
+                    "assetId": "4e66aa537abf1bdec8ecc508324891ac"
+                }
+            },
+            "shortId": "zVvo7"
+        },
+        {
+            "name": "Test Cases",
+            "id": "Tk_5MSCREeeSVwJCrBEADA",
+            "slug": "test-cases",
+            "content": {
+                "org.coursera.ondemand.reference.AssetReferenceContent": {
+                    "assetId": "7c84e5c5249eb551d95444c172592274"
+                }
+            },
+            "shortId": "a4I28"
+        }
+    ]
+}

coursera/test/fixtures/json/supplement-programming-immediate-instructions-empty-instructions.json

@@ -0,0 +1,18 @@
+{
+    "elements": [
+        {
+            "id": "Gtv4Xb1-EeS-ViIACwYKVQ~8f3qT",
+            "itemId": "8f3qT",
+            "courseId": "Gtv4Xb1-EeS-ViIACwYKVQ",
+            "assignmentInstructions": {
+                "definition": {
+                    "dtdId": "",
+                    "value": "<co-content><text/><text/><text/><text/><text/><text/></co-content>"
+                },
+                "typeName": "cml"
+            }
+        }
+    ],
+    "linked": {},
+    "paging": {}
+}

coursera/test/fixtures/json/supplement-programming-immediate-instructions-no-instructions.json

@@ -0,0 +1,6 @@
+{
+  "elements": [
+  ],
+  "paging": null,
+  "linked": null
+}

coursera/test/fixtures/json/supplement-programming-immediate-instructions-one-asset.json

@@ -0,0 +1,18 @@
+{
+    "elements": [
+        {
+            "id": "Gtv4Xb1-EeS-ViIACwYKVQ~e4hZk",
+            "itemId": "e4hZk",
+            "courseId": "Gtv4Xb1-EeS-ViIACwYKVQ",
+            "assignmentInstructions": {
+                "definition": {
+                    "dtdId": "",
+                    "value": "<co-content><text><asset id=\"yeJ7Q8VAEeWPRQ4YsSEORQ\" name=\"statement-pca\" extension=\"pdf\" assetType=\"generic\"/>. </text></co-content>"
+                },
+                "typeName": "cml"
+            }
+        }
+    ],
+    "linked": {},
+    "paging": {}
+}

coursera/test/fixtures/json/video-output-1-all.json

@@ -0,0 +1,16 @@
+{
+    "zh-CN.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=eNyKwEu_aMQtn7bg0mUj6uIyVZvjahFSE5x2CrbOXOU&fileExtension=txt",
+    "en.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=afqFhv9FWfxxEeSka8PCA4ihiyX3g2Z6K4jWJPFlcdo&fileExtension=srt",
+    "zh-CN.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=nmGzGoF4oNLv28ZDLUtX5dF4xPXUABgym76XMs4UzDE&fileExtension=srt",
+    "en.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=2Z37WW5Rc7GoT0eft1vdK0HX5imBqoZTKULMTiZ2EjM&fileExtension=txt",
+    "hi.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=qk--Ptsc4w3u6c-5BFPO9vhjyczMHzlSqUOQskjbfZ0&fileExtension=srt",
+    "es.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=Ts5QKzu0jwhUafwsaHk7RKoQJK26d4_bzrX2M6iuRaQ&fileExtension=srt",
+    "pl.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=mcaMPGeK3J7Fn9RRwnuVFnHkyr1COFnLXYKVkUbyfSg&fileExtension=srt",
+    "ja.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=huh5qtCJVj4rEJnsJ6D7MJdCcqN-s9cMd-M6xlSicLc&fileExtension=srt",
+    "pt-BR.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=XYyDJ71d9gl3HOqNplyJeEr7Wd2UhU3DhT-9w_Yudzs&fileExtension=srt",
+    "hi.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=earWLk_RUi3K5UpZfEVOlBgOcpSE9efXz2njRKu31rQ&fileExtension=txt",
+    "es.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=sd6_C14J-qEkvvbqNTgI8W5eUCvOKwW6RzHcz8yF2Jk&fileExtension=txt",
+    "pl.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=sFwO_BWNlhZEDHsXYkFlnOEtHBIX8lSsVGIOLIHeZZ0&fileExtension=txt",
+    "ja.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=WMhDBDbF6SiBuvRwg_QEkglLSK36bj8_5y6kZ9z94YY&fileExtension=txt",
+    "pt-BR.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=uQaL2V2AJ_Wp5dlCZH1HeyTU_AQo9VdJ2cphUhG8yxk&fileExtension=txt"
+}

coursera/test/fixtures/json/video-output-1-en.json

@@ -0,0 +1,4 @@
+{
+    "en.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=afqFhv9FWfxxEeSka8PCA4ihiyX3g2Z6K4jWJPFlcdo&fileExtension=srt",
+    "en.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=2Z37WW5Rc7GoT0eft1vdK0HX5imBqoZTKULMTiZ2EjM&fileExtension=txt"
+}

coursera/test/fixtures/json/video-output-1.json

@@ -0,0 +1,6 @@
+{
+    "zh-CN.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=eNyKwEu_aMQtn7bg0mUj6uIyVZvjahFSE5x2CrbOXOU&fileExtension=txt",
+    "en.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=afqFhv9FWfxxEeSka8PCA4ihiyX3g2Z6K4jWJPFlcdo&fileExtension=srt",
+    "zh-CN.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=nmGzGoF4oNLv28ZDLUtX5dF4xPXUABgym76XMs4UzDE&fileExtension=srt",
+    "en.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=2Z37WW5Rc7GoT0eft1vdK0HX5imBqoZTKULMTiZ2EjM&fileExtension=txt"
+}

coursera/test/fixtures/json/video-output-2.json

@@ -0,0 +1,6 @@
+{
+    "zh-TW.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=-sOeJbk_bICP9OMfbtkjLuwUAIZZcjGasIMk8JO6n0Q&fileExtension=srt",
+    "en.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=xhMK0SSslbfwxl-vzjAXy-bd_iQQTY9iAIrNP4QHxq4&fileExtension=txt",
+    "en.srt": "https://api.coursera.org/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=nO6NGCExQ5FO0aFFnr_YVXtd_lVW4JQaT34WS9tJi6c&fileExtension=srt",
+    "zh-TW.txt": "https://api.coursera.org/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=O9DKhZW6bOsI7ncNZIZPBMXmsreSrgulhGf3eyTCULo&fileExtension=txt"
+}

coursera/test/fixtures/json/video-reply-1.json

@@ -0,0 +1,47 @@
+{
+    "sources": [
+        {
+            "resolution": "540p",
+            "formatSources": {
+                "video/webm": "https://d3c33hcgiwev3.cloudfront.net/20.1-Conclusion-SummaryAndThankYou.63149a70b22b11e4aca907c8d9623f2b/full/540p/index.webm?Expires=1495238400&Signature=Oj2j7hCpfrpp1ugtZHjRITM9D4MjaOJm2x34ecUPGH2nm~BIvt6RY25XpKgCFZ0qbIK01eloymAUolfupBwzJYIjwANxibwpIJ3bX43dtxnoy1dRh2F1YZoZ6lbPCVOSCxODJFod7bZPCuqRTfXvK6X6F0o-IzkbXy8myk6G5Js_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A",
+                "video/mp4": "https://d3c33hcgiwev3.cloudfront.net/20.1-Conclusion-SummaryAndThankYou.63149a70b22b11e4aca907c8d9623f2b/full/540p/index.mp4?Expires=1495238400&Signature=TW8nTCrNKfrBGLCloVNFvNB~7qsNXaRI8T~gBUCxyxxPzumARwdw8W9ONVd-8j2TrI8Zvm~j4ysS4UedtLTKynDewxOzjrbsVc3HRBLTqrcQNjjLkG9vzbrGz2wUMMRUwX6qlwT8xFTVuNjh7-W72gq83bzk4eyaALHO~YKXvxk_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A"
+            }
+        }
+    ],
+    "subtitles": {
+        "hi": "/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=qk--Ptsc4w3u6c-5BFPO9vhjyczMHzlSqUOQskjbfZ0&fileExtension=srt",
+        "en": "/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=afqFhv9FWfxxEeSka8PCA4ihiyX3g2Z6K4jWJPFlcdo&fileExtension=srt",
+        "zh-CN": "/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=nmGzGoF4oNLv28ZDLUtX5dF4xPXUABgym76XMs4UzDE&fileExtension=srt",
+        "es": "/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=Ts5QKzu0jwhUafwsaHk7RKoQJK26d4_bzrX2M6iuRaQ&fileExtension=srt",
+        "pl": "/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=mcaMPGeK3J7Fn9RRwnuVFnHkyr1COFnLXYKVkUbyfSg&fileExtension=srt",
+        "ja": "/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=huh5qtCJVj4rEJnsJ6D7MJdCcqN-s9cMd-M6xlSicLc&fileExtension=srt",
+        "pt-BR": "/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=XYyDJ71d9gl3HOqNplyJeEr7Wd2UhU3DhT-9w_Yudzs&fileExtension=srt"
+    },
+    "playlists": {
+        "hls": "https://d3c33hcgiwev3.cloudfront.net/assetMasterHlsPlaylists.v1/DB-HfUh-EeWWUA71mMib3w?expiry=1495238400000&hmac=kaIpHBiD8US9yrHVsABKCkRPRgIzsBA7tWzB-PHEqhY&mediaCdn=cloudfront"
+    },
+    "subtitlesVtt": {
+        "hi": "/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=4S0NgqfShX81v0QJckTU0IbeAJJnD9m_iZcTYJltbNc&fileExtension=vtt",
+        "en": "/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=lMlV1hdtArRLJvePvHHqFJhekv1Gs-P6WzPQz_TEnzE&fileExtension=vtt",
+        "zh-CN": "/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=H7jCFYLCxt9yS5y5YiLkUGwd3-0fWiioGGVVwiYFfjo&fileExtension=vtt",
+        "es": "/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=nTa427_IEx68vJJYeNQvNQhWQaNOSwJkWUqfAYW2tYI&fileExtension=vtt",
+        "pl": "/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=jPndFFisYDio-2FgSWp-vMVOg9Ybx-Zh4tODJsHvUWY&fileExtension=vtt",
+        "ja": "/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=m9-S0joHnqjaCJ72qKLtL199dsngS9zoP1jmvI7_AA4&fileExtension=vtt",
+        "pt-BR": "/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=ePd5ZewhjQBA5J6QybTLY-U8yEz2FlIqZuvCE7gdYxQ&fileExtension=vtt"
+    },
+    "subtitlesTxt": {
+        "hi": "/api/subtitleAssetProxy.v1/v2OWSJUVSqCjlkiVFRqgng?expiry=1495238400000&hmac=earWLk_RUi3K5UpZfEVOlBgOcpSE9efXz2njRKu31rQ&fileExtension=txt",
+        "en": "/api/subtitleAssetProxy.v1/GgGZN65HQkyBmTeuR2JMsw?expiry=1495238400000&hmac=2Z37WW5Rc7GoT0eft1vdK0HX5imBqoZTKULMTiZ2EjM&fileExtension=txt",
+        "zh-CN": "/api/subtitleAssetProxy.v1/UKEuZoMQRcChLmaDEMXAsA?expiry=1495238400000&hmac=eNyKwEu_aMQtn7bg0mUj6uIyVZvjahFSE5x2CrbOXOU&fileExtension=txt",
+        "es": "/api/subtitleAssetProxy.v1/jtYTHsSQToaWEx7EkJ6G4A?expiry=1495238400000&hmac=sd6_C14J-qEkvvbqNTgI8W5eUCvOKwW6RzHcz8yF2Jk&fileExtension=txt",
+        "pl": "/api/subtitleAssetProxy.v1/RGtowSWPQxSraMElj3MUbA?expiry=1495238400000&hmac=sFwO_BWNlhZEDHsXYkFlnOEtHBIX8lSsVGIOLIHeZZ0&fileExtension=txt",
+        "ja": "/api/subtitleAssetProxy.v1/758f7ykrRcWfH-8pK3XFHw?expiry=1495238400000&hmac=WMhDBDbF6SiBuvRwg_QEkglLSK36bj8_5y6kZ9z94YY&fileExtension=txt",
+        "pt-BR": "/api/subtitleAssetProxy.v1/1kRk9rXlSSeEZPa15aknhQ?expiry=1495238400000&hmac=uQaL2V2AJ_Wp5dlCZH1HeyTU_AQo9VdJ2cphUhG8yxk&fileExtension=txt"
+    },
+    "posters": [
+        {
+            "url": "https://d3c33hcgiwev3.cloudfront.net/imageAssetProxy.v1/20.1-Conclusion-SummaryAndThankYou.63149a70b22b11e4aca907c8d9623f2b/thumbnails/540p/0.jpg?expiry=1495238400000&hmac=LuHD_gGzBtwVeNIQqABnGU69sWteFl8xdMozFAsGPco",
+            "resolution": "540p"
+        }
+    ]
+}

coursera/test/fixtures/json/video-reply-2.json

@@ -0,0 +1,77 @@
+{
+    "posters": [
+        {
+            "url": "https://d3c33hcgiwev3.cloudfront.net/imageAssetProxy.v1/qHTMkA4fEeW2rSIAC2yC6g.processed/thumbnails/540p/0.jpg?expiry=1495238400000&hmac=ES_Ho42kS5yI6VWTPDEY2LNWyvBMOoKwoGXJHb6LqnU",
+            "resolution": "540p"
+        }
+    ],
+    "subtitles": {
+        "en": "/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=nO6NGCExQ5FO0aFFnr_YVXtd_lVW4JQaT34WS9tJi6c&fileExtension=srt",
+        "fr": "/api/subtitleAssetProxy.v1/wAkpEeN1SE6JKRHjdWhOTw?expiry=1495238400000&hmac=DeXzNpOf_7RhvGBaqMWqud5J96PoIh6At1eSDWF1RUM&fileExtension=srt",
+        "lt": "/api/subtitleAssetProxy.v1/qDmNTUOsQoG5jU1DrEKBtw?expiry=1495238400000&hmac=ifyDt77JEmZu5SgJ0nubLGsy6JV9d2IAzNvZdVevBcE&fileExtension=srt",
+        "ko": "/api/subtitleAssetProxy.v1/BatzP4LfQA6rcz-C35AOXQ?expiry=1495238400000&hmac=U70Yc3wumD-G_XE3AYEISRbqtjtl9WSOqQMjlHI2OGM&fileExtension=srt",
+        "hi": "/api/subtitleAssetProxy.v1/-v2-jzNFSxq9vo8zRVsa5Q?expiry=1495238400000&hmac=teLyzOnnfT0P8sZVvD4O8fFDgd0RojSYCxB3n6RvTk0&fileExtension=srt",
+        "fa": "/api/subtitleAssetProxy.v1/MVu-cT46QyqbvnE-OvMq0g?expiry=1495238400000&hmac=gQgvsV9WFOC-cbOuUjLEIepCtka_W8fup6mltVcnJq8&fileExtension=srt",
+        "es": "/api/subtitleAssetProxy.v1/_mNV07xnT_ejVdO8Z2_3Ig?expiry=1495238400000&hmac=fC2H9Lajgsm_WkxnsuNaL6TIfgEzpfhdY2dIJhNasMI&fileExtension=srt",
+        "he": "/api/subtitleAssetProxy.v1/keFd5T42SI6hXeU-NkiO1w?expiry=1495238400000&hmac=DiJ-h_TfbyqgajhWVb-1302MXPoqD5x0JEachin0c3Q&fileExtension=srt",
+        "ar": "/api/subtitleAssetProxy.v1/vHGgsHVsQ2CxoLB1bKNg9w?expiry=1495238400000&hmac=pmjuKhL8-SNhzXi8FAETaJcakRt5S1yqay-G--r4C0I&fileExtension=srt",
+        "bn": "/api/subtitleAssetProxy.v1/hi47OjiORsmuOzo4jqbJBQ?expiry=1495238400000&hmac=tcHBX4hMne23haoOlu1olxiKa7M1n_CSOX4eOmH0U14&fileExtension=srt",
+        "pl": "/api/subtitleAssetProxy.v1/lBH5IsEBRySR-SLBAQcknw?expiry=1495238400000&hmac=QHDKKoNSt2A9Bd4PfRiR82OJwSAsCQmbxZEZoFhLnIg&fileExtension=srt",
+        "tr": "/api/subtitleAssetProxy.v1/jdBjiWbSSlWQY4lm0rpVzw?expiry=1495238400000&hmac=5WoVi4jm3274ClP3U4JZAIhJt-V5ulNHkNtUACqgR6w&fileExtension=srt",
+        "zh-TW": "/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=-sOeJbk_bICP9OMfbtkjLuwUAIZZcjGasIMk8JO6n0Q&fileExtension=srt",
+        "hr": "/api/subtitleAssetProxy.v1/k_PHTdx0Rw-zx03cdOcPyw?expiry=1495238400000&hmac=EeciDHPlRVHxIgsgzSJ0uAOjnaatGmqt4bw1hAyGh9A&fileExtension=srt",
+        "id": "/api/subtitleAssetProxy.v1/xIop0OAYTKaKKdDgGKym3g?expiry=1495238400000&hmac=BT9i5NUXcz5RTOUzjMTK8NOQciU5o-gGI6rTxjNWhbM&fileExtension=srt"
+    },
+    "sources": [
+        {
+            "formatSources": {
+                "video/webm": "https://d3c33hcgiwev3.cloudfront.net/qHTMkA4fEeW2rSIAC2yC6g.processed/full/540p/index.webm?Expires=1495238400&Signature=iAKFovwfSpnnEflg4blQajE9Tle2peDHKv0ScvXSK5rMVBQnX9SWq8M2CvsawxPwKdA2Xo02lOZBi~-5I5F6fCag9mzzlV-8Q-dxVoS1yWZLu7HtGROStDSwOiJYvGoLGSgS2dT0dGYG4rNJ9hxQmElQzBCOYkeQsP6lIsh0Ejg_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A",
+                "video/mp4": "https://d3c33hcgiwev3.cloudfront.net/qHTMkA4fEeW2rSIAC2yC6g.processed/full/540p/index.mp4?Expires=1495238400&Signature=SkYBCgVvj~W2eeSAKjtX4igYZ2WLKq8crzqLbFqgDbZxhH48jW3nXZNWB5~H6ev0EIHkSAbMSQWP4xUGKhzGhcciL8B9jB8LjI180wsSv1jfNknCP9S5p9vd1mdCidheNmJftBtIjfr54m8CgFYUqe1WAo2aLUzRimiS~nf8kxk_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A"
+            },
+            "resolution": "540p"
+        },
+        {
+            "formatSources": {
+                "video/webm": "https://d3c33hcgiwev3.cloudfront.net/qHTMkA4fEeW2rSIAC2yC6g.processed/full/360p/index.webm?Expires=1495238400&Signature=CBaZCNZGbCJ9UpJdWyfGA~KwtUx4UxqnwM8v6GS7T2fsynUVexCfjBcE7IMowDzWa~GZ-jdOI43~s5e4kVc4hbiNOaQoZ-p-te1AffsRJaMhlhI529vxfWUQJGhO3bUGbn9Az9ueSehoe8WLojtHHb5q-IIr53MX-rftl~d3srI_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A",
+                "video/mp4": "https://d3c33hcgiwev3.cloudfront.net/qHTMkA4fEeW2rSIAC2yC6g.processed/full/360p/index.mp4?Expires=1495238400&Signature=dt9-1ARw8I5U1IrIKJGbQzy5MkCqjySGXutT~KFNx8~~UD0v3T0cFwUG3ggLhL3lkyMAztA-dTpARKfi2igKgq6Q8qfcnTfs8~iu5Ayt1vRZVHDfgomlasB3aElmOHB7WaWkQbZJCChXYlVgg2fDKLGxMcfEGf797AzzDrhEFBY_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A"
+            }
+        }
+    ],
+    "subtitlesTxt": {
+        "en": "/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=xhMK0SSslbfwxl-vzjAXy-bd_iQQTY9iAIrNP4QHxq4&fileExtension=txt",
+        "fr": "/api/subtitleAssetProxy.v1/wAkpEeN1SE6JKRHjdWhOTw?expiry=1495238400000&hmac=Y9ysEdDIlObbFPyLBWorn1XEOeJ57TEYPxWsnzE3x5Q&fileExtension=txt",
+        "lt": "/api/subtitleAssetProxy.v1/qDmNTUOsQoG5jU1DrEKBtw?expiry=1495238400000&hmac=92ArkEZV3O5nkxp5f7pxyUClVWbywOWjJ8DFE86foKA&fileExtension=txt",
+        "ko": "/api/subtitleAssetProxy.v1/BatzP4LfQA6rcz-C35AOXQ?expiry=1495238400000&hmac=aVqHA-CIZDni9UrmUGB6aR2VjDstYzdquDrOvzusKbc&fileExtension=txt",
+        "hi": "/api/subtitleAssetProxy.v1/-v2-jzNFSxq9vo8zRVsa5Q?expiry=1495238400000&hmac=fN1qhDyCL5aMzYRW2NbkfNyikEypjzB57LJtO9QXb2Q&fileExtension=txt",
+        "fa": "/api/subtitleAssetProxy.v1/MVu-cT46QyqbvnE-OvMq0g?expiry=1495238400000&hmac=JLJVoBvCluUXGXqzDti9uaW0gDjCsRWQIBOqxARAM9w&fileExtension=txt",
+        "es": "/api/subtitleAssetProxy.v1/_mNV07xnT_ejVdO8Z2_3Ig?expiry=1495238400000&hmac=FzDR-l0J3CTljW9aywGiBn56TTWdmzh1TEYzsfmdceo&fileExtension=txt",
+        "he": "/api/subtitleAssetProxy.v1/keFd5T42SI6hXeU-NkiO1w?expiry=1495238400000&hmac=xwD0e-3s7FwTbDuQn-nNVIi9eoAueviOZl4Ezofd_rY&fileExtension=txt",
+        "ar": "/api/subtitleAssetProxy.v1/vHGgsHVsQ2CxoLB1bKNg9w?expiry=1495238400000&hmac=it_tXtSCiX5oX9MV9VDZQuz1hj5IFZOohCAyYOn8pR4&fileExtension=txt",
+        "bn": "/api/subtitleAssetProxy.v1/hi47OjiORsmuOzo4jqbJBQ?expiry=1495238400000&hmac=fZebyPIojlQlJL3HuHkEOgoQHlwPsJJ5YEC4Pd_a4Sg&fileExtension=txt",
+        "pl": "/api/subtitleAssetProxy.v1/lBH5IsEBRySR-SLBAQcknw?expiry=1495238400000&hmac=jmygEnmsUvBv4-sDUbYZK0MsJht9Mg24AAyaI5iMhmg&fileExtension=txt",
+        "tr": "/api/subtitleAssetProxy.v1/jdBjiWbSSlWQY4lm0rpVzw?expiry=1495238400000&hmac=kgkFFJSswv5HLtPMkjV7rsgLQZzoSBpWHbIffW1FUKc&fileExtension=txt",
+        "zh-TW": "/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=O9DKhZW6bOsI7ncNZIZPBMXmsreSrgulhGf3eyTCULo&fileExtension=txt",
+        "hr": "/api/subtitleAssetProxy.v1/k_PHTdx0Rw-zx03cdOcPyw?expiry=1495238400000&hmac=DO3oN6U9JBwZcScxzOsIAI8Nn2CTaGnlWGi4pAxjDEE&fileExtension=txt",
+        "id": "/api/subtitleAssetProxy.v1/xIop0OAYTKaKKdDgGKym3g?expiry=1495238400000&hmac=HIQ_jyC6_xWBpz4dCF6hxiG5ay1tVSJwQJF8LSCo0gk&fileExtension=txt"
+    },
+    "subtitlesVtt": {
+        "en": "/api/subtitleAssetProxy.v1/r3LdPY_CTUqy3T2Pwu1KVQ?expiry=1495238400000&hmac=WtAfot596syGVoSQ-UJ-QpyWQWqhSQ4auwDRijJy7IM&fileExtension=vtt",
+        "fr": "/api/subtitleAssetProxy.v1/wAkpEeN1SE6JKRHjdWhOTw?expiry=1495238400000&hmac=KDSprAsOxTLlNvgUVon4RRUAAN7BhOd6rRuTm8KWEU0&fileExtension=vtt",
+        "lt": "/api/subtitleAssetProxy.v1/qDmNTUOsQoG5jU1DrEKBtw?expiry=1495238400000&hmac=Hwu1AvpVFsSZUrlSh-VxQ2Rvj6dJ3pgu1_VXDflnH-k&fileExtension=vtt",
+        "ko": "/api/subtitleAssetProxy.v1/BatzP4LfQA6rcz-C35AOXQ?expiry=1495238400000&hmac=I1QSwiy02AWEChUfrmw5C8XKrDReprSmP0mqBv-ipno&fileExtension=vtt",
+        "hi": "/api/subtitleAssetProxy.v1/-v2-jzNFSxq9vo8zRVsa5Q?expiry=1495238400000&hmac=p6JE3hXLwYbJmlTCQf9vZtIO0Gsg8TVAvPjobSFU0eg&fileExtension=vtt",
+        "fa": "/api/subtitleAssetProxy.v1/MVu-cT46QyqbvnE-OvMq0g?expiry=1495238400000&hmac=BkasYNlGSg-zm2GSKqcDLzfwzISJu6agsq1qirTV3xU&fileExtension=vtt",
+        "es": "/api/subtitleAssetProxy.v1/_mNV07xnT_ejVdO8Z2_3Ig?expiry=1495238400000&hmac=zlOw3ifj1lIy4i7GNROw5muCIBidW2MCFnSHRHZ5vhI&fileExtension=vtt",
+        "he": "/api/subtitleAssetProxy.v1/keFd5T42SI6hXeU-NkiO1w?expiry=1495238400000&hmac=9kbtSLEO8NqxGhzvRMUEJGIZ0rlUQ0IO2F_Un-Bpmj0&fileExtension=vtt",
+        "ar": "/api/subtitleAssetProxy.v1/vHGgsHVsQ2CxoLB1bKNg9w?expiry=1495238400000&hmac=YnlfoVEGoQ7LAtnEjrMFXfRPFy5F0-tq_C7s_Vlqf8o&fileExtension=vtt",
+        "bn": "/api/subtitleAssetProxy.v1/hi47OjiORsmuOzo4jqbJBQ?expiry=1495238400000&hmac=J0pV67Lu5PRMeSP_6Mk-pN6CdlguHW024MYQDf1ZOjI&fileExtension=vtt",
+        "pl": "/api/subtitleAssetProxy.v1/lBH5IsEBRySR-SLBAQcknw?expiry=1495238400000&hmac=knptQ0ipo3LojO72PQawrSjdiy6VqBjlFHX62ECyFPg&fileExtension=vtt",
+        "tr": "/api/subtitleAssetProxy.v1/jdBjiWbSSlWQY4lm0rpVzw?expiry=1495238400000&hmac=9gYXktWObxjtSPjkiBB3qo__LZhsRzWoTEPc32uRxhA&fileExtension=vtt",
+        "zh-TW": "/api/subtitleAssetProxy.v1/j8femXUVQaGH3pl1FYGh-Q?expiry=1495238400000&hmac=coTFw8DgESXYsiMqySKkp1JXRfUkwx2fBY_lniYl-i0&fileExtension=vtt",
+        "hr": "/api/subtitleAssetProxy.v1/k_PHTdx0Rw-zx03cdOcPyw?expiry=1495238400000&hmac=OUzErWIwafoewJ97evxPxAdJRiHgTUQEkPydMHHJElk&fileExtension=vtt",
+        "id": "/api/subtitleAssetProxy.v1/xIop0OAYTKaKKdDgGKym3g?expiry=1495238400000&hmac=rt0Afn6mQCoiOAVn258RUI0qiXnInP73f7DM3qldWYY&fileExtension=vtt"
+    },
+    "playlists": {
+        "hls": "https://d3c33hcgiwev3.cloudfront.net/assetMasterHlsPlaylists.v1/qHTMkA4fEeW2rSIAC2yC6g?expiry=1495238400000&hmac=PwXl2HLjLXI3lkJ4YeUu1pNM9Y8XG39j2QzOLFwQ8F4&mediaCdn=cloudfront"
+    }
+}

coursera/test/test_api.py

@@ -10,34 +10,192 @@ from mock import patch, Mock
 from coursera import api
 from coursera import define
 
-from coursera.test.utils import slurp_fixture
+from coursera.test.utils import slurp_fixture, links_to_plain_text
 from coursera.utils import BeautifulSoup
 
+from requests.exceptions import HTTPError
+from requests import Response
+
 
 @pytest.fixture
 def course():
     course = api.CourseraOnDemand(
-        session=None, course_id='0', course_name='test_course')
+        session=Mock(cookies={}), course_id='0', course_name='test_course')
     return course
 
 
+@patch('coursera.api.get_page')
+def test_extract_links_from_programming_http_error(get_page, course):
+    """
+    This test checks that downloader skips locked programming assignments
+    instead of throwing an error. (Locked == returning 403 error code)
+    """
+    locked_response = Response()
+    locked_response.status_code = define.HTTP_FORBIDDEN
+    get_page.side_effect = HTTPError('Mocked HTTP error',
+                                     response=locked_response)
+    assert None == course.extract_links_from_programming('0')
+
+
+@patch('coursera.api.get_page')
+def test_extract_links_from_exam_http_error(get_page, course):
+    """
+    This test checks that downloader skips locked exams
+    instead of throwing an error. (Locked == returning 403 error code)
+    """
+    locked_response = Response()
+    locked_response.status_code = define.HTTP_FORBIDDEN
+    get_page.side_effect = HTTPError('Mocked HTTP error',
+                                     response=locked_response)
+    assert None == course.extract_links_from_exam('0')
+
+
+@patch('coursera.api.get_page')
+def test_extract_links_from_supplement_http_error(get_page, course):
+    """
+    This test checks that downloader skips locked supplements
+    instead of throwing an error. (Locked == returning 403 error code)
+    """
+    locked_response = Response()
+    locked_response.status_code = define.HTTP_FORBIDDEN
+    get_page.side_effect = HTTPError('Mocked HTTP error',
+                                     response=locked_response)
+    assert None == course.extract_links_from_supplement('0')
+
+
+@patch('coursera.api.get_page')
+def test_extract_links_from_lecture_http_error(get_page, course):
+    """
+    This test checks that downloader skips locked lectures
+    instead of throwing an error. (Locked == returning 403 error code)
+    """
+    locked_response = Response()
+    locked_response.status_code = define.HTTP_FORBIDDEN
+    get_page.side_effect = HTTPError('Mocked HTTP error',
+                                     response=locked_response)
+    assert None == course.extract_links_from_lecture('fake_course_id', '0')
+
+
+@patch('coursera.api.get_page')
+def test_extract_links_from_quiz_http_error(get_page, course):
+    """
+    This test checks that downloader skips locked quizzes
+    instead of throwing an error. (Locked == returning 403 error code)
+    """
+    locked_response = Response()
+    locked_response.status_code = define.HTTP_FORBIDDEN
+    get_page.side_effect = HTTPError('Mocked HTTP error',
+                                     response=locked_response)
+    assert None == course.extract_links_from_quiz('0')
+
+
+@patch('coursera.api.get_page')
+def test_extract_references_poll_http_error(get_page, course):
+    """
+    This test checks that downloader skips locked programming assignments
+    instead of throwing an error. (Locked == returning 403 error code)
+    """
+    locked_response = Response()
+    locked_response.status_code = define.HTTP_FORBIDDEN
+    get_page.side_effect = HTTPError('Mocked HTTP error',
+                                     response=locked_response)
+    assert None == course.extract_references_poll()
+
+
+@patch('coursera.api.get_page')
+def test_extract_links_from_reference_http_error(get_page, course):
+    """
+    This test checks that downloader skips locked resources
+    instead of throwing an error. (Locked == returning 403 error code)
+    """
+    locked_response = Response()
+    locked_response.status_code = define.HTTP_FORBIDDEN
+    get_page.side_effect = HTTPError('Mocked HTTP error',
+                                     response=locked_response)
+    assert None == course.extract_links_from_reference('0')
+
+
+@patch('coursera.api.get_page')
+def test_extract_links_from_programming_immediate_instructions_http_error(
+        get_page, course):
+    """
+    This test checks that downloader skips locked programming immediate instructions
+    instead of throwing an error. (Locked == returning 403 error code)
+    """
+    locked_response = Response()
+    locked_response.status_code = define.HTTP_FORBIDDEN
+    get_page.side_effect = HTTPError('Mocked HTTP error',
+                                     response=locked_response)
+    assert (
+        None == course.extract_links_from_programming_immediate_instructions('0'))
+
+
 @patch('coursera.api.get_page')
 def test_ondemand_programming_supplement_no_instructions(get_page, course):
-    no_instructions = slurp_fixture('json/supplement-programming-no-instructions.json')
+    no_instructions = slurp_fixture(
+        'json/supplement-programming-no-instructions.json')
     get_page.return_value = json.loads(no_instructions)
 
     output = course.extract_links_from_programming('0')
     assert {} == output
 
 
+@patch('coursera.api.get_page')
+@pytest.mark.parametrize(
+    "input_filename,expected_output", [
+        ('peer-assignment-instructions-all.json', 'intro Review criteria section'),
+        ('peer-assignment-instructions-no-title.json', 'intro section'),
+        ('peer-assignment-instructions-only-introduction.json', 'intro'),
+        ('peer-assignment-instructions-only-sections.json', 'Review criteria section'),
+        ('peer-assignment-no-instructions.json', ''),
+    ]
+)
+def test_ondemand_from_peer_assignment_instructions(
+        get_page, course, input_filename, expected_output):
+    instructions = slurp_fixture('json/%s' % input_filename)
+    get_page.return_value = json.loads(instructions)
+
+    output = course.extract_links_from_peer_assignment('0')
+    assert expected_output == links_to_plain_text(output)
+
+
+@patch('coursera.api.get_page')
+def test_ondemand_from_programming_immediate_instructions_no_instructions(
+        get_page, course):
+    no_instructions = slurp_fixture(
+        'json/supplement-programming-immediate-instructions-no-instructions.json')
+    get_page.return_value = json.loads(no_instructions)
+
+    output = course.extract_links_from_programming_immediate_instructions('0')
+    assert {} == output
+
+
 @patch('coursera.api.get_page')
 def test_ondemand_programming_supplement_empty_instructions(get_page, course):
-    empty_instructions = slurp_fixture('json/supplement-programming-empty-instructions.json')
+    empty_instructions = slurp_fixture(
+        'json/supplement-programming-empty-instructions.json')
     get_page.return_value = json.loads(empty_instructions)
     output = course.extract_links_from_programming('0')
 
     # Make sure that SOME html content has been extracted, but remove
-    # it immeditely because it's a hassle to properly prepare test input
+    # it immediately because it's a hassle to properly prepare test input
+    # for it. FIXME later.
+    assert 'html' in output
+    del output['html']
+
+    assert {} == output
+
+
+@patch('coursera.api.get_page')
+def test_ondemand_programming_immediate_instructions_empty_instructions(
+        get_page, course):
+    empty_instructions = slurp_fixture(
+        'json/supplement-programming-immediate-instructions-empty-instructions.json')
+    get_page.return_value = json.loads(empty_instructions)
+    output = course.extract_links_from_programming_immediate_instructions('0')
+
+    # Make sure that SOME html content has been extracted, but remove
+    # it immediately because it's a hassle to properly prepare test input
     # for it. FIXME later.
     assert 'html' in output
     del output['html']
@@ -51,14 +209,50 @@ def test_ondemand_programming_supplement_one_asset(get_page, course):
     one_asset_url = slurp_fixture('json/asset-urls-one.json')
     asset_json = json.loads(one_asset_url)
     get_page.side_effect = [json.loads(one_asset_tag),
-                                 json.loads(one_asset_url)]
+                            json.loads(one_asset_url)]
 
     expected_output = {'pdf': [(asset_json['elements'][0]['url'],
-                               'statement-pca')]}
+                                'statement-pca')]}
     output = course.extract_links_from_programming('0')
 
     # Make sure that SOME html content has been extracted, but remove
-    # it immeditely because it's a hassle to properly prepare test input
+    # it immediately because it's a hassle to properly prepare test input
+    # for it. FIXME later.
+    assert 'html' in output
+    del output['html']
+
+    assert expected_output == output
+
+
+@patch('coursera.api.get_page')
+def test_extract_references_poll(get_page, course):
+    """
+    Test extracting course references.
+    """
+    get_page.side_effect = [
+        json.loads(slurp_fixture('json/references-poll-reply.json'))
+    ]
+    expected_output = json.loads(
+        slurp_fixture('json/references-poll-output.json'))
+    output = course.extract_references_poll()
+    assert expected_output == output
+
+
+@patch('coursera.api.get_page')
+def test_ondemand_programming_immediate_instructions_one_asset(get_page, course):
+    one_asset_tag = slurp_fixture(
+        'json/supplement-programming-immediate-instructions-one-asset.json')
+    one_asset_url = slurp_fixture('json/asset-urls-one.json')
+    asset_json = json.loads(one_asset_url)
+    get_page.side_effect = [json.loads(one_asset_tag),
+                            json.loads(one_asset_url)]
+
+    expected_output = {'pdf': [(asset_json['elements'][0]['url'],
+                                'statement-pca')]}
+    output = course.extract_links_from_programming_immediate_instructions('0')
+
+    # Make sure that SOME html content has been extracted, but remove
+    # it immediately because it's a hassle to properly prepare test input
     # for it. FIXME later.
     assert 'html' in output
     del output['html']
@@ -68,17 +262,19 @@ def test_ondemand_programming_supplement_one_asset(get_page, course):
 
 @patch('coursera.api.get_page')
 def test_ondemand_programming_supplement_three_assets(get_page, course):
-    three_assets_tag = slurp_fixture('json/supplement-programming-three-assets.json')
+    three_assets_tag = slurp_fixture(
+        'json/supplement-programming-three-assets.json')
     three_assets_url = slurp_fixture('json/asset-urls-three.json')
     get_page.side_effect = [json.loads(three_assets_tag),
-                                 json.loads(three_assets_url)]
+                            json.loads(three_assets_url)]
 
-    expected_output = json.loads(slurp_fixture('json/supplement-three-assets-output.json'))
+    expected_output = json.loads(slurp_fixture(
+        'json/supplement-three-assets-output.json'))
     output = course.extract_links_from_programming('0')
     output = json.loads(json.dumps(output))
 
     # Make sure that SOME html content has been extracted, but remove
-    # it immeditely because it's a hassle to properly prepare test input
+    # it immediately because it's a hassle to properly prepare test input
     # for it. FIXME later.
     assert 'html' in output
     del output['html']
@@ -88,12 +284,15 @@ def test_ondemand_programming_supplement_three_assets(get_page, course):
 
 @patch('coursera.api.get_page')
 def test_extract_links_from_lecture_assets_typename_asset(get_page, course):
-    open_course_assets_reply = slurp_fixture('json/supplement-open-course-assets-reply.json')
-    api_assets_v1_reply = slurp_fixture('json/supplement-api-assets-v1-reply.json')
+    open_course_assets_reply = slurp_fixture(
+        'json/supplement-open-course-assets-reply.json')
+    api_assets_v1_reply = slurp_fixture(
+        'json/supplement-api-assets-v1-reply.json')
     get_page.side_effect = [json.loads(open_course_assets_reply),
-                                 json.loads(api_assets_v1_reply)]
+                            json.loads(api_assets_v1_reply)]
 
-    expected_output = json.loads(slurp_fixture('json/supplement-extract-links-from-lectures-output.json'))
+    expected_output = json.loads(slurp_fixture(
+        'json/supplement-extract-links-from-lectures-output.json'))
     assets = ['giAxucdaEeWJTQ5WTi8YJQ']
     output = course._extract_links_from_lecture_assets(assets)
     output = json.loads(json.dumps(output))
@@ -107,14 +306,20 @@ def test_extract_links_from_lecture_assets_typname_url_and_asset(get_page, cours
     links both from typename == 'asset' and == 'url'.
     """
     get_page.side_effect = [
-        json.loads(slurp_fixture('json/supplement-open-course-assets-typename-url-reply-1.json')),
-        json.loads(slurp_fixture('json/supplement-open-course-assets-typename-url-reply-2.json')),
-        json.loads(slurp_fixture('json/supplement-open-course-assets-typename-url-reply-3.json')),
-        json.loads(slurp_fixture('json/supplement-open-course-assets-typename-url-reply-4.json')),
-        json.loads(slurp_fixture('json/supplement-open-course-assets-typename-url-reply-5.json')),
+        json.loads(slurp_fixture(
+            'json/supplement-open-course-assets-typename-url-reply-1.json')),
+        json.loads(slurp_fixture(
+            'json/supplement-open-course-assets-typename-url-reply-2.json')),
+        json.loads(slurp_fixture(
+            'json/supplement-open-course-assets-typename-url-reply-3.json')),
+        json.loads(slurp_fixture(
+            'json/supplement-open-course-assets-typename-url-reply-4.json')),
+        json.loads(slurp_fixture(
+            'json/supplement-open-course-assets-typename-url-reply-5.json')),
     ]
 
-    expected_output = json.loads(slurp_fixture('json/supplement-extract-links-from-lectures-url-asset-output.json'))
+    expected_output = json.loads(slurp_fixture(
+        'json/supplement-extract-links-from-lectures-url-asset-output.json'))
     assets = ['Yry0spSKEeW8oA5fR3afVQ',
               'kMQyUZSLEeWj-hLVp2Pm8w',
               'xkAloZmJEeWjYA4jOOgP8Q']
@@ -122,6 +327,7 @@ def test_extract_links_from_lecture_assets_typname_url_and_asset(get_page, cours
     output = json.loads(json.dumps(output))
     assert expected_output == output
 
+
 @patch('coursera.api.get_page')
 def test_list_courses(get_page, course):
     """
@@ -130,29 +336,66 @@ def test_list_courses(get_page, course):
     get_page.side_effect = [
         json.loads(slurp_fixture('json/list-courses-input.json'))
     ]
-    expected_output = json.loads(slurp_fixture('json/list-courses-output.json'))
+    expected_output = json.loads(
+        slurp_fixture('json/list-courses-output.json'))
     expected_output = expected_output['courses']
     output = course.list_courses()
     assert expected_output == output
 
 
+@pytest.mark.parametrize(
+    "input_filename,output_filename,subtitle_language,video_id", [
+        ('video-reply-1.json', 'video-output-1.json',
+            'en,zh-CN|zh-TW', "None"),
+        ('video-reply-1.json', 'video-output-1-en.json',
+            'zh-TW', "None"),
+        ('video-reply-1.json', 'video-output-1-en.json',
+            'en', "None"),
+        ('video-reply-1.json', 'video-output-1-all.json',
+            'all', "None"),
+        ('video-reply-1.json', 'video-output-1-all.json',
+            'zh-TW,all|zh-CN', "None"),
+        ('video-reply-2.json', 'video-output-2.json',
+            'en,zh-CN|zh-TW', "None"),
+    ]
+)
+def test_extract_subtitles_from_video_dom(input_filename, output_filename, subtitle_language, video_id):
+    video_dom = json.loads(slurp_fixture('json/%s' % input_filename))
+    expected_output = json.loads(slurp_fixture('json/%s' % output_filename))
+    course = api.CourseraOnDemand(
+        session=Mock(cookies={}), course_id='0', course_name='test_course')
+    actual_output = course._extract_subtitles_from_video_dom(
+        video_dom, subtitle_language, video_id)
+    actual_output = json.loads(json.dumps(actual_output))
+    assert actual_output == expected_output
+
+
 @pytest.mark.parametrize(
     "input_filename,output_filename", [
         ('empty-input.json', 'empty-output.txt'),
-        ('answer-text-replaced-with-span-input.json', 'answer-text-replaced-with-span-output.txt'),
-        ('question-type-textExactMatch-input.json', 'question-type-textExactMatch-output.txt'),
+        ('answer-text-replaced-with-span-input.json',
+         'answer-text-replaced-with-span-output.txt'),
+        ('question-type-textExactMatch-input.json',
+         'question-type-textExactMatch-output.txt'),
         ('question-type-regex-input.json', 'question-type-regex-output.txt'),
-        ('question-type-mathExpression-input.json', 'question-type-mathExpression-output.txt'),
+        ('question-type-mathExpression-input.json',
+         'question-type-mathExpression-output.txt'),
         ('question-type-checkbox-input.json', 'question-type-checkbox-output.txt'),
         ('question-type-mcq-input.json', 'question-type-mcq-output.txt'),
-        ('question-type-singleNumeric-input.json', 'question-type-singleNumeric-output.txt'),
+        ('question-type-singleNumeric-input.json',
+         'question-type-singleNumeric-output.txt'),
+        ('question-type-reflect-input.json', 'question-type-reflect-output.txt'),
+        ('question-type-mcqReflect-input.json',
+         'question-type-mcqReflect-output.txt'),
         ('question-type-unknown-input.json', 'question-type-unknown-output.txt'),
         ('multiple-questions-input.json', 'multiple-questions-output.txt'),
     ]
 )
 def test_quiz_exam_to_markup_converter(input_filename, output_filename):
-    quiz_json = json.loads(slurp_fixture('json/quiz-to-markup/%s' % input_filename))
-    expected_output = slurp_fixture('json/quiz-to-markup/%s' % output_filename).strip()
+    quiz_json = json.loads(slurp_fixture(
+        'json/quiz-to-markup/%s' % input_filename))
+    expected_output = slurp_fixture(
+        'json/quiz-to-markup/%s' % output_filename).strip()
 
     converter = api.QuizExamToMarkupConverter(session=None)
     actual_output = converter(quiz_json).strip()
@@ -168,17 +411,34 @@ class TestMarkupToHTMLConverter:
     STYLE = None
 
     def setup_method(self, test_method):
-        self.STYLE = self._p(define.INSTRUCTIONS_HTML_INJECTION)
+        self.STYLE = self._p(
+            "".join([define.INSTRUCTIONS_HTML_INJECTION_PRE,
+                     define.INSTRUCTIONS_HTML_MATHJAX_URL,
+                     define.INSTRUCTIONS_HTML_INJECTION_AFTER])
+        )
         self.markup_to_html = api.MarkupToHTMLConverter(session=None)
 
+        ALTERNATIVE_MATHJAX_CDN = "https://alternative/mathjax/cdn.js"
+        self.STYLE_WITH_ALTER = self._p(
+            "".join([define.INSTRUCTIONS_HTML_INJECTION_PRE,
+                     ALTERNATIVE_MATHJAX_CDN,
+                     define.INSTRUCTIONS_HTML_INJECTION_AFTER])
+        )
+        self.markup_to_html_with_alter_mjcdn = api.MarkupToHTMLConverter(
+            session=None, mathjax_cdn_url=ALTERNATIVE_MATHJAX_CDN)
+
     def test_empty(self):
         output = self.markup_to_html("")
-        assert self._p("""
+        output_with_alter_mjcdn = self.markup_to_html_with_alter_mjcdn("")
+        markup = """
         <meta charset="UTF-8"/>
-        """) + self.STYLE == output
+        """
+        assert self._p(markup) + self.STYLE == output
+        assert self._p(markup) + \
+            self.STYLE_WITH_ALTER == output_with_alter_mjcdn
 
     def test_replace_text_tag(self):
-        output = self.markup_to_html("""
+        markup = """
         <co-content>
         <text>
             Test<text>Nested</text>
@@ -187,8 +447,8 @@ class TestMarkupToHTMLConverter:
             Test2
         </text>
         </co-content>
-        """)
-        assert self._p("""
+        """
+        result = """
         <meta charset="UTF-8"/>
         <co-content>
         <p>
@@ -198,7 +458,12 @@ class TestMarkupToHTMLConverter:
             Test2
         </p>
         </co-content>\n
-        """) + self.STYLE == output
+        """
+        output = self.markup_to_html(markup)
+        output_with_alter_mjcdn = self.markup_to_html_with_alter_mjcdn(markup)
+        assert self._p(result) + self.STYLE == output
+        assert self._p(result) + \
+            self.STYLE_WITH_ALTER == output_with_alter_mjcdn
 
     def test_replace_heading(self):
         output = self.markup_to_html("""
@@ -261,7 +526,8 @@ class TestMarkupToHTMLConverter:
             'nodata': Mock(data=None, content_type='image/png')
         }
         mock_asset_retriever.__call__ = Mock(return_value=None)
-        mock_asset_retriever.__getitem__  = Mock(side_effect=replies.__getitem__)
+        mock_asset_retriever.__getitem__ = Mock(
+            side_effect=replies.__getitem__)
         self.markup_to_html._asset_retriever = mock_asset_retriever
 
         output = self.markup_to_html("""
@@ -292,7 +558,8 @@ class TestMarkupToHTMLConverter:
             'bWTK9sYwEeW7AxLLCrgDQQ': Mock(data=b'b', content_type='unknown')
         }
         mock_asset_retriever.__call__ = Mock(return_value=None)
-        mock_asset_retriever.__getitem__  = Mock(side_effect=replies.__getitem__)
+        mock_asset_retriever.__getitem__ = Mock(
+            side_effect=replies.__getitem__)
         self.markup_to_html._asset_retriever = mock_asset_retriever
 
         output = self.markup_to_html("""
@@ -330,6 +597,7 @@ def test_quiz_converter():
     with open('quiz.html', 'w') as file:
         file.write(result)
 
+
 def test_quiz_converter_all():
     pytest.skip()
     import os
@@ -343,8 +611,8 @@ def test_quiz_converter_all():
     markup_to_html = api.MarkupToHTMLConverter(session=session)
 
     path = 'quiz_json'
-    for filename in ['quiz-audio.json']: #os.listdir(path):
-    # for filename in ['all_question_types.json']:
+    for filename in ['quiz-audio.json']:  # os.listdir(path):
+        # for filename in ['all_question_types.json']:
         # if 'YV0W4' not in filename:
         #     continue
         # if 'QVHj1' not in filename:
@@ -360,6 +628,7 @@ def test_quiz_converter_all():
         with open('quiz_html/' + filename + '.html', 'w') as f:
             f.write(result)
 
+
 def create_session():
     from coursera.coursera_dl import get_session
     from coursera.credentials import get_credentials
@@ -385,10 +654,14 @@ def test_asset_retriever(get_reply, get_page):
                  'vdqUTz61Eea_CQ5dfWSAjQ']
 
     expected_output = [
-        api.Asset(id="bWTK9sYwEeW7AxLLCrgDQQ", name="M111.mp3", type_name="audio", url="url4", content_type="image/png", data="<...>"),
-        api.Asset(id="VceKeChKEeaOMw70NkE3iw", name="09_graph_decomposition_problems_1.pdf", type_name="pdf", url="url7", content_type="image/png", data="<...>"),
-        api.Asset(id="VcmGXShKEea4ehL5RXz3EQ", name="09_graph_decomposition_starter_files_1.zip", type_name="generic", url="url2", content_type="image/png", data="<...>"),
-        api.Asset(id="vdqUTz61Eea_CQ5dfWSAjQ", name="Capture.PNG", type_name="image", url="url9", content_type="image/png", data="<...>"),
+        api.Asset(id="bWTK9sYwEeW7AxLLCrgDQQ", name="M111.mp3", type_name="audio",
+                  url="url4", content_type="image/png", data="<...>"),
+        api.Asset(id="VceKeChKEeaOMw70NkE3iw", name="09_graph_decomposition_problems_1.pdf",
+                  type_name="pdf", url="url7", content_type="image/png", data="<...>"),
+        api.Asset(id="VcmGXShKEea4ehL5RXz3EQ", name="09_graph_decomposition_starter_files_1.zip",
+                  type_name="generic", url="url2", content_type="image/png", data="<...>"),
+        api.Asset(id="vdqUTz61Eea_CQ5dfWSAjQ", name="Capture.PNG",
+                  type_name="image", url="url9", content_type="image/png", data="<...>"),
     ]
 
     retriever = api.AssetRetriever(session=None)

coursera/test/test_commandline.py

@@ -0,0 +1,23 @@
+"""
+Test command line module.
+"""
+
+from coursera import commandline
+from coursera.test import test_workflow
+
+
+def test_class_name_arg_required():
+    args = {'list_courses': False, 'version': False}
+    mock_args = test_workflow.MockedCommandLineArgs(**args)
+    assert commandline.class_name_arg_required(mock_args)
+
+
+def test_class_name_arg_not_required():
+    not_required_cases = [
+        {'list_courses': True, 'version': False},
+        {'list_courses': False, 'version': True},
+        {'list_courses': True, 'version': True},
+    ]
+    for args in not_required_cases:
+        mock_args = test_workflow.MockedCommandLineArgs(**args)
+        assert not commandline.class_name_arg_required(mock_args)

coursera/test/test_parsing.py

@@ -65,7 +65,7 @@ def test_that_we_parse_and_write_json_correctly(get_page, json_path):
 def get_old_style_video(monkeypatch):
     pytest.skip()
     """
-    Mock some methods that would, otherwise, create repeateadly many web
+    Mock some methods that would, otherwise, create repeatedly many web
     requests.
 
     More specifically, we mock:
@@ -139,7 +139,7 @@ def test_get_on_demand_supplement_url_accumulates_assets(mocked):
     output = course.extract_links_from_supplement('element_id')
 
     # Make sure that SOME html content has been extracted, but remove
-    # it immeditely because it's a hassle to properly prepare test input
+    # it immediately because it's a hassle to properly prepare test input
     # for it. FIXME later.
     assert 'html' in output
     del output['html']

coursera/test/test_utils.py

@@ -34,7 +34,7 @@ from coursera.utils import total_seconds, is_course_complete
         ('Week 3: Data and Abstraction', 'Week_3-_Data_and_Abstraction'),
         ('  (Week 1) BRANDING:  Marketing Strategy and Brand Positioning',
          'Week_1_BRANDING-__Marketing_Strategy_and_Brand_Positioning'),
-        ('test & " adfas', 'test___adfas'),
+        ('test & " adfas', 'test__-_adfas'),  # `"` were changed first to `-`
         (' ', ''),
         ('☂℮﹩т ω☤☂ℌ Ṳᾔ☤ḉ◎ⅾε', '__')
     ]
@@ -54,7 +54,7 @@ def test_clean_filename(unclean, clean):
          'Week 3- Data and Abstraction'),
         ('  (Week 1) BRANDING:  Marketing Strategy and Brand Positioning',
          '  (Week 1) BRANDING-  Marketing Strategy and Brand Positioning'),
-        ('test & " adfas', 'test & " adfas'),
+        ('test & " adfas', 'test & - adfas'),  # `"` are forbidden on Windows
         (' ', u'\xa0'),
         ('☂℮﹩т ω☤☂ℌ Ṳᾔ☤ḉ◎ⅾε', '☂℮﹩т ω☤☂ℌ Ṳᾔ☤ḉ◎ⅾε')
     ]

coursera/test/test_workflow.py

@@ -37,7 +37,7 @@ class MockedFailingDownloader(Downloader):
         raise self._exception_to_throw
 
 
-TEST_URL = "https://www.coursera.org/api/test-url"
+TEST_URL = "https://api.coursera.org/api/test-url"
 
 
 def make_test_modules():
@@ -110,7 +110,7 @@ def test_iter_modules():
         (0, '01_section1'),
         (0, normpath('test_class/01_section1/01_module1')),
         (0, 'lecture1', 'en.txt', 'title'),
-        ('en.txt', 'https://www.coursera.org/api/test-url', 'title')
+        ('en.txt', 'https://api.coursera.org/api/test-url', 'title')
     ]
     collected_output = []
 
@@ -138,7 +138,7 @@ def test_walk_modules():
         (0, '01_section1',
          0, normpath('test_class/01_section1/01_module1'),
          0, 'lecture1', normpath('test_class/01_section1/01_module1/01_lecture1_title.en.txt'),
-         'https://www.coursera.org/api/test-url')]
+         'https://api.coursera.org/api/test-url')]
     collected_output = []
 
     for module, section, lecture, resource in _walk_modules(

coursera/test/utils.py

@@ -2,9 +2,43 @@
 Helper functions that are only used in tests.
 """
 import os
+import re
 from io import open
 
+from six import iteritems
+
+from coursera.define import IN_MEMORY_MARKER
+from coursera.utils import BeautifulSoup
+
 
 def slurp_fixture(path):
     return open(os.path.join(os.path.dirname(__file__),
                              "fixtures", path), encoding='utf8').read()
+
+
+def links_to_plain_text(links):
+    """
+    Converts extracted links into text and cleans up extra whitespace. Only HTML
+    sections are converted. This is a helper to be used in tests.
+
+    @param links: Links obtained from such methods as extract_links_from_peer_assignment.
+    @type links: @see CourseraOnDemand._extract_links_from_text
+
+    @return: HTML converted to plain text with extra space removed.
+    @rtype: str
+    """
+    result = []
+    for filetype, contents in iteritems(links):
+        if filetype != 'html':
+            continue
+
+        for content, _prefix in contents:
+            if content.startswith(IN_MEMORY_MARKER):
+                content = content[len(IN_MEMORY_MARKER):]
+
+            soup = BeautifulSoup(content)
+            [script.extract() for script in soup(["script", "style"])]
+            text = re.sub(r'[ \t\r\n]+', ' ', soup.get_text()).strip()
+            result.append(text)
+
+    return ''.join(result)

coursera/utils.py

@@ -8,6 +8,7 @@ import os
 import re
 import sys
 import time
+import json
 import errno
 import random
 import string
@@ -41,7 +42,9 @@ else:
 from .define import COURSERA_URL, WINDOWS_UNC_PREFIX
 
 # Force us of bs4 with html.parser
-BeautifulSoup = lambda page: BeautifulSoup_(page, 'html.parser')
+
+
+def BeautifulSoup(page): return BeautifulSoup_(page, 'html.parser')
 
 
 if six.PY2:
@@ -55,6 +58,16 @@ else:
         return x
 
 
+def spit_json(obj, filename):
+    with open(filename, 'w') as file_object:
+        json.dump(obj, file_object, indent=4)
+
+
+def slurp_json(filename):
+    with open(filename) as file_object:
+        return json.load(file_object)
+
+
 def is_debug_run():
     """
     Check whether we're running with DEBUG loglevel.
@@ -106,13 +119,24 @@ def clean_filename(s, minimal_change=False):
     s = unquote_plus(s)
 
     # Strip forbidden characters
+    # https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx
     s = (
         s.replace(':', '-')
         .replace('/', '-')
+        .replace('<', '-')
+        .replace('>', '-')
+        .replace('"', '-')
+        .replace('\\', '-')
+        .replace('|', '-')
+        .replace('?', '-')
+        .replace('*', '-')
         .replace('\x00', '-')
-        .replace('\n', '')
+        .replace('\n', ' ')
     )
 
+    # Remove trailing dots and spaces; forbidden on Windows
+    s = s.rstrip(' .')
+
     if minimal_change:
         return s
 

deploy/.netrc

@@ -1 +0,0 @@
-machine coursera-dl login <user> password <pass>

deploy/Dockerfile

@@ -1,14 +0,0 @@
-FROM ubuntu:14.04
-MAINTAINER Dmitry Senin <seninds@gmail.com>
-
-RUN apt-get update
-RUN DEBIAN_FRONTEND=noninteractive apt-get install -y git build-essential libssl-dev libffi-dev
-RUN DEBIAN_FRONTEND=noninteractive apt-get install -y python-pip python-dev
-RUN pip install ndg-httpsclient
-
-COPY .netrc /root/.netrc
-RUN chmod 0600 /root/.netrc
-
-RUN cd /root && git clone https://github.com/coursera-dl/coursera.git
-RUN cd /root/coursera && pip install -r requirements.txt
-RUN cd /usr/bin && ln -s /root/coursera/coursera-dl coursera-dl

deploy/README.md

@@ -1,10 +0,0 @@
-# How to launch the container
-
-1. [optional] Insert your username and password in the `.netrc` file if you
-   plan to use the `-n` optionof `coursera-dl` (edit template in this
-   directory).
-2. Build Docker image:
-   `./build.sh`
-3. Run Docker container to download courses A, B and C:
-   `./download.sh A B C`
-4. All courses will be downloaded in directory `~/courses`

deploy/build.sh

@@ -1,7 +0,0 @@
-#!/bin/sh
-
-if groups | grep -q "docker" ; then
-    docker build --tag coursera-img --rm .
-else
-    sudo docker build --tag coursera-img --rm .
-fi

deploy/download.sh

@@ -1,15 +0,0 @@
-#!/bin/sh
-
-COURSES=$*
-
-if [ ! -e ~/courses ]; then
-    mkdir ~/courses
-fi
-
-if groups | grep -q "docker" ; then
-    docker run --rm --name coursera -v ~/courses:/courses coursera-img \
-               coursera-dl -n --path /courses $COURSES
-else
-    sudo docker run --rm --name coursera -v ~/courses:/courses coursera-img \
-                    coursera-dl -n --path /courses $COURSES
-fi

requirements.txt

@@ -1,6 +1,8 @@
 beautifulsoup4>=4.1.3
 requests>=2.10.0
 six>=1.5.0
-urllib3>=1.10
+urllib3>=1.23
 pyasn1>=0.1.7
 keyring>=4.0
+configargparse>=0.12.0
+attrs==18.1.0

setup.py

@@ -10,6 +10,8 @@ from __future__ import print_function
 import os.path
 import subprocess
 import sys
+# For compatibility with Python2.7
+from io import open
 
 from setuptools import setup
 
@@ -48,7 +50,7 @@ def read_file(filename, alt=None):
     lines = None
 
     try:
-        with open(filename) as f:
+        with open(filename, encoding='utf-8') as f:
             lines = f.read()
     except IOError:
         lines = [] if alt is None else alt
@@ -58,9 +60,8 @@ def read_file(filename, alt=None):
 generate_readme_rst()
 
 long_description = read_file(
-    'README.rst',
-    'Generate README.rst from README.md via pandoc!\n\nExample: '
-    'pandoc --from=markdown --to=rst --output=README.rst README.md'
+    'README.md',
+    'Cannot read README.md'
 )
 requirements = read_file('requirements.txt')
 dev_requirements = read_file('requirements-dev.txt')
@@ -72,12 +73,11 @@ trove_classifiers = [
     'License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)',
     'Operating System :: OS Independent',
     'Programming Language :: Python :: 2',
-    'Programming Language :: Python :: 2.6',
     'Programming Language :: Python :: 2.7',
     'Programming Language :: Python :: 3',
-    'Programming Language :: Python :: 3.3',
     'Programming Language :: Python :: 3.4',
     'Programming Language :: Python :: 3.5',
+    'Programming Language :: Python :: 3.6',
     'Programming Language :: Python :: Implementation :: CPython',
     'Programming Language :: Python :: Implementation :: PyPy',
     'Programming Language :: Python',
@@ -88,7 +88,7 @@ setup(
     name='coursera-dl',
     version=__version__,
     maintainer='Rogério Theodoro de Brito',
-    maintainer_email='rbrito@ime.usp.br',
+    maintainer_email='rbrito@gmail.com',
 
     license='LGPL',
     url='https://github.com/coursera-dl/coursera-dl',
@@ -100,7 +100,9 @@ setup(
 
     description='Script for downloading Coursera.org videos and naming them.',
     long_description=long_description,
-    keywords=['coursera-dl', 'coursera', 'download', 'education', 'MOOCs', 'video'],
+    long_description_content_type='text/markdown',
+    keywords=['coursera-dl', 'coursera',
+              'download', 'education', 'MOOCs', 'video'],
     classifiers=trove_classifiers,
 
     packages=["coursera"],

tox.ini

@@ -1,5 +1,5 @@
 [tox]
-envlist = py26,py27,py33,py34,py35
+envlist = py26,py27,py33,py34,py35,py36
 
 [testenv]
 downloadcache = .tox/_download/
@@ -14,9 +14,19 @@ deps =
     six>=1.5.0
     urllib3>=1.10
     keyrings.alt>=1.1
+    configargparse>=0.12.0
 
 commands = py.test -v --junitxml={envlogdir}/result.xml coursera/test
 # Original command: install_command = pip install {opts} {packages}
 # {opts} is remove to prevent passing option "--download-cache" to pip
 # which is already gone.
 install_command = pip install {packages}
+
+# Notes for developers. Depending on your system configuration,
+# you may find this bash function useful to run before running tox:
+#
+# activate_pyenv () {
+#         export PYENV_ROOT="$HOME/.pyenv"
+#         export PATH="$PYENV_ROOT/bin:$PATH"
+#         eval "$(pyenv init -)"
+# }