Some JSON documents for some courses in Coursera's New Platform don't always
have an `assetId` attribute right below a `definition` attribute.
Since we convert a JSON document to a Python dictionary, we can't
(unfortunately) just go ahead and access something like
`element['definition']['assetId']`.
Important aside: It seems, during my tests with the `graph-analytics` course
that the method `_get_open_course_asset_ids` only gets called with such
"defective" JSON documents, which *may* indicate that the `if` filtering in
the list comprehension is always necessary and the lists returned by
`_get_open_course_asset_ids` are always empty after filtering (again, at
least with this course; haven't tested with others).
We should investigate things better and see what kind of JSON documents get
us there and, if possible, categorize the documents that Coursera spits so
that we have a better idea of what they are sending us and how to behave.
One (wild) idea would be to have a method for each possible type of JSON
document returning the data structures that we need.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
Unless I'm wrong (which I may quite likely be), the tests will fail if the
keyring backend is not properly configured. Including `keyrings.alt` is
likely to make tests pass even if we don't have a multitude of keyring
backends available.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
We use an SVG file instead of a PNG file, which should look better on some
devices (yes, I know, this is a frivolous change).
Unfortunately, the URL registered there is still `coursera-dl/coursera` and
not the new one (`coursera-dl/coursera-dl`), since the site gives me an HTTP
500 error whenever I click on the button to update the repository
information, and this seems to be the recommended update procedure.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
Unfortunately, with this move, we lost the history of the project.
On the other hand, the site doesn't seem to have any easy to spot place to
update the URL (the one that it has is not editable).
At least we now point the site to the correct repository. The previously
computed "GPA" was only 1.67 and we have a substantial amount of new tests
now.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
Python 3.5 was released some time ago and new Linux distributions will have
it as the default interpreter, and we should make sure that it is supported.
Furthermore, Python 2 is on the way out of future releases of said Linux
distributions (e.g., Debian, Fedora and so on).
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
I don't have pypy right now installed on my computer. I plan to add it back
when Python 3 is supported.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
Extract more supplementary files.
From the pull request:
----
This pull request adds extraction of supplementary files (e.g. PDFs) from lecture units.
There are many ways on the new Coursera platform to provide supplementary files. Different course makers use different facilities for that. As a result, it gets complicated and intricate during the
parsing/extraction. Thank got we only have to work with json structure, not (almost never) HTML.
This pull request is the second attempt to fix#425.
----
Thanks @balta2ar for the great contribution! Closes#425.
It appears that there are many ways to provide PDFs in a lecture:
1. Add <a href> tags into the text of a "supplement" lecture unit
2. Add <asset> tags into the text of a "supplement" lecture unit
3. Add "assets" field to the content.definition of a "lecture" lecture unit
This commit implements extraction of files that were added with the
third option.
This requires:
1. Call api/openCourseAssets.v1 on each such asset to receive another
list of asset ids (lets call them sub ids)
2. Call api/assets.v1 on each sub asset id to receive name and url
of an asset.
Weirdly, asset ids there are present in content.definition.assets, contain
"@1" characters at the end. Only after they are cut off, method
api/openCourseAssets.v1 returns proper results.
It appears that normal way to put a supplement file into a page
is to put it inside an <asset> tag. However, in certain courses
files can be found in <a href> tags in a text of an assignment.
added download: all-lang-subtitles and subscripts to on-demand classes
From the pull request message:
----
> I added feature of downloading additional files for on-demand classes:
>
> * subscripts for video
> * special value "all" for download subtitles and subscripts of all available languages
> (it's changed filename, for example: file.srt->file.en.srt).
----
Thanks @klimenko for the nice contribution!
Add extraction of supplementary materials in on-demand courses.
From the pull request message:
----
> Supplementary materials are usually located on the first page of a
> module. These materials include: pdf, zip, ipynb, csv and other files.
> The code uses undocumented but "reverse engineered" (in browser) method:
> https://www.coursera.org/api/onDemandSupplements.v1. The code handles
> cases when there are multiple files of different formats (extensions)
> on the same page and downloads them all. This clearly benefits to the users
> which will now automatically get all the necessary files for practical
> assignments.
>
>One remarkable downside of this change is that new lecture is now added
> to the beginning of the list of lectures. It means that lectures will
> be renumbered, which will cause downloads of old lectures (even if they
> are already downloaded). I suggest removing ongoing on-demand courses
> and downloading them again (Internet connection permitting). Alternatively,
> one can use hardlink utility to find duplicate files and turn them into
> hardlinks to save up disk space.
>
> Another minor flaw is that occasionally supplementary HTML pages are
> downloaded and saved to filenames without extensions, e.g.
>
>> 01_reading-recommending-songs-assignment_graphlab.SFrame.groupby.html#graphlab.SFrame.groupby
>
>Technically, extension here is "groupby" but that looks ugly. I consider
> this to be a very minor issue. Anyway, suggestions are welcome.
----
Thanks to @balta2ar for the outstanding work with the support for on-demand
courses!
I also moved a couple of methods into a separate class CourseraOnDemand
to encapsulate all the tricks that are performed to extract links to
different types of resources.
Supplementary materials are usually located on the first page of a
module. These materials include: pdf, zip, ipynb, csv and other files.
The code uses undocumented but "reverse engineered" (in browser) method:
https://www.coursera.org/api/onDemandSupplements.v1. The code handles
cases when there are multiple files of different formats (extensions)
on the same page and downloads them all. This clearly benefits to the users
which will now automatically get all the necessary files for practical
assignments.
One remarkable downside of this change is that new lecture is now added
to the beginning of the list of lectures. It means that lectures will
be renumbered, which will cause downloads of old lectures (even if they
are already downloaded). I suggest removing ongoing on-demand courses
and downloading them again (Internet connection permitting). Alternatively,
one can use hardlink utility to find duplicate files and turn them into
hardlinks to save up disk space.
Another minor flaw is that occasionally supplementary HTML pages are
downloaded and saved to filenames without extensions, e.g.
> 01_reading-recommending-songs-assignment_graphlab.SFrame.groupby.html#graphlab.SFrame.groupby
Technically, extension here is "groupby" but that looks ugly. I consider
this to be a very minor issue. Anyway, suggestions are welcome.
Deprecate on demand option.
This nice pull request from @balta2ar stops the madness of having
the users know which course is of which type (which they really shouldn't
have to know, frankly).
From the original pull request message:
----
Users are very confused about `--on-demand` option. Sometimes it is required, sometimes it is not. This pull request deprecates the option. Instead, the script tries to detect course type automatically by trying all course types consecutively.
There are no tests for the feature but I think they should definitely be added. I'll do that once I figure out how to do it best. Please share your advices. For better testability, I moved consecutive course tries into a separate function.
----
This closes#387, closes#411, closes#412.
* Add link to coursera help page describing course types
* Print type of the course we're about to start downloading
(for possible future debuging and troubleshooting)
We have received a lot of issues where people are confused about
distinction between old-style and new-style (on-demand) courses.
Sometimes --on-demand option is required, sometimes it's not.
This patch deprecates --on-demand option in favour of automatic
detection. The detection is straight forward: first, try to treat
the course as an old-style, and if the course is not found, try to
treat it as a new-style (on-demand).
Corresponding messages were adjusted (login method). Help message
for --on-demand option now advices to remove that option.
Selectable resolution (fixes#389)
From my original pull request:
> This is a prototype to select the video resolution of courses. It,
> unfortunately, only works for on-demand courses (since I don't believe
> that regular courses offer videos in resolutions other than 540p yet).
>
> The code here sucks a lot, since the program started "growing organically"
> and we have some functions with more than 10 parameters, which is a real
> madness (and we are essentially, just mimicking passing the command line
> arguments object to the functions, but in a very poor style: "unboxing"
> the parameters).
