added download: all-lang-subtitles and subscripts to on-demand classes
From the pull request message:
----
> I added feature of downloading additional files for on-demand classes:
>
> * subscripts for video
> * special value "all" for download subtitles and subscripts of all available languages
> (it's changed filename, for example: file.srt->file.en.srt).
----
Thanks @klimenko for the nice contribution!
Add extraction of supplementary materials in on-demand courses.
From the pull request message:
----
> Supplementary materials are usually located on the first page of a
> module. These materials include: pdf, zip, ipynb, csv and other files.
> The code uses undocumented but "reverse engineered" (in browser) method:
> https://www.coursera.org/api/onDemandSupplements.v1. The code handles
> cases when there are multiple files of different formats (extensions)
> on the same page and downloads them all. This clearly benefits to the users
> which will now automatically get all the necessary files for practical
> assignments.
>
>One remarkable downside of this change is that new lecture is now added
> to the beginning of the list of lectures. It means that lectures will
> be renumbered, which will cause downloads of old lectures (even if they
> are already downloaded). I suggest removing ongoing on-demand courses
> and downloading them again (Internet connection permitting). Alternatively,
> one can use hardlink utility to find duplicate files and turn them into
> hardlinks to save up disk space.
>
> Another minor flaw is that occasionally supplementary HTML pages are
> downloaded and saved to filenames without extensions, e.g.
>
>> 01_reading-recommending-songs-assignment_graphlab.SFrame.groupby.html#graphlab.SFrame.groupby
>
>Technically, extension here is "groupby" but that looks ugly. I consider
> this to be a very minor issue. Anyway, suggestions are welcome.
----
Thanks to @balta2ar for the outstanding work with the support for on-demand
courses!
I also moved a couple of methods into a separate class CourseraOnDemand
to encapsulate all the tricks that are performed to extract links to
different types of resources.
Supplementary materials are usually located on the first page of a
module. These materials include: pdf, zip, ipynb, csv and other files.
The code uses undocumented but "reverse engineered" (in browser) method:
https://www.coursera.org/api/onDemandSupplements.v1. The code handles
cases when there are multiple files of different formats (extensions)
on the same page and downloads them all. This clearly benefits to the users
which will now automatically get all the necessary files for practical
assignments.
One remarkable downside of this change is that new lecture is now added
to the beginning of the list of lectures. It means that lectures will
be renumbered, which will cause downloads of old lectures (even if they
are already downloaded). I suggest removing ongoing on-demand courses
and downloading them again (Internet connection permitting). Alternatively,
one can use hardlink utility to find duplicate files and turn them into
hardlinks to save up disk space.
Another minor flaw is that occasionally supplementary HTML pages are
downloaded and saved to filenames without extensions, e.g.
> 01_reading-recommending-songs-assignment_graphlab.SFrame.groupby.html#graphlab.SFrame.groupby
Technically, extension here is "groupby" but that looks ugly. I consider
this to be a very minor issue. Anyway, suggestions are welcome.
Deprecate on demand option.
This nice pull request from @balta2ar stops the madness of having
the users know which course is of which type (which they really shouldn't
have to know, frankly).
From the original pull request message:
----
Users are very confused about `--on-demand` option. Sometimes it is required, sometimes it is not. This pull request deprecates the option. Instead, the script tries to detect course type automatically by trying all course types consecutively.
There are no tests for the feature but I think they should definitely be added. I'll do that once I figure out how to do it best. Please share your advices. For better testability, I moved consecutive course tries into a separate function.
----
This closes#387, closes#411, closes#412.
* Add link to coursera help page describing course types
* Print type of the course we're about to start downloading
(for possible future debuging and troubleshooting)
We have received a lot of issues where people are confused about
distinction between old-style and new-style (on-demand) courses.
Sometimes --on-demand option is required, sometimes it's not.
This patch deprecates --on-demand option in favour of automatic
detection. The detection is straight forward: first, try to treat
the course as an old-style, and if the course is not found, try to
treat it as a new-style (on-demand).
Corresponding messages were adjusted (login method). Help message
for --on-demand option now advices to remove that option.
Selectable resolution (fixes#389)
From my original pull request:
> This is a prototype to select the video resolution of courses. It,
> unfortunately, only works for on-demand courses (since I don't believe
> that regular courses offer videos in resolutions other than 540p yet).
>
> The code here sucks a lot, since the program started "growing organically"
> and we have some functions with more than 10 parameters, which is a real
> madness (and we are essentially, just mimicking passing the command line
> arguments object to the functions, but in a very poor style: "unboxing"
> the parameters).
Python 2 and Python 3 disagree on the exception thrown when
`subprocess.call` can't find the program to execute.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
This is utterly disgusting and we desperately need a refactoring right now
to avoid this code duplication madness!
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
This is clearly marked to be only for on-demand videos and only a few
resolutions allowed (but we actually don't filter the resolutions, since,
say, coursera may start offering 1080p videos and the user might very well
try to use that before we have an opportunity to update the program).
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
This is still not yet hooked up with any command-line option, but that's the
easiest step.
Notice that we use '540p' as the default value for the resolution, to match
the value for regular, time-based, not on-demand courses.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
We should probably substitute the video_id thing with a human-readable
string, like the name of the lecture. It is too ugly and uninformative the
way that it is at present.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
Due to popular demand, here are a few selected hints on how to get started
contributing with git/github in very few steps.
I may include more commands at a future date, depending on the reaction to
these notes.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
[skip-ci]
This is a cropped version of what Victor Westmann (@victorwestmann) sent me
privately via e-mail.
About the "cropped" part above: it is a cropped version because his source
file had other proposals, but we are keeping only the one that we will use.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
While not strictly true, it will save me a lot of headaches compared to the
situation of people using very old dependencies. Generally, it is preferred
to have newer versions than to have older dependencies.
I don't know if `pip` supports the equivalent of Ruby's Gemfiles `~=`
operator to specify versions that don't break API compatibility with the
version specified.
I saw that mentioned in [PEP-400][pep400], but I don't know if it is
implemented already.
[pep440]: https://www.python.org/dev/peps/pep-0440/#compatible-release
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
While not strictly required (I use Python 2 or Python 3 about 50% of the
time each), Python 3 has better SSL/TLS support, especially after all the
attacks on SSLv3.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
We have seen a significant rise in people asking if the program supports
On-Demand courses (or even claiming that it doesn't work).
See issues #386, #380, #379, #333, among many others.
This ability was contributed by @MXAmin quite some time ago and we even have
it documented in our README file, right there at the top, in the "Features"
section.
Hopefully this makes it more explicit that we *do support* such courses.
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
This commit makes execution of setup.py generate the README.rst file by
calling pandoc.
We only stop execution of the program if README.md can not be converted into
a README.rst file, in which case we execute the show a message to the user.
I hope that, in the future, PyPI can accept Markdown files and that we can
stop having to convert between formats (and, also, get a more faithful
reproduction of the intended markup, since some of the conversions are,
unfortunately, lossy).
Signed-off-by: Rogério Brito <rbrito@ime.usp.br>
