Mirrors/coursera-dl
1
0
Fork 0
mirror of https://github.com/coursera-dl/coursera-dl.git synced 2026-07-17 16:35:35 +00:00
Code Issues Projects Releases Wiki Activity
1131 commits 7 branches 23 tags 2.6 MiB
Commit graph

1131 commits

This branch
This branch
All branches
Author SHA1 Message Date
Yuri Bochkarev
9d29d1d19e Limit file name and extension length before downloading 
This is a quick and dirty solution. We need to come up with
something more general.

related #488
 2016-06-20 12:38:12 +03:00
Yuri Bochkarev
00c3042b8f Fix typo 2016-06-19 17:24:55 +03:00
Yuri Bochkarev
0ba2c5e72a Ignore HTTP errors when downloading about page in old-style course 
When running with option --process_local_page, this prevents from
downloading the course.

related #504
 2016-06-18 18:32:12 +03:00
Yuri Bochkarev
f4957293a3 Mention Chrome extension downloader in the README.md 2016-06-18 18:34:31 +04:00
Yuri Bochkarev
bc2614a9c7 Only prepend UNC prefix when necessary 2016-06-18 16:22:50 +03:00
Yuri Bochkarev
2b0e150b3d Add "Found 0 sections and 0 lectures on this page" Troubleshooting section 2016-06-18 16:44:12 +04:00
Yuri Bochkarev
c8b1c83d88 Add Table of Contents to README.md 
For easier navigation.
 2016-06-17 21:48:05 +04:00
Yuri Bochkarev
3942b81e74 Fix link to coverage in README.md header 2016-06-17 21:38:41 +04:00
Yuri Bochkarev
4eb1883986 Bump version (0.5.2 -> 0.6.0) 2016-06-17 20:30:03 +03:00
Yuri Bochkarev
623dec5d97 Fix links to PyPI badges in the header of README.md 2016-06-17 21:10:29 +04:00
Yuri Bochkarev
a9b8c718bf Mention "Failed to create process" error on Windows in README.md 
Thanks to @lorendunlop for providing the solution!

related #500
related #509
 2016-06-17 20:00:56 +03:00
Yuri Bochkarev
c2f318a571 Rename PyPI package name from "coursera" to "coursera-dl" 
This name corresponds to the name of our github repository.
Hopefully there will be less confusion about names.

I can't thank enough the author of "coursera-dl" PyPI package @dgorissen
who generously gave up that package to us. Thank you!
 2016-06-17 19:47:23 +03:00
Yuri Bochkarev
e9f62e6baa Fix some of pylint warnings 2016-06-17 18:14:09 +03:00
Yuri Bochkarev
126b058fe4 Update CHANGELOG.md 2016-06-17 17:57:57 +03:00
Yuri Bochkarev
b282c35961 Add workaround for "Filename is too long" on Windows 
The trick is to prepend "\\?\" to the file name and use absolute
paths. See MSDN docs for the details:
https://msdn.microsoft.com/en-us/library/aa365247%28v=vs.85%29.aspx#maxpath

fix #483
fix #476
fix #414
fix #406
 2016-06-17 17:50:11 +03:00
Yuri Bochkarev
0f7ea38427 Update CHANGELOG.md 2016-06-15 19:00:47 +03:00
Yuri Bochkarev
dcb06acf45 Merge branch 'master' of https://github.com/coursera-dl/coursera-dl 2016-06-14 21:30:23 +03:00
Yuri Bochkarev
65a2efe603 Add stricter filename cleaning to OnDemand course parser 
We've had numerous bug reports because OnDemand course parser
is leaving too many illegal characters behind. This commit adds
more calls to `clean_filename` in OnDemand parser.

fix #486
fix #481
fix #443
 2016-06-14 21:20:57 +03:00
Yuri Bochkarev
8e8ffb5de1 Merge pull request #498 from ReadmeCritic/master 
Update README URLs based on HTTP redirects
 2016-06-14 14:41:57 +03:00
Yuri Bochkarev
a2eae31f12 Shorten progress output and make it hierarchical-looking 
Berofe:
Processing lecture welcome-to-the-course/introduction/welcome
Processing lecture welcome-to-the-course/introduction/programming-resources
Processing lecture earthquakes-programming-and-interfaces/searching-earthquake-data/module-learning-outcomes-resources
Processing lecture earthquakes-programming-and-interfaces/searching-earthquake-data/introduction
Processing lecture earthquakes-programming-and-interfaces/searching-earthquake-data/relationships-between-classes

After:
Processing module  welcome-to-the-course
Processing section     introduction
Processing lecture         welcome
Processing lecture         programming-resources
Processing module  earthquakes-programming-and-interfaces
Processing section     searching-earthquake-data
Processing lecture         module-learning-outcomes-resources
Processing lecture         introduction
Processing lecture         relationships-between-classes
 2016-06-14 14:29:38 +03:00
Yuri Bochkarev
0ee048d24a Add "--ignore-http-errors" option 
This option allows internal downloader to ignore HTTP errors. Specifically,
requests.exceptions.RequestException is catched, thus all its descendants
can be ignored.

Please not that this option is only applicable to internal resource downloader.
Nor does this affect course syllabus parsing stage nor external downloaders.
Errors occured in the above scenarious are not handled.

If option is enabled, list of failed URLs is accumulated and printed after
downloading is finished.

fixes #489
fixes #453
 2016-06-14 12:16:02 +03:00
Yuri Bochkarev
1fd96c0e01 Refactor part of download_lectures into two functions 2016-06-13 22:25:01 +03:00
Yuri Bochkarev
0878df2c09 Add "--disable-url-skipping" option 
Related: #489, #486, #481, #414, #453
 2016-06-13 21:13:04 +03:00
Yuri Bochkarev
57e3cf02c1 Enhance URL filtering, print skipped URLs 
Certain URLs are grabbed during the parsing stage, but are
absolutely useless and should not be downloaded. The script
has been enhanced to a certain amount of heuristics to detect
URLs that should be skipped.

Skipped URLs are printed after the course has finished
downloading.
 2016-06-13 20:51:21 +03:00
Yuri Bochkarev
2df9dae934 More URL cleaning 2016-06-13 20:00:20 +03:00
Yuri Bochkarev
55d81f7dc4 Print meaningful lecture names instead of IDs 
Before:
Gathering video URLs for video_id <IAMfh21DEeWwmQpgO3010Q>.
Proceeding with download of resolution 540p of <IAMfh21DEeWwmQpgO3010Q>.
Gathering subtitle URLs for video_id <IAMfh21DEeWwmQpgO3010Q>.
Gathering transcript URLs for video_id <IAMfh21DEeWwmQpgO3010Q>.

After:
Processing lecture week-1-prediction-errors-and-cross-validation/week-1/welcome-to-practical-machine-learning
Processing lecture week-1-prediction-errors-and-cross-validation/week-1/syllabus
Processing lecture week-1-prediction-errors-and-cross-validation/week-1/pre-course-survey
Processing lecture week-1-prediction-errors-and-cross-validation/week-1/prediction-motivation
 2016-06-13 19:22:07 +03:00
Yuri Bochkarev
5618814bae Remove garbage from URL when extracting file extension 
Sometimes links may look as follows:
https://d28rh4a8wq0iu5.cloudfront.net/internetgiants/W1Microsoft.pdf?response-content-type=application%2Foctet-stream&a=1&response-content-disposition=attachment

Before we call os.path.splitext, we need to remove everything
after '?' and leaving only path part of the URL.
 2016-06-13 19:02:14 +03:00
ReadmeCritic
0275686899 Update README URLs based on HTTP redirects 2016-06-13 07:51:50 -07:00
Yuri Bochkarev
bdfd3738b3 Strip white space from extensions and URLs more in API 
For example, work-starter-not-harder course contains URLs
with trailing white space.
 2016-06-13 17:26:07 +03:00
Yuri Bochkarev
a0dd68e92d Refactor m3u playlist creation into a function 2016-06-12 22:19:50 +03:00
Yuri Bochkarev
2763681bf0 Add handler for SSLError exception 
The handler will print a URL where a user can find instructions on
how to fix SSL issue.

fix #480
fix #478
fix #377
fix #330
fix #329
 2016-06-12 20:01:15 +03:00
Yuri Bochkarev
074d03f297 Add Troubleshooting section for SSLError 2016-06-12 20:29:04 +04:00
Yuri Bochkarev
255abae04f Merge pull request #492 from balta2ar/more-programming-assignments 
Extract more programming assignments
 2016-06-12 13:34:06 +03:00
Yuri Bochkarev
495c4ae2e2 Fix conflicts 2016-06-12 13:21:07 +03:00
Yuri Bochkarev
0626cbdf41 Inject MathJax JS to render math in locally-saved instructions 
You still need the Internet to render math when you open instructions
later.
 2016-06-12 13:15:21 +03:00
Yuri Bochkarev
6bb46e9a0d Add missing import json 2016-06-12 13:15:21 +03:00
Yuri Bochkarev
b222d04230 Refactor common pattern into get_page_json utility function 2016-06-12 13:15:21 +03:00
Yuri Bochkarev
50684adc4a Download and base64 encode images in instructions 
Images are downloaded, base64-encoded and are inserted into
<img> tags. This allows for later offline viewing.
 2016-06-12 13:10:44 +03:00
Yuri Bochkarev
06c7a6d174 Extract instructions from programming assignments as well 2016-06-12 13:10:44 +03:00
Yuri Bochkarev
b5b42b6310 Extract text instructions from supplementary lectures 
Parsed and downloader have been extended to support instruction
text extraction. Instructions are very important because they
often contain the description of the task, URLs and so on.

Instructions are converted into a more HTML friendly markup.
 2016-06-12 13:10:44 +03:00
Yuri Bochkarev
2d527c391b Save some intermediate JSONs to files when running in DEBUG mode 2016-06-12 13:08:16 +03:00
Yuri Bochkarev
eedcb7ad6d Extract additional another variation of programming assignments 
There are courses (e.g. data-structures, algorithms-on-graphs) that
have almost empty modules in api/opencourse.v1/course response. Such
modules look as follows:

{ // this is a module
    "id": "QgCuM", // <- lecture id
    "slug": "programming-assignment",
    "name": "Programming Assignment",
    "timeCommitment": 0,
    "elements": []
}

Their elements are empty so it's not straightforward to extract
supplementary materials (such as zip archives, PDFs) from them.
To do that, one needs to call api/onDemandCourseMaterials.v1
and parse the result. There you will see something like this:

{
  "id": "AUd0k",
  "moduleId": "0MGvs",
  "lessonId": "QgCuM",
  "name": "Programming Assignment 1: Decomposition of Graphs",
  "slug": "programming-assignment-1-decomposition-of-graphs",
  "timeCommitment": 10800000,
  "content": {
    "typeName": "gradedProgramming",
    "definition": {
      "programmingAssignmentId": "zHzR5yhHEeaE0BKOcl4zJQ@2",
      "gradingWeight": 20
    }
  },
  "isLocked": true,
  "itemLockedReasonCode": "PREMIUM",
  "trackId": "core"
},

Result is substituted into the list of lectures and is parsed
as a normal lecture.
 2016-06-12 13:07:16 +03:00
Yuri Bochkarev
2dd8cdcc4a Merge pull request #497 from benjamintanweihao/patch-1 
Fix typo
 2016-06-12 11:05:08 +03:00
Benjamin Tan Wei Hao
f453566773 Fix typo 
dowload -> download
 2016-06-12 09:41:36 +08:00
Yuri Bochkarev
3edbb835e9 Merge pull request #496 from FiloSottile/master 
Detect old style pages redirecting to on-demand
 2016-06-11 22:20:24 +03:00
Yuri Bochkarev
adaac3b513 Merge pull request #493 from raonyguimaraes/master 
Filter our URLs that start with "mailto:" (fix #487)
 2016-06-11 22:11:00 +03:00
Filippo Valsorda
c849d1dc0f Detect old style pages redirecting to on-demand 
Fixes #495
 2016-06-11 20:03:30 +01:00
Raony Guimarães
aa9f8ec4ff fix issue #487 2016-06-11 11:59:11 +02:00
Yuri Bochkarev
8333a925e5 Add first version of CHANGELOG.md 2016-06-10 19:55:38 +03:00
Yuri Bochkarev
edae0980b3 Add option --downloader-arguments 
This option allows user to pass additional arguments to the downloader.

Usage:
```
coursera-dl --aria --downloader-arguments='--max-download-limit=5K' -n --path . <course_name>
coursera-dl --wget --downloader-arguments='--limit-rate=5k' -n --path . parprog1 <course_name>
```

fix #464
 2016-06-10 19:31:01 +03:00