uppy/packages/@uppy
Mikael Finstad 56339fc3e5
Rewrite Companion providers to use streams to allow simultaneous upload/download without saving to disk (#3159)
* rewrite to async/await

* Only fetch size (HEAD) if needed #3034

* Update packages/@uppy/companion/src/server/controllers/url.js

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

* Change HEAD to GET in getURLMeta

and abort request immediately upon response headers received
https://github.com/transloadit/uppy/issues/3034#issuecomment-908059234

* fix lint

* fix lint

* cut off length of file names

or else we get
"MetadataTooLarge: Your metadata headers exceed the maximum allowed metadata size" in tus / S3

* try to fix flaky test

* remove iife and cleanup code a bit

* fix lint by reordering code

* rename Uploader to MultipartUploader

* Rewrite Uploader to use fs-capacitor #3098

This allows for upload to start almost immediately without having to first download the file.
And it allows for uploading bigger files, because transloadit assembly will not timeout,
as it will get upload progress events all the time.
No longer need for illusive progress.
Also fix eslint warnings and simplify logic

Still TODO: TUS pause/resume has a bug:
https://github.com/tus/tus-js-client/issues/275

* add comment in dev Dashboard and pull out variable

* fix a bug where remote xhr upload would ignore progress events in the UI

* fix bug where s3 multipart cancel wasn't working

* fix also cancel for xhr

* Rewrite providers to use streams

This removes the need for disk space as data will be buffered in memory and backpressure will be respected
https://github.com/transloadit/uppy/issues/3098#issuecomment-907763809
All providers "download" methods will now return a { stream } which can be consumed by uploader.

Also:
- Remove capacitor (no longer needed)
- Change Provider/SearchProvider API to async (Breaking change for custom companion providers)
- Fix the case with unknown length streams (zoom / google drive). Need to be downloaded first
- rewrite controllers deauth-callback, thumbnail, list, logout to async
- getURLMeta: make sure size is never NaN (NaN gets converted to null in JSON.stringify when sent to client but not when used in backend)
- fix purest mock (it wasn't returning statusCode on('response'))
- add missing http mock for "request" for THUMBNAIL_URL and http://url.myendpoint.com/file (these request errors were never caught by tests previously)
- "upload functions with tus protocol" test: move filename checking to new test where size is null. Fix broken expects
- fix some lint

* Implement streamingUpload flag

COMPANION_STREAMING_UPLOAD
Default to false due to backward compatibility
If set to true, will start to upload files at the same time as dowlnoading them, by piping the streams

- Also implement progress for downloading too
- and fix progress duplication logic
- fix test that assumed file was fully downloaded after first progress event

* rearrange validation logic

* add COMPANION_STREAMING_UPLOAD to env.test.sh too

* implement maxFileSize option in companion

for both unknown length and known length downloads

* fix bug

* fix memory leak when non 200 status

streams were being kept

* fix lint

* Add backward-compatibility for companion providers

Implement a new static field "version" on providers, which when not set to 2,
will cause a compatibility layer to be added for supporting old callback style provider api

also fix some eslint and rename some vars

* document new provider API

* remove static as it doesn't work on node 10

* try to fix build issue

* degrade to node 14 in github actions

due to hitting this error: https://github.com/nodejs/node/issues/40030
https://github.com/transloadit/uppy/pull/3159/checks?check_run_id=3544858518

* pull out duplicated logic into reusable function

* fix lint

* make methods private

* re-add unsplash download_location request

got lost in merge

* add try/catch

as suggested https://github.com/transloadit/uppy/pull/3159#discussion_r727149263

* Only set default chunkSize if needed

for being more compliant with previous behavior when streamingUpload = false

* Improve flaky test

Trying to fix this error:

FAIL packages/@uppy/utils/src/delay.test.js
  ● delay › should reject when signal is aborted

    expect(received).toBeLessThan(expected)

    Expected: < 70
    Received:   107

      32 |     const time = Date.now() - start
      33 |     expect(time).toBeGreaterThanOrEqual(30)
    > 34 |     expect(time).toBeLessThan(70)
         |                  ^
      35 |   })
      36 | })
      37 |

      at Object.<anonymous> (packages/@uppy/utils/src/delay.test.js:34:18)

https://github.com/transloadit/uppy/runs/3984613454?check_suite_focus=true

* Apply suggestions from code review

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

* fix review feedback & lint

* Apply suggestions from code review

Co-authored-by: Merlijn Vos <merlijn@soverin.net>

* remove unneeded ts-ignore

* Update packages/@uppy/companion/src/server/controllers/url.js

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

* Update packages/@uppy/companion/src/server/Uploader.js

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

* reduce nesting

* fix lint

* optimize promisify

https://github.com/transloadit/uppy/pull/3159#discussion_r738160576

* Update packages/@uppy/companion/test/__tests__/uploader.js

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Co-authored-by: Merlijn Vos <merlijn@soverin.net>
2021-11-01 16:35:25 +07:00
..
angular meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
aws-s3 meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
aws-s3-multipart Rewrite Companion providers to use streams to allow simultaneous upload/download without saving to disk (#3159) 2021-11-01 16:35:25 +07:00
box meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
companion Rewrite Companion providers to use streams to allow simultaneous upload/download without saving to disk (#3159) 2021-11-01 16:35:25 +07:00
companion-client meta: use workspace:^ instead of workspace:* (#3278) 2021-10-27 17:53:11 +02:00
core chore: Add FileRemoveReason type (#3283) 2021-10-31 19:23:13 +01:00
dashboard meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
drag-drop meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
drop-target meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
dropbox meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
facebook meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
file-input meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
form meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
golden-retriever meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
google-drive meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
image-editor meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
informer meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
instagram meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
locales Release 2021-10-14 21:58:29 +01:00
onedrive meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
progress-bar meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
provider-views meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
react meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
react-native meta: use workspace:^ instead of workspace:* (#3278) 2021-10-27 17:53:11 +02:00
redux-dev-tools meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
robodog meta: use workspace:^ instead of workspace:* (#3278) 2021-10-27 17:53:11 +02:00
screen-capture meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
status-bar meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
store-default Release 2021-10-14 21:58:29 +01:00
store-redux Release 2021-10-14 21:58:29 +01:00
svelte meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
thumbnail-generator meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
transloadit meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
tus meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
unsplash meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
url meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
utils Rewrite Companion providers to use streams to allow simultaneous upload/download without saving to disk (#3159) 2021-11-01 16:35:25 +07:00
vue meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
webcam meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00
xhr-upload Rewrite Companion providers to use streams to allow simultaneous upload/download without saving to disk (#3159) 2021-11-01 16:35:25 +07:00
zoom meta: use workspace:^ in peerDependencies (#3278) 2021-10-27 17:53:16 +02:00