New concept "simple auth" - authentication that happens immediately (in one http request) without redirecting to any third party.
uppyAuthToken initially used to simply contain an encrypted & json encoded OAuth2 access_token for a specific provider. Then we added refresh tokens as well inside uppyAuthToken #4448. Now we also allow storing other state or parameters needed for that specific provider, like username, password, host name, webdav URL etc... This is needed for providers like webdav, ftp etc, where the user needs to give some more input data while authenticating
Companion:
- `providerTokens` has been renamed to `providerUserSession` because it now includes not only tokens, but a user's session with a provider.
Companion `Provider` class:
- New `hasSimpleAuth` static boolean property - whether this provider uses simple auth
- uppyAuthToken expiry default 24hr again for providers that don't support refresh tokens
- make uppyAuthToken expiry configurable per provider - new `authStateExpiry` static property (defaults to 24hr)
- new static property `grantDynamicToUserSession`, allows providers to specify which state from Grant `dynamic` to include into the provider's `providerUserSession`.
Co-authored-by: Mikael Finstad <finstaden@gmail.com>
Co-authored-by: Nick Rutten <2504906+nickrttn@users.noreply.github.com>
Co-authored-by: Murderlon <merlijn@soverin.net>
Co-authored-by: Artur Paikin <artur@arturpaikin.com>
* respond with 500 if unhandled upload error
400 seems inappropriate
* add todo
* fix companion download error
respond with correct response code when calling /get
and request to the upstream server fail
also add a unit test for this
* forward 429 from provider
allows us to retry it
* rename fn
* add a way to test refresh tokens
* fix race condtiion with refreshing token
* implement retry and fix socket
- Make sure we don't instantiate a new Provider for every file uploaded (reuse the original provider instead) - this caused the refresh token synchronization not to work
- Retry most errors when uploading (`/get`) using companion, but for HTTP, only retry certain HTTP status codes
- Reimplement the websocket code to be more stable and retry as well
* remove unused socket wrapper
* remove useFastRemoteRetry
because it isn't supported in Companion as far as I can tell
* fix error handling
* remove unneeded logic
* apply review suggestion
* retry awaiting for companion socket too
* retry whole operation instead of individually
or else we risk getting stuck retrying just half of the operation, which might never recover
* improvements
- rewrite so we can use p-retry for exponential backoff for socket reconnect too
- improve logging
- simplify/reuse code
* Update packages/@uppy/utils/src/fetchWithNetworkError.js
Co-authored-by: Antoine du Hamel <antoine@transloadit.com>
* Explain requestClient Symbol with comment
Co-authored-by: Mikael Finstad <finstaden@gmail.com>
* Update packages/@uppy/provider-views/src/View.js
* fixes
preventing socket.send when the socket is not in the connected state
catch errors in handlers
* log if trying to refresh with no token
* log whether we got a refresh token
* dont log retrying when in reality not
* always prompt: consent
for google drive
* add signal to post
* refactor away eventmanager
and event handler error catching
* don't fail with upload-error
if a file is canceled
* fix expect
* add capabilities support
* make requestClient non-enumerable
---------
Co-authored-by: Antoine du Hamel <antoine@transloadit.com>
Co-authored-by: Merlijn Vos <merlijn@soverin.net>
* Add VirtualList to ProviderView
* Import correctly from lib
* fixup! Import correctly from lib
* fixup! fixup! Import correctly from lib
* Really do it this time
* Only virtual list for lists
* fixup! Only virtual list for lists
* Fix scroll
* Set hardcoded width/height for img, as recommened
---------
Co-authored-by: Artur Paikin <artur@arturpaikin.com>
Fixes: https://github.com/transloadit/uppy/issues/4272
Co-authored-by: R <r@r.com>
Co-authored-by: Mikael Finstad <finstaden@gmail.com>
Co-authored-by: Antoine du Hamel <antoine@transloadit.com>
* When file is removed (or all are canceled), controller.abort queued requests
* add controller.abort to all other remote uploaders too
* fix the queue
---------
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* show how many files are added when loading
remake of https://github.com/transloadit/uppy/pull/4388
* add french (cherry pick)
* implement concurrent file listing
* refactor / fix lint
* refactor/reduce duplication
* pull out totals validation
don't do it for every file added, as it's very slow
instead do the check at the end when all files are added.
this allows us to easily work with 10k+ files
fixes#4389
* Update packages/@uppy/core/src/Uppy.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* make restricter.validate validate everything
instead make more specific methods for sub-validation
also rename validateTotals to validateAggregateRestrictions
* improve errors and user feedback
- handle errors centrally so that we can limit the amount of toasts (informers) sent to the users (prevent flooding hundreds/thousands of them)
- introduce FileRestrictionError which is a restriction error for a specific file
- introduce isUserFacing field for RestrictionError
* fix performance issue reintroduced
* improvements
- show "%{count} additional restrictions were not fulfilled" for any restriction errors more than 4
- refactor/rename methods
- improve ghost logic/comments
* improve performance when uploading
- introduce new event "upload-start" that can contain multiple files
- make a new patchFilesState method to allow updating more files
- unify "upload-start" logic in all plugins (send it before files start uploading)
- defer slicing buffer until we need the data
- refactor to reuse code
* fix e2e build issue
* try to upgrade cypress
maybe it fixes the error
* Revert "fix e2e build issue"
This reverts commit ff3e580c0f.
* upgrade parcel
* move mutation logic to end
* remove FileRestrictionError
merge it with RestrictionError
* fix silly bug
looks like the e2e tests are doing its job 👏
---------
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* fix race condtion when adding files
don't call addFolder from listAllFiles
because that would call addFile before all folders were loaded
also remove unused selectedFolders state
* fix also the case of adding multiple folders
* fix todo: remove SharedHandler
* remove loaderWrapper
* fix logic
* fix the last race condition
* fix broken duplicate file check
* fix logic
* prettiyfy loop
* add user feedback
so they know that something is happening
* run corepack yarn run build:locale-pack
* Revert "run corepack yarn run build:locale-pack"
This reverts commit 85548ce2fc.
* Revert "add user feedback"
This reverts commit 4060019c35.
* use async generator instead of p-map
* re-fix race-condition
* remove providerFileToId
as suggested by @arturi
* use addFiles instead of addFile
* rename function
* use provider-supplied file ID
instead of generating an ID, for providers that we whitelsit
this allows adding the same time many times (with a different path)
* call core directly
* improve dev dashboard
* disable experimental getAsFileSystemHandle
it seems to be causing problems when dropping folders with subfolders in newest chrome
e.g a folder with 50 files and a subfolder which in turn has another 50 files
also refactor and document the code more
* Update packages/@uppy/provider-views/src/ProviderView/ProviderView.jsx
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* mov eto utils
---------
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* workaround chrome crash
fixes#4133
* Update packages/@uppy/utils/src/getDroppedFiles/utils/webkitGetAsEntryApi/index.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
---------
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
`webkitGetAsEntry` is a non-standard/deprecated API, replacing it with
`getAsFileSystemHandle` when available.
This also work around a Chromium bug with symlinks.
Fixes: https://github.com/transloadit/uppy/issues/3505.
The `ErrorWithCause` class uses the same signature as the builtin Error
does, but it supports the `{ cause }` option in older environments, and
it propagates the `isNetworkError` property from the underlying error.
* @uppy/tus: pause all requests in response to server rate limiting
When the remote server responds with HTTP 429, all requests are
paused for a while in the hope that it can resolve the rate limiting.
Failed requests are also now queued up after the retry delay. Before
that, they were simply scheduled which would sometimes end up
overflowing the `limit` option.
* Address review comments
* fix requests bypassing queue pause state
* Auto rate limiting
* fix `RateLimitedQueue`
* rewrite to async/await
* Only fetch size (HEAD) if needed #3034
* Update packages/@uppy/companion/src/server/controllers/url.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* Change HEAD to GET in getURLMeta
and abort request immediately upon response headers received
https://github.com/transloadit/uppy/issues/3034#issuecomment-908059234
* fix lint
* fix lint
* cut off length of file names
or else we get
"MetadataTooLarge: Your metadata headers exceed the maximum allowed metadata size" in tus / S3
* try to fix flaky test
* remove iife and cleanup code a bit
* fix lint by reordering code
* rename Uploader to MultipartUploader
* Rewrite Uploader to use fs-capacitor #3098
This allows for upload to start almost immediately without having to first download the file.
And it allows for uploading bigger files, because transloadit assembly will not timeout,
as it will get upload progress events all the time.
No longer need for illusive progress.
Also fix eslint warnings and simplify logic
Still TODO: TUS pause/resume has a bug:
https://github.com/tus/tus-js-client/issues/275
* add comment in dev Dashboard and pull out variable
* fix a bug where remote xhr upload would ignore progress events in the UI
* fix bug where s3 multipart cancel wasn't working
* fix also cancel for xhr
* Rewrite providers to use streams
This removes the need for disk space as data will be buffered in memory and backpressure will be respected
https://github.com/transloadit/uppy/issues/3098#issuecomment-907763809
All providers "download" methods will now return a { stream } which can be consumed by uploader.
Also:
- Remove capacitor (no longer needed)
- Change Provider/SearchProvider API to async (Breaking change for custom companion providers)
- Fix the case with unknown length streams (zoom / google drive). Need to be downloaded first
- rewrite controllers deauth-callback, thumbnail, list, logout to async
- getURLMeta: make sure size is never NaN (NaN gets converted to null in JSON.stringify when sent to client but not when used in backend)
- fix purest mock (it wasn't returning statusCode on('response'))
- add missing http mock for "request" for THUMBNAIL_URL and http://url.myendpoint.com/file (these request errors were never caught by tests previously)
- "upload functions with tus protocol" test: move filename checking to new test where size is null. Fix broken expects
- fix some lint
* Implement streamingUpload flag
COMPANION_STREAMING_UPLOAD
Default to false due to backward compatibility
If set to true, will start to upload files at the same time as dowlnoading them, by piping the streams
- Also implement progress for downloading too
- and fix progress duplication logic
- fix test that assumed file was fully downloaded after first progress event
* rearrange validation logic
* add COMPANION_STREAMING_UPLOAD to env.test.sh too
* implement maxFileSize option in companion
for both unknown length and known length downloads
* fix bug
* fix memory leak when non 200 status
streams were being kept
* fix lint
* Add backward-compatibility for companion providers
Implement a new static field "version" on providers, which when not set to 2,
will cause a compatibility layer to be added for supporting old callback style provider api
also fix some eslint and rename some vars
* document new provider API
* remove static as it doesn't work on node 10
* try to fix build issue
* degrade to node 14 in github actions
due to hitting this error: https://github.com/nodejs/node/issues/40030https://github.com/transloadit/uppy/pull/3159/checks?check_run_id=3544858518
* pull out duplicated logic into reusable function
* fix lint
* make methods private
* re-add unsplash download_location request
got lost in merge
* add try/catch
as suggested https://github.com/transloadit/uppy/pull/3159#discussion_r727149263
* Only set default chunkSize if needed
for being more compliant with previous behavior when streamingUpload = false
* Improve flaky test
Trying to fix this error:
FAIL packages/@uppy/utils/src/delay.test.js
● delay › should reject when signal is aborted
expect(received).toBeLessThan(expected)
Expected: < 70
Received: 107
32 | const time = Date.now() - start
33 | expect(time).toBeGreaterThanOrEqual(30)
> 34 | expect(time).toBeLessThan(70)
| ^
35 | })
36 | })
37 |
at Object.<anonymous> (packages/@uppy/utils/src/delay.test.js:34:18)
https://github.com/transloadit/uppy/runs/3984613454?check_suite_focus=true
* Apply suggestions from code review
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* fix review feedback & lint
* Apply suggestions from code review
Co-authored-by: Merlijn Vos <merlijn@soverin.net>
* remove unneeded ts-ignore
* Update packages/@uppy/companion/src/server/controllers/url.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* Update packages/@uppy/companion/src/server/Uploader.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* reduce nesting
* fix lint
* optimize promisify
https://github.com/transloadit/uppy/pull/3159#discussion_r738160576
* Update packages/@uppy/companion/test/__tests__/uploader.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Co-authored-by: Merlijn Vos <merlijn@soverin.net>
* rewrite to async/await
* Only fetch size (HEAD) if needed #3034
* Update packages/@uppy/companion/src/server/controllers/url.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* Change HEAD to GET in getURLMeta
and abort request immediately upon response headers received
https://github.com/transloadit/uppy/issues/3034#issuecomment-908059234
* fix lint
* fix lint
* cut off length of file names
or else we get
"MetadataTooLarge: Your metadata headers exceed the maximum allowed metadata size" in tus / S3
* try to fix flaky test
* remove iife and cleanup code a bit
* fix lint by reordering code
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* fix: AbortController is not defined
* use globalThis
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* add stylelint with stylelint-config-standard
* autofix style issues
* fix more linting issues
* add stylelint-scss
* fix two more
* add stylelint-config-rational-order and autofix order
https://github.com/constverum/stylelint-config-rational-order
