* @uppy/tus: pause all requests in response to server rate limiting
When the remote server responds with HTTP 429, all requests are
paused for a while in the hope that it can resolve the rate limiting.
Failed requests are also now queued up after the retry delay. Before
that, they were simply scheduled which would sometimes end up
overflowing the `limit` option.
* Address review comments
* fix requests bypassing queue pause state
* Auto rate limiting
* fix `RateLimitedQueue`
* rewrite to async/await
* Only fetch size (HEAD) if needed #3034
* Update packages/@uppy/companion/src/server/controllers/url.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* Change HEAD to GET in getURLMeta
and abort request immediately upon response headers received
https://github.com/transloadit/uppy/issues/3034#issuecomment-908059234
* fix lint
* fix lint
* cut off length of file names
or else we get
"MetadataTooLarge: Your metadata headers exceed the maximum allowed metadata size" in tus / S3
* try to fix flaky test
* remove iife and cleanup code a bit
* fix lint by reordering code
* rename Uploader to MultipartUploader
* Rewrite Uploader to use fs-capacitor #3098
This allows for upload to start almost immediately without having to first download the file.
And it allows for uploading bigger files, because transloadit assembly will not timeout,
as it will get upload progress events all the time.
No longer need for illusive progress.
Also fix eslint warnings and simplify logic
Still TODO: TUS pause/resume has a bug:
https://github.com/tus/tus-js-client/issues/275
* add comment in dev Dashboard and pull out variable
* fix a bug where remote xhr upload would ignore progress events in the UI
* fix bug where s3 multipart cancel wasn't working
* fix also cancel for xhr
* Rewrite providers to use streams
This removes the need for disk space as data will be buffered in memory and backpressure will be respected
https://github.com/transloadit/uppy/issues/3098#issuecomment-907763809
All providers "download" methods will now return a { stream } which can be consumed by uploader.
Also:
- Remove capacitor (no longer needed)
- Change Provider/SearchProvider API to async (Breaking change for custom companion providers)
- Fix the case with unknown length streams (zoom / google drive). Need to be downloaded first
- rewrite controllers deauth-callback, thumbnail, list, logout to async
- getURLMeta: make sure size is never NaN (NaN gets converted to null in JSON.stringify when sent to client but not when used in backend)
- fix purest mock (it wasn't returning statusCode on('response'))
- add missing http mock for "request" for THUMBNAIL_URL and http://url.myendpoint.com/file (these request errors were never caught by tests previously)
- "upload functions with tus protocol" test: move filename checking to new test where size is null. Fix broken expects
- fix some lint
* Implement streamingUpload flag
COMPANION_STREAMING_UPLOAD
Default to false due to backward compatibility
If set to true, will start to upload files at the same time as dowlnoading them, by piping the streams
- Also implement progress for downloading too
- and fix progress duplication logic
- fix test that assumed file was fully downloaded after first progress event
* rearrange validation logic
* add COMPANION_STREAMING_UPLOAD to env.test.sh too
* implement maxFileSize option in companion
for both unknown length and known length downloads
* fix bug
* fix memory leak when non 200 status
streams were being kept
* fix lint
* Add backward-compatibility for companion providers
Implement a new static field "version" on providers, which when not set to 2,
will cause a compatibility layer to be added for supporting old callback style provider api
also fix some eslint and rename some vars
* document new provider API
* remove static as it doesn't work on node 10
* try to fix build issue
* degrade to node 14 in github actions
due to hitting this error: https://github.com/nodejs/node/issues/40030https://github.com/transloadit/uppy/pull/3159/checks?check_run_id=3544858518
* pull out duplicated logic into reusable function
* fix lint
* make methods private
* re-add unsplash download_location request
got lost in merge
* add try/catch
as suggested https://github.com/transloadit/uppy/pull/3159#discussion_r727149263
* Only set default chunkSize if needed
for being more compliant with previous behavior when streamingUpload = false
* Improve flaky test
Trying to fix this error:
FAIL packages/@uppy/utils/src/delay.test.js
● delay › should reject when signal is aborted
expect(received).toBeLessThan(expected)
Expected: < 70
Received: 107
32 | const time = Date.now() - start
33 | expect(time).toBeGreaterThanOrEqual(30)
> 34 | expect(time).toBeLessThan(70)
| ^
35 | })
36 | })
37 |
at Object.<anonymous> (packages/@uppy/utils/src/delay.test.js:34:18)
https://github.com/transloadit/uppy/runs/3984613454?check_suite_focus=true
* Apply suggestions from code review
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* fix review feedback & lint
* Apply suggestions from code review
Co-authored-by: Merlijn Vos <merlijn@soverin.net>
* remove unneeded ts-ignore
* Update packages/@uppy/companion/src/server/controllers/url.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* Update packages/@uppy/companion/src/server/Uploader.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* reduce nesting
* fix lint
* optimize promisify
https://github.com/transloadit/uppy/pull/3159#discussion_r738160576
* Update packages/@uppy/companion/test/__tests__/uploader.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Co-authored-by: Merlijn Vos <merlijn@soverin.net>
* rewrite to async/await
* Only fetch size (HEAD) if needed #3034
* Update packages/@uppy/companion/src/server/controllers/url.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* Change HEAD to GET in getURLMeta
and abort request immediately upon response headers received
https://github.com/transloadit/uppy/issues/3034#issuecomment-908059234
* fix lint
* fix lint
* cut off length of file names
or else we get
"MetadataTooLarge: Your metadata headers exceed the maximum allowed metadata size" in tus / S3
* try to fix flaky test
* remove iife and cleanup code a bit
* fix lint by reordering code
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* fix: AbortController is not defined
* use globalThis
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
* add stylelint with stylelint-config-standard
* autofix style issues
* fix more linting issues
* add stylelint-scss
* fix two more
* add stylelint-config-rational-order and autofix order
https://github.com/constverum/stylelint-config-rational-order
We can now use `Object.fromEntries`, `Object.values`, and `Symbol`.
Also changes polyfill suggestion for legacy browsers to use `core-js`
which includes almost all we need and more.
* Remove IE polyfills and special casing
* dashboard: remove `resize-observer-polyfill`
* deps: remove url-parse to use global `URL` instead
* Update documentation for browser support
* Add a migration guide
* Update e2e test platforms
* Remove polyfills from examples and add those missing in CDN bundle
* Use Node.js v16.x on CI because of `AbortController`
* Update bundle builder to produce two builds: `uppy.js` without polyfills, `uppy.legacy.js` with polyfills.
* Upgrade create-react-app deps
* remove null coalescing operator as it's not supported by the build chain
* disable Safari testing (for now)
Co-authored-by: Renée Kooi <renee@kooi.me>
* Translator: refactor interpolate
Fixes types and small code edits:
`<regex>[Symbol.split](<string>)` is equivalent to
`String.prototype.split.call(<string>, <regex>)` and potentially more
readable (the downside it's not compatible with IE, which may be why we
currently use the latter).
Refs: https://github.com/transloadit/uppy/pull/2899
* Update packages/@uppy/utils/src/Translator.js
* Revert RegExp Symbol for IE compat
* move findIndex to @uppy/utils, add tests
* use Array.findIndex instead of Array.indexOf in uppy.removePlugin
because strict comparison started failing with Vue3 Proxies
* dashboard: fix modal position on rtl pages, fixes#2495
* Align text to the right on right-to-left pages
* core: fix default text alignment on right-to-left pages
* dashboard: fix margin in provider browser on right-to-left pages
* dashboard: use direction-aware offset properties
* provider-views: use logical offset CSS properties
* status-bar: use logical offset CSS properties
* dashboard: fix addMore icon margin on RTL pages
* dashboard: some more logical properties
* build: compile logical properties to old CSS
* dashboard: set dir attribute if page did not provide one
* status-bar: set fallback text direction if not provided
* support direction option for inline and modal dashboard
* use companion.uppy.io on deploy previews
* provider-views: fix RTL alignment of file icons
* put the dir attribute on a wrapper div
* fix icon alignment in RTL provider views
* informer: fix ? alignment in RTL mode
* add passesRestrictions(file) and utilize it in provider-views
* disallow selecting more than maxNumberOfFiles
* add restriction reason and include folders
* let’s leave folder out of this for now — otherwise not only you can’t select them, but navigate too
* disallow selecting if totalCurrentSelectionFileSize is over maxTotalFileSize
* handle maxNumberOfFiles not being set 🙈
* fix totalCurrentSelectionFileSize and handle no file.name
* support grid in restrictions
* refactor the restriction reason logic, rename to validateRestrictions, wrap icon in a container
* add uppyFileSizes 🙈
* get rid of canSelectMore in favor of validateRestrictions (file, files)
* handle shift key bulk selection
* remove unused props
* use infoTimeout
I added support to load thumbnails when uploading AVIF images, by adding 'avif' to the regex. I also updated the unit test.
Note that support for this is dependant on browser support, and currently only Chrome 85+ and Firefox 77+ support it (Firefox requires flag to be set to allow it)
* Move @uppy/utils/lib/prettyBytes -> @transloadit/prettier-bytes so we can more easily re-use it across projects
* Switch to MIT licensed @transloadit/prettier-bytes
* lockfile fun
* build: add dep for website/inject.js
Co-authored-by: Renée Kooi <renee@kooi.me>
* Add NetworkError error type and isNetworkError utility check
* Detect network errors in tus and xhr-upload
Co-Authored-By: Ifedapo .A. Olarewaju <ifedapoolarewaju@gmail.com>
* add NetworkError check to RequestClient in companion-client
* return false if !xhr 🙈
* move catch so that only errors from _checkMinNumberOfFiles are caught — errors from uploads are logged in `upload-error` already
//cc @goto-bus-stop
* Add NetworkError to MiniXHRUpload in aws-s3
//cc @goto-bus-stop
* Add tests
//cc @ifedapoolarewaju
* pass xhr to NetworkError
* add logging in catch of uppy.upload()
* start error message with “This looks like a network error”
* originalRequest --> request, return NetworkError right away
* check for NetworkError in `delete` as well
do we need it in preflight too, @ifedapoolarewaju?
* remove redundant error message re-declaration
* _showOrLogErrorAndThrow, but don’t showInformer
Co-authored-by: Ifedapo .A. Olarewaju <ifedapoolarewaju@gmail.com>
* aws-s3: handle actual upload internally instead of deferring to XHRUpload
* aws-s3: add a comment explaining the mess
* aws-s3: move XHRUpload code into a separate class
* utils: support prioritization in rate limiting queue
* aws-s3: remove old implementation
* core,status-bar: account for pre/postprocessing files in progress calculations
* changelog: add entry for this PR
* utils: Support cancellation in `RateLimitedQueue#wrapPromiseFunction`
* s3: remove preprocessing stage, only use upload-* events
* utils: remove console.log
* aws-s3: comment out duplicate `upload-started`
* xhr-upload: reorder `ProgressTimeout` creation for readability
* aws-s3: apply 7d972e0 to MiniXHRUpload
* aws-s3: apply 4ed7508 to MiniXHRUpload
* webcam: request a video mime type listed in `allowedFileTypes`
* webcam: stop recording when we are about to reach the max file size
* webcam: request an image mime type listed in `allowedFileTypes`
* docs: add Webcam#preferredImageMimeType
* locale: add `recordingStoppedMaxSize` string
* changelog: check webcam restrictions
Co-authored-by: Artur Paikin <artur@arturpaikin.com>
* ProviderViews: Improve Instagram’s “video thumbnail”
* Update all provider icons to nicer round ones
* Change “drop files” copy depending on number of acquirers and Dashboard width
* Translator: support multiple components in options
When the source contains multiple placeholders for interpolation, we should ignore chunks that are not strings, because those can be JSX objects and will be otherwise incorrectly turned into strings.
Without this condition we’d get this: [object Object] hello [object Object] my <button>
* Remove unused locale strings
* Rever renderSingleInlineProvider, use Browse button instead of link, tweak styles
* Try “Browse My Device” button
* Introduce “Dark Mode“: new Dashboard option 'theme': 'light', 'dark' or 'auto'
* Refactor to also add uninstall/removeListener for the darkModeMediaQuery; remove all bindings in favor of class properties
* More dark mode styles: Providers, FileCard, tweaking focus, inputs, etc
* My Device: mobile and desktop icons
* add isMobileDevice capability, very simple detection
* More dark mode styles
* refactor
* switch back to single simple my device icon
* chrome 78 for android doesn’t support document.querySelector 👌
* nobr --> span white-space: nowrap
* Major style tweaks for mobile provider list, show dashed borders only when DnD is supported
* Add color after “import from”
just for Russian for now, because I’m thinking if this should be a part of the language string or html? @goto-bus-stop @kvz, what do you think?
* improve isMobileDevice, add tests
* don’t set the isMobileDevice capability in core for now
* add checks for typeof window !== 'undefined', otherwise fails in Node/RN
* update locales
* refactor renderDropFilesSubtitle
* Fix translations that did not respect word order
Fixes#2069
We were still using raw string concatenation in these two places. To
maintain backwards compatibility, this commit adds _new_ translations
that are substitution-based instead of concatenation-based. The old
translation is passed in as a possible substitution parameter
`backwardsCompat`, and the default (English) translations use this
parameter.
With this, if you are using a custom locale that overrides eg. the
`exceedsSize` string, Uppy will first get your `exceedsSize`
translation, and then use the default `exceedsSize2` translation which
concatenates that result and the file size. So it works the same as it
did in the past.
If a translation requires a different word order, it can override the
`exceedsSize2` translation instead.
```js
new Uppy({
locale: {
strings: {
exceedsSize2: 'Maximum file size of %{size} exceeded',
poweredBy2: '%{uppy} made this!'
}
}
})
```
In 2.0, we can remove all the old strings and only use `poweredBy2` (and
probably rename them to `poweredBy` too, heh)
* website: add missing dashboard strings to docs
* locales: remove unused second plural form from EN and NL
* build: allow different numbers of plural forms
* utils: add test for new throwing behaviour
* docs: insert PR url
* locales: revert ko_KR changes
* changelog: add 2.0 backlog entry
* locales: var → const
* docs: add brief note that translators should supply a pluralize function
* locales: update en_US with new template
* fix HEIGHT_MD = 400
Co-authored-by: Artur Paikin <artur@arturpaikin.com>
