* stream upload unknown size files
behind a new option streamingUploadSizeless
COMPANION_STREAMING_UPLOAD_SIZELESS
for tus
* allow for all upload protocols
seems to be working
closes#5305
* refactor
and fix bug where progress was not always emitted
* fix type
* fix progress throttling
only do it on total progress
* Improve progress in UI
- only show progress percent and total bytes for files that we know the size of. (but all files will still be included in number of files)
- use `null` as an unknown value for progress and ETA, allowing us to remove ETA from UI when unknown
- `percentage` make use of `undefined` when progress is not yet known - don't show percentage in UI when unknown
- add a new state field `progress` that's the same as `totalProgress` but can also be `null`
* fix build error
* format
* fix progress when upload complete
* use execa for companion load balancer
if not, then it leaves zombie companion instances running in the background when e2e stops
have to be manually killed before running e2e again
* update docs and tests for new state.progress
* revert progress/totalProgress
* improve doc
* remove option streamingUploadSizeless
we agreed that this can be considered not a breaking change
* change progress the to "of unknown"
* revert
* remove companion doc
* add e2e test
* fix typo
* escape regex in default companionAllowedHosts
- also fail early if invalid regex supplied
- and add unit tests
* Remove todo comment
Co-authored-by: Mikael Finstad <finstaden@gmail.com>
---------
Co-authored-by: Merlijn Vos <merlijn@soverin.net>
* initial poc
* improvements
- split into two plugins
- implement photos picker
- auto login
- save access token in local storage
- document
- handle photos/files picked and send to companion
- add new hook useStore for making it easier to use localStorage data in react
- add new hook useUppyState for making it easier to use uppy state from react
- add new hook useUppyPluginState for making it easier to plugin state from react
- fix css error
* implement picker in companion
* type todo
* fix ts error
which occurs in dev when js has been built before build:ts gets called
* reuse docs
* imrpve type safety
* simplify async wrapper
* improve doc
* fix lint
* fix build error
* check if token is valid
* fix broken logging code
* pull logic out from react component
* remove docs
* improve auth ui
* fix bug
* remove unused useUppyState
* try to fix build error
* don't close socket when pausing
because it prevented us from canceling an upload that is paused
note that we here sacrifice the ability to pause uploads and have other uploads take over the paused upload's rate limit queue spot.
this is sacrificed because we will soon remove the ability to pause/resume uploads anyways
(cherry picked from commit e4dbe77d2f)
* add migration guide entry
* ProviderView.tsx - fix onedrive breadcrumbs
* providers - correct ch-unch-indeterminate states
* providers - made .breadcrumbs derived from .partialTree
* everywhere - { files, folders, isChecked } => .partialTree
GoogleDrive
- travelling down into folders works
- checking a file works
- breadcrumbs DONT work
* GoogleDrive - made breadcrumbs work
* .getFolder() - remove the `name` argument
* <Breadcrumbs/> - refactors "/"
* Instagram - made files get fetched onScroll
* clearSelection() - recover the functionality
* GoogleDrive - recover custom `.toggleCheckbox()` functionality
* providers - recover `.isDisabled` functionality
* <SearchProviderView/> - made Unsplash use .partialTree
* Facebook - change `.files, .folders` => `.partialTree`
* everywhere - we don't need to ! `partialTreeFile.data` anymore
* <ProviderView/> - implement folder caching
* <View/> - enable shift-clicking
* everywhere - get rid of unnecessary `.getNextFolder()`
* everywhere - fixing types
* <ProviderView/> - rename `requestPath` to `folderId`
* all providers - get rid of `.onFirstRender()`
* provider views - get rid of `.onFirstRender()`
* <ProviderView/> - make the root folder cacheable too
* TEMP - setup for working with FOLDERS + LAZY_LOADING
* <ProviderView/> - get rid of `.#listFilesAndFolders`
* <ProviderView/> - make `this.nextPagePath` per-folder
* everywhere - more refined types
* types - reintroduce `StatusInPartialTree`
* <SearchProviderView/> - made Unsplash work with the new structure
* <ProviderView/> - preemptive cleaning of `.absDirPath` and `.relDirPath`
* <ProviderView/> - give `.nextPagePath` a rigorous type
* <ProviderView/> - make `.nextPagePath` & `.cached` a composite key
* <ListItem/> - remove unnecessary indirection level
* css - factor out `.statusClassName`
* everywhere - refactor `.validateRestrictions()`
* nOfSelectedFiles - make "Selected (n)" as smart as possible
* <ProviderViews/> - prevent shift-clicking from highlighting file names
* `.validateRestrictions()` - make it accept a `CompanionFile` instead of `PartialTree`'s file
* `.getFolder()` - simplify code
* everywhere - account for `restrictions` in `.partialTree`
* `PartialTreeUtils.ts` - factor out `getPartialTreeAfterTogglingCheckboxes()`
* `PartialTreeUtils.ts` - factor out `clickOnFolder()`
* `PartialTreeUtils.ts` - factor out `getPartialTreeAfterScroll()`
* `PartialTreeUtils.ts` - rename methods
* `.donePicking()` - implement using recursion
* `.donePicking()` - integrate with `<ProviderView/>`
* `donePicking()` - show notifications after addition
* `#list()` - get rid of unnecessary indirection
* ProviderView.tsx - add `signal` everywhere, reduce try/catch indents everywhere
* `handleError()` - make error handling uniform
* `state.isSearchVisible` - remove, it's just not used anywhere
* state - reuse default state
* state - reset state on close panel (like we discussed in the uppy call)
* methods - remove unnecessary indirection in state setting
* `<CloseWrapper/>` - remove CloseWrappers, this is unnecessary indirection now too
* `this.requestClientId` - remove, again - this was unnecessary indirection
* `getTagFile()` - factor out into a separate file
* `recordShiftKeyPress()` - fix chaotic shift-clicking in Grid providers, remove endless prop drilling while we're at it
* `getNOfSelectedFiles.ts`, `filterItems.ts` - factor out, this removes props drilling
* <Browser/> - pass `displayedPartialTree` right away (because Search&NormalProvider have wildly different logics!)
* `searchTerm`, `filterInput` - we only need one of these of course!
* <SearchProvider/> - fix the issue where `afterToggleCheckbox()` thinks we should always filter by `searchString`
* <SearchProvider/> - remove `this.nextPageQuery`
Also: fix the issue where <SearchProvider/> upon searching for "ocean" and then "pajama" would just be adding pajama pictures after the ocean ones
* <Browser/> - remove unnecessary prop indirection
Typescript didn't actually know some of these props aren't used (removed those now)! It only discovers unused props upon normal props passing, like we do now.
* <SearchFilterInput/> - make the form controlled, hugely simplifies everything
* `filterItems.ts` - move to <ProviderView/>, because it's only used there
* /utils/PartialTreeUtils.ts - put every util in a separate file
* `shouldHandleScroll.ts` - factor out into a util
This brings all references to `this.isHandlingScroll` into a single place, and makes `shouldHandleScroll()` a self-contained simple function
* this.state - make sure state is reset 1. on cancel 2. on close
* `this.xxx` - never leave `this.xxx` variables undefined
* `this.username` - should be in `this.state`
Also - when there is no username, stop showing the little dot
* `SearchProviderPluginState` type - simplify this type, never leave state vars undefined
* <Header/> - remove completely unnecessary indirection, remove unused props
* Facebook.tsx - more sane `viewOptions` code
* providers - properly type `opts`
* `this.isShiftKeyPressed` - move this variable into <Browser/>
* `this.handleError()` - move to /utils
* `this.isHandlingScroll` - move to child classes
* `this.registerRequestClient()` - move to child classes
* `this.lastCheckbox` - move to child classes
* `this.setLoading()` - move to child classes
* `this.validateRestrictions()` - move to utils
* types - fully simplify provider types, remove `View.ts` parent class
* index.d.ts - we're not using `OnFirstRenderer` anymore
* <ProviderView/>, <SearchProviderView/> - more precise typing for options
* package.json - remove nanoid
* GoogleDrive - make shift-clicking work
* everywhere - fix types across uppy
* `afterToggleCheckbox.ts` - less redundant args, pass `ourItem.id` instead of `ourItem`
* tests - create `afterToggleCheckbox()` tests
* `getClickedRange.ts` - decouple `getClickedRange()` from `afterToggleCheckbox()`
* tests - wrote tests for `afterToggleCheckbox.ts`
* tests - wrote tests for `afterClickOnFolder.ts`
* everywhere - finally rename `getFolder` => `openFolder`
* tests - wrote tests for `afterScrollFolder.ts`
* getPaths.ts - make `absDirPath`, `relDirPath` work like in docs & add tests for that
* injectPaths.ts - improve performance
* getTagFile.ts - handle path injection all in one place
* getTagFile.ts - refactor
Just makes it easier to read the structure of TagFile
* fill.ts - `provider.list(currentPath, { signal })` => `apiList`
(remove the dependency on provider, just pass a callback)
* tests - wrote tests for `fill.ts`
* tests - wrote tests for `getNOfSelectedFiles.ts`
* everywhere - change `JSON.stringify()` => `clone()`
* `PartialTreeUtils.ts` - more consistent function naming + alphabetical order in tests
* `donePicking()` - superseded a notification to i18n one
* GoogleDrive - make the shared drive checkable
* `Item.tsx` - standardize names; remove unnecessary question marks from props
* ProviderView.tsx - clicking "Cancel" should make all files "unchecked"
* everywhere - move `document.getSelection()?.removeAllRanges()` to <Browser/> to avoid repetition
* everywhere - standardize names and types of passed props
* <Browser/> - only leave "list of files" to the browser
Moves stuff closer to where it's used, prevents props drilling
* TEMP - easier pageSize for alex to play with
When it's set to 5 pages you have to reduce the browser window to make it scrollable
* everywhere - only handle individual-file restrictions
* everywhere - add aggregate restrictions on top
* SearchProvider, NormalProvider - unite the way we addFiles()
Same notifications, same code, same everything
* `getTagFile.ts` - pass fewer arguments
* `addFiles.ts` - move conversion to tagFiles into `addFiles()`
* `uppy.validateRestrictions()` - remove legacy method
* `uppy.validateAggregateRestrictions()` - make aggregate restricter report aggregate error
* <FooterActions/> css - make aggregate errors look nice
* `PartialTreeUtils/index.test.ts` - accommodate tests to the latest changes
* tests - make all uppy tests work
* prettiness - run `yarn format`
* prettiness - run `yarn lint:fix`
* package.json - add `vitest` as a dev dependency
* eslint - fixing 1
eslint - fixing 2
eslint - fixing 3
* <SearchFilterInput/> - add default props as per eslint
* <SearchFilterInput/> - rename to <SearchInput/>
* eslint - fixing 4 (clone.ts)
* Uppy.ts - rewrite `partialTree` docs
* eslint - fixing 5
* eslint - fixing 6
* `getBreadcrumbs.ts` - factor out
* tests - fixing 7
* everywhere - remove `.toReversed()`, because it's not yet supported in all browsers
* dev/Dashboard.js - restore to pristine version
* prettiness - run `yarn format`
* fixing 8 (`yarn run build:ts`)
* fixing 9 (run `corepack yarn`)
* prettier - undo indentation harm done by prettier
* `getBreadcrumbs()` - add tests, and rewrite to avoid using `.toReversed()`
Clarification: `.toReversed()` is no supported by all browsers
* `<SearchInput/>` - make it work for eslint
* everywhere - remove `eslint-disable react/require-default-props`
* <GridItem/>, <ListItem/> - refactor to avoid prop drilling
* <ListItem/> - disable checkboxes for GoogleDrive team drives
See #5232
* merge (fixing up some lines from the previous merge)
* merge (fixing up some lines from the previous merge)
* everywhere - remove TEMP development values
* `this.validateSingleFile()` - switch to `.restrictionError`
* `afterToggleCheckbox.ts` - refactor, add comments
* `afterToggleCheckbox.ts` - refactor to use ids instead of whole objects
* `afterToggleCheckbox.ts` - try to satisfy prettier
* fixing 10 (try to satisfy `npx webpack`)
* fixing 11 (try to satisfy `npx webpack`)
* Antoine: use Math.min & Math.max in `getClickedRange()`
Co-authored-by: Antoine du Hamel <antoine@transloadit.com>
* fixing 12 (run `yarn run format`)
* `clone.ts` - rename to `shallowClone.ts`
* Antoine: in `package.json`, move `devDependencies` up
* Antoine: rename `getNOfSelectedFiles()` to `getNumberOfSelectedFiles()`
* `getNumberOfSelectedFiles()` - better comments
* Antoine: remove `<form/>` tag
* Antoine: change `{}` to `Object.create(null)`, write tests
* Antoine: `<SearchInput/>` - return dynamic <form/> element
* `<SearchInput/>` - return `buttonCSSClassName`
* `GoogleDrive.tsx` - make team drive checkboxes visible
* merge (more)
* Mifi: update packages/@uppy/provider-views/src/ProviderView/ProviderView.tsx
Co-authored-by: Mikael Finstad <finstaden@gmail.com>
* merge (more changes)
* `Facebook.tsx`, `GooglePhotos.tsx` - render in 'grid' style on per-folder basis
* `<GridItem/>` - use the `.thumbnail` whenever possible (improves image quality, adds video icons)
* `prettier` - ensure `PartialTree` is always strongly indented in tests
---------
Co-authored-by: Antoine du Hamel <antoine@transloadit.com>
Co-authored-by: Mikael Finstad <finstaden@gmail.com>
We sometimes where importing source files, which may not be available in
production. Instead we should target extension-less paths in `lib/`
folders so the generated `.d.ts` files are used.
New concept "simple auth" - authentication that happens immediately (in one http request) without redirecting to any third party.
uppyAuthToken initially used to simply contain an encrypted & json encoded OAuth2 access_token for a specific provider. Then we added refresh tokens as well inside uppyAuthToken #4448. Now we also allow storing other state or parameters needed for that specific provider, like username, password, host name, webdav URL etc... This is needed for providers like webdav, ftp etc, where the user needs to give some more input data while authenticating
Companion:
- `providerTokens` has been renamed to `providerUserSession` because it now includes not only tokens, but a user's session with a provider.
Companion `Provider` class:
- New `hasSimpleAuth` static boolean property - whether this provider uses simple auth
- uppyAuthToken expiry default 24hr again for providers that don't support refresh tokens
- make uppyAuthToken expiry configurable per provider - new `authStateExpiry` static property (defaults to 24hr)
- new static property `grantDynamicToUserSession`, allows providers to specify which state from Grant `dynamic` to include into the provider's `providerUserSession`.
According to https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests, we have so many preflight requests because we add `Uppy-Versions` header and `Content-Type: application/json` to each request. It's particularly not useful for for requests with no body / an empty body because the remote Companion doesn't check `Content-Type`.
It was added in b25f243627, and since f0f1105ef0, it is only used for log
purposes.
Co-authored-by: Mikael Finstad <finstaden@gmail.com>
* respond with 500 if unhandled upload error
400 seems inappropriate
* add todo
* fix companion download error
respond with correct response code when calling /get
and request to the upstream server fail
also add a unit test for this
* forward 429 from provider
allows us to retry it
* rename fn
* add a way to test refresh tokens
* fix race condtiion with refreshing token
* implement retry and fix socket
- Make sure we don't instantiate a new Provider for every file uploaded (reuse the original provider instead) - this caused the refresh token synchronization not to work
- Retry most errors when uploading (`/get`) using companion, but for HTTP, only retry certain HTTP status codes
- Reimplement the websocket code to be more stable and retry as well
* remove unused socket wrapper
* remove useFastRemoteRetry
because it isn't supported in Companion as far as I can tell
* fix error handling
* remove unneeded logic
* apply review suggestion
* retry awaiting for companion socket too
* retry whole operation instead of individually
or else we risk getting stuck retrying just half of the operation, which might never recover
* improvements
- rewrite so we can use p-retry for exponential backoff for socket reconnect too
- improve logging
- simplify/reuse code
* Update packages/@uppy/utils/src/fetchWithNetworkError.js
Co-authored-by: Antoine du Hamel <antoine@transloadit.com>
* Explain requestClient Symbol with comment
Co-authored-by: Mikael Finstad <finstaden@gmail.com>
* Update packages/@uppy/provider-views/src/View.js
* fixes
preventing socket.send when the socket is not in the connected state
catch errors in handlers
* log if trying to refresh with no token
* log whether we got a refresh token
* dont log retrying when in reality not
* always prompt: consent
for google drive
* add signal to post
* refactor away eventmanager
and event handler error catching
* don't fail with upload-error
if a file is canceled
* fix expect
* add capabilities support
* make requestClient non-enumerable
---------
Co-authored-by: Antoine du Hamel <antoine@transloadit.com>
Co-authored-by: Merlijn Vos <merlijn@soverin.net>
that would be triggered when uploading multiple files in parallell
if a 401 occurs, it would cause multiple refresh-token requests to be fired in parallel
also only call refresh-token if we have a token
and reset the token if refresh-token fails with 401
* @uppy/companion-client,@uppy/provider-views: make authentication optional in ProviderView
* try different approach
* Move authAborted translation to companion-client
* @uppy/companion-client: do not reject login promise on events from incorrect source, but always return after rejections
While these events should be ignored they might be unrelated and should not reject the whole login
* Revert "Move authAborted translation to companion-client"
This reverts commit ce7f90e455.
* Check unused core locale strings in companion-client
---------
Co-authored-by: Murderlon <merlijn@soverin.net>
* fix multiple race conditions
multiple race conditions / bugs:
- If you open a folder in e.g. facebook and you scroll down, triggering the infinity scroll, and then go back (one folder up) before the infinity scroll has finished loading, you could end up with the contents of the scrolled folder instead of the previous folder
- if you select a folder structure (e.g. deep structure with many files) and then click "cancel" while loading, it will still continue loading recursively in the background even after cancelled and potentially cause more issues down the road
* Apply suggestions from code review
Co-authored-by: Antoine du Hamel <antoine@transloadit.com>
* merge conditions
---------
Co-authored-by: Antoine du Hamel <antoine@transloadit.com>
Co-authored-by: Murderlon <merlijn@soverin.net>
This also has the upside of letting the JS engine garbage collect the
underlying `WebSocket` instance.
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
