Golden retriever refactor and UppyFile type improvements (#5978)

Probably best reviewed commit by commit.

I also split UppyFile into two intefaces distinguished by the `isRemote`
boolean:
- LocalUppyFile
- RemoteUppyFile

Also:
- Removed the TagFile type
- Don't re-upload completed files - fixes #5930
- Clean up stored files on `complete` event *only* if *all* files
succeeded (no failed files). this allows the user to retry failed files
if the browser & upload get interrupted - fixes #5927, closes #5955
- Only set `isGhost` for non-successful files. it doesn't make sense for
successfully uploaded files to be ghosted because they're already done.
#5930

fixes #6013

---------

Co-authored-by: Prakash <qxprakash@gmail.com>
This commit is contained in:
Mikael Finstad 2025-10-17 23:17:40 +08:00 committed by GitHub
parent 1fbb95da2b
commit 0c16fe44b9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
59 changed files with 1311 additions and 940 deletions

8
.changeset/aws-s3.md Normal file
View file

@ -0,0 +1,8 @@
---
"@uppy/aws-s3": patch
---
- Make `file.data` nullable - Because for ghosts it will be `undefined` and we don't have any type to distinguish ghosts from other (local) files. This caused a crash, because we didn't check for `undefined` everywhere (when trying to store a blob that was `undefined`). This means we have to add null checks in some packages
- Split UppyFile into two intefaces distinguished by the `isRemote` boolean:
- LocalUppyFile
- RemoteUppyFile

View file

@ -0,0 +1,7 @@
---
"@uppy/companion-client": patch
---
- Split UppyFile into two intefaces distinguished by the `isRemote` boolean:
- LocalUppyFile
- RemoteUppyFile

6
.changeset/components.md Normal file
View file

@ -0,0 +1,6 @@
---
"@uppy/components": patch
---
- Make `file.data` nullable - Because for ghosts it will be `undefined` and we don't have any type to distinguish ghosts from other (local) files. This caused a crash, because we didn't check for `undefined` everywhere (when trying to store a blob that was `undefined`). This means we have to add null checks in some packages
- Move `restore-confirmed` from `onUploadStart` event listener to `startUpload`, else it would cause `restore-confirmed` to be triggered even if there is no `recoveredState` to recover

8
.changeset/compressor.md Normal file
View file

@ -0,0 +1,8 @@
---
"@uppy/compressor": patch
---
- Make `file.data` nullable - Because for ghosts it will be `undefined` and we don't have any type to distinguish ghosts from other (local) files. This caused a crash, because we didn't check for `undefined` everywhere (when trying to store a blob that was `undefined`). This means we have to add null checks in some packages
- Split UppyFile into two intefaces distinguished by the `isRemote` boolean:
- LocalUppyFile
- RemoteUppyFile

20
.changeset/core.md Normal file
View file

@ -0,0 +1,20 @@
---
"@uppy/core": patch
"@uppy/utils": patch
---
- Make `file.data` nullable - Because for ghosts it will be `undefined` and we don't have any type to distinguish ghosts from other (local) files. This caused a crash, because we didn't check for `undefined` everywhere (when trying to store a blob that was `undefined`)
- Introduce new field `progress`.`complete`: if there is a post-processing step, set it to `true` once post processing is complete. If not, set it to `true` once upload has finished.
- Throw a proper `Nonexistent upload` error message if trying to upload a non-existent upload, instead of TypeError
- Rewrite `Uppy.upload()` - this fixes two bugs:
1. No more duplicate emit call when this.#restricter.validateMinNumberOfFiles throws (`#informAndEmit` and `this.emit('error')`)
2. 'restriction-failed' now also gets correctly called when `checkRequiredMetaFields` check errors.
- Don't re-upload completed files #5930
- Split UppyFile into two intefaces distinguished by the `isRemote` boolean:
- LocalUppyFile
- RemoteUppyFile
- Remove TagFile type - Use UppyFile instead.
- Make `name` required on UppyFile (it is in reality always set)
- Fix bug: `RestrictionError` sometimes thrown with a `file` property that was *not* a `UppyFile`, but a `File`. This would happen if someone passed a `File` instead of a `MinimalRequiredUppyFile` into `core.addFile` (which is valid to do according to our API)
- Improve some log messages
- Simplify Uppy `postprocess-complete` handler

5
.changeset/dashboard.md Normal file
View file

@ -0,0 +1,5 @@
---
"@uppy/dashboard": patch
---
- Remove `restore-canceled` event as it was not being used.

View file

@ -0,0 +1,20 @@
---
"@uppy/golden-retriever": patch
---
- **Internal inter-package breaking change:** Remove hacky internal event `restore:get-data` that would send a function as its event data (to golden retriever for it to call the function to receive data from it). Add instead `restore:plugin-data-changed` that publishes data when it changes. This means that **old versions of `@uppy/transloadit` are not compatible with newest version of `@uppy/golden-retriever` (and vice versa)**.
- Large internal refactor of Golden Retriever
- Use `state-update` handler to trigger save to local storage and blobs, instead of doing it in various other event handlers (`complete`, `upload-success`, `file-removed`, `file-editor:complete`, `file-added`). this way we don't miss any state updates. also simplifies the code a lot. this fixes:
- Always store blob when it changes - this fixes the bug when using the compressor plugin, it would store the uncompressed original blob (like when using image editor plugin)
- Add back throttle: but throttling must happen on the actual local storage save calls inside MetaDataStore, *not* the handleStateUpdate function, so we don't miss any state updates (and end up with inconsistent data). Note that there is still a race condition where if the user removes a file (causing the blob to be deleted), then quickly reloads the page before the throttled save has happened, the file will be restored but the blob will be missing, so it will become a ghost. this is probably not a big problem though. need to disable throttling when running tests (add it as an option to the plugin)
- Fix implicit `any` types in #restore filesWithBlobs
- Don't error when saving indexedDB file that already exists (make it idempotent)
- Fix bug: Golden Retriever was not deleting from IndexedDbStore if ServiceWorkerStore exists, causing a storage leak
- Remove unused Golden Retriever cleanup.ts
- Clean up stored files on `complete` event *only* if *all* files succeeded (no failed files). this allows the user to retry failed files if they get interrupted - fixes #5927, closes #5955
- Only set `isGhost` for non-successful files - it doesn't make sense for successfully uploaded files to be ghosted because they're already done. #5930
- Add `upload-success` event handler `handleFileUploaded`: this handler will remove blobs of files that have successfully uploaded. this prevents leaking blobs when an upload with multiple files gets interrupted (but some files have uploaded successfully), because `#handleUploadComplete` (which normally does the cleanup) doesn't get called untill *all* files are complete.
- Fix `file-editor:complete` potential race condition: it would delete and add at the same time (without first awaiting delete operation)
- Fix: Don't double `setState` when restoring
- Improve types in golden retriever and MetaDataStore
- MetaDataStore: move old state expiry to from `constructor` to `load()`

View file

@ -0,0 +1,8 @@
---
"@uppy/image-editor": patch
---
- Make `file.data` nullable - Because for ghosts it will be `undefined` and we don't have any type to distinguish ghosts from other (local) files. This caused a crash, because we didn't check for `undefined` everywhere (when trying to store a blob that was `undefined`). This means we have to add null checks in some packages
- Split UppyFile into two intefaces distinguished by the `isRemote` boolean:
- LocalUppyFile
- RemoteUppyFile

View file

@ -0,0 +1,5 @@
---
"@uppy/provider-views": patch
---
- Rename `getTagFile` to `companionFileToUppyFile`

View file

@ -0,0 +1,8 @@
---
"@uppy/thumbnail-generator": patch
---
- Make `file.data` nullable - Because for ghosts it will be `undefined` and we don't have any type to distinguish ghosts from other (local) files. This caused a crash, because we didn't check for `undefined` everywhere (when trying to store a blob that was `undefined`). This means we have to add null checks in some packages
- Split UppyFile into two intefaces distinguished by the `isRemote` boolean:
- LocalUppyFile
- RemoteUppyFile

View file

@ -0,0 +1,9 @@
---
"@uppy/transloadit": patch
---
- **Internal inter-package breaking change:** Remove hacky internal event `restore:get-data` that would send a function as its event data (to golden retriever for it to call the function to receive data from it). Add instead `restore:plugin-data-changed` that publishes data when it changes. This means that **old versions of `@uppy/transloadit` are not compatible with newest version of `@uppy/golden-retriever` (and vice versa)**.
- Minor internal refactoring in order to make sure that we will always emit `restore:plugin-data-changed` whenever assembly state changes
- Split UppyFile into two intefaces distinguished by the `isRemote` boolean:
- LocalUppyFile
- RemoteUppyFile

8
.changeset/tus.md Normal file
View file

@ -0,0 +1,8 @@
---
"@uppy/tus": patch
---
- Make `file.data` nullable - Because for ghosts it will be `undefined` and we don't have any type to distinguish ghosts from other (local) files. This caused a crash, because we didn't check for `undefined` everywhere (when trying to store a blob that was `undefined`). This means we have to add null checks in some packages
- Split UppyFile into two intefaces distinguished by the `isRemote` boolean:
- LocalUppyFile
- RemoteUppyFile

8
.changeset/url.md Normal file
View file

@ -0,0 +1,8 @@
---
"@uppy/url": patch
---
- Make `file.data` nullable - Because for ghosts it will be `undefined` and we don't have any type to distinguish ghosts from other (local) files. This caused a crash, because we didn't check for `undefined` everywhere (when trying to store a blob that was `undefined`). This means we have to add null checks in some packages
- Split UppyFile into two intefaces distinguished by the `isRemote` boolean:
- LocalUppyFile
- RemoteUppyFile

View file

@ -186,6 +186,7 @@ export class HTTPCommunicationQueue<M extends Meta, B extends Body> {
// where we just miss a new result so we loop here until we get nothing back,
// at which point it's out turn to create a new cache entry.
for (;;) {
if (file.data == null) throw new Error('File data is empty')
cachedResult = this.#cache.get(file.data)
if (cachedResult == null) break
try {
@ -199,6 +200,7 @@ export class HTTPCommunicationQueue<M extends Meta, B extends Body> {
const promise = this.#createMultipartUpload(this.#getFile(file), signal)
const abortPromise = () => {
if (file.data == null) throw new Error('File data is empty')
promise.abort(signal.reason)
this.#cache.delete(file.data)
}
@ -208,10 +210,12 @@ export class HTTPCommunicationQueue<M extends Meta, B extends Body> {
async (result) => {
signal.removeEventListener('abort', abortPromise)
this.#setS3MultipartState(file, result)
if (file.data == null) throw new Error('File data is empty')
this.#cache.set(file.data, result)
},
() => {
signal.removeEventListener('abort', abortPromise)
if (file.data == null) throw new Error('File data is empty')
this.#cache.delete(file.data)
},
)
@ -220,6 +224,7 @@ export class HTTPCommunicationQueue<M extends Meta, B extends Body> {
}
async abortFileUpload(file: UppyFile<M, B>): Promise<void> {
if (file.data == null) throw new Error('File data is empty')
const result = this.#cache.get(file.data)
if (result == null) {
// If the createMultipartUpload request never was made, we don't
@ -325,6 +330,7 @@ export class HTTPCommunicationQueue<M extends Meta, B extends Body> {
}
restoreUploadFile(file: UppyFile<M, B>, uploadIdAndKey: UploadResult): void {
if (file.data == null) throw new Error('File data is empty')
this.#cache.set(file.data, uploadIdAndKey)
}

View file

@ -65,6 +65,7 @@ describe('AwsS3Multipart', () => {
})
const createFile = (size: number): UppyFile<any, any> => ({
name: '',
size,
data: new Blob(),
extension: '',

View file

@ -6,7 +6,13 @@ import {
type PluginOpts,
type Uppy,
} from '@uppy/core'
import type { Body, Meta, RequestOptions, UppyFile } from '@uppy/utils'
import type {
Body,
LocalUppyFile,
Meta,
RequestOptions,
UppyFile,
} from '@uppy/utils'
import {
createAbortError,
filterFilesToEmitUploadStarted,
@ -26,7 +32,7 @@ import type {
} from './utils.js'
import { throwIfAborted } from './utils.js'
interface MultipartFile<M extends Meta, B extends Body> extends UppyFile<M, B> {
type MultipartFile<M extends Meta, B extends Body> = UppyFile<M, B> & {
s3Multipart: UploadResult
}
@ -813,7 +819,7 @@ export default class AwsS3Multipart<
return this.uppy.getFile(file.id) || file
}
#uploadLocalFile(file: UppyFile<M, B>) {
#uploadLocalFile(file: LocalUppyFile<M, B>) {
return new Promise<undefined | string>((resolve, reject) => {
const onProgress = (bytesUploaded: number, bytesTotal: number) => {
const latestFile = this.uppy.getFile(file.id)
@ -852,6 +858,8 @@ export default class AwsS3Multipart<
resolve(undefined)
}
if (file.data == null) throw new Error('File data is empty')
const upload = new MultipartUploader<M, B>(file.data, {
// .bind to pass the file object to each handler.
companionComm: this.#companionCommunicationQueue,
@ -916,9 +924,9 @@ export default class AwsS3Multipart<
#getCompanionClientArgs(file: UppyFile<M, B>) {
return {
...file.remote?.body,
...('remote' in file && file.remote?.body),
protocol: 's3-multipart',
size: file.data.size,
size: file.data!.size,
metadata: file.meta,
}
}

View file

@ -1,5 +1,11 @@
import type Uppy from '@uppy/core'
import type { Body, Meta, RequestOptions, UppyFile } from '@uppy/utils'
import type {
Body,
Meta,
RemoteUppyFile,
RequestOptions,
UppyFile,
} from '@uppy/utils'
import {
ErrorWithCause,
fetchWithNetworkError,
@ -76,14 +82,14 @@ async function handleJSONResponse<ResJson>(res: Response): Promise<ResJson> {
throw new HttpError({ statusCode: res.status, message: errMsg })
}
function emitSocketProgress(
uploader: { uppy: Uppy<any, any> },
function emitSocketProgress<M extends Meta, B extends Body>(
uploader: { uppy: Uppy<M, B> },
progressData: {
progress: string // pre-formatted percentage number as a string
bytesTotal: number
bytesUploaded: number
},
file: UppyFile<any, any>,
file: UppyFile<M, B>,
): void {
const { progress, bytesUploaded, bytesTotal } = progressData
if (progress) {
@ -236,7 +242,7 @@ export default class RequestClient<M extends Meta, B extends Body> {
* uploading or is otherwise done (failed, canceled)
*/
async uploadRemoteFile(
file: UppyFile<M, B>,
file: RemoteUppyFile<M, B>,
reqBody: Record<string, unknown>,
options: { signal: AbortSignal; getQueue: () => any },
): Promise<void> {
@ -262,7 +268,7 @@ export default class RequestClient<M extends Meta, B extends Body> {
async (
...args: [
{
file: UppyFile<M, B>
file: RemoteUppyFile<M, B>
postBody: Record<string, unknown>
signal: AbortSignal
},
@ -304,7 +310,7 @@ export default class RequestClient<M extends Meta, B extends Body> {
this.uppy.setFileState(file.id, { serverToken })
return this.#awaitRemoteFileUpload({
file: this.uppy.getFile(file.id), // re-fetching file because it might have changed in the meantime
file: this.uppy.getFile(file.id) as RemoteUppyFile<M, B>, // re-fetching file because it might have changed in the meantime
queue: getQueue(),
signal,
})
@ -334,7 +340,7 @@ export default class RequestClient<M extends Meta, B extends Body> {
postBody,
signal,
}: {
file: UppyFile<M, B>
file: RemoteUppyFile<M, B>
postBody: Record<string, unknown>
signal: AbortSignal
}): Promise<string> => {
@ -364,7 +370,7 @@ export default class RequestClient<M extends Meta, B extends Body> {
queue,
signal,
}: {
file: UppyFile<M, B>
file: RemoteUppyFile<M, B>
queue: any
signal: AbortSignal
}): Promise<void> {

View file

@ -36,6 +36,7 @@ export default function Thumbnail(props: ThumbnailProps) {
if (props.file.isRemote) {
return props.file.preview
}
if (props.file.data == null) throw new Error('File data is empty')
return URL.createObjectURL(props.file.data)
}, [props.file.data, props.images, props.file.isRemote, props.file.preview])

View file

@ -1,11 +1,10 @@
import prettierBytes from '@transloadit/prettier-bytes'
import type { DefinePluginOpts, PluginOpts } from '@uppy/core'
import { BasePlugin, type Uppy } from '@uppy/core'
import type { Body, Meta, UppyFile } from '@uppy/utils'
import type { Body, LocalUppyFile, Meta, UppyFile } from '@uppy/utils'
// @ts-ignore
import { getFileNameAndExtension, RateLimitedQueue } from '@uppy/utils'
import CompressorJS from 'compressorjs'
import locale from './locale.js'
declare module '@uppy/core' {
@ -64,8 +63,9 @@ export default class Compressor<
let totalCompressedSize = 0
const compressedFiles: UppyFile<M, B>[] = []
const compressAndApplyResult = this.#RateLimitedQueue.wrapPromiseFunction(
async (file: UppyFile<M, B>) => {
async (file: LocalUppyFile<M, B>) => {
try {
if (file.data == null) throw new Error('File data is empty')
const compressedBlob = await this.compress(file.data)
const compressedSavingsSize = file.data.size - compressedBlob.size
this.uppy.log(
@ -119,8 +119,8 @@ export default class Compressor<
// Some browsers (Firefox) add blobs with empty file type, when files are
// added from a folder. Uppy auto-detects type from extension, but leaves the original blob intact.
// However, Compressor.js failes when file has no type, so we set it here
if (!file.data.type) {
file.data = file.data.slice(0, file.data.size, file.type)
if (!file.data!.type) {
file.data = file.data!.slice(0, file.data!.size, file.type)
}
if (!file.type?.startsWith('image/')) {

View file

@ -19,10 +19,11 @@ export type Restrictions = {
*/
export type ValidateableFile<M extends Meta, B extends Body> = Pick<
UppyFile<M, B>,
'type' | 'extension' | 'size' | 'name'
'type' | 'extension' | 'size'
// Both UppyFile and CompanionFile need to be passable as a ValidateableFile
// CompanionFile's do not have `isGhost`, so we mark it optional.
> & { isGhost?: boolean }
> &
Partial<Pick<UppyFile<M, B>, 'name' | 'isGhost'>>
const defaultOptions = {
maxFileSize: null,

View file

@ -813,6 +813,7 @@ describe('src/Core', () => {
expect(core.getFile(fileId).progress).toEqual({
percentage: 0,
bytesUploaded: false,
complete: true,
bytesTotal: 17175,
uploadComplete: false,
uploadStarted: null,
@ -1941,7 +1942,7 @@ describe('src/Core', () => {
const core = new Core()
let proceedUpload: (value?: unknown) => void
let finishUpload: (value?: unknown) => void
const promise = new Promise((resolve) => {
const progressPromise = new Promise((resolve) => {
proceedUpload = resolve
})
const finishPromise = new Promise((resolve) => {
@ -1949,7 +1950,7 @@ describe('src/Core', () => {
})
core.addUploader(async ([id]) => {
core.emit('upload-start', [core.getFile(id)])
await promise
await progressPromise
// @ts-ignore deprecated
core.emit('upload-progress', core.getFile(id), {
bytesTotal: 3456,
@ -1971,10 +1972,11 @@ describe('src/Core', () => {
// @ts-ignore
core[Symbol.for('uppy test: updateTotalProgress')]()
const uploadStartedPromise = new Promise((resolve) =>
core.once('upload-start', resolve),
)
const uploadPromise = core.upload()
await Promise.all([
new Promise((resolve) => core.once('upload-start', resolve)),
])
await uploadStartedPromise
expect(core.getFiles()[0].size).toBeNull()
expect(core.getFiles()[0].progress).toMatchObject({
@ -1986,7 +1988,7 @@ describe('src/Core', () => {
// @ts-ignore
proceedUpload()
// wait for progress event
await promise
await progressPromise
expect(core.getFiles()[0].size).toBeNull()
expect(core.getFiles()[0].progress).toMatchObject({

View file

@ -10,10 +10,13 @@ import type {
FileProgressStarted,
I18n,
Locale,
LocalUppyFile,
Meta,
MinimalRequiredUppyFile,
OptionalPluralizeLocale,
RemoteUppyFile,
UppyFile,
UppyFileId,
} from '@uppy/utils'
import {
getFileNameAndExtension,
@ -199,15 +202,18 @@ export type UnknownSearchProviderPlugin<
provider: CompanionClientSearchProvider
}
// for better readability
export type UploadId = string
export interface UploadResult<M extends Meta, B extends Body> {
successful?: UppyFile<M, B>[]
failed?: UppyFile<M, B>[]
uploadID?: string
uploadID?: UploadId
[key: string]: unknown
}
interface CurrentUpload<M extends Meta, B extends Body> {
fileIDs: string[]
fileIDs: UppyFileId[]
step: number
result: UploadResult<M, B>
}
@ -215,6 +221,10 @@ interface CurrentUpload<M extends Meta, B extends Body> {
// TODO: can we use namespaces in other plugins to populate this?
interface Plugins extends Record<string, Record<string, unknown> | undefined> {}
type UppyFilesMap<M extends Meta, B extends Body> = {
[key: UppyFileId]: UppyFile<M, B>
}
export interface State<M extends Meta, B extends Body>
extends Record<string, unknown> {
meta: M
@ -225,13 +235,16 @@ export interface State<M extends Meta, B extends Body>
isMobileDevice?: boolean
darkMode?: boolean
}
currentUploads: Record<string, CurrentUpload<M, B>>
currentUploads: Record<UploadId, CurrentUpload<M, B>>
allowNewUpload: boolean
recoveredState: null | Required<Pick<State<M, B>, 'files' | 'currentUploads'>>
/** `recoveredState` is a special version of state in which the files don't have any data (because the data was never stored) */
recoveredState:
| (Omit<Pick<State<M, B>, 'currentUploads'>, 'files'> & {
files: Record<UppyFileId, Omit<UppyFile<M, B>, 'data'>>
})
| null
error: string | null
files: {
[key: string]: UppyFile<M, B>
}
files: UppyFilesMap<M, B>
info: Array<{
isHidden?: boolean
type: LogLevel
@ -323,9 +336,8 @@ export interface _UppyEventMap<M extends Meta, B extends Body> {
progress: NonNullable<FileProgressStarted['preprocess']>,
) => void
progress: (progress: number) => void
restored: (pluginData: any) => void
restored: (pluginData: unknown) => void
'restore-confirmed': () => void
'restore-canceled': () => void
'restriction-failed': (file: UppyFile<M, B> | undefined, error: Error) => void
'resume-all': () => void
'retry-all': (files: UppyFile<M, B>[]) => void
@ -575,7 +587,7 @@ export class Uppy<
},
]),
),
},
} as UppyFilesMap<M, B>,
})
}
@ -609,7 +621,7 @@ export class Uppy<
...(newOpts as UppyOptions<M, B>),
restrictions: {
...this.opts.restrictions,
...(newOpts?.restrictions as Restrictions),
...newOpts?.restrictions,
},
}
@ -971,34 +983,43 @@ export class Uppy<
/**
* Create a file state object based on user-provided `addFile()` options.
*/
#transformFile(fileDescriptorOrFile: File | UppyFile<M, B>): UppyFile<M, B> {
#transformFile(
fileDescriptorOrFile: File | MinimalRequiredUppyFile<M, B>,
): UppyFile<M, B> {
// Uppy expects files in { name, type, size, data } format.
// If the actual File object is passed from input[type=file] or drag-drop,
// we normalize it to match Uppy file object
const file = (
const file =
fileDescriptorOrFile instanceof File
? {
? ({
name: fileDescriptorOrFile.name,
type: fileDescriptorOrFile.type,
size: fileDescriptorOrFile.size,
data: fileDescriptorOrFile,
}
: fileDescriptorOrFile
) as UppyFile<M, B>
meta: {},
isRemote: false,
source: undefined,
preview: undefined,
} as const)
: (fileDescriptorOrFile as MinimalRequiredUppyFile<M, B> &
(
| Pick<LocalUppyFile<M, B>, 'isRemote' | 'data'>
| Pick<RemoteUppyFile<M, B>, 'isRemote' | 'remote' | 'data'>
))
const fileType = getFileType(file)
const fileName = getFileName(fileType, file)
const fileExtension = getFileNameAndExtension(fileName).extension
const id = getSafeFileId(file, this.getID())
const meta = file.meta || {}
meta.name = fileName
meta.type = fileType
const meta = {
...file.meta,
name: fileName,
type: fileType,
}
// `null` means the size is unknown.
const size = Number.isFinite(file.data.size)
? file.data.size
: (null as never)
const size = Number.isFinite(file.data.size) ? file.data.size : null
return {
source: file.source || '',
@ -1010,7 +1031,6 @@ export class Uppy<
...meta,
},
type: fileType,
data: file.data,
progress: {
percentage: 0,
bytesUploaded: false,
@ -1020,8 +1040,16 @@ export class Uppy<
},
size,
isGhost: false,
isRemote: file.isRemote || false,
remote: file.remote,
...(file.isRemote
? {
isRemote: true,
remote: file.remote,
data: file.data,
}
: {
isRemote: false,
data: file.data,
}),
preview: file.preview,
}
}
@ -1040,7 +1068,9 @@ export class Uppy<
}
}
#checkAndUpdateFileState(filesToAdd: UppyFile<M, B>[]): {
#checkAndUpdateFileState(
filesToAdd: (File | MinimalRequiredUppyFile<M, B>)[],
): {
nextFilesState: State<M, B>['files']
validFilesToAdd: UppyFile<M, B>[]
errors: RestrictionError<M, B>[]
@ -1056,17 +1086,20 @@ export class Uppy<
try {
let newFile = this.#transformFile(fileToAdd)
this.#assertNewUploadAllowed(newFile)
// If a file has been recovered (Golden Retriever), but we were unable to recover its data (probably too large),
// users are asked to re-select these half-recovered files and then this method will be called again.
// In order to keep the progress, meta and everything else, we keep the existing file,
// but we replace `data`, and we remove `isGhost`, because the file is no longer a ghost now
const isGhost = existingFiles[newFile.id]?.isGhost
if (isGhost) {
const existingFileState = existingFiles[newFile.id]
const existingFile = existingFiles[newFile.id]
const isGhost = existingFile?.isGhost
if (isGhost && !newFile.isRemote) {
if (newFile.data == null) throw new Error('File data is missing')
newFile = {
...existingFileState,
...existingFile,
isGhost: false,
data: fileToAdd.data,
data: newFile.data,
}
this.log(
`Replaced the blob in the restored ghost file: ${newFile.name}, ${newFile.id}`,
@ -1090,7 +1123,7 @@ export class Uppy<
this.i18n('noDuplicates', {
fileName: newFile.name ?? this.i18n('unnamed'),
}),
{ file: fileToAdd },
{ file: newFile },
)
}
@ -1099,7 +1132,7 @@ export class Uppy<
// Dont show UI info for this error, as it should be done by the developer
throw new RestrictionError(
'Cannot add the file because onBeforeFileAdded returned false.',
{ isUserFacing: false, file: fileToAdd },
{ isUserFacing: false, file: newFile },
)
} else if (
typeof onBeforeFileAddedResult === 'object' &&
@ -1149,10 +1182,8 @@ export class Uppy<
* and start an upload if `autoProceed === true`.
*/
addFile(file: File | MinimalRequiredUppyFile<M, B>): UppyFile<M, B>['id'] {
this.#assertNewUploadAllowed(file as UppyFile<M, B>)
const { nextFilesState, validFilesToAdd, errors } =
this.#checkAndUpdateFileState([file as UppyFile<M, B>])
this.#checkAndUpdateFileState([file])
const restrictionErrors = errors.filter((error) => error.isRestriction)
this.#informAndEmit(restrictionErrors)
@ -1182,10 +1213,8 @@ export class Uppy<
* Programmatic users should usually still use `addFile()` on individual files.
*/
addFiles(fileDescriptors: MinimalRequiredUppyFile<M, B>[]): void {
this.#assertNewUploadAllowed()
const { nextFilesState, validFilesToAdd, errors } =
this.#checkAndUpdateFileState(fileDescriptors as UppyFile<M, B>[])
this.#checkAndUpdateFileState(fileDescriptors)
const restrictionErrors = errors.filter((error) => error.isRestriction)
this.#informAndEmit(restrictionErrors)
@ -1445,6 +1474,9 @@ export class Uppy<
this.setState(defaultUploadState)
}
/**
* Retry a specific file that has errored.
*/
retryUpload(fileID: string): Promise<UploadResult<M, B> | undefined> {
this.setFileState(fileID, {
error: null,
@ -1690,7 +1722,7 @@ export class Uppy<
uploadComplete: false,
bytesUploaded: 0,
bytesTotal: file.size,
} as FileProgressStarted,
} satisfies FileProgressStarted,
},
]),
)
@ -1711,16 +1743,18 @@ export class Uppy<
}
const currentProgress = this.getFile(file.id).progress
const needsPostProcessing = this.#postProcessors.size > 0
this.setFileState(file.id, {
progress: {
...currentProgress,
postprocess:
this.#postProcessors.size > 0
? {
mode: 'indeterminate',
}
: undefined,
postprocess: needsPostProcessing
? {
mode: 'indeterminate',
}
: undefined,
uploadComplete: true,
...(!needsPostProcessing && { complete: true }),
percentage: 100,
bytesUploaded: currentProgress.bytesTotal,
} as FileProgressStarted,
@ -1784,25 +1818,25 @@ export class Uppy<
})
})
this.on('postprocess-complete', (file) => {
if (file == null || !this.getFile(file.id)) {
this.on('postprocess-complete', (fileIn) => {
const file = fileIn && this.getFile(fileIn.id)
if (file == null) {
this.log(
`Not setting progress for a file that has been removed: ${file?.id}`,
`Not setting progress for a file that has been removed: ${fileIn?.id}`,
)
return
}
const files = {
...this.getState().files,
}
files[file.id] = {
...files[file.id],
progress: {
...files[file.id].progress,
},
}
delete files[file.id].progress.postprocess
this.setState({ files })
const { postprocess: _deleted, ...newProgress } = file.progress
this.patchFilesState({
[file.id]: {
progress: {
...newProgress,
complete: true as const,
},
},
})
})
this.on('restored', () => {
@ -2059,7 +2093,7 @@ export class Uppy<
/** @protected */
getRequestClientForFile<Client>(file: UppyFile<M, B>): Client {
if (!file.remote)
if (!('remote' in file && file.remote))
throw new Error(
`Tried to get RequestClient for a non-remote file ${file.id}`,
)
@ -2077,13 +2111,7 @@ export class Uppy<
* Restore an upload by its ID.
*/
async restore(uploadID: string): Promise<UploadResult<M, B> | undefined> {
this.log(`Core: attempting to restore upload "${uploadID}"`)
if (!this.getState().currentUploads[uploadID]) {
this.#removeUpload(uploadID)
throw new Error('Nonexistent upload')
}
this.log(`Core: Running restored upload "${uploadID}"`)
const result = await this.#runUpload(uploadID)
this.emit('complete', result!)
return result
@ -2163,8 +2191,8 @@ export class Uppy<
*
*/
#removeUpload(uploadID: string): void {
const currentUploads = { ...this.getState().currentUploads }
delete currentUploads[uploadID]
const { [uploadID]: _deleted, ...currentUploads } =
this.getState().currentUploads
this.setState({
currentUploads,
@ -2181,6 +2209,9 @@ export class Uppy<
}
let currentUpload = getCurrentUpload()
if (!currentUpload) {
throw new Error('Nonexistent upload')
}
const steps = [
...this.#preProcessors,
@ -2204,7 +2235,13 @@ export class Uppy<
},
})
const { fileIDs } = currentUpload
// when restoring (e.g. using golden retriever), we don't need to re-upload already successfully uploaded files
// so let's exclude them here:
// https://github.com/transloadit/uppy/issues/5930
const fileIDs = currentUpload.fileIDs.filter((fileID) => {
const file = this.getFile(fileID)
return !file.progress.uploadComplete
})
// TODO give this the `updatedUpload` object as its only parameter maybe?
// Otherwise when more metadata may be added to the upload this would keep getting more parameters
@ -2299,10 +2336,8 @@ export class Uppy<
const onBeforeUploadResult = this.opts.onBeforeUpload(files)
if (onBeforeUploadResult === false) {
return Promise.reject(
new Error(
'Not starting the upload because onBeforeUpload returned false',
),
throw new Error(
'Not starting the upload because onBeforeUpload returned false',
)
}
@ -2315,52 +2350,38 @@ export class Uppy<
})
}
return Promise.resolve()
.then(() => this.#restricter.validateMinNumberOfFiles(files))
.catch((err) => {
this.#informAndEmit([err])
throw err
})
.then(() => {
if (!this.#checkRequiredMetaFields(files)) {
throw new RestrictionError(this.i18n('missingRequiredMetaField'))
}
})
.catch((err) => {
// Doing this in a separate catch because we already emited and logged
// all the errors in `checkRequiredMetaFields` so we only throw a generic
// missing fields error here.
throw err
})
.then(async () => {
const { currentUploads } = this.getState()
// get a list of files that are currently assigned to uploads
const currentlyUploadingFiles = Object.values(currentUploads).flatMap(
(curr) => curr.fileIDs,
try {
this.#restricter.validateMinNumberOfFiles(files)
if (!this.#checkRequiredMetaFields(files)) {
throw new RestrictionError(this.i18n('missingRequiredMetaField'))
}
const { currentUploads } = this.getState()
// get a list of files that are currently assigned to uploads
const currentlyUploadingFiles = Object.values(currentUploads).flatMap(
(curr) => curr.fileIDs,
)
const waitingFileIDs = Object.keys(files).filter((fileID) => {
const file = this.getFile(fileID)
// if the file hasn't started uploading and hasn't already been assigned to an upload..
return (
file &&
!file.progress.uploadStarted &&
!currentlyUploadingFiles.includes(fileID)
)
const waitingFileIDs: string[] = []
Object.keys(files).forEach((fileID) => {
const file = this.getFile(fileID)
// if the file hasn't started uploading and hasn't already been assigned to an upload..
if (
!file.progress.uploadStarted &&
currentlyUploadingFiles.indexOf(fileID) === -1
) {
waitingFileIDs.push(file.id)
}
})
const uploadID = this.#createUpload(waitingFileIDs)
const result = await this.#runUpload(uploadID)
this.emit('complete', result!)
return result
})
.catch((err) => {
this.emit('error', err)
this.log(err, 'error')
throw err
})
const uploadID = this.#createUpload(waitingFileIDs)
const result = await this.#runUpload(uploadID)
this.emit('complete', result!)
return result
} catch (err) {
this.#informAndEmit([err])
throw err
}
}
}

View file

@ -1,133 +0,0 @@
// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html
exports[`src/Core > plugins > should not be able to add a plugin that has no id 1`] = `"Your plugin must have an id"`;
exports[`src/Core > plugins > should not be able to add a plugin that has no type 1`] = `"Your plugin must have a type"`;
exports[`src/Core > plugins > should not be able to add an invalid plugin 1`] = `"Expected a plugin class, but got object. Please verify that the plugin was imported and spelled correctly."`;
exports[`src/Core > plugins > should prevent the same plugin from being added more than once 1`] = `
"Already found a plugin named 'TestSelector1'. Tried to use: 'TestSelector1'.
Uppy plugins must have unique \`id\` options. See https://uppy.io/docs/plugins/#id."
`;
exports[`src/Core > uploading a file > should only upload files that are not already assigned to another upload id 1`] = `
{
"failed": [],
"successful": [
{
"data": Uint8Array [],
"extension": "jpg",
"id": "uppy-foo/jpg-1e-image/jpeg",
"isRemote": false,
"meta": {
"name": "foo.jpg",
"type": "image/jpeg",
},
"name": "foo.jpg",
"preview": undefined,
"progress": {
"bytesTotal": null,
"bytesUploaded": 0,
"percentage": 0,
"uploadComplete": false,
"uploadStarted": null,
},
"remote": "",
"size": null,
"source": "vi",
"type": "image/jpeg",
},
{
"data": Uint8Array [],
"extension": "jpg",
"id": "uppy-bar/jpg-1e-image/jpeg",
"isRemote": false,
"meta": {
"name": "bar.jpg",
"type": "image/jpeg",
},
"name": "bar.jpg",
"preview": undefined,
"progress": {
"bytesTotal": null,
"bytesUploaded": 0,
"percentage": 0,
"uploadComplete": false,
"uploadStarted": null,
},
"remote": "",
"size": null,
"source": "vi",
"type": "image/jpeg",
},
],
"uploadID": "cjd09qwxb000dlql4tp4doz8h",
}
`;
exports[`src/Core plugins should not be able to add a plugin that has no id 1`] = `"Your plugin must have an id"`;
exports[`src/Core plugins should not be able to add a plugin that has no type 1`] = `"Your plugin must have a type"`;
exports[`src/Core plugins should not be able to add an invalid plugin 1`] = `"Expected a plugin class, but got object. Please verify that the plugin was imported and spelled correctly."`;
exports[`src/Core plugins should prevent the same plugin from being added more than once 1`] = `
"Already found a plugin named 'TestSelector1'. Tried to use: 'TestSelector1'.
Uppy plugins must have unique \`id\` options. See https://uppy.io/docs/plugins/#id."
`;
exports[`src/Core uploading a file should only upload files that are not already assigned to another upload id 1`] = `
{
"failed": [],
"successful": [
{
"data": Uint8Array [],
"extension": "jpg",
"id": "uppy-foo/jpg-1e-image/jpeg",
"isRemote": false,
"meta": {
"name": "foo.jpg",
"type": "image/jpeg",
},
"name": "foo.jpg",
"preview": undefined,
"progress": {
"bytesTotal": null,
"bytesUploaded": 0,
"percentage": 0,
"uploadComplete": false,
"uploadStarted": null,
},
"remote": "",
"size": null,
"source": "jest",
"type": "image/jpeg",
},
{
"data": Uint8Array [],
"extension": "jpg",
"id": "uppy-bar/jpg-1e-image/jpeg",
"isRemote": false,
"meta": {
"name": "bar.jpg",
"type": "image/jpeg",
},
"name": "bar.jpg",
"preview": undefined,
"progress": {
"bytesTotal": null,
"bytesUploaded": 0,
"percentage": 0,
"uploadComplete": false,
"uploadStarted": null,
},
"remote": "",
"size": null,
"source": "jest",
"type": "image/jpeg",
},
],
"uploadID": "cjd09qwxb000dlql4tp4doz8h",
}
`;

View file

@ -34,7 +34,6 @@ exports[`src/Core > uploading a file > should only upload files that are not alr
"uploadComplete": false,
"uploadStarted": null,
},
"remote": undefined,
"size": null,
"source": "vi",
"type": "image/jpeg",
@ -58,7 +57,6 @@ exports[`src/Core > uploading a file > should only upload files that are not alr
"uploadComplete": false,
"uploadStarted": null,
},
"remote": undefined,
"size": null,
"source": "vi",
"type": "image/jpeg",

View file

@ -39,7 +39,6 @@ declare module '@uppy/core' {
'dashboard:file-edit-start': DashboardFileEditStartCallback<M, B>
'dashboard:file-edit-complete': DashboardFileEditCompleteCallback<M, B>
'dashboard:close-panel': (id: string | undefined) => void
'restore-canceled': GenericEventCallback
}
}
@ -926,10 +925,6 @@ export default class Dashboard<M extends Meta, B extends Body> extends UIPlugin<
}
}
private handleCancelRestore = () => {
this.uppy.emit('restore-canceled')
}
#generateLargeThumbnailIfSingleFile = () => {
if (this.opts.disableThumbnailGenerator) {
return
@ -1246,7 +1241,6 @@ export default class Dashboard<M extends Meta, B extends Body> extends UIPlugin<
showNativeVideoCameraButton: this.opts.showNativeVideoCameraButton,
nativeCameraFacingMode: this.opts.nativeCameraFacingMode,
singleFileFullScreen: this.opts.singleFileFullScreen,
handleCancelRestore: this.handleCancelRestore,
handleRequestThumbnail: this.handleRequestThumbnail,
handleCancelThumbnail: this.handleCancelThumbnail,
// drag props

View file

@ -114,7 +114,6 @@ type DashboardUIProps<M extends Meta, B extends Body> = {
showNativeVideoCameraButton: boolean
nativeCameraFacingMode: 'user' | 'environment' | ''
singleFileFullScreen: boolean
handleCancelRestore: () => void
handleRequestThumbnail: (file: UppyFile<M, B>) => void
handleCancelThumbnail: (file: UppyFile<M, B>) => void
isDraggingOver: boolean

View file

@ -38,7 +38,8 @@ const renderFileName = (props: {
const renderAuthor = (props: { file: UppyFile<any, any> }) => {
const { author } = props.file.meta
const providerName = props.file.remote?.providerName
const providerName =
'remote' in props.file ? props.file.remote?.providerName : undefined
const dot = `\u00B7`
if (!author) {

View file

@ -1,4 +1,4 @@
import type { Body, Meta, Uppy, UppyFile } from '@uppy/core'
import type { Body, Meta, State, Uppy, UppyFile } from '@uppy/core'
import type { I18n } from '@uppy/utils'
import { emaFilter } from '@uppy/utils'
import type { ComponentChild } from 'preact'
@ -24,7 +24,7 @@ type StatusBarProps<M extends Meta, B extends Body> = {
function getUploadingState(
error: unknown,
isAllComplete: boolean,
recoveredState: any,
recoveredState: State<any, any>['recoveredState'],
files: Record<string, UppyFile<any, any>>,
): StatusBarUIProps<any, any>['uploadState'] {
if (error) {
@ -102,7 +102,6 @@ export default class StatusBar<
)
// We don't set `#lastUpdateTime` at this point because the upload won't
// actually resume until the user asks for it.
this.props.uppy.emit('restore-confirmed')
return
}
@ -187,10 +186,15 @@ export default class StatusBar<
return Math.round(filteredETA / 100) / 10
}
startUpload = (): ReturnType<Uppy<M, B>['upload']> => {
return this.props.uppy.upload().catch((() => {
// Error logged in Core
}) as () => undefined)
startUpload = (): void => {
const { recoveredState } = this.props.uppy.getState()
if (recoveredState) {
this.props.uppy.emit('restore-confirmed')
} else {
this.props.uppy.upload().catch((() => {
// Error logged in Core
}) as () => undefined)
}
}
render(): ComponentChild {

View file

@ -9,7 +9,9 @@
],
"scripts": {
"build": "tsc --build tsconfig.build.json",
"typecheck": "tsc --build"
"typecheck": "tsc --build",
"test": "vitest run --silent='passed-only'",
"test:e2e": "vitest run --project browser"
},
"keywords": [
"file uploader",
@ -47,6 +49,10 @@
"@uppy/core": "workspace:^"
},
"devDependencies": {
"typescript": "^5.8.3"
"@uppy/dashboard": "workspace:^",
"@uppy/xhr-upload": "workspace:^",
"@vitest/browser": "^3.2.4",
"typescript": "^5.8.3",
"vitest": "^3.2.4"
}
}

View file

@ -1,4 +1,4 @@
import type { UppyFile } from '@uppy/utils'
import type { UppyFileId } from '@uppy/utils'
const indexedDB =
typeof window !== 'undefined' &&
@ -23,7 +23,7 @@ const MiB = 0x10_00_00
/**
* Set default `expires` dates on existing stored blobs.
*/
function migrateExpiration(store: IDBObjectStore) {
function migrateExpiration(store: IDBObjectStore): void {
const request = store.openCursor()
request.onsuccess = (event) => {
const cursor = (event.target as IDBRequest).result
@ -78,6 +78,8 @@ function waitForRequest<T>(request: IDBRequest): Promise<T> {
})
}
type AddFilePayload = { id: UppyFileId; data: Blob }
type IndexedDBStoredFile = {
id: string
fileID: string
@ -137,14 +139,14 @@ class IndexedDBStore {
return Promise.resolve(this.#ready)
}
key(fileID: string): string {
key(fileID: UppyFileId): string {
return `${this.name}!${fileID}`
}
/**
* List all file blobs currently in the store.
*/
async list(): Promise<Record<string, IndexedDBStoredFile['data']>> {
async list(): Promise<Record<UppyFileId, IndexedDBStoredFile['data']>> {
const db = await this.#ready
const transaction = db.transaction([STORE_NAME], 'readonly')
const store = transaction.objectStore(STORE_NAME)
@ -197,8 +199,8 @@ class IndexedDBStore {
/**
* Save a file in the store.
*/
async put<T>(file: UppyFile<any, any>): Promise<T> {
if (file.data.size > this.opts.maxFileSize) {
async put<T>(file: AddFilePayload): Promise<T> {
if (file.data.size != null && file.data.size > this.opts.maxFileSize) {
throw new Error('File is too big to store.')
}
const size = await this.getSize()
@ -220,7 +222,7 @@ class IndexedDBStore {
/**
* Delete a file blob from the store.
*/
async delete(fileID: string): Promise<unknown> {
async delete(fileID: UppyFileId): Promise<unknown> {
const db = await this.#ready
const transaction = db.transaction([STORE_NAME], 'readwrite')
const request = transaction.objectStore(STORE_NAME).delete(this.key(fileID))

View file

@ -1,28 +1,21 @@
import type { Body, Meta, State as UppyState } from '@uppy/core'
import type { LocalUppyFile, RemoteUppyFile, UppyFileId } from '@uppy/utils'
import throttle from 'lodash/throttle.js'
// we don't want to store blobs in localStorage
type FileWithoutData<M extends Meta, B extends Body> =
| Omit<LocalUppyFile<M, B>, 'data'>
| Omit<RemoteUppyFile<M, B>, 'data'>
export type StoredState<M extends Meta, B extends Body> = {
expires: number
metadata: {
currentUploads: UppyState<M, B>['currentUploads']
files: UppyState<M, B>['files']
files: Record<UppyFileId, FileWithoutData<M, B>>
pluginData: Record<string, unknown>
}
}
/**
* Get uppy instance IDs for which state is stored.
*/
function findUppyInstances(): string[] {
const instances: string[] = []
for (let i = 0; i < localStorage.length; i++) {
const key = localStorage.key(i)
if (key?.startsWith('uppyState:')) {
instances.push(key.slice('uppyState:'.length))
}
}
return instances
}
/**
* Try to JSON-parse a string, return null on failure.
*/
@ -39,68 +32,104 @@ function maybeParse<M extends Meta, B extends Body>(
type MetaDataStoreOptions = {
storeName: string
expires?: number
throttleTime?: number
}
const prefix = 'uppyState:'
const getItemKey = (name: string): string => `${prefix}${name}`
function expireOldState(): void {
const existingKeys: string[] = []
for (let i = 0; i < localStorage.length; i++) {
const key = localStorage.key(i)
if (key?.startsWith(prefix)) {
existingKeys.push(key)
}
}
const now = Date.now()
existingKeys.forEach((key) => {
const data = localStorage.getItem(key)
if (!data) return
const obj = maybeParse(data)
if (obj?.expires && obj.expires < now) {
localStorage.removeItem(key)
}
})
}
let cleanedUp = false
export default class MetaDataStore<M extends Meta, B extends Body> {
opts: Required<MetaDataStoreOptions>
name: string
// biome doesn't seem to support #fields
#saveThrottled!: typeof this.save
constructor(opts: MetaDataStoreOptions) {
this.opts = {
expires: 24 * 60 * 60 * 1000, // 24 hours
throttleTime: 500,
...opts,
}
this.name = `uppyState:${opts.storeName}`
this.name = getItemKey(opts.storeName)
if (!cleanedUp) {
cleanedUp = true
MetaDataStore.cleanup()
}
this.#saveThrottled =
this.opts.throttleTime === 0
? this.save
: throttle(this.save, this.opts.throttleTime, {
leading: true,
trailing: true,
})
}
#state: StoredState<M, B> | null | undefined
/**
*
*/
load(): StoredState<M, B>['metadata'] | null {
const savedState = localStorage.getItem(this.name)
if (!savedState) return null
const data = maybeParse<M, B>(savedState)
if (!data) return null
load = (): StoredState<M, B>['metadata'] | undefined => {
expireOldState()
const savedState = localStorage.getItem(this.name)
if (!savedState) return undefined
const data = maybeParse<M, B>(savedState)
if (!data) return undefined
this.#state = data
return data.metadata
}
save(metadata: Record<string, unknown>): void {
const expires = Date.now() + this.opts.expires
const state = JSON.stringify({
metadata,
expires,
})
get = (): StoredState<M, B>['metadata'] | undefined => {
return this.#state?.metadata
}
private save = (): void => {
if (this.#state === null) {
localStorage.removeItem(this.name)
return
}
const state = JSON.stringify(this.#state)
localStorage.setItem(this.name, state)
}
/**
* Remove all expired state.
* Save the given metadata to localStorage, along with an expiry timestamp.
* If metadata is null, remove any existing stored state.
*
* @param metadata - The metadata to store, or null to clear the stored state.
*/
static cleanup(instanceID?: string): void {
if (instanceID) {
localStorage.removeItem(`uppyState:${instanceID}`)
return
}
set = (metadata: StoredState<M, B>['metadata'] | null): void => {
this.#state =
metadata === null
? null
: {
metadata,
expires: Date.now() + this.opts.expires,
}
const instanceIDs = findUppyInstances()
const now = Date.now()
instanceIDs.forEach((id) => {
const data = localStorage.getItem(`uppyState:${id}`)
if (!data) return
const obj = maybeParse(data)
if (!obj) return
if (obj.expires && obj.expires < now) {
localStorage.removeItem(`uppyState:${id}`)
}
})
this.#saveThrottled()
}
}

View file

@ -1,10 +1,11 @@
/// <reference lib="webworker" />
import type { UppyFileId } from '@uppy/utils'
declare const self: ServiceWorkerGlobalScope
type StoreName = string
type FileId = string
type CachedStore = Record<FileId, Blob>
type CachedStore = Record<UppyFileId, Blob>
type FileCache = Record<StoreName, CachedStore>
const fileCache: FileCache = Object.create(null)
@ -22,7 +23,7 @@ self.addEventListener('activate', (event: ExtendableEvent) => {
event.waitUntil(self.clients.claim())
})
type AllFilesMessage = {
export type AllFilesMessage = {
type: 'uppy/ALL_FILES'
store: StoreName
files: CachedStore
@ -36,13 +37,13 @@ function sendMessageToAllClients(msg: AllFilesMessage): void {
})
}
type AddFilePayload = { id: FileId; data: Blob }
export type AddFilePayload = { id: UppyFileId; data: Blob }
function addFile(store: StoreName, file: AddFilePayload): void {
getCache(store)[file.id] = file.data
}
function removeFile(store: StoreName, fileID: FileId): void {
function removeFile(store: StoreName, fileID: UppyFileId): void {
delete getCache(store)[fileID]
}
@ -54,9 +55,9 @@ function getFiles(store: StoreName): void {
})
}
type IncomingMessage =
export type IncomingMessage =
| { type: 'uppy/ADD_FILE'; store: StoreName; file: AddFilePayload }
| { type: 'uppy/REMOVE_FILE'; store: StoreName; fileID: FileId }
| { type: 'uppy/REMOVE_FILE'; store: StoreName; fileID: UppyFileId }
| { type: 'uppy/GET_FILES'; store: StoreName }
self.addEventListener('message', (event: ExtendableMessageEvent) => {

View file

@ -1,4 +1,9 @@
import type { Body, Meta, UppyFile } from '@uppy/utils'
import type { Body, Meta, UppyFile, UppyFileId } from '@uppy/utils'
import type {
AddFilePayload,
AllFilesMessage,
IncomingMessage,
} from './ServiceWorker.js'
const isSupported =
typeof navigator !== 'undefined' && 'serviceWorker' in navigator
@ -28,7 +33,7 @@ type ServiceWorkerStoreOptions = {
storeName: string
}
class ServiceWorkerStore<M extends Meta, B extends Body> {
class ServiceWorkerStore {
#ready: void | Promise<void>
name: string
@ -46,11 +51,11 @@ class ServiceWorkerStore<M extends Meta, B extends Body> {
return Promise.resolve(this.#ready)
}
async list(): Promise<ServiceWorkerStoredFile<M, B>[]> {
async list(): Promise<Record<UppyFileId, Blob>> {
await this.#ready
return new Promise((resolve, reject) => {
const onMessage = (event: MessageEvent) => {
return new Promise<Record<UppyFileId, Blob>>((resolve, reject) => {
const onMessage = (event: MessageEvent<AllFilesMessage>) => {
if (event.data.store !== this.name) {
return
}
@ -69,26 +74,26 @@ class ServiceWorkerStore<M extends Meta, B extends Body> {
navigator.serviceWorker.controller!.postMessage({
type: 'uppy/GET_FILES',
store: this.name,
})
} satisfies IncomingMessage)
})
}
async put(file: UppyFile<any, any>): Promise<void> {
async put(file: AddFilePayload): Promise<void> {
await this.#ready
navigator.serviceWorker.controller!.postMessage({
type: 'uppy/ADD_FILE',
store: this.name,
file,
})
} satisfies IncomingMessage)
}
async delete(fileID: string): Promise<void> {
async delete(fileID: UppyFileId): Promise<void> {
await this.#ready
navigator.serviceWorker.controller!.postMessage({
type: 'uppy/REMOVE_FILE',
store: this.name,
fileID,
})
} satisfies IncomingMessage)
}
}

View file

@ -1,10 +0,0 @@
import IndexedDBStore from './IndexedDBStore.js'
import MetaDataStore from './MetaDataStore.js'
/**
* Clean old blobs without needing to import all of Uppy.
*/
export default function cleanup(): void {
MetaDataStore.cleanup()
IndexedDBStore.cleanup()
}

View file

@ -0,0 +1,231 @@
import Uppy, { type UppyEventMap, type UppyOptions } from '@uppy/core'
import Dashboard from '@uppy/dashboard'
import XHRUpload from '@uppy/xhr-upload'
import { page, userEvent } from '@vitest/browser/context'
import { beforeEach, describe, expect } from 'vitest'
import '@uppy/core/css/style.css'
import '@uppy/dashboard/css/style.css'
import { HttpResponse, http } from 'msw'
import GoldenRetriever from './index.js'
import { test } from './test-extend.js'
function createUppy(
{ withPageReload = false }: { withPageReload?: boolean } = {},
opts?: Partial<UppyOptions<any, any>>,
) {
if (withPageReload) {
// simulate page reload
document.body.innerHTML = ''
}
const root = document.createElement('div')
document.body.appendChild(root)
return new Uppy().use(Dashboard, {
target: root,
inline: true,
...opts,
})
}
beforeEach(async () => {
// @ts-expect-error dunno
GoldenRetriever[Symbol.for('uppy test: throttleTime')] = 0
// clear any previously restored files so they don't interfere with tests
new Uppy().use(GoldenRetriever).clear()
document.body.innerHTML = ''
})
const createMockFile = ({
size,
name = `${Date.now()}.txt`,
type = 'text/plain',
}: {
name?: string
size: number
type?: string
}) => new File(['a'.repeat(size)], name, { type })
describe('Golden retriever', () => {
test('Restore files', async ({ worker }) => {
worker.use(
http.post('http://localhost/upload', () => HttpResponse.json({})),
)
let uppy = createUppy().use(GoldenRetriever)
const fileInput = document.querySelector('.uppy-Dashboard-input')!
const file = createMockFile({ size: 50000 })
await userEvent.upload(fileInput, file)
// reload page and recreate Uppy instance
uppy = createUppy({ withPageReload: true })
.use(GoldenRetriever)
.use(XHRUpload, {
endpoint: 'http://localhost/upload',
})
await new Promise((resolve) => uppy.once('restored', resolve))
expect(uppy.getFiles().length).toBe(1)
await expect.element(page.getByText(file.name)).toBeVisible()
// Start the upload
await page.getByRole('button', { name: 'Upload 1 file' }).click()
await expect
.element(page.getByText('Complete', { exact: true }))
.toBeVisible()
// reload page and recreate Uppy instance and wait for golden retriever to initialize
uppy = createUppy({ withPageReload: true })
const promise = new Promise<void>((resolve) =>
uppy.once('plugin-added', (plugin) => {
if (plugin.id === 'GoldenRetriever') resolve()
}),
)
uppy.use(GoldenRetriever)
await promise
// make sure that the restored file is cleared after successful upload
expect(uppy.getFiles().length).toBe(0)
})
test('Should not re-upload completed files', async ({ worker }) => {
let requestAt = 0
worker.use(
http.post('http://localhost/upload', () => {
if (requestAt === 0) {
requestAt += 1
return HttpResponse.json({})
}
// never reply to subsequent requests (leave them hanging)
}),
)
let uppy = createUppy().use(GoldenRetriever).use(XHRUpload, {
endpoint: 'http://localhost/upload',
})
const fileInput = document.querySelector('.uppy-Dashboard-input')!
await userEvent.upload(fileInput, [
createMockFile({ size: 50000 }),
createMockFile({ size: 50000 }),
])
const uploadFirstFileCompletePromise = new Promise<void>((resolve) =>
uppy.once('upload-success', () => resolve()),
)
// Start the upload
await page.getByRole('button', { name: 'Upload 2 files' }).click()
await uploadFirstFileCompletePromise
// reload page and recreate Uppy instance
uppy = createUppy({ withPageReload: true })
.use(GoldenRetriever)
.use(XHRUpload, {
endpoint: 'http://localhost/upload',
})
await new Promise((resolve) => uppy.once('restored', resolve))
worker.resetHandlers()
requestAt = 0 // reset request counter
worker.use(
http.post('http://localhost/upload', () => {
if (requestAt === 0) {
requestAt += 1
return HttpResponse.json({})
}
// don't allow more than 1 request
return new HttpResponse({ status: 400 })
}),
)
// Start the upload
await page.getByRole('button', { name: 'Upload 2 files' }).click()
await expect
.element(page.getByText('Complete', { exact: true }))
.toBeVisible()
expect(uppy.getFiles().length).toBe(2)
})
test('Should not clean up files upon completion if there were failed uploads and it should only make the failed file a ghost', async ({
worker,
}) => {
let requestAt = 0
let respondSecondRequest: (() => void) | undefined
worker.use(
http.post('http://localhost/upload', async () => {
if (requestAt === 0) {
requestAt += 1
return HttpResponse.json({})
}
await new Promise<void>((resolve) => {
respondSecondRequest = resolve
})
return new HttpResponse({ status: 400 })
}),
)
let uppy = createUppy().use(GoldenRetriever).use(XHRUpload, {
endpoint: 'http://localhost/upload',
})
const fileInput = document.querySelector('.uppy-Dashboard-input')!
await userEvent.upload(fileInput, [
createMockFile({ size: 50000 }),
createMockFile({ size: 50000 }),
])
const uploadFirstFileCompletePromise = new Promise<void>((resolve) =>
uppy.once('upload-success', () => resolve()),
)
// Start the upload
await page.getByRole('button', { name: 'Upload 2 files' }).click()
// wait for the first file to finish uploading
await uploadFirstFileCompletePromise
// let the second file fail
respondSecondRequest!()
const uploadCompletePromise = new Promise<
Parameters<UppyEventMap<any, any>['complete']>[0]
>((resolve) => uppy.once('complete', resolve))
const completedFiles = await uploadCompletePromise
expect(completedFiles.successful?.length).toBe(1)
expect(completedFiles.failed?.length).toBe(1)
const fileIds = uppy.getFiles().map((f) => f.id)
// Simulate ghosting of the files by deleting it from store(s)
// @ts-expect-error
;(uppy.getPlugin('GoldenRetriever') as GoldenRetriever<any, any>)[
Symbol.for('uppy test: deleteBlobs')
](fileIds)
// reload page and recreate Uppy instance
uppy = createUppy({ withPageReload: true }).use(GoldenRetriever)
await new Promise((resolve) => uppy.once('restored', resolve))
// make sure that the failed files are still there
expect(uppy.getFiles().length).toBe(2)
const errorFile = uppy.getFiles().find((f) => f.error)
expect(errorFile).toBeDefined()
expect(errorFile!.isGhost).toBeTruthy()
const successfulFile = uppy
.getFiles()
.find((f) => f.progress.uploadComplete)
expect(successfulFile).toBeDefined()
// even though the successful file was deleted from store(s), it should not be a ghost
expect(successfulFile!.isGhost).toBeFalsy()
})
})

View file

@ -3,24 +3,21 @@ import type {
DefinePluginOpts,
Meta,
PluginOpts,
UploadResult,
State,
Uppy,
UppyFile,
} from '@uppy/core'
import { BasePlugin } from '@uppy/core'
import throttle from 'lodash/throttle.js'
import type { UppyFileId } from '@uppy/utils'
import packageJson from '../package.json' with { type: 'json' }
import IndexedDBStore from './IndexedDBStore.js'
import MetaDataStore from './MetaDataStore.js'
import ServiceWorkerStore, {
type ServiceWorkerStoredFile,
} from './ServiceWorkerStore.js'
import ServiceWorkerStore from './ServiceWorkerStore.js'
declare module '@uppy/core' {
// biome-ignore lint/correctness/noUnusedVariables: must be defined
export interface UppyEventMap<M extends Meta, B extends Body> {
// TODO: remove this event
'restore:get-data': (fn: (data: Record<string, unknown>) => void) => void
'restore:plugin-data-changed': (data: Record<string, unknown>) => void
}
}
@ -56,386 +53,367 @@ export default class GoldenRetriever<
> extends BasePlugin<Opts, M, B> {
static VERSION = packageJson.version
MetaDataStore: MetaDataStore<M, B>
#metaDataStore: MetaDataStore<M, B>
ServiceWorkerStore: ServiceWorkerStore<M, B> | null
#serviceWorkerStore: ServiceWorkerStore | undefined
IndexedDBStore: IndexedDBStore
#indexedDBStore: IndexedDBStore
savedPluginData?: Record<string, unknown>
// @ts-expect-error for tests
static [Symbol.for('uppy test: throttleTime')]: number | undefined
constructor(uppy: Uppy<M, B>, opts?: GoldenRetrieverOptions) {
super(uppy, { ...defaultOptions, ...opts })
this.type = 'debugger'
this.id = this.opts.id || 'GoldenRetriever'
this.MetaDataStore = new MetaDataStore({
this.#metaDataStore = new MetaDataStore({
expires: this.opts.expires,
storeName: uppy.getID(),
throttleTime:
// @ts-expect-error for tests
GoldenRetriever[Symbol.for('uppy test: throttleTime')] ?? undefined,
})
this.ServiceWorkerStore = null
if (this.opts.serviceWorker) {
this.ServiceWorkerStore = new ServiceWorkerStore({
this.#serviceWorkerStore = new ServiceWorkerStore({
storeName: uppy.getID(),
})
}
this.IndexedDBStore = new IndexedDBStore({
this.#indexedDBStore = new IndexedDBStore({
expires: this.opts.expires,
...(this.opts.indexedDB || {}),
storeName: uppy.getID(),
})
this.saveFilesStateToLocalStorage = throttle(
this.saveFilesStateToLocalStorage.bind(this),
500,
{ leading: true, trailing: true },
)
this.restoreState = this.restoreState.bind(this)
this.loadFileBlobsFromServiceWorker =
this.loadFileBlobsFromServiceWorker.bind(this)
this.loadFileBlobsFromIndexedDB = this.loadFileBlobsFromIndexedDB.bind(this)
this.onBlobsLoaded = this.onBlobsLoaded.bind(this)
}
restoreState(): void {
const savedState = this.MetaDataStore.load()
if (savedState) {
this.uppy.log('[GoldenRetriever] Recovered some state from Local Storage')
this.uppy.setState({
currentUploads: savedState.currentUploads || {},
files: savedState.files || {},
recoveredState: savedState,
})
this.savedPluginData = savedState.pluginData
}
}
saveFilesStateToLocalStorage(): void {
// File objects that are currently waiting: they've been selected,
// but aren't yet being uploaded.
const waitingFiles = this.uppy
.getFiles()
.filter((file) => !file.progress || !file.progress.uploadStarted)
// File objects that are currently being uploaded. If a file has finished
// uploading, but the other files in the same batch have not, the finished
// file is also returned.
const uploadingFiles = Object.values(this.uppy.getState().currentUploads)
.map((currentUpload) =>
currentUpload.fileIDs.map((fileID) => {
const file = this.uppy.getFile(fileID)
return file != null ? [file] : [] // file might have been removed
}),
)
.flat(2)
const allFiles = [...waitingFiles, ...uploadingFiles]
// unique by file.id
const fileToSave = Object.values(
Object.fromEntries(allFiles.map((file) => [file.id, file])),
)
// If all files have been removed by the user, clear recovery state
if (fileToSave.length === 0) {
if (this.uppy.getState().recoveredState !== null) {
this.uppy.setState({ recoveredState: null })
}
MetaDataStore.cleanup(this.uppy.opts.id)
async #restore(): Promise<void> {
const recoveredState = this.#metaDataStore.load()
if (!recoveredState) {
return
}
// We dontt need to store file.data on local files, because the actual blob will be restored later,
// and we want to avoid having weird properties in the serialized object.
// Also adding file.isRestored to all files, since they will be restored from local storage
const filesToSaveWithoutData = Object.fromEntries(
fileToSave.map((fileInfo) => [
fileInfo.id,
fileInfo.isRemote
? {
...fileInfo,
isRestored: true,
}
: {
...fileInfo,
isRestored: true,
data: null,
preview: null,
},
]),
const currentUploads = recoveredState.currentUploads || {}
const files = recoveredState.files || {}
this.uppy.log(
`[GoldenRetriever] Recovered ${Object.keys(currentUploads).length} current uploads and ${Object.keys(files).length} files from Local Storage`,
)
const pluginData = {}
// TODO Remove this,
// Other plugins can attach a restore:get-data listener that receives this callback.
// Plugins can then use this callback (sync) to provide data to be stored.
this.uppy.emit('restore:get-data', (data) => {
Object.assign(pluginData, data)
})
const { currentUploads } = this.uppy.getState()
this.MetaDataStore.save({
currentUploads,
files: filesToSaveWithoutData,
pluginData,
})
}
loadFileBlobsFromServiceWorker(): Promise<
ServiceWorkerStoredFile<M, B> | Record<string, unknown>
> {
if (!this.ServiceWorkerStore) {
return Promise.resolve({})
if (Object.keys(recoveredState.files).length <= 0) {
this.uppy.log(
'[GoldenRetriever] No files need to be loaded, restored only processing state...',
)
return
}
return this.ServiceWorkerStore.list()
.then((blobs) => {
const numberOfFilesRecovered = Object.keys(blobs).length
if (numberOfFilesRecovered > 0) {
this.uppy.log(
`[GoldenRetriever] Successfully recovered ${numberOfFilesRecovered} blobs from Service Worker!`,
)
return blobs
}
this.uppy.log(
'[GoldenRetriever] No blobs found in Service Worker, trying IndexedDB now...',
)
return {}
})
.catch((err) => {
this.uppy.log(
'[GoldenRetriever] Failed to recover blobs from Service Worker',
'warning',
)
this.uppy.log(err)
return {}
})
}
loadFileBlobsFromIndexedDB(): ReturnType<IndexedDBStore['list']> {
return this.IndexedDBStore.list()
.then((blobs) => {
const numberOfFilesRecovered = Object.keys(blobs).length
if (numberOfFilesRecovered > 0) {
this.uppy.log(
`[GoldenRetriever] Successfully recovered ${numberOfFilesRecovered} blobs from IndexedDB!`,
)
return blobs
}
this.uppy.log('[GoldenRetriever] No blobs found in IndexedDB')
return {}
})
.catch((err) => {
this.uppy.log(
'[GoldenRetriever] Failed to recover blobs from IndexedDB',
'warning',
)
this.uppy.log(err)
return {}
})
}
onBlobsLoaded(blobs: Record<string, Blob>): void {
const obsoleteBlobs: string[] = []
const updatedFiles = { ...this.uppy.getState().files }
const [serviceWorkerBlobs, indexedDbBlobs] = await Promise.all([
this.#loadFileBlobsFromServiceWorker(),
this.#loadFileBlobsFromIndexedDB(),
])
const blobs = {
...serviceWorkerBlobs,
...indexedDbBlobs,
}
// Loop through blobs that we can restore, add blobs to file objects
Object.keys(blobs).forEach((fileID) => {
const originalFile = this.uppy.getFile(fileID)
if (!originalFile) {
obsoleteBlobs.push(fileID)
return
}
const filesWithBlobs: Record<
UppyFileId,
UppyFile<M, B>
> = Object.fromEntries(
Object.entries(files).map(
([fileID, file]): [UppyFileId, UppyFile<M, B>] => {
if (file.isRemote) {
return [
fileID,
{
...file,
isRestored: true,
data: { size: null }, // todo shouldn't we save/restore the size too?
},
]
}
const cachedData = blobs[fileID]
const updatedFileData = {
data: cachedData,
isRestored: true,
isGhost: false,
}
updatedFiles[fileID] = { ...originalFile, ...updatedFileData }
})
// Loop through files that we cant restore fully — we only have meta, not blobs,
// set .isGhost on them, also set isRestored to all files
Object.keys(updatedFiles).forEach((fileID) => {
if (updatedFiles[fileID].data === null) {
updatedFiles[fileID] = {
...updatedFiles[fileID],
isGhost: true,
}
}
})
const blob: Blob | undefined = blobs[fileID]
return [
fileID,
!file.progress.uploadComplete && blob == null
? // if we dont have the blob (and the file is not completed uploading), mark the file as a ghost
{
...file,
isRestored: true,
isGhost: true,
data: undefined,
}
: {
...file,
isRestored: true,
isGhost: false,
data: blob,
},
]
},
),
)
this.uppy.setState({
files: updatedFiles,
recoveredState,
currentUploads,
files: filesWithBlobs,
})
this.uppy.emit('restored', this.savedPluginData)
this.uppy.emit('restored', recoveredState.pluginData) // must adhere to PersistentState interface in Transloadit
const obsoleteBlobs = Object.keys(blobs).filter((fileID) => !files[fileID])
if (obsoleteBlobs.length) {
this.deleteBlobs(obsoleteBlobs)
.then(() => {
try {
this.uppy.log(
`[GoldenRetriever] Cleaning up ${obsoleteBlobs.length} old files`,
)
await this.#deleteBlobs(obsoleteBlobs)
} catch (err) {
this.uppy.log(
`[GoldenRetriever] Could not clean up ${obsoleteBlobs.length} old files`,
'warning',
)
this.uppy.log(err)
}
}
}
#patchMetadata = ({
pluginData,
...patch
}: Partial<NonNullable<ReturnType<MetaDataStore<M, B>['load']>>>): void => {
const existing = this.#metaDataStore.get()
this.#metaDataStore.set({
...(existing ?? {
currentUploads: {},
files: {},
}),
...patch,
pluginData: {
// pluginData is keyed by plugin id, so we merge instead of replace
...existing?.pluginData,
...pluginData,
},
})
}
async #loadFileBlobsFromServiceWorker(): Promise<Record<UppyFileId, Blob>> {
if (!this.#serviceWorkerStore) {
return {}
}
try {
const blobs = await this.#serviceWorkerStore.list()
const numberOfFilesRecovered = Object.keys(blobs).length
this.uppy.log(
numberOfFilesRecovered > 0
? `[GoldenRetriever] Successfully recovered ${numberOfFilesRecovered} blobs from Service Worker!`
: '[GoldenRetriever] No blobs found in Service Worker',
)
return blobs
} catch (err) {
this.uppy.log(
'[GoldenRetriever] Failed to recover blobs from Service Worker',
'warning',
)
this.uppy.log(err)
return {}
}
}
async #loadFileBlobsFromIndexedDB(): ReturnType<IndexedDBStore['list']> {
try {
const blobs = await this.#indexedDBStore.list()
const numberOfFilesRecovered = Object.keys(blobs).length
this.uppy.log(
numberOfFilesRecovered > 0
? `[GoldenRetriever] Successfully recovered ${numberOfFilesRecovered} blobs from IndexedDB!`
: '[GoldenRetriever] No blobs found in IndexedDB',
)
return blobs
} catch (err) {
this.uppy.log(
'[GoldenRetriever] Failed to recover blobs from IndexedDB',
'warning',
)
this.uppy.log(err)
return {}
}
}
async #deleteBlobs(fileIDs: UppyFileId[]): Promise<void> {
await Promise.all(
fileIDs.map(async (id) => {
try {
await Promise.all([
this.#serviceWorkerStore?.delete(id),
this.#indexedDBStore.delete(id),
])
} catch (err) {
this.uppy.log(
`[GoldenRetriever] Cleaned up ${obsoleteBlobs.length} old files`,
)
})
.catch((err) => {
this.uppy.log(
`[GoldenRetriever] Could not clean up ${obsoleteBlobs.length} old files`,
`[GoldenRetriever] Could not remove file ${id} from all stores`,
'warning',
)
this.uppy.log(err)
})
}
}
async deleteBlobs(fileIDs: string[]): Promise<void> {
await Promise.all(
fileIDs.map(
(id) =>
this.ServiceWorkerStore?.delete(id) ??
this.IndexedDBStore?.delete(id),
),
}
}),
)
if (fileIDs.length > 0) {
this.uppy.log(`[GoldenRetriever] Removed ${fileIDs.length} blobs`)
}
}
addBlobToStores = (file: UppyFile<M, B>): void => {
if (file.isRemote) return
async [Symbol.for('uppy test: deleteBlobs')](fileIDs: UppyFileId[]) {
return this.#deleteBlobs(fileIDs)
}
if (this.ServiceWorkerStore) {
this.ServiceWorkerStore.put(file).catch((err) => {
this.uppy.log('[GoldenRetriever] Could not store file', 'warning')
#addBlobToStores = async (file: UppyFile<M, B>): Promise<void> => {
const { id, data, isRemote } = file
if (isRemote || data == null) return
await Promise.all([
this.#serviceWorkerStore?.put({ id, data }).catch((err) => {
this.uppy.log(
'[GoldenRetriever] Could not store file in Service Worker',
'warning',
)
this.uppy.log(err)
})
}),
this.#indexedDBStore.put({ id, data }).catch((err) => {
// idempotent; assume "Key already exists in the object store"
if (
err instanceof Event &&
err.target instanceof IDBRequest &&
err.target.error?.name === 'ConstraintError'
) {
return
}
this.uppy.log(
'[GoldenRetriever] Could not store file in IndexedDB',
'warning',
)
this.uppy.log(err)
}),
])
}
#handleStateUpdate = (
prevState: State<M, B>,
nextState: State<M, B>,
patch: Partial<State<M, B>> | undefined,
): void => {
if (nextState.currentUploads !== prevState.currentUploads) {
const { currentUploads } = this.uppy.getState()
this.#patchMetadata({ currentUploads })
}
this.IndexedDBStore.put(file).catch((err) => {
this.uppy.log('[GoldenRetriever] Could not store file', 'warning')
this.uppy.log(err)
})
}
if (nextState.files !== prevState.files) {
// If all files have completed *successfully*, remove the whole stored restoration state.
// This makes sure that if the upload was only partially successful, the user can still restore and upload the remaining files.
// Here are some scenarios we have to take into account:
// todo (make unit/e2e tests for these scenarios)
// - the user removes all uploads one by one (once all are removed, we should not restore anything after reloading page)
// - the user uploads files with Transloadit plugin enabled, uploads complete successfully, and the user refreshes the page while the assembly is still running. golden retriever should then restore the files, and the ongoing assembly should progress
// - once a file finishes uploading successfully, it should have it its blob removed (even if a post processing step remains). if not successful upload it should not be removed
if (
Object.values(prevState.files).some((f) => !f.progress.complete) &&
(Object.values(nextState.files).length === 0 ||
Object.values(nextState.files).every(
(f) => f.progress.complete && !f.error,
))
) {
this.uppy.log(
`[GoldenRetriever] All files have been uploaded and processed successfully, clearing recovery state`,
)
this.uppy.setState({ recoveredState: null })
this.#metaDataStore.set(null)
} else {
// We dont want to store file.data on local files, because the actual blob is too large and should therefore stored separately,
// and we want to avoid having weird properties in the serialized object (like file.preview).
const filesWithoutBlobs = Object.fromEntries(
Object.entries(nextState.files).map(
([fileID, { data, preview, ...fileInfo }]) => [fileID, fileInfo],
),
)
this.#patchMetadata({ files: filesWithoutBlobs })
}
removeBlobFromStores = (file: UppyFile<M, B>): void => {
if (this.ServiceWorkerStore) {
this.ServiceWorkerStore.delete(file.id).catch((err) => {
this.uppy.log('[GoldenRetriever] Failed to remove file', 'warning')
this.uppy.log(err)
const addedFiles = Object.values(nextState.files).filter(
(nextFile) => prevState.files[nextFile.id] == null,
)
const editedFileBlobs = Object.values(nextState.files).flatMap(
(nextFile) => {
const prevFile = prevState.files[nextFile.id]
if (prevFile != null && nextFile.data !== prevFile.data)
return [nextFile]
return []
},
)
const deletedFiles = Object.values(prevState.files).filter((prevFile) => {
const nextFile = nextState.files[prevFile.id]
// also treat successfully uploaded files as deleted (when it comes to deleting their blob)
return (
nextFile == null ||
(nextFile.progress.uploadComplete &&
!prevFile.progress.uploadComplete)
)
})
const blobsToDelete = [...deletedFiles, ...editedFileBlobs]
const blobsToAdd = [...addedFiles, ...editedFileBlobs]
;(async () => {
// delete old blobs that have been removed, or edited
await this.#deleteBlobs(blobsToDelete.map((f) => f.id))
// add new blobs for new files and edited files
for (const blob of blobsToAdd) {
await this.#addBlobToStores(blob)
}
if (blobsToAdd.length > 0) {
this.uppy.log(`[GoldenRetriever] Added ${blobsToAdd.length} blobs`)
}
})()
}
this.IndexedDBStore.delete(file.id).catch((err) => {
this.uppy.log('[GoldenRetriever] Failed to remove file', 'warning')
this.uppy.log(err)
})
}
replaceBlobInStores = (file: UppyFile<M, B>): void => {
this.removeBlobFromStores(file)
this.addBlobToStores(file)
}
handleRestoreConfirmed = (): void => {
#handleRestoreConfirmed = (): void => {
this.uppy.log('[GoldenRetriever] Restore confirmed, proceeding...')
// start all uploads again when file blobs are restored
const { currentUploads } = this.uppy.getState()
if (currentUploads) {
if (Object.keys(currentUploads).length > 0) {
this.uppy.resumeAll()
Object.keys(currentUploads).forEach((uploadId) => {
this.uppy.restore(uploadId)
})
}
this.uppy.setState({ recoveredState: null })
}
abortRestore = (): void => {
this.uppy.log('[GoldenRetriever] Aborting restore...')
const fileIDs = Object.keys(this.uppy.getState().files)
this.deleteBlobs(fileIDs)
.then(() => {
this.uppy.log(`[GoldenRetriever] Removed ${fileIDs.length} files`)
})
.catch((err) => {
this.uppy.log(
`[GoldenRetriever] Could not remove ${fileIDs.length} files`,
'warning',
)
this.uppy.log(err)
})
this.uppy.cancelAll()
this.uppy.setState({ recoveredState: null })
MetaDataStore.cleanup(this.uppy.opts.id)
}
handleComplete = ({ successful }: UploadResult<M, B>): void => {
const fileIDs = successful!.map((file) => file.id)
this.deleteBlobs(fileIDs)
.then(() => {
this.uppy.log(
`[GoldenRetriever] Removed ${successful!.length} files that finished uploading`,
)
})
.catch((err) => {
this.uppy.log(
`[GoldenRetriever] Could not remove ${successful!.length} files that finished uploading`,
'warning',
)
this.uppy.log(err)
})
this.uppy.setState({ recoveredState: null })
MetaDataStore.cleanup(this.uppy.opts.id)
}
restoreBlobs = (): void => {
if (this.uppy.getFiles().length > 0) {
Promise.all([
this.loadFileBlobsFromServiceWorker(),
this.loadFileBlobsFromIndexedDB(),
]).then((resultingArrayOfObjects) => {
const blobs = {
...resultingArrayOfObjects[0],
...resultingArrayOfObjects[1],
} as Record<string, Blob>
this.onBlobsLoaded(blobs)
})
} else {
this.uppy.log(
'[GoldenRetriever] No files need to be loaded, only restoring processing state...',
)
// if there are no current uploads, but there were files added just start a new upload with the current files
this.uppy.upload()
}
this.uppy.setState({ recoveredState: null })
}
#handlePluginDataChanged = (data: Record<string, unknown>): void => {
this.#patchMetadata({ pluginData: data })
}
install(): void {
this.restoreState()
this.restoreBlobs()
this.#restore()
this.uppy.on('file-added', this.addBlobToStores)
// @ts-expect-error this is typed in @uppy/image-editor and we can't access those types.
this.uppy.on('file-editor:complete', this.replaceBlobInStores)
this.uppy.on('file-removed', this.removeBlobFromStores)
// TODO: the `state-update` is bad practise. It fires on any state change in Uppy
// or any state change in any of the plugins. We should to able to only listen
// for the state changes we need, somehow.
this.uppy.on('state-update', this.saveFilesStateToLocalStorage)
this.uppy.on('restore-confirmed', this.handleRestoreConfirmed)
this.uppy.on('restore-canceled', this.abortRestore)
this.uppy.on('complete', this.handleComplete)
this.uppy.on('state-update', this.#handleStateUpdate)
this.uppy.on('restore-confirmed', this.#handleRestoreConfirmed)
this.uppy.on('restore:plugin-data-changed', this.#handlePluginDataChanged)
}
uninstall(): void {
this.uppy.off('file-added', this.addBlobToStores)
// @ts-expect-error this is typed in @uppy/image-editor and we can't access those types.
this.uppy.off('file-editor:complete', this.replaceBlobInStores)
this.uppy.off('file-removed', this.removeBlobFromStores)
this.uppy.off('state-update', this.saveFilesStateToLocalStorage)
this.uppy.off('restore-confirmed', this.handleRestoreConfirmed)
this.uppy.off('restore-canceled', this.abortRestore)
this.uppy.off('complete', this.handleComplete)
this.uppy.off('state-update', this.#handleStateUpdate)
this.uppy.off('restore-confirmed', this.#handleRestoreConfirmed)
this.uppy.off('restore:plugin-data-changed', this.#handlePluginDataChanged)
}
}

View file

@ -0,0 +1,3 @@
import { setupWorker } from 'msw/browser'
export const worker = setupWorker()

View file

@ -0,0 +1,22 @@
import { test as testBase } from 'vitest'
import { worker } from './mocks/browser.js'
export const test = testBase.extend<{ worker: typeof worker }>({
worker: [
// biome-ignore lint/correctness/noEmptyPattern: dunno
async ({}, use) => {
// Start the worker before the test.
await worker.start()
// Expose the worker object on the test's context.
await use(worker)
// Remove any request handlers added in individual test cases.
// This prevents them from affecting unrelated tests.
worker.resetHandlers()
},
{
auto: true,
},
],
})

View file

@ -0,0 +1,29 @@
import { defineConfig } from 'vitest/config'
export default defineConfig({
test: {
projects: [
{
test: {
name: 'unit',
include: [
'src/**/*.test.{ts,tsx}',
'!src/**/*.browser.test.{ts,tsx}',
],
environment: 'jsdom',
},
},
{
test: {
name: 'browser',
include: ['src/**/*.browser.test.{ts,tsx}'],
browser: {
enabled: true,
provider: 'playwright',
instances: [{ browser: 'chromium' }],
},
},
},
],
},
})

View file

@ -1,5 +1,5 @@
import type { Body, Meta, UppyFile } from '@uppy/core'
import type { I18n } from '@uppy/utils'
import type { Body, Meta } from '@uppy/core'
import type { I18n, LocalUppyFile } from '@uppy/utils'
import Cropper from 'cropperjs'
import { Component } from 'preact'
import type ImageEditor from './ImageEditor.js'
@ -9,7 +9,7 @@ import limitCropboxMovementOnMove from './utils/limitCropboxMovementOnMove.js'
import limitCropboxMovementOnResize from './utils/limitCropboxMovementOnResize.js'
type Props<M extends Meta, B extends Body> = {
currentImage: UppyFile<M, B>
currentImage: LocalUppyFile<M, B>
storeCropperInstance: (cropper: Cropper) => void
opts: ImageEditor<M, B>['opts']
i18n: I18n
@ -375,6 +375,7 @@ export default class Editor<M extends Meta, B extends Body> extends Component<
render() {
const { currentImage, opts } = this.props
const { actions } = opts
if (currentImage.data == null) throw new Error('File data is empty')
const imageURL = URL.createObjectURL(currentImage.data)
return (

View file

@ -55,7 +55,7 @@ const getCheckedFilesWithPaths = (
const absDirPath = `/${absFolders.map((i) => i.data.name).join('/')}`
const relDirPath =
relFolders.length === 1
? // Must return `undefined` (which later turns into `null` in `.getTagFile()`)
? // Must return `undefined` (which later turns into `null` in `.companionFileToUppyFile()`)
// (https://github.com/transloadit/uppy/pull/4537#issuecomment-1629136652)
undefined
: relFolders.map((i) => i.data.name).join('/')

View file

@ -5,31 +5,31 @@ import type {
CompanionClientSearchProvider,
CompanionFile,
Meta,
TagFile,
UppyFileNonGhost,
} from '@uppy/utils'
import { getSafeFileId } from '@uppy/utils'
import getTagFile from './getTagFile.js'
import companionFileToUppyFile from './companionFileToUppyFile.js'
const addFiles = <M extends Meta, B extends Body>(
companionFiles: CompanionFile[],
plugin: UnknownPlugin<M, B>,
provider: CompanionClientProvider | CompanionClientSearchProvider,
): void => {
const tagFiles: TagFile<M>[] = companionFiles.map((f) =>
getTagFile<M, B>(f, plugin, provider),
const uppyFiles = companionFiles.map((f) =>
companionFileToUppyFile<M, B>(f, plugin, provider),
)
const filesToAdd: TagFile<M>[] = []
const filesAlreadyAdded: TagFile<M>[] = []
tagFiles.forEach((tagFile) => {
const filesToAdd: UppyFileNonGhost<M, B>[] = []
const filesAlreadyAdded: UppyFileNonGhost<M, B>[] = []
uppyFiles.forEach((file) => {
if (
plugin.uppy.checkIfFileAlreadyExists(
getSafeFileId(tagFile, plugin.uppy.getID()),
getSafeFileId(file, plugin.uppy.getID()),
)
) {
filesAlreadyAdded.push(tagFile)
filesAlreadyAdded.push(file)
} else {
filesToAdd.push(tagFile)
filesToAdd.push(file)
}
})

View file

@ -5,24 +5,27 @@ import type {
CompanionClientSearchProvider,
CompanionFile,
Meta,
TagFile,
RemoteUppyFile,
} from '@uppy/utils'
// TODO: document what is a "tagFile" or get rid of this concept
const getTagFile = <M extends Meta, B extends Body>(
const companionFileToUppyFile = <M extends Meta, B extends Body>(
file: CompanionFile,
plugin: UnknownPlugin<M, B>,
provider: CompanionClientProvider | CompanionClientSearchProvider,
): TagFile<M> => {
const tagFile: TagFile<any> = {
): RemoteUppyFile<M, B> => {
const name = file.name || file.id
return {
id: file.id,
source: plugin.id,
name: file.name || file.id,
name,
type: file.mimeType,
isRemote: true,
data: file,
preview: file.thumbnail || undefined,
// @ts-expect-error TODO: fixme
meta: {
// name, // todo shouldn't this be here?
authorName: file.author?.name,
authorUrl: file.author?.url,
// We need to do this `|| null` check, because null value
@ -45,8 +48,6 @@ const getTagFile = <M extends Meta, B extends Body>(
requestClientId: provider.provider,
},
}
return tagFile
}
export default getTagFile
export default companionFileToUppyFile

View file

@ -21,7 +21,7 @@ const ETAFilterHalfLife = 2000
function getUploadingState(
error: unknown,
isAllComplete: boolean,
recoveredState: any,
recoveredState: unknown,
files: Record<string, UppyFile<any, any>>,
): StatusBarUIProps<any, any>['uploadState'] {
if (error) {

View file

@ -1,6 +1,6 @@
import type { DefinePluginOpts, UIPluginOptions, Uppy } from '@uppy/core'
import { UIPlugin } from '@uppy/core'
import type { Body, Meta, UppyFile } from '@uppy/utils'
import type { Body, LocalUppyFile, Meta, UppyFile } from '@uppy/utils'
import { dataURItoBlob, isObjectURL, isPreviewSupported } from '@uppy/utils'
// @ts-ignore untyped
import { rotation } from 'exifr/dist/mini.esm.mjs'
@ -194,10 +194,11 @@ export default class ThumbnailGenerator<
}
createThumbnail(
file: UppyFile<M, B>,
file: LocalUppyFile<M, B>,
targetWidth: number | null,
targetHeight: number | null,
): Promise<string> {
if (file.data == null) throw new Error('File data is empty')
const originalUrl = URL.createObjectURL(file.data)
const onload = new Promise<HTMLImageElement>((resolve, reject) => {

View file

@ -46,7 +46,7 @@ class TransloaditAssembly extends Emitter {
#sse: EventSource | null = null
status: AssemblyResponse
#status: AssemblyResponse
pollInterval: ReturnType<typeof setInterval> | null
@ -56,7 +56,7 @@ class TransloaditAssembly extends Emitter {
super()
// The current assembly status.
this.status = assembly
this.#status = assembly
// The interval timer for full status updates.
this.pollInterval = null
// Whether this assembly has been closed (finished or errored)
@ -78,6 +78,14 @@ class TransloaditAssembly extends Emitter {
this.close()
}
get status(): AssemblyResponse {
return this.#status
}
set status(status: AssemblyResponse) {
this.#status = status
this.emit('status', status)
}
#connectServerSentEvents() {
this.#sse = new EventSource(
`${this.status.websocket_url}?assembly=${this.status.assembly_id}`,
@ -110,22 +118,24 @@ class TransloaditAssembly extends Emitter {
})
this.#sse.addEventListener('assembly_upload_finished', (e) => {
const file = JSON.parse(e.data) as AssemblyFile
this.status.uploads ??= []
this.status.uploads.push(file)
const file: AssemblyFile = JSON.parse(e.data)
this.status = {
...this.status,
uploads: [...(this.status.uploads ?? []), file],
}
this.emit('upload', file)
})
this.#sse.addEventListener('assembly_result_finished', (e) => {
const [stepName, rawResult] = JSON.parse(e.data) as [
string,
AssemblyResult,
]
this.status.results ??= {}
this.status.results[stepName] ??= []
this.status.results[stepName].push(rawResult)
this.emit('result', stepName, rawResult)
const [stepName, result]: [string, AssemblyResult] = JSON.parse(e.data)
this.status = {
...this.status,
results: {
...this.status.results,
[stepName]: [...(this.status.results?.[stepName] ?? []), result],
},
}
this.emit('result', stepName, result)
})
this.#sse.addEventListener('assembly_execution_progress', (e) => {
@ -201,7 +211,6 @@ class TransloaditAssembly extends Emitter {
// Avoid updating if we closed during this request's lifetime.
if (this.closed) return
this.emit('status', status)
if (diff) {
this.updateStatus(status)

View file

@ -8,7 +8,12 @@ import type {
} from '@uppy/core'
import { BasePlugin } from '@uppy/core'
import Tus, { type TusDetailedError, type TusOpts } from '@uppy/tus'
import { ErrorWithCause, hasProperty, RateLimitedQueue } from '@uppy/utils'
import {
ErrorWithCause,
hasProperty,
RateLimitedQueue,
type RemoteUppyFile,
} from '@uppy/utils'
import type {
AssemblyStatus,
AssemblyStatusResult,
@ -93,9 +98,9 @@ declare module '@uppy/core' {
// biome-ignore lint/correctness/noUnusedVariables: must be defined
export interface UppyEventMap<M extends Meta, B extends Body> {
// We're also overriding the `restored` event as it is now populated with Transloadit state.
restored: (pluginData: Record<string, TransloaditState>) => void
'restore:get-data': (
setData: (arg: Record<string, PersistentState>) => void,
restored: (pluginData: Record<string, PersistentState>) => void
'restore:plugin-data-changed': (
pluginData: Record<string, PersistentState | undefined>,
) => void
'transloadit:assembly-created': (
assembly: AssemblyResponse,
@ -130,7 +135,11 @@ declare module '@uppy/core' {
}
declare module '@uppy/utils' {
export interface UppyFile<M extends Meta, B extends Body> {
export interface LocalUppyFile<M extends Meta, B extends Body> {
transloadit?: { assembly: string }
tus?: TusOpts<M, B>
}
export interface RemoteUppyFile<M extends Meta, B extends Body> {
transloadit?: { assembly: string }
tus?: TusOpts<M, B>
}
@ -230,7 +239,7 @@ export default class Transloadit<
client: Client<M, B>
assembly?: Assembly
#assembly?: Assembly
#watcher!: AssemblyWatcher<M, B>
@ -332,22 +341,22 @@ export default class Transloadit<
// remote, because this is the criteria to identify remote files.
// We only replace the hostname for Transloadit's companions, so that
// people can also self-host them while still using Transloadit for encoding.
let { remote } = file
let remote: RemoteUppyFile<M, B>['remote'] | undefined
if (
file.remote &&
status.companion_url &&
TL_COMPANION.test(file.remote.companionUrl)
) {
const newHost = status.companion_url.replace(/\/$/, '')
const path = file.remote.url
.replace(file.remote.companionUrl, '')
.replace(/^\//, '')
if ('remote' in file && file.remote) {
;({ remote } = file)
remote = {
...file.remote,
companionUrl: newHost,
url: `${newHost}/${path}`,
if (status.companion_url && TL_COMPANION.test(file.remote.companionUrl)) {
const newHost = status.companion_url.replace(/\/$/, '')
const path = file.remote.url
.replace(file.remote.companionUrl, '')
.replace(/^\//, '')
remote = {
...file.remote,
companionUrl: newHost,
url: `${newHost}/${path}`,
}
}
}
@ -481,6 +490,33 @@ export default class Transloadit<
)
}
/**
* Allows Golden Retriever plugin to serialize the Assembly status so we can restore it later
*/
#handleAssemblyStatusUpdate = (
assemblyResponse: AssemblyResponse | undefined,
) => {
this.uppy.emit('restore:plugin-data-changed', {
[this.id]: assemblyResponse ? { assemblyResponse } : undefined,
})
}
get assembly() {
return this.#assembly
}
set assembly(newAssembly: Assembly | undefined) {
if (!newAssembly && this.assembly) {
this.assembly.off('status', this.#handleAssemblyStatusUpdate)
}
this.#assembly = newAssembly
this.#handleAssemblyStatusUpdate(newAssembly?.status)
if (newAssembly) {
newAssembly.on('status', this.#handleAssemblyStatusUpdate)
}
}
/**
* Used when `importFromUploadURLs` is enabled: adds files to the Assembly
* once they have been fully uploaded.
@ -607,22 +643,10 @@ export default class Transloadit<
}
}
/**
* Custom state serialization for the Golden Retriever plugin.
* It will pass this back to the `_onRestored` function.
*/
#getPersistentData = (
setData: (arg: Record<string, PersistentState>) => void,
) => {
if (this.assembly) {
setData({ [this.id]: { assemblyResponse: this.assembly.status } })
}
}
#onRestored = (pluginData: Record<string, unknown>) => {
const savedState = (
pluginData?.[this.id] ? pluginData[this.id] : {}
) as PersistentState
#onRestored = (pluginData: Record<string, PersistentState>) => {
const savedState: {
assemblyResponse?: PersistentState['assemblyResponse']
} = pluginData?.[this.id] ? pluginData[this.id] : {}
const previousAssembly = savedState.assemblyResponse
if (!previousAssembly) {
@ -685,8 +709,9 @@ export default class Transloadit<
}
})
this.assembly = new Assembly(previousAssembly, this.#rateLimitedQueue)
this.assembly.status = previousAssembly
const assembly = new Assembly(previousAssembly, this.#rateLimitedQueue)
assembly.status = previousAssembly
this.assembly = assembly
this.setPluginState({ files, results })
return files
}
@ -718,7 +743,6 @@ export default class Transloadit<
#connectAssembly(assembly: Assembly, ids: UppyFile<M, B>['id'][]) {
const { status } = assembly
const id = ensureAssemblyId(status)
this.assembly = assembly
assembly.on('upload', (file: AssemblyFile) => {
this.#onFileUploadComplete(id, file)
@ -810,6 +834,7 @@ export default class Transloadit<
this.uppy.emit('preprocess-complete', file)
})
this.#createAssemblyWatcher(ensureAssemblyId(assembly.status))
this.assembly = assembly
this.#connectAssembly(assembly, fileIDs)
} catch (err) {
fileIDs.forEach((fileID) => {
@ -954,7 +979,6 @@ export default class Transloadit<
})
}
this.uppy.on('restore:get-data', this.#getPersistentData)
this.uppy.on('restored', this.#onRestored)
this.setPluginState({

View file

@ -14,6 +14,7 @@ import {
getAllowedMetaFields,
hasProperty,
isNetworkError,
type LocalUppyFile,
NetworkError,
RateLimitedQueue,
} from '@uppy/utils'
@ -97,7 +98,10 @@ type Opts<M extends Meta, B extends Body> = DefinePluginOpts<
>
declare module '@uppy/utils' {
export interface UppyFile<M extends Meta, B extends Body> {
export interface LocalUppyFile<M extends Meta, B extends Body> {
tus?: TusOpts<M, B>
}
export interface RemoteUppyFile<M extends Meta, B extends Body> {
tus?: TusOpts<M, B>
}
}
@ -204,7 +208,9 @@ export default class Tus<M extends Meta, B extends Body> extends BasePlugin<
* up a spot in the queue.
*
*/
#uploadLocalFile(file: UppyFile<M, B>): Promise<tus.Upload | string> {
async #uploadLocalFile(
file: LocalUppyFile<M, B>,
): Promise<tus.Upload | string> {
this.resetUploaderReferences(file.id)
// Create a new tus upload
@ -439,6 +445,7 @@ export default class Tus<M extends Meta, B extends Body> extends BasePlugin<
uploadOptions.metadata = meta
if (file.data == null) throw new Error('File data is empty')
upload = new tus.Upload(file.data, uploadOptions)
this.uploaders[file.id] = upload
const eventManager = new EventManager(this.uppy)
@ -539,11 +546,11 @@ export default class Tus<M extends Meta, B extends Body> extends BasePlugin<
}
return {
...file.remote?.body,
...('remote' in file && file.remote.body),
endpoint: opts.endpoint,
uploadUrl: opts.uploadUrl,
protocol: 'tus',
size: file.data.size,
size: file.data!.size,
headers: opts.headers,
metadata: file.meta,
}

View file

@ -4,7 +4,11 @@ import {
} from '@uppy/companion-client'
import type { Body, Meta } from '@uppy/core'
import { UIPlugin, type Uppy } from '@uppy/core'
import type { LocaleStrings, TagFile } from '@uppy/utils'
import type {
LocaleStrings,
MinimalRequiredUppyFile,
RemoteUppyFile,
} from '@uppy/utils'
import { toArray } from '@uppy/utils'
// biome-ignore lint/style/useImportType: h is not a type
import { type ComponentChild, h } from 'preact'
@ -145,7 +149,8 @@ export default class Url<M extends Meta, B extends Body> extends UIPlugin<
try {
const meta = await this.getMeta(url)
const tagFile: TagFile<M> = {
const file: Omit<RemoteUppyFile<M, B>, 'name' | 'meta' | 'body'> &
Pick<MinimalRequiredUppyFile<M, B>, 'name' | 'meta'> = {
meta: optionalMeta,
source: this.id,
name: meta.name || getFileNameFromUrl(url),
@ -154,6 +159,7 @@ export default class Url<M extends Meta, B extends Body> extends UIPlugin<
size: meta.size,
},
isRemote: true,
// @ts-expect-error TODO: should this be removed? the types say it's not needed
body: {
url,
},
@ -170,7 +176,7 @@ export default class Url<M extends Meta, B extends Body> extends UIPlugin<
this.uppy.log('[Url] Adding remote file')
try {
return this.uppy.addFile(tagFile)
return this.uppy.addFile(file)
} catch (err) {
if (!err.isRestriction) {
this.uppy.log(err)

View file

@ -14,12 +14,25 @@ export type FileProcessingInfo =
// TODO explore whether all of these properties need to be optional
interface FileProgressBase {
/**
* When `uploadComplete` is `true`, the upload has completed successfully (no error).
* This is a bit confusing, because elsewhere in the code, we use the term "complete" when we mean an upload has completed either with success or failure.
* Note that it does not mean that post-processing steps are done, for example a Transloadit assembly may still be running.
* TODO rename to `uploadSuccess` or similar
*/
uploadComplete?: boolean
percentage?: number // undefined if we don't know the percentage (e.g. for files with `bytesTotal` null)
// note that Companion will send `bytesTotal` 0 if unknown size (not `null`).
// this is not perfect because some files can actually have a size of 0,
// and then we might think those files have an unknown size
// todo we should change this in companion
/**
* When `true`, the file upload completed uploading *and* post-processing (if post-processing steps exit) either successfully or failed.
*/
complete?: true
/** `undefined` if we don't know the percentage (e.g. for files with `bytesTotal` null) */
percentage?: number
/**
* note that Companion will send `bytesTotal` 0 if unknown size (not `null`).
* this is not perfect because some files can actually have a size of 0,
* and then we might think those files have an unknown size
* todo we should change this in companion
*/
bytesTotal: number | null
preprocess?: FileProcessingInfo
postprocess?: FileProcessingInfo

View file

@ -6,29 +6,21 @@ export type Body = Record<string, unknown>
export type InternalMetadata = { name: string; type?: string }
export interface UppyFile<M extends Meta, B extends Body> {
data: Blob | File
// for better readability instead of using Record<string, something>
export type UppyFileId = string
interface UppyFileBase<M extends Meta, B extends Body> {
error?: string | null
extension: string
id: string
id: UppyFileId
isPaused?: boolean
isRestored?: boolean
isRemote: boolean
isGhost: boolean
meta: InternalMetadata & M
name?: string
name: string
preview?: string
progress: FileProgress
missingRequiredMetaFields?: string[]
remote?: {
body?: Record<string, unknown>
companionUrl: string
host?: string
provider?: string
providerName?: string
requestClientId: string
url: string
}
serverToken?: string | null
size: number | null
source?: string
@ -42,54 +34,54 @@ export interface UppyFile<M extends Meta, B extends Body> {
}
}
export interface LocalUppyFile<M extends Meta, B extends Body>
extends UppyFileBase<M, B> {
isRemote: false
data: Blob | File | undefined
}
export interface LocalUppyFileNonGhost<M extends Meta, B extends Body>
extends UppyFileBase<M, B> {
isRemote: false
isGhost: false
data: Blob | File
}
export interface RemoteUppyFile<M extends Meta, B extends Body>
extends UppyFileBase<M, B> {
data: { size: number | null }
isRemote: true
remote: {
body?: Record<string, unknown>
companionUrl: string
host?: string
provider?: string
providerName?: string
requestClientId: string
url: string
}
}
export type UppyFile<M extends Meta, B extends Body> =
| LocalUppyFile<M, B>
| RemoteUppyFile<M, B>
// TODO use this type in more places, so we don't have to check for data not being null/undefined everywhere
/**
* For when you know the file is not a ghost, and data is definitely present.
*/
export type UppyFileNonGhost<M extends Meta, B extends Body> =
| LocalUppyFileNonGhost<M, B>
| RemoteUppyFile<M, B>
/*
* The user facing type for UppyFile used in uppy.addFile() and uppy.setOptions()
*/
export type MinimalRequiredUppyFile<M extends Meta, B extends Body> = Required<
Pick<UppyFile<M, B>, 'name'>
> &
Partial<
Omit<UppyFile<M, B>, 'name' | 'data' | 'meta'>
> & { data: NonNullable<UppyFile<M, B>['data']> } & Partial<
Omit<UppyFile<M, B>, 'name' | 'meta' | 'data'>
// We want to omit the 'meta' from UppyFile because of internal metadata
// (see InternalMetadata in `UppyFile.js`), as when adding a new file
// that is not required.
> & { meta?: M; data: { size: number | null } }
/*
* We are not entirely sure what a "tag file" is.
* It is used as an intermidiate type between `CompanionFile` and `UppyFile`
* in `@uppy/provider-views` and `@uppy/url`.
* TODO: remove this in favor of UppyFile
*/
export type TagFile<M extends Meta> = {
id?: string
source: string
name: string
type: string
isRemote: boolean
preview?: string
data: {
size: number | null
}
body?: {
url?: string
fileId?: string
}
meta?: {
authorName?: string
authorUrl?: string
relativePath?: string | null
absolutePath?: string
} & M
remote: {
companionUrl: string
url: string
body: {
fileId: string
url?: string
}
providerName?: string
provider?: string
requestClientId: string
}
}
> & { meta?: M }

View file

@ -1,5 +1,11 @@
import getFileType from './getFileType.js'
import type { MinimalRequiredUppyFile, UppyFile } from './UppyFile.js'
import type {
Body,
LocalUppyFile,
Meta,
RemoteUppyFile,
UppyFile,
} from './UppyFile.js'
function encodeCharacter(character: string): string {
return character.charCodeAt(0).toString(32)
@ -19,9 +25,11 @@ function encodeFilename(name: string): string {
* Takes a file object and turns it into fileID, by converting file.name to lowercase,
* removing extra characters and adding type, size and lastModified
*/
export default function generateFileID(
file: Omit<MinimalRequiredUppyFile<any, any>, 'name'> &
Pick<UppyFile<any, any>, 'name'>,
export default function generateFileID<M extends Meta, B extends Body>(
file: Partial<Pick<UppyFile<M, B>, 'id' | 'type' | 'name'>> &
Pick<UppyFile<M, B>, 'data'> & {
meta?: { relativePath?: unknown }
},
instanceId: string,
): string {
// It's tempting to do `[items].filter(Boolean).join('-')` here, but that
@ -40,7 +48,7 @@ export default function generateFileID(
id += `-${encodeFilename(file.meta.relativePath.toLowerCase())}`
}
if (file.data.size !== undefined) {
if (file.data?.size !== undefined) {
id += `-${file.data.size}`
}
if ((file.data as File).lastModified !== undefined) {
@ -52,9 +60,10 @@ export default function generateFileID(
// If the provider has a stable, unique ID, then we can use that to identify the file.
// Then we don't have to generate our own ID, and we can add the same file many times if needed (different path)
function hasFileStableId(
file: Omit<MinimalRequiredUppyFile<any, any>, 'name'> &
Pick<UppyFile<any, any>, 'name'>,
function hasFileStableId<M extends Meta, B extends Body>(
file:
| Pick<LocalUppyFile<M, B>, 'isRemote'>
| Pick<RemoteUppyFile<M, B>, 'isRemote' | 'remote'>,
): boolean {
if (!file.isRemote || !file.remote) return false
// These are the providers that it seems like have stable IDs for their files. The other's I haven't checked yet.
@ -65,12 +74,16 @@ function hasFileStableId(
'facebook',
'unsplash',
])
return stableIdProviders.has(file.remote.provider as any)
return stableIdProviders.has(file.remote.provider!)
}
export function getSafeFileId(
file: Omit<MinimalRequiredUppyFile<any, any>, 'name'> &
Pick<UppyFile<any, any>, 'name'>,
export function getSafeFileId<M extends Meta, B extends Body>(
file: Partial<Pick<UppyFile<M, B>, 'id' | 'type'>> &
Pick<UppyFile<M, B>, 'data'> &
(
| Pick<RemoteUppyFile<M, B>, 'isRemote' | 'remote'>
| Pick<LocalUppyFile<M, B>, 'isRemote'>
) & { meta?: { relativePath?: unknown } },
instanceId: string,
): string {
if (hasFileStableId(file)) return file.id!

View file

@ -96,10 +96,14 @@ export { default as truncateString } from './truncateString.js'
export type {
Body,
InternalMetadata,
LocalUppyFile,
LocalUppyFileNonGhost,
Meta,
MinimalRequiredUppyFile,
TagFile,
RemoteUppyFile,
UppyFile,
UppyFileId,
UppyFileNonGhost,
} from './UppyFile.js'
export { default as UserFacingApiError } from './UserFacingApiError.js'

View file

@ -8,7 +8,7 @@ import type {
Uppy,
} from '@uppy/core'
import { UIPlugin } from '@uppy/core'
import type { LocaleStrings } from '@uppy/utils'
import type { LocaleStrings, LocalUppyFileNonGhost } from '@uppy/utils'
import { canvasToBlob, getFileTypeExtension, mimeTypes } from '@uppy/utils'
import { isMobile } from 'is-mobile'
// biome-ignore lint/style/useImportType: h is not a type
@ -581,11 +581,12 @@ export default class Webcam<M extends Meta, B extends Body> extends UIPlugin<
}
try {
const tagFile = await this.getImage()
this.capturedMediaFile = tagFile
const file = await this.getImage()
this.capturedMediaFile = file
if (file.data == null) throw new Error('File data is empty')
// Create object URL for preview
const capturedSnapshotUrl = URL.createObjectURL(tagFile.data as Blob)
const capturedSnapshotUrl = URL.createObjectURL(file.data)
this.setPluginState({ capturedSnapshot: capturedSnapshotUrl })
this.captureInProgress = false
} catch (error) {
@ -597,7 +598,9 @@ export default class Webcam<M extends Meta, B extends Body> extends UIPlugin<
}
}
getImage(): Promise<MinimalRequiredUppyFile<M, B>> {
async getImage(): Promise<
Pick<LocalUppyFileNonGhost<M, B>, 'data' | 'name'>
> {
const video = this.getVideoElement()
if (!video) {
return Promise.reject(

View file

@ -17,8 +17,10 @@ import {
getAllowedMetaFields,
internalRateLimitedQueue,
isNetworkError,
type LocalUppyFile,
NetworkError,
RateLimitedQueue,
type RemoteUppyFile,
} from '@uppy/utils'
import packageJson from '../package.json' with { type: 'json' }
import locale from './locale.js'
@ -70,7 +72,10 @@ export interface XhrUploadOpts<M extends Meta, B extends Body>
export type { XhrUploadOpts as XHRUploadOptions }
declare module '@uppy/utils' {
export interface UppyFile<M extends Meta, B extends Body> {
export interface LocalUppyFile<M extends Meta, B extends Body> {
xhrUpload?: XhrUploadOpts<M, B>
}
export interface RemoteUppyFile<M extends Meta, B extends Body> {
xhrUpload?: XhrUploadOpts<M, B>
}
}
@ -111,8 +116,14 @@ function buildResponseError(
* because we might have detected a more accurate file type in Uppy
* https://stackoverflow.com/a/50875615
*/
function setTypeInBlob<M extends Meta, B extends Body>(file: UppyFile<M, B>) {
const dataWithUpdatedType = file.data.slice(0, file.data.size, file.meta.type)
function setTypeInBlob<M extends Meta, B extends Body>(
file: LocalUppyFile<M, B>,
) {
const dataWithUpdatedType = file.data!.slice(
0,
file.data!.size,
file.meta.type,
)
return dataWithUpdatedType
}
@ -326,7 +337,7 @@ export default class XHRUpload<
})
}
createFormDataUpload(file: UppyFile<M, B>, opts: Opts<M, B>): FormData {
createFormDataUpload(file: LocalUppyFile<M, B>, opts: Opts<M, B>): FormData {
const formPost = new FormData()
this.addMetadata(formPost, file.meta, opts)
@ -342,7 +353,10 @@ export default class XHRUpload<
return formPost
}
createBundledUpload(files: UppyFile<M, B>[], opts: Opts<M, B>): FormData {
createBundledUpload(
files: LocalUppyFile<M, B>[],
opts: Opts<M, B>,
): FormData {
const formPost = new FormData()
const { meta } = this.uppy.getState()
@ -363,7 +377,7 @@ export default class XHRUpload<
return formPost
}
async #uploadLocalFile(file: UppyFile<M, B>) {
async #uploadLocalFile(file: LocalUppyFile<M, B>) {
const events = new EventManager(this.uppy)
const controller = new AbortController()
const uppyFetch = this.requests.wrapPromiseFunction(async () => {
@ -400,7 +414,7 @@ export default class XHRUpload<
}
}
async #uploadBundle(files: UppyFile<M, B>[]) {
async #uploadBundle(files: LocalUppyFile<M, B>[]) {
const controller = new AbortController()
const uppyFetch = this.requests.wrapPromiseFunction(async () => {
const optsFromState = this.uppy.getState().xhrUpload ?? {}
@ -441,7 +455,7 @@ export default class XHRUpload<
}
}
#getCompanionClientArgs(file: UppyFile<M, B>) {
#getCompanionClientArgs(file: RemoteUppyFile<M, B>) {
const opts = this.getOptions(file)
const allowedMetaFields = getAllowedMetaFields(
opts.allowedMetaFields,
@ -534,7 +548,7 @@ export default class XHRUpload<
)
}
await this.#uploadBundle(filesFiltered)
await this.#uploadBundle(filesFiltered as LocalUppyFile<M, B>[])
} else {
await this.#uploadFiles(filesFiltered)
}

View file

@ -17,7 +17,7 @@ switch (window.location.pathname.toLowerCase()) {
if ('serviceWorker' in navigator) {
navigator.serviceWorker
.register('/sw.js')
.register('./sw.js', { type: 'module' })
.then((registration) => {
console.log(
'ServiceWorker registration successful with scope: ',

View file

@ -1,61 +1 @@
/* globals clients */
const fileCache = Object.create(null)
function getCache(name) {
if (!fileCache[name]) {
fileCache[name] = Object.create(null)
}
return fileCache[name]
}
self.addEventListener('install', (event) => {
console.log('Installing Uppy Service Worker...')
event.waitUntil(Promise.resolve().then(() => self.skipWaiting()))
})
self.addEventListener('activate', (event) => {
event.waitUntil(self.clients.claim())
})
function sendMessageToAllClients(msg) {
clients.matchAll().then((clients) => {
clients.forEach((client) => {
client.postMessage(msg)
})
})
}
function addFile(store, file) {
getCache(store)[file.id] = file.data
console.log('Added file blob to service worker cache:', file.data)
}
function removeFile(store, fileID) {
delete getCache(store)[fileID]
console.log('Removed file blob from service worker cache:', fileID)
}
function getFiles(store) {
sendMessageToAllClients({
type: 'uppy/ALL_FILES',
store,
files: getCache(store),
})
}
self.addEventListener('message', (event) => {
switch (event.data.type) {
case 'uppy/ADD_FILE':
addFile(event.data.store, event.data.file)
break
case 'uppy/REMOVE_FILE':
removeFile(event.data.store, event.data.fileID)
break
case 'uppy/GET_FILES':
getFiles(event.data.store)
break
default:
}
})
import '@uppy/golden-retriever/lib/ServiceWorker'

View file

@ -12,6 +12,20 @@ const config = {
jsx: 'automatic',
jsxImportSource: 'preact',
},
build: {
rollupOptions: {
input: {
main: './index.html',
sw: './sw.js',
},
output: {
entryFileNames: (chunk) => {
if (chunk.name === 'sw') return 'sw.js' // force predictable filename
return 'assets/[name].[hash].js'
},
},
},
},
resolve: {
alias: [
{

View file

@ -11009,9 +11009,13 @@ __metadata:
version: 0.0.0-use.local
resolution: "@uppy/golden-retriever@workspace:packages/@uppy/golden-retriever"
dependencies:
"@uppy/dashboard": "workspace:^"
"@uppy/utils": "workspace:^"
"@uppy/xhr-upload": "workspace:^"
"@vitest/browser": "npm:^3.2.4"
lodash: "npm:^4.17.21"
typescript: "npm:^5.8.3"
vitest: "npm:^3.2.4"
peerDependencies:
"@uppy/core": "workspace:^"
languageName: unknown
@ -11437,7 +11441,7 @@ __metadata:
languageName: unknown
linkType: soft
"@uppy/xhr-upload@workspace:*, @uppy/xhr-upload@workspace:packages/@uppy/xhr-upload":
"@uppy/xhr-upload@workspace:*, @uppy/xhr-upload@workspace:^, @uppy/xhr-upload@workspace:packages/@uppy/xhr-upload":
version: 0.0.0-use.local
resolution: "@uppy/xhr-upload@workspace:packages/@uppy/xhr-upload"
dependencies: