# A Million Files From a Browser Tab

> A progress bar fix turned into a browser-native upload engine for a million files, with crash recovery, multi-tab coordination, and resumable transfers.

- Author: Vasyl Semiliak (https://www.vasylsemiliak.com)
- Published: 2026-10-11
- URL: https://www.vasylsemiliak.com/projects/a-million-files-from-a-browser-tab
- Tags: architecture, object-storage, web-workers, file-system-access-api, typescript, s3, sqlite, web-locks, broadcast-channel

_The [web version](https://www.vasylsemiliak.com/projects/a-million-files-from-a-browser-tab) of this article has an interactive demo here._

## It started as a progress bar

A geospatial platform takes in LiDAR point clouds, panoramic imagery, and trajectory data. These are large binary files, and every dataset on the platform enters through the upload. I came in to fix the progress bar, which reported numbers nobody trusted, and to look at why large uploads failed so often.

The upload that existed was the simplest thing that could work. The backend mirrored the three S3 multipart calls as its own API: start an upload, send a part, complete the upload. The frontend read a file into memory, looped over the parts on the main thread, and posted each one. For a 200 MB file that is fine. For a 4 GB file it read the whole file into an ArrayBuffer first, so memory went to several gigabytes and the tab died on weaker machines. A part that failed at 95% restarted the file from byte zero. A network drop, an expired token, or a closed tab lost everything in flight. And while all this ran on the main thread, the UI stuttered.

The fix I had in mind was to move the upload loop into a Web Worker and slice each file by byte range instead of reading it whole. That would stop the tab from loading whole files and leave the main thread free for the UI, which covered everything I had been asked for.

## One worker, then two

The worker version was better straight away. [`File.slice()`](https://developer.mozilla.org/en-US/docs/Web/API/Blob/slice) returns a reference to a byte range, not a copy, so the worker never loads a whole file, and the UI stopped freezing.

Then a second problem showed up. Every file needs preparation before it can go to S3: a sanitized name, a size check, a type check, and a classification by format so the processing pipeline knows what it is getting. With a few hundred files that happened in the upload worker and nobody noticed. With tens of thousands it blocked the uploads behind it. So the preparation moved to its own worker: one worker discovers, validates, and queues files, the other uploads them, and the main thread only hands over file references and renders the result.

That version handled about 10 to 20 thousand files per session, depending on the user’s machine and browser, and for a while it looked like the job was done.

## The numbers changed

Then the product side came back with the real expectations: up to a million files per session, with single files of up to 100 GB. And these sessions take hours, so an upload had to survive a reload, a re-login, a closed browser, and time offline. The user had to keep working in the app while it ran. Users open the app in several tabs, and the tabs could not fight over the same upload.

The obvious answer to all of that is a desktop app, but the platform is web-first and a native client was off the table.

I started listing what could go wrong at that scale. Progress lives in JavaScript variables, so a crash erases it. A reload drops every `File` object, so even if I knew where each upload stopped, I had no way to reopen the files. A second tab would start its own copy of the engine against the same S3 upload IDs. And a queue of a million files is far too big to hold as one React state object and copy on every update. By the end of that list I was looking at a system design problem, and my part-time work for this client became full-time. I led the project from there, owned the frontend end to end, and worked closely with the backend and infrastructure teams.

## What the libraries could do

I did not want to build this from scratch, so I spent time on the existing options. Uppy and FilePond are good tools, and at a smaller scale either would have done. I also looked at tus, the AWS SDK’s own multipart helper, and the older chunked uploaders. The question was always the same five things: a million-file queue, 100 GB files over S3 multipart, resume after a reload, resume after a crash without re-selecting the files, and more than one tab.

| Library | 1M files | 100 GB | Reload | Crash | Tabs |
| --- | --- | --- | --- | --- | --- |
| Uppy 6 (Aug 2026, after the build) | unknown | yes | partial | partial | no |
| Uppy 5 (at decision time) | unknown | partial | partial | partial | no |
| Amplify Storage (uploadData) | no | yes | partial | no | no |
| AWS SDK (lib-storage Upload) | no | yes | no | no | no |
| better-upload (presigned multipart) | unknown | yes | no | no | no |
| tus-js-client (needs a tus server) | no | no | partial | no | no |
| FilePond (own chunk protocol) | unknown | no | no | no | no |
| Fine Uploader (and Evaporate.js, unmaintained) | unknown | yes | partial | no | no |
| Flow.js (and Resumable.js, unmaintained) | unknown | no | partial | no | no |
| Our engine (Chromium only) | yes | yes | yes | yes | yes |

_Each cell has a note in the [web version](https://www.vasylsemiliak.com/projects/a-million-files-from-a-browser-tab)._

[Uppy](https://uppy.io/docs/aws-s3/) came closest. Its S3 plugin does multipart, and [Golden Retriever](https://uppy.io/docs/golden-retriever/) restores state after a reload. But Golden Retriever stores file contents, not a way to reopen the file on disk. Files up to 10 MiB go to IndexedDB by default. Larger files go through a Service Worker, and the docs are clear about what that means:

> Service Worker storage is _quite_ temporary though, and doesn’t persist across browser crashes or restarts.
>
> — [Uppy docs, Golden Retriever](https://uppy.io/docs/golden-retriever/)

After a crash, a user with 50 multi-gigabyte files selects all 50 again. The metadata also expires after 24 hours by default. When I did the research, the S3 plugin could not continue a multipart upload after a restore at all ([#5230](https://github.com/transloadit/uppy/issues/5230)), and a part that failed for good aborted the whole multipart upload, so the retry started from part one ([#5762](https://github.com/transloadit/uppy/issues/5762)). Uppy keeps the queue in one in-memory state object. The oldest report on large queues is from 2019, when [about 30k files took three minutes to render](https://github.com/transloadit/uppy/issues/1908), and Uppy has done a lot of work on that since. Nothing in the docs says what happens at 100k or a million.

[tus-js-client](https://github.com/tus/tus-js-client/blob/main/docs/usage.md) resumes properly, but it needs a tus server in the middle, and after a reload the user re-selects the file so the client can match its fingerprint. The [AWS SDK’s `Upload` helper](https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-storage/README.md) does multipart and nothing else. It has no resume API, and on error it aborts the multipart upload unless you ask it not to. Flow.js and Resumable.js resume by asking the server about every chunk, and neither has shipped a release since 2020.

Two things were missing from every library I looked at. None of them persists file handles, so none can reopen a file after a crash. None of them coordinates tabs. There is no Web Locks and no BroadcastChannel anywhere in the Uppy source, and I found no other upload library that uses them either.

Uppy 6.0 [shipped in August 2026](https://uppy.io/blog/), after this work, with an S3 plugin rewritten from scratch. It closed the resume bug, and its retry path now asks S3 which parts exist and uploads only the missing ones. That is real progress. The other gaps remain as of October 2026, and when I had to decide, in early 2026, none of it existed yet, so we built our own.

## Chromium only, on purpose

One of the first decisions I made with the team was to support Chrome and the other Chromium browsers only. That includes Edge, Arc, Brave, and Opera, and it excludes Firefox and Safari.

Crash recovery depends on the [File System Access API](https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API): the file pickers, file handles you can store, and the permission model around them. Only Chromium ships it. Mozilla’s [standards position](https://github.com/mozilla/standards-positions/issues/154) on the API is negative, and Safari has none of it. Both browsers do ship the Origin Private File System with [SyncAccessHandle](https://developer.mozilla.org/en-US/docs/Web/API/FileSystemSyncAccessHandle), so the database layer would run there. The file handles would not, and without them crash recovery means asking the user to select every file again.

A cross-browser fallback would have shipped a worse product under the same name, so in other browsers the upload panel does not mount and the user sees a message that explains why. The team accepted that early.

## What we built

The engine runs entirely in the browser. I designed the architecture after the research above, and Claude Opus 4.5 did the heavy lifting from there: it was the first model that knew the newer Chromium APIs well enough to generate a working backbone for the workers, the SQLite layer, and the tab coordination, which the team and I then reviewed and refined.

### Two workers, no main-thread work

All upload logic runs in dedicated [Web Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API). One worker discovers and prepares files, and the other uploads them. The main thread runs the React UI and nothing else. It does not read files, call S3, or touch the database, so the app stays at 60 fps while a huge upload runs in the background. If the React app crashes, the workers can be respawned and continue from the last checkpoint.

### One tab runs the engine

Every open tab on the same origin starts its own copy of the app, and only one of them can run the engine. Two engines would upload the same files against the same S3 upload IDs and write to the same database. The engine uses the [Web Locks API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API) to elect a leader, and only the leader runs the upload workers and owns the database. The other tabs show live progress that the leader sends over [BroadcastChannel](https://developer.mozilla.org/en-US/docs/Web/API/Broadcast_Channel_API).

When the leader tab closes, the browser releases its lock, so there is no heartbeat and no stale leader. The next tab in the queue gets the lock within seconds, checks for interrupted uploads, resumes them, and shows the user a brief toast. Two tabs never run the engine at the same time.

The lock makes the election simple. Managing the leader was harder: every other tab mirrors a leader that can close at any moment, and a new leader has to pick up exactly where the old one stopped. [Notion’s post on running SQLite in the browser](https://www.notion.com/blog/how-we-sped-up-notion-in-the-browser-with-wasm-sqlite) was the starting point for this. They pick an active tab through a SharedWorker and use a Web Lock to notice when it dies. We let the lock do the election itself, which is less code and fits an engine that only one tab should ever run.

### SQLite in the browser

Every in-flight upload is checkpointed in [SQLite compiled to WebAssembly](https://sqlite.org/wasm/doc/trunk/index.md). It runs in its own worker, apart from the two upload workers, and stores its file in the [Origin Private File System](https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system). The discovery worker sends it new files in batches to queue, and the upload worker sends it each finished part to checkpoint. That gives the engine real queries over a million rows instead of key-value lookups, and the data survives tab crashes and browser restarts.

SQLite’s WASM build has two OPFS storage backends, described in its [persistence docs](https://sqlite.org/wasm/doc/trunk/persistence.md). The classic `opfs` VFS needs SharedArrayBuffer, which means cross-origin isolation headers on the whole site, and those headers can break third-party embeds. The `opfs-sahpool` VFS needs no headers and is faster, but it allows one connection per database, and a second tab that tries to open it fails. Only the leader tab opens the database, so that limit cost us nothing, and we went with the pool.

File handles live separately in [IndexedDB](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API). After a browser crash, a closed tab, or a power cut, the engine knows which files were mid-upload and where each one stopped, reopens them from disk, and continues without asking the user to select them again. At most, Chrome asks for file access once more.

### Discovery and multipart

Users drag in files or whole folder trees, or use the picker. Both paths run the same code. It walks directories, classifies files by type, rejects unsupported formats, and queues the rest. Files appear in the list as the walker finds them, and the UI says “47 files skipped (unsupported format)” instead of dropping them silently.

The upload worker splits each file into parts and sends them with S3 multipart upload. S3 allows [10,000 parts per object](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html), so a 100 GB file needs parts of at least 10 MB, and the part size scales with the file. Each completed part is checkpointed. On resume, the engine checks its records against the parts S3 actually has and sends only the rest. A file that crashed at 95% re-uploads 5%.

### Failure handling

We listed every failure we could think of and gave each one a defined response: network drops, server errors, S3 throttling, expired tokens, file permissions revoked by the browser, files moved or deleted after selection, and crashes during active uploads. The engine retries the transient ones on its own, with the retry state in the database, so a crash during a backoff wait loses nothing. When a failure needs the user, like re-granting file access, the UI says what happened and what to do. A failure that hits a whole directory pauses the engine instead of failing files one by one.

### Progress and ETA

The progress bar this project started with became a problem of its own. Speed and ETA depend on conditions the engine does not control: the network slows down or drops, S3 throttles requests, retries wait out a backoff, tokens expire, and the browser can revoke file access mid-session. A naive calculation jumps with each of these. We smooth both values, and between files the display holds the last estimate instead of dropping to zero.

## Known limitations

Three limitations remain. File System Access permissions last for the session by default. Since Chrome 122 the user can pick [“Allow on every visit”](https://developer.chrome.com/blog/persistent-permissions-for-the-file-system-access-api), but the app cannot make that choice for them. OPFS quota is per origin and managed by the browser, so a long upload history needs periodic cleanup of completed records. And the SQLite pool allows one open connection, so a tab that is not the leader cannot read the database directly and depends on what the leader broadcasts.

## Where it landed

The engine takes a million files in one session and single files of up to 100 GB. Someone can close a laptop, open it the next day, and watch the upload continue from the same part. The company runs no upload server, because the browser talks to S3 directly and nothing else touches the bytes. The demo at the top of this page shows what that looks like from the user’s side.

Since launch, multi-gigabyte uploads finish without anyone starting over. Customers were impressed, and the support team was happy to see the difference.

## What I took from it

The requirements arrived one at a time, each after I had designed for the one before, and each one added a part: two workers, a SQLite pool in OPFS, handles in IndexedDB, a Web Lock, and a BroadcastChannel. I now write the numbers down first: how many files, how large, what must survive, and which browsers. Every one of those answers changed this architecture.

I was also surprised by how far the web platform has come. You can store a file handle and reopen it after a crash, run a SQL database in a worker on a private file system fast enough for a million rows, and let the browser release a lock for you when a tab dies. Five years ago this would have been an Electron app without much discussion, and the only real cost of doing it in the browser today was dropping Firefox and Safari.