Skip to content

Chunked/partial uploads

Andrej Shadura requested to merge partial-uploads into main

Some proxies have a limitation where they can only cope with HTTP requests that are less than 4G in size (for file uploads). While this is enough for most uploads, it's not enough for everyone. In particular, texlive-extra builds result in a putjob payload about 5.7G in size.

Given the absence of a standard way to do this (there’s only an abandoned internet draft titled Partial Uploads in HTTP: https://datatracker.ietf.org/doc/html/draft-wright-http-partial-upload-01), use a custom Patch-Content-Range header to specify which chunk is being uploaded. Upon a complete upload, issue a HEAD request; the server calculates MD5 of the file sets ETag to the calculated value. After the upload is verified, the client issues a putjob request with no payload but chunked_upload=1 set; the proxy server uses that to distinguish it from non-chunked uploads

sequence diagram

Task: https://phabricator.apertis.org/T8655

Edited by Andrej Shadura

Merge request reports