Chunked/partial uploads

Some proxies have a limitation where they can only cope with HTTP requests that are less than 4G in size (for file uploads). While this is enough for most uploads, it's not enough for everyone. In particular, texlive-extra builds result in a putjob payload about 5.7G in size.

Given the absence of a standard way to do this (there’s only an abandoned internet draft titled Partial Uploads in HTTP: https://datatracker.ietf.org/doc/html/draft-wright-http-partial-upload-01), use a custom Patch-Content-Range header to specify which chunk is being uploaded. Upon a complete upload, issue a HEAD request; the server calculates MD5 of the file sets ETag to the calculated value. After the upload is verified, the client issues a putjob request with no payload but chunked_upload=1 set; the proxy server uses that to distinguish it from non-chunked uploads

Task: https://phabricator.apertis.org/T8655

added 11 commits

• 899f564b - WIP: Partial/chunked uploads
• 5b684ef4 - Add a stub for chunked uploads
• a45e230d - Add the chunked uploads blueprint to the server app
• 6d0a33f5 - Add HEAD + ETag calculation
• 88101a4c - Add typing stubs for xmltodict
• 392782d5 - Implement the upload client
• 60e1762c - Verify the hashsum after the upload
• b29f832f - Add a test for chunked upload over HTTP
• ab815eb2 - Integrate chunked uploads into the client
• 724f6c40 - Split streaming and file upload routines out of upload_data
• 22226173 - Support chunked uploads on the server side

Compare with previous version

added 8 commits

• 5f764c3e - Add HEAD + ETag calculation
• 697c2b9f - Add typing stubs for xmltodict
• 97df34bc - Implement the upload client
• 546e7bbc - Verify the hashsum after the upload
• 7a4d6870 - Add a test for chunked upload over HTTP
• 2da21c44 - Integrate chunked uploads into the client
• 6fd2b6ce - Split streaming and file upload routines out of upload_data
• 2368e08f - Support chunked uploads on the server side

Compare with previous version

added 7 commits

• bd5382e6 - Add typing stubs for xmltodict
• 30855eab - Implement the upload client
• 281ad6bb - Verify the hashsum after the upload
• c79ceac8 - Add a test for chunked upload over HTTP
• 73238a49 - Integrate chunked uploads into the client
• d85b4783 - Split streaming and file upload routines out of upload_data
• f733aeef - Support chunked uploads on the server side

Compare with previous version

added 11 commits

• acee5ca5 - WIP: Partial/chunked uploads
• 414fe90b - Add a stub for chunked uploads
• 69a3a650 - Add the chunked uploads blueprint to the server app
• 2120fc94 - Add HEAD + ETag calculation
• 5b0eb0e9 - Add typing stubs for xmltodict
• 8b7033fa - Implement the upload client
• db5b103e - Verify the hashsum after the upload
• 2c2047e8 - Add a test for chunked upload over HTTP
• 85cfaf05 - Integrate chunked uploads into the client
• 1f66e45f - Split streaming and file upload routines out of upload_data
• d74a5698 - Support chunked uploads on the server side

Compare with previous version

added 13 commits

• bdb7a2d3 - Server-side part of the partial/chunked uploads
• 02b20c77 - Add a stub for chunked uploads
• e049b2f8 - Add the chunked uploads blueprint to the server app
• 89e43486 - Add HEAD + ETag calculation
• 6a1a8617 - Add typing stubs for xmltodict
• 51f2c945 - Implement the upload client
• f9004a8a - Verify the hashsum after the upload
• 0c0ea60b - Add a test for chunked upload over HTTP
• 9c30dcba - Integrate chunked uploads into the client
• 6da96ffc - Split streaming and file upload routines out of upload_data
• 7aee09c3 - Support chunked uploads on the server side
• c173248e - Add a full test for a 5 GB upload
• 60a37ad8 - Move the simple upload test under tests/upload

Compare with previous version

added 1 commit

• 9af5835b - Remove loop= parameter from asyncio.gather() call

Compare with previous version

changed the description

changed the description

added 1 commit

• 485cc20d - Remove loop= parameter from asyncio.gather() call

Compare with previous version

changed title from WIP: Chunked/partial uploads to Chunked/partial uploads

resolved all threads

added 1 commit

• 22295cf7 - fixup! Add a test for chunked upload over HTTP

Compare with previous version

marked this merge request as draft from 22295cf7

added 7 commits

• cec9e00c - Add a test for chunked upload over HTTP
• d56d7bd1 - Integrate chunked uploads into the client
• a7396add - Split streaming and file upload routines out of upload_data
• 1fc17f34 - Support chunked uploads on the server side
• 2d52f21a - Add a full test for a 5 GB upload
• add11c65 - Move the simple upload test under tests/upload
• db30ffdc - Remove loop= parameter from asyncio.gather() call

Compare with previous version