Multipart HTTP transfer mode proposal

This is a proposal for a new transfer mode, designed to support multi-part HTTP uploads. This is a protocol extension to Git LFS, defining a new transfer mode to be implemented by Git LFS clients and servers in addition to the current basic transfer mode.

This proposal is based on the experimental multipart-basic transfer mode originally implemented by datopian/giftless.

Reasoning

Many storage vendors and cloud vendors today offer an API to upload files in "parts" or "chunks", using multiple HTTP requests, allowing improved stability and performance. This is especially handy when files are multiple gigabytes in size, and a failure during the upload of a file would require re-uploading it, which could be extremely time consuming.

The purpose of the multipart transfer mode is to allow Git LFS servers and client facilitate direct-to-storage uploads for backends supporting multipart or chunked uploads.

As the APIs offered by storage vendors differ greatly, multipart transfer mode will offer abstraction over most of these complexities in hope of supporting as many storage vendors as possible.

Terminology

Throughout this document, the following terms are in use:

LFS Server - The HTTP server to which the LFS batch request is sent
Client or LFS Client - a client using the Git LFS protocol to push large files to storage via an LFS server
Storage Backend - The HTTP server handling actual storage; This may or may not be the same server as the LFS server, and for the purpose of this document, typically it is not. A typical implementation of this protocol would have the Storage Backend be a cloud storage service such as Amazon S3 or Google Cloud Storage.

Design Goals

Abstract vendor specific API and flow into a generic protocol
Remain as close as possible to the basic transfer API
Work at least with the multi-part APIs of Amazon S3, Google Cloud Storage and Azure Blob Storage,
Define how uploads can be resumed by re-doing parts and not-redoing parts that were uploaded successfully (this may be vendor specific and not always supported)
Do not require any state to be maintained in the server side

High Level Protocol Specs

The name of the transfer is multipart
Batch requests are the same as basic requests except that {"transfers": ["multipart", "basic"]} is the expected transfers value. Clients MUST retain basic as the fallback transfer mode to ensure compatibility with servers not implementing this extension.
{"operation": "download"} replies work exactly like basic download request with no change
{"operation": "upload"} replies will break the upload into several actions:
- parts (optional), a list of zero or more part upload actions
- verify (optional), an action to verify the file is in storage, similar to basic upload verify actions
- abort (optional), an action to abort the upload and clean up all unfinished chunks and state
Just like basic transfers, if the file fully exists and is committed to storage, no actions will be provided in the reply and the upload can simply be skipped
If a verify action is provided, calling it is required and not optional. In some cases, this endpoint may be used to finalize the upload.
An empty or missing list of parts with a verify action may mean all parts have been uploaded but verify still needs to be called by the client.
Authentication and authorization behave just like with the basic protocol.

Action Objects

Each one of the parts and the verify and abort actions contain instructions for the client on how to send a request performing a the particular action. These are similar to basic transfer adapter actions but may include some common, as well as action specific additional parameters.

All actions allow href, header and expires_in parameters just like basic transfer actions.

`parts` actions

Each parts action should include the pos and size attributes, in addition to the attributes specified above:

pos indicate the position in bytes within the file in which the part should begin. If not specified, 0 (that is the beginning of the file) is assumed.
size is the size of the part in bytes. If size is omitted, default to read until the end of file.
If both pos and size are omitted, the action is expected to be a single-part upload of the entire file

In addition, parts actions may include the following parameters:

method, with PUT as the default method if none is specified. This allows customizing the HTTP method used when uploading object parts.
want_digest to specify an expected HTTP Digest header, as described below.

`verify` action

The verify action is similar to basic transfer mode verify, with the following additional parameters:

params - an object with additional parameters to send to the server when sending the verify request. These parameters are to be sent to the server exactly as provided, as the value of the params JSON attribute.

`abort` action

The abort action may include the method attribute as specified for parts actions above.

Batch Request / Response Examples

Upload Batch Request

The following is a ~10mb file upload request:

json

{
  "transfers": ["multipart", "basic"],
  "operation": "upload",
  "objects": [
    {
      "oid": "20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e",
      "size": 10000000
    }
  ]
}

Upload Batch Response

The following is a response for the same request, given an imaginary storage backend:

json

{
  "transfer": "multipart",
  "objects": [
    {
      "oid": "20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e",
      "size": 10000000,
      "actions": {
        "parts": [
          {
            "href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=0",
            "header": {
              "Authorization": "Bearer someauthorizationtokenwillbesethere"
            },
            "pos": 0,
            "size": 2500000,
            "expires_in": 86400
          },
          {
            "href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=1",
            "header": {
              "Authorization": "Bearer someauthorizationtokenwillbesethere"
            },
            "pos": 2500000,
            "size": 2500000,
            "expires_in": 86400
          },
          {
            "href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=2",
            "header": {
              "Authorization": "Bearer someauthorizationtokenwillbesethere"
            },
            "pos": 5000000,
            "size": 2500000,
            "expires_in": 86400
          },
          {
            "href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=3",
            "header": {
              "Authorization": "Bearer someauthorizationtokenwillbesethere"
            },
            "pos": 7500000,
            "expires_in": 86400
          }
        ],
        "verify": {
          "href": "https://lfs.mycompany.example/myorg/myrepo/multipart/verify",
          "authenticated": true,
          "header": {
            "Authorization": "Basic 123abc123abc123abc123abc123="
          },
          "expires_in": 86400,
          "params": {
            "uploadId": "20492a4d0d84",
            "partIds": [0, 1, 2, 3]
          }
        },
        "abort": {
          "href": "https://storage.cloud.example/storage/upload/20492a4d0d84",
          "authenticated": true,
          "header": {
            "Authorization": "Basic 123abc123abc123abc123abc123="
          },
          "method": "DELETE",
          "expires_in": 86400
        }
      }
    }
  ]
}

`verify` request example

Given the batch response above, after all parts have been uploaded the client should send the following verify request to https://lfs.mycompany.example/myorg/myrepo/multipart/verify:

POST /myorg/myrepo/multipart/verify
Host: lfs.mycompany.example
Authorization: Basic 123abc123abc123abc123abc123=
Content-type: application/json

{
  "oid": "20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e",
  "size": 10000000,
  "params": {
    "uploadId": "20492a4d0d84",
    "partIds": [0, 1, 2, 3]
  }
}

Assuming that all parts were uploaded successfully, the server should respond with a 200 OK response.

`abort` request example

Given the batch response above, the client may choose to cancel the upload by sending the following abort request to https://storage.cloud.example/storage/upload/20492a4d0d84:

> DELETE /storage/upload/20492a4d0d84
> Host: storage.cloud.example
> Content-length: 0

Uploaded Part Digest

Some storage backends will support, or even require, uploading clients to send a digest of the uploaded part when uploading the part. This is a useful capability even if not required, as it allows backends to validate each part separately as it is uploaded.

To support this, parts request objects may include a want_digest value, which is expected to be a list of digest algorithms in the same format of the Want-Digest HTTP header specified by RFC-3230.

Any cryptographically secure digest algorithm registered with IANA via the process outlined in RFC-3230 may be specified in want_digest. Algorithms considered cryptographically insecure, including MD5 and SHA-1, should not be accepted. Namely, contentMD5 is not an accepted value of want_digest.

If one or more digest algorithms with non-zero q-value is specified in want_digest, clients should select a favored supported algorithm, calculate the part digest using that algorithm, and send it when uploading the part using the Digest HTTP header as specified by RFC-3230 section 4.3.1.

While clients may include the part digest calculated using more than one algorithm, this is typically not required and should be avoided.

Note that if want_digest is specified but the client cannot support any of the requested algorithms, the client may still choose to continue uploading parts without sending a Digest header. However, the storage server may choose to reject the request in such cases.

Uploaded Part Digest Example

Examples of a batch response with `want_digest` in the reply

With SHA-512 as a preferred algorithm, and SHA-256 as a less preferred option if SHA-512 is not possible:

json

{
  "actions": {
    "parts": [
      {
        "href": "https://storage.cloud.example/storage/upload/20492a4d0d84?part=3",
        "header": {
          "Authorization": "Bearer someauthorizationtokenwillbesethere"
        },
        "pos": 7500001,
        "want_digest": "sha-512;q=1.0, sha-256;q=0.5"
      }
    ]
  }
}

Example of part upload request send to the storage server

Following on the want_digest value specified in the last example, the client should now send the following headers to the storage server when uploading the part, assuming SHA-512 is supported:

HTTP/1.1 PUT /storage/upload/20492a4d0d84?part=3
Authorization: Bearer someauthorizationtokenwillbesethere
Digest: SHA-512=thvDyvhfIqlvFe+A9MYgxAfm1q5thvDyvhfIqlvFe+A9MYgxAfm1q5=

Expected HTTP Responses

For each one of the parts, as well as abort and verify requests sent by the client, the following responses are to be expected:

Any response with a 20x status code is to be considered by clients as successful. This ambiguity is by design, to support variances between vendors (which may use 200 or 201 to indicate a successful upload, for example).
Any other response is to be considered as an error, and it is up to the client to decide whether the request should be retried or not. Implementors are encouraged to follow standard HTTP error status code guidelines.

`batch` replies for partially uploaded content

When content was already partially uploaded, the server is expected to return a normal reply but omit request and parts which do not need to be repeated. If the entire file has been uploaded, it is expected that no actions value will be returned, in which case clients should simply skip the upload.

However, if parts of the file were successfully uploaded while others weren't, it is expected that a normal reply would be returned, but with less parts to send.

`verify` HTTP 409 errors

An HTTP 409 error on verify requests typically indicates that the file could not be fully committed or verified. In this case, clients should follow the following process to try and recover from the error:

retry the batch request to see if any parts of the file were not uploaded yet. If there are still parts to upload (i.e. parts is not empty), proceed to upload them and re-do verify
If parts is empty, it is possible that the file exists in storage but is corrupt / has wrong size. In this case it is recommended to issue an abort and re-attempt the same upload again
It is recommended to take special note of the number of retries, to avoid infinite recovery attempt loops

Additional Considerations

Chunk sizing

It is up to the LFS server to decide the size of each file chunk.

Action lifetime considerations

As multipart uploads tend to require much more time than simple uploads, it is recommended to allow for longer expires_in values than one would consider for basic uploads. It is possible that the process of uploading a single object in multiple parts may take several hours from batch to verify.

Falling back to `basic` transfer for small files

Using multipart upload APIs has some complexity and speed overhead. For this reason, if a client specifies support for both multipart and basic transfer modes in a batch request, and the object(s) uploaded are small enough to fit in a single part upload, servers may choose to respond with a basic transfer mode even if multipart is supported:

For example a small (2mb) upload batch request:

{
  "transfers": ["multipart", "basic"],
  "operation": "upload",
  "objects": [
    {
      "oid": "13aea96040f2133033d103008d5d96cfe98b3361f7202d77bea97b2424a7a6cd",
      "size": 2000000
    }
  ]
}

May be responded with:

{
  "transfer": "basic",
  "objects": [
    ...
  ]
}

Even if the server does support multipart, as basic can be preferable in this case.

Implementation Notes

Hiding initialization / commit complexities from clients

While part requests are typically quite similar between vendors, the specifics of multipart upload initialization and commit procedures are very specific to vendors. For this reason, in many cases, it will be up to the LFS server to take care of initialization and commit code. This is fine, as long as actual uploaded data is sent directly to the storage backend.

For example, in the case of Amazon S3:

All requests need to have an "upload ID" token which is obtained in an initial request
When finalizing the upload, a special "commit" request need to be sent, listing all uploaded part IDs.

These are very hard to abstract in a way that would allow clients to send them directly to the server. In addition, as we do not want to maintain any state in the server, there is a need to make two requests when finalizing the upload: one to fetch a list of uploaded chunks, and another to send this list to the S3 finalization endpoint.

For this reason, it is expected that any initialization actions will be handled by the Git LFS server during the batch request handling. In most cases, the verify action will also be responsible for any finalization / commit actions. The params attribute of the verify action is designed specifically to transfer some vendor-specific "state" between initialization and finalization of the upload process.

Implementing Complex `abort` actions

Some storage backends will accept a simple DELETE or POST request to a URL, with no request body, in order to abort the upload. In such cases, abort may refer directly to the storage backend. However, in cases where aborting the upload requires more complex logic or some payload in the request body, abort actions should point to an endpoint of the LFS server, and it should be up to the LFS server to abort the upload and clean up any partially uploaded parts.

As abort requests do not have a body, any parameters required by the LFS server in order to complete the request should be passed as part of the URL in the href parameter.

It should be noted that clients will not always be able to abort partial uploads cleanly. Implementors are expected to ensure proper cleanup of partially uploaded files via other means, such as a periodical cron job that locates uncommitted uploaded parts and deletes them.

Multipart HTTP transfer mode proposal

Multipart HTTP transfer mode proposal

Reasoning

Terminology

Design Goals

High Level Protocol Specs

Action Objects

parts actions

verify action

abort action

Batch Request / Response Examples

Upload Batch Request

Upload Batch Response

verify request example

abort request example

Uploaded Part Digest

Uploaded Part Digest Example

Examples of a batch response with want_digest in the reply

Example of part upload request send to the storage server

Expected HTTP Responses

batch replies for partially uploaded content

verify HTTP 409 errors

Additional Considerations

Chunk sizing

Action lifetime considerations

Falling back to basic transfer for small files

Implementation Notes

Hiding initialization / commit complexities from clients

Implementing Complex abort actions

`parts` actions

`verify` action

`abort` action

`verify` request example

`abort` request example

Examples of a batch response with `want_digest` in the reply

`batch` replies for partially uploaded content

`verify` HTTP 409 errors

Falling back to `basic` transfer for small files

Implementing Complex `abort` actions