rfd/0179-api-versioning.md
Discussion around viable options to help version our evolving web API
The Web API/Client have generally been seen as one-in-the-same with the proxy service. While that is somewhat true due to the fact that the proxy serves the web client, there are scenarios where multiple versions of the proxy can exist in a cluster. This means that the web api should be thought of as a separate "component" when discussing our backward compatibility promise.
We recently updated a few resources to be paginated in the web UI instead of sending the entire list. This example is taken from a recent feature update to the Access Requests list in the Web UI and highlights the issue at hand.
Access Requests in the web UI used a client-side paginated table, and fetched all requests in a single API call.
// example handler
func AccessRequestHandler() ([]AccessRequest, error) {
accessRequests := []AccessRequest
return accessRequests,nil
}
Some users had too many access requests to send over the fetch and the page
would break, becoming unusable for larger users. The solution was to server
side paginate the Access Requests request. The response used to just be a list
of Access Requests, and it has then changed to an object with a list of
requests and nextKey as fields.
// example handler
func AccessRequestHandler() ([]AccessRequest, error) {
accessRequests := []AccessRequest
nextKey := "hi_im_next"
return AccessRequestResponse{
requests: accessRequests,
nextKey: nextKey,
}, nil
}
This breaks backward compatibility because a web client of version n-1 would be expecting a list of requests only, and a web client of n hitting a proxy of n-1 would be expecting to receive the paginated list.
The solution in this particular case was
limit) and decide which shape to sendThis may work, but it highlights a couple of things:
Using a standard guideline for our api service/web client should help us evolve our API overtime in a backward compatible way.
We should stop stripping the api prefix, and add a new endpoint with a incremented prefix to accommodate the new response shape. This will allow us to implement the following guidelines.
// regex to match /v1, /v2, /v3, etc
apiPrefixRegex := regexp.MustCompile(`^/v(\d+)/webapi`)
// request is going to the API?
if matches := apiPrefixRegex.FindStringSubmatch(r.URL.Path); matches != nil {
version := matches[1]
// Strip the prefix only if it's v1
if version == "1" {
newPath := strings.TrimPrefix(r.URL.Path, "/v1")
r.URL.Path = newPath
}
h.ServeHTTP(w, r)
}
We can continue to strip the prefix if the requested path includes /v1 so we can leave our current response handlers "versionless"
and not have to update the entire api server.
// keep this
h.GET("/webapi/tokens", h.WithAuth(h.getTokens))
// and add this with a new version
h.GET("/v2/webapi/tokens", h.WithAuth(h.getTokensV2))
A proxy server version N must support a web client of version N and N-1. A web client of version N might hit a proxy of version N-1. Follow the guidelines below to understand if a new versioned endpoint is needed to support this edge case.
Please follow these guidelines for common scenarios:
You can also use this checklist to make sure you've covered the backward compatibility promise
Creating a new endpoint that previously didn't exist is ok. This endpoint should
be prefixed with /v1 if using the REST API paradigm. The response shape should
always return an object, even if that object would contain only one field.
Prefer pagination for resources that can be paginated, and always create the
response as if they could be paginated in the future.
Bad
func (h *Handler) getFoos() (interface{}, error) {
foos := []string{"foo1", "foo2", "foo3"}
return foos, nil
}
Good
type GetFoosResponse struct {
Foos []string
NextKey string
ContainsDuplicates bool
}
func (h *Handler) getFoos() (interface{}, error) {
foos := []string{"foo1", "foo2", "foo3"}
return GetFoosResponse{
Foos: foos,
ContainsDuplicates: true
}, nil
}
All endpoints should be defined in their own objects inside the cfg.api object in appropriate project's config.ts file, like in this example.
This helps us tell apart same paths but with different HTTP verbs.
Example of a new single endpoint:
user: {
create: '/v1/webapi/users',
}
Example of an endpoint with same paths but with different verbs:
user: {
create: '/v1/webapi/users',
update: '/v1/webapi/users',
}
Example of creating a version N endpoint:
// Note: only mark the old endpoint for deletion later if we need to
// handle a fallback, otherwise we can delete the old endpoint at the
// introduction of a v2 endpoint.
user: {
// TODO(<your-github-handle>): DELETE IN 18.0 - replaced by /v2/webapi/users
create: '/v1/webapi/users',
createV2: '/v2/webapi/users',
}
An endpoint can be removed in a major version n+2, where n is the last major version where the endpoint was used.
Mark endpoints that needs to be removed with:
// TODO(<your-github-handle>): DELETE IN 18.0
h.GET("/webapi/tokens", h.WithAuth(h.getTokens))
Example 1: v17 no longer uses GET /webapi/foos which was last used in v16. The endpoint can be removed in v18.
Example 2: v4.2.0 no longer uses GET /webapi/bars which was last used in v4.1.3. The endpoint can be removed in v6, so that v5 still supports clients using v4.1.3 and v4.2.0.
Adding a new field to a response is OK as long as that field has no effect on the previously existing fields. Any field that was previously in a response should stay populated in the new response, even if that creates duplicate data. An existing field cannot have its type changed. A field should not be removed until the backward compatibility promise has been 'fulfilled'. Follow removing an endpoint guidelines for removing fields.
Bad
type GetFoosResponse struct {
Foos []string
NextKey string
// ContainsDuplicates bool <--- previously existing field made redundant by DuplicateCount
DuplicateCount int
}
func (h *Handler) getFoos() (interface{}, error) {
foos := []string{"foo1", "foo2", "foo3", "foo3"}
return GetFoosResponse{
Foos: foos,
// ContainsDuplicates must be populated to support older clients.
DuplicateCount: 1
}, nil
}
Good
type GetFoosResponse struct {
Foos []string
NextKey string
// TODO(avatus): DELETE IN 18.
// Deprecated: Use `DuplicateCount` instead.
ContainsDuplicates bool
DuplicateCount int
}
func (h *Handler) getFoos() (interface{}, error) {
foos := []string{"foo1", "foo2", "foo3", "foo3"}
return GetFoosResponse{
Foos: foos,
// ContainsDuplicates must be populated to support older clients.
ContainsDuplicates: true
DuplicateCount: 1
}, nil
}
If a response shape must change entirely, prefer creating a new endpoint
By default, prefer a new versioned endpoint when updating the request shape. For
example, you have an API listFoos(startKey: string) and you add a new request
param connectedOnly: bool to it. If you hit a proxy of n-1, the response will
still be ok, just the new param will be silently dropped and you will get all
"foos".
A Request body can be updated and reuse the current endpoint with a new field as long as the omission of the new field would not cause a breaking change (i.e., an old client sending the old request without the new field). If the handler requires this new field in order to function, then a new endpoint must be created with a version increment.
The existing request adding in the new MyNewThing field
type MyRequest struct {
Limit int
Query string
MyNewThing string // <-- new field
}
if the handler has code along the lines of this
if req.MyNewThing == "" {
return nil, err("BAD REQUEST")
}
then that is a BREAKING CHANGE and should thus have a new versioned handler.
If a feature requires a new endpoint, it must properly handle the case of a 404. This means making the "expected" 404 error clear to the user that an endpoint is not found due to a version mismatch (and preferably which minimum version is needed) rather than an ambiguous "Not Found" error. This works mostly for a "resources" endpoint where it is expected to return an empty list if something isn't found (i.e., "no access requests"), but a new endpoint that returns a single resource should not assume that a 404 means anything other than a resource is not found. In this particular case, there isn't much graceful work we can do here. Graceful handling of this error state should be best effort.
If the updated feature cannot function without receiving new fields from the API (for example, receiving a response from a proxy version N-1), refer to the API guidelines about creating a new versioned endpoint. If the feature itself is degraded but still usable (be conservative with your discretion on this/ask product), a warning alert should be shown to the user with information about
An example of this in recent memory is Pinned Resources. The "pinned resources" tab was available in clients that may have been communicating with proxies older than the required version. So a check was made to see if that proxy supported pinned resources, and if it wasn't, made the error known to the user that "Pinned Resources is supported by proxies version 14.1+" (or something like that).