doc/development/api_styleguide.md
This style guide recommends best practices for API development.
We offer two types of API to our customers:
To reduce the technical burden of supporting two APIs in parallel, they should share implementations as much as possible. For example, they could share the same services.
See the frontend guide on details on which API to use when developing in the frontend.
Don't use instance variables, there is no need for them (we don't need to access them as we do in Rails views), local variables are fine.
Always use an Entity to present the endpoint's payload.
Every exposed field in an entity must include or reference a valid type.
When exposing a field that references another entity, use the using option.
The using option only accepts a constant that points to an API::Entities class.
A good example is as follows,
expose :project, using: ::API::Entities::BasicProjectDetails
Field types must be specified as strings. The following types are accepted:
| Category | Types |
|---|---|
| Scalar | Integer, Float, BigDecimal, Numeric, Date, DateTime, Time, String, Symbol, Boolean |
| Structures | Hash, Array, Set |
| Special | JSON, File |
| Entity references | Any API::Entities::* class (as a string) |
Field types should be defined in the documentation hash:
expose :id, documentation: { type: 'Integer', example: 1 }
expose :name, documentation: { type: 'String', example: 'John Doe' }
expose :active, documentation: { type: 'Boolean', example: true }
expose :project, documentation: { type: 'API::Entities::BasicProject'}
Some foundational entities like UserBasic, ProjectIdentity, and Commit
are embedded or nested across many API endpoints. Adding a single expose
call to one of these entities inflates the JSON response of every endpoint
that uses it, directly or transitively. For example, adding a single expose
call to UserBasic would affect 212 endpoints, and to CustomAttribute 238.
To prevent uncontrolled growth of API response payloads, a set of
high-impact entities is protected by the
API/EntityExposureGrowth
RuboCop cop. The cop maintains an allowlist of permitted fields per entity in
api_entity_exposure_baseline.yml.
Any new expose call added to a protected entity that is not in the allowlist
triggers an offense.
class User < UserBasic)
and embedding (expose :author, using: UserBasic). A single field added to
UserBasic cascades to User, UserPublic, and every entity that embeds it.Instead of adding fields to a high-impact entity, create a feature-bounded entity: a new, purpose-built entity class that is used only by the endpoints that need the new field.
The simplest approach is to create a new entity that inherits from the foundational one and adds the fields you need:
# bad - adds :notification_email to every endpoint using UserBasic (212 endpoints)
module API
module Entities
class UserBasic < UserSafe
expose :state
expose :avatar_url
expose :web_url
expose :notification_email # <-- new field inflates 212 endpoint responses
end
end
end
# good - create a domain-scoped entity used only by the endpoints that need it
module API
module Entities
module Ci
class JobOwner < UserBasic
expose :notification_email, documentation: { type: 'String', example: '[email protected]' }
end
end
end
end
Name the entity after what it represents in its domain context (for example,
Ci::JobOwner), not after the fields it contains (for example,
UserWithNotificationEmail). A name like UserWithNotificationEmail invites
reuse across unrelated domains, which re-creates the cascade problem. A
domain-scoped name keeps the entity focused on a single use case.
Then use the new entity only in the endpoints that need it:
# In your API endpoint file
desc 'List CI job owners' do
detail 'Returns the owners of CI jobs with notification details.'
success Entities::Ci::JobOwner
tags ['ci']
end
get ':id/ci/job_owners' do
owners = find_job_owners(params[:id])
present owners, with: Entities::Ci::JobOwner
end
The allowlist in
api_entity_exposure_baseline.yml
records the permitted field names for each protected entity. You should
not manually edit the allowlist to add new fields; instead, create a
feature-bounded entity as described above.
If you believe a field genuinely belongs on a high-impact entity (for example, it is needed by the vast majority of consumers), open a discussion with the API Platform team to evaluate the trade-offs before proceeding.
Each new or updated API endpoint must come with documentation. The docs should be in the same merge request, or, if strictly necessary, in a follow-up with the same milestone as the original merge request.
See the Documentation Style Guide RESTful API page for details on documenting API resources in Markdown as well as in OpenAPI definition files.
Every method must be described using the Grape DSL
(see environments.rb
for a good example):
desc for the method summary. This must include a summary string no more than 120 characters.detail for each desc block. This must be a string.success for each desc block. This defines the success response.tags for each desc block. This should be a string, or array of strings.params for the method parameters. This acts as description,
validation, and coercion of the parametersA good example is as follows:
desc 'Get all broadcast messages' do
detail 'This feature was introduced in GitLab 8.12.'
success Entities::System::BroadcastMessage
tags ['broadcast_messages']
end
params do
optional :page, type: Integer, desc: 'Current page number'
optional :per_page, type: Integer, desc: 'Number of messages per page'
end
get do
messages = System::BroadcastMessage.all
present paginate(messages), with: Entities::System::BroadcastMessage
end
Every endpoint must include a summary string in the desc block.
The summary describes the operation on the REST resource and is used in the generated OpenAPI documentation.
The summary should:
The summary must:
A good example is as follows:
desc 'Get a specific environment' do
detail 'Returns environment details. This feature was introduced in GitLab 18.12.'
success Entities::Environment
end
A bad example is as follows:
desc 'Get a specific environment. Returns environment details. This feature was introduced in GitLab 18.12.' do
detail 'Only available to authenticated project owner.'
success Entities::Environment
end
Every endpoint must have a detail value for each desc block. The value must be a string.
The detail should describe any additional details not covered by the desc such as:
This feature is gated by the :feature\_flag\_symbol feature flag.Do not include lifecycle terms like "experiment", "experimental", "general availability", "GA", or "beta" in the
detail or desc summary strings. Use route_setting :lifecycle instead.
For more information, see Marking endpoint lifecycle.
Every endpoint must have a success value for each desc block.
The value should accurately describe a success response for the endpoint.
Do not use the http_codes option to document the success response.
The success option accepts either:
Grape::Entity class directlyWhen using the hash form, the following options are available:
| Option | Type | Required | Description |
|---|---|---|---|
code | Integer | No | The HTTP status code. Although it is not required always specify the intended response code. |
model | Entities::* | Required for JSON responses | The Grape::Entity class returned in response body. Without a model, no response schema or examples are emitted in the OpenAPI spec. Omit only for responses with no body, such as 204 No Content or redirects. |
message | String | No | A short description of the response. |
is_array | Boolean | No | Set to true if the response is an array of the model. Only needed in the hash form. Wrapping the entity class in an array (success [Entities::MyEntity]) is equivalent. |
example | Hash | No | A single inline example of the response body. Mutually exclusive with examples. Requires model. |
examples | Hash | No | Named examples of the response body. Mutually exclusive with example. Requires model. |
Format the success value based on what the endpoint returns:
If the endpoint responds with an object, pass the Grape::Entity class directly or using the model: option:
# Direct form
success Entities::System::BroadcastMessage
# Hash form
success code: 200, model: Entities::System::BroadcastMessage
If the endpoint responds with a collection, either wrap the entity class in an array or use is_array: true in the hash form. Both are equivalent:
# Direct form
success [Entities::System::BroadcastMessage]
# Hash form — use when you also need to specify other options
success code: 200, model: Entities::System::BroadcastMessage, is_array: true
If the endpoint does not respond with an object, include a status code and message:
success code: 204, message: 'Record was deleted'
If the endpoint returns multiple possible success codes, pass an array:
success [
{ code: 200, model: Entities::Security::VulnerabilityScanning::SbomScan },
{ code: 202, message: 'Scan in progress' }
]
If no example: or examples: is provided, and a model: is defined, an example is
generated automatically — either from documentation: { example: ... } values on the
entity fields, or from field types if no field-level examples are defined.
If the endpoint responds with an object and you want to illustrate a complete response
body or provide multiple possible response body examples, use example: for a single
inline value or examples: for multiple named scenarios.
Both require model: and are mutually exclusive:
# Single example
success code: 200, model: Entities::System::BroadcastMessage,
example: {
id: 1,
message: 'Scheduled maintenance at 23:00',
starts_at: '2024-03-01T23:00:00.000Z',
ends_at: '2024-03-02T01:00:00.000Z',
active: false
}
# Multiple named examples
success code: 200, model: Entities::System::BroadcastMessage,
examples: {
active_message: {
summary: 'An active broadcast message',
value: {
id: 1,
message: 'Scheduled maintenance at 23:00',
starts_at: '2024-03-01T23:00:00.000Z',
ends_at: '2024-03-02T01:00:00.000Z',
active: true
}
},
expired_message: {
summary: 'An expired broadcast message',
value: {
id: 2,
message: 'Maintenance complete',
starts_at: '2024-03-01T23:00:00.000Z',
ends_at: '2024-03-02T01:00:00.000Z',
active: false
}
}
}
When deprecating an endpoint, add the following to the desc block:
deprecated true option.
This sets the standard OpenAPI deprecated: true flag on the operation.detail option.Do not use route_setting :lifecycle for deprecated endpoints. Unlike experiment and
beta stages, deprecation is natively supported by the OpenAPI specification through
the deprecated field, which deprecated true maps to directly.
desc 'Get legacy broadcast messages' do
detail 'Deprecated in GitLab 17.0. Use /api/v4/broadcast_messages instead.'
deprecated true
success Entities::System::BroadcastMessage
tags ['broadcast_messages']
end
Together, these make the deprecation programmatically discoverable in the OpenAPI specification.
When an endpoint is not yet generally available, use route_setting :lifecycle to
indicate its development stage. Valid values are :experiment and :beta.
Do not put lifecycle information in the desc summary or detail strings.
The API/LifecycleInDescription RuboCop cop enforces this rule.
For generally available endpoints, omit route_setting :lifecycle.
# bad -- Specifies "experimental" in "detail"
desc 'Get all widgets' do
detail 'This feature is experimental.'
tags %w[widgets]
end
# good -- Specifies "experiment" as route_setting
route_setting :lifecycle, :experiment
desc 'Get all widgets' do
detail 'Introduced in GitLab 18.10.'
tags %w[widgets]
end
# good -- Specifies "beta" as route_setting
route_setting :lifecycle, :beta
desc 'Get all widgets' do
detail 'Introduced in GitLab 18.10.'
tags %w[widgets]
end
The route_setting :lifecycle value is included in the generated OpenAPI specification
as the x-gitlab-lifecycle vendor extension. This makes the lifecycle status
programmatically discoverable.
For more information about development stages, see development stages and support.
Every endpoint must have at least one value defined in tags per desc block.
The tags should describe the type of objects being acted upon in the API call, in their plural form.
In most cases, the filename of the API is sufficient but can also be too granular.
audit_eventsusersclusterscommit (singular)epic_management (coupled to a product category, not an entity)If the correct name for a tag is not clear, speak to technical writers for guidance.
We must not make breaking changes to our REST API v4, even in major GitLab releases. See what is a breaking change and what is not a breaking change.
Our REST API maintains its own versioning independent of GitLab versioning.
The current REST API version is 4. Because we commit to follow semantic versioning for our REST API, we cannot make breaking changes to it. A major version change for our REST API (most likely, 5) is currently not planned, or scheduled.
The exception is API features that are marked as experimental or beta. These features can be removed or changed at any time.
The following sections suggest alternatives to making breaking changes.
If a feature changes, we should aim to accommodate backwards-compatibility without making a breaking change to the API.
Instead of introducing a breaking change, change the API controller layer to adapt to the feature change in a way that does not present any change to the API consumer.
For example, we renamed the merge request WIP feature to Draft. To accomplish the change, we:
draft field to the API response.work_in_progress field.Customers did not experience any disruption to their existing API integrations.
When a feature that an endpoint interfaced with is removed in a major GitLab version, we must maintain a balance between API backwards-compatibility and returning a result the user can rely on.
Choose the appropriate approach based on the context:
Silent degradation - Use when an error would disrupt broader functionality:
null or []).Error response - Use when a feature has been fully removed:
404 Not Found when the removed feature was the primary purpose of the endpoint.The key principle is that existing customer API integrations should degrade gracefully where possible, while providing clear feedback when a feature is no longer available. The endpoints continue to respond with the same fields and accept the same arguments, although the underlying feature interaction is no longer operational.
The intended changes must be documented ahead of time following the v4 deprecation guide.
For example, when we removed an application setting, we kept the old API field which now returns a sensible static value.
Some examples of breaking changes are:
Number, String, Boolean, Array, or Object type to another type.500.Some examples of non-breaking changes:
500 status code to any supported status code (this is a bugfix).You can add API elements as experimental and beta features. They must be additive changes, otherwise they are categorized as a breaking change.
API elements marked as experiment or beta are exempt from the breaking changes policy, and can be changed or removed at any time without prior notice.
While in the experiment status:
route_setting :lifecycle, :experiment before the endpoint. For more information, see Marking endpoint lifecycle.404 Not Found.hidden option).While in the beta status:
route_setting :lifecycle, :beta before the endpoint. For more information, see Marking endpoint lifecycle.When the feature becomes generally available:
route_setting :lifecycle from the endpoint.Grape allows you to access only the parameters that have been declared by your
params block. It filters out the parameters that have been passed, but are not
allowed. For more details, see the Ruby Grape documentation for declared().
By default declared(params) includes parameters that were defined in all
parent namespaces. For more details, see the Ruby Grape documentation for include_parent_namespaces.
In most cases you should exclude parameters from the parent namespaces:
declared(params, include_parent_namespaces: false)
declared(params)You should always use declared(params) when you pass the parameters hash as
arguments to a method call.
For instance:
# bad
User.create(params) # imagine the user submitted `admin=1`... :)
# good
User.create(declared(params, include_parent_namespaces: false).to_h)
[!note]
declared(params)return aHashie::Mashobject, on which you must call.to_h.
But we can use params[key] directly when we access single elements.
For instance:
# good
Model.create(foo: params[:foo])
With Grape v1.3+, Array types must be defined with a coerce_with
block, or parameters, fails to validate when passed a string from an
API request. See the
Grape upgrading documentation
for more details.
Prior to Grape v1.3.3, Array parameters with nil values would
automatically be coerced to an empty Array. However, due to
this pull request in v1.3.3, this
is no longer the case. For example, suppose you define a PUT /test
request that has an optional parameter:
optional :user_ids, type: Array[Integer], coerce_with: ::API::Validations::Types::CommaSeparatedToIntegerArray.coerce, desc: 'The user ids for this rule'
Usually, a request to PUT /test?user_ids would cause Grape to pass
params of { user_ids: nil }.
This may introduce errors with endpoints that expect a blank array and
do not handle nil inputs properly. To preserve the previous behavior,
there is a helper method coerce_nil_params_to_array! that is used
in the before block of all API calls:
before do
coerce_nil_params_to_array!
end
With this change, a request to PUT /test?user_ids causes Grape to
pass params to be { user_ids: [] }.
There is an open issue in the Grape tracker to make this easier.
All REST API endpoints that accept file content must use Workhorse-assisted uploads.
See the Workhorse uploads documentation for implementation details.
For non-200 HTTP responses, use the provided helpers in lib/api/helpers.rb to ensure correct behavior (like not_found! or no_content!). These throw inside Grape and abort the execution of your endpoint.
For DELETE requests, you should also generally use the destroy_conditionally! helper which by default returns a 204 No Content response on success, or a 412 Precondition Failed response if the given If-Unmodified-Since header is out of range. This helper calls #destroy on the passed resource, but you can also implement a custom deletion method by passing a block.
When defining a new API route, use the correct HTTP request method.
PATCH and PUTIn a Rails application, both the PATCH and PUT request methods are routed to
the update method in controllers. With Grape, the framework we use to write
the GitLab API, you must explicitly set the PATCH or PUT HTTP verb for an
endpoint that does updates.
If the endpoint updates all attributes of a given resource, use the
PUT request
method. If the endpoint updates some attributes of a given resource, use the
PATCH
request method.
Here is a good example for PATCH: PATCH /projects/:id/protected_branches/:name
Here is a good example for PUT: PUT /projects/:id/merge_requests/:merge_request_iid/approve
Often, a good PUT endpoint only has ids and a verb (in the example above, "approve").
Or, they only have a single value and represent a key/value pair.
The Rails blog
has a detailed explanation of why PATCH is usually the most apt verb for web
API endpoints that perform an update.
Because we support installing GitLab under a relative URL, one must take this
into account when using API path helpers generated by Grape. Any such API path
helper usage must be in wrapped into the expose_path helper call.
For instance:
- endpoint = expose_path(api_v4_projects_issues_related_merge_requests_path(id: @project.id, issue_iid: @issue.iid))
In order to validate some parameters in the API request, we validate them before sending them further (say Gitaly). The following are the custom validators, which we have added so far and how to use them. We also wrote a guide on how you can add a new custom validator.
FilePath:
GitLab supports various functionalities where we need to traverse a file path.
The FilePath validator
validates the parameter value for different cases. Mainly, it checks whether a
path is relative and does it contain ../../ relative traversal using
File::Separator or not, and whether the path is absolute, for example
/etc/passwd/. By default, absolute paths are not allowed. However, you can optionally pass in an allowlist for allowed absolute paths in the following way:
requires :file_path, type: String, file_path: { allowlist: ['/foo/bar/', '/home/foo/', '/app/home'] }
Git SHA:
The Git SHA validator
checks whether the Git SHA parameter is a valid SHA.
It checks by using the regex mentioned in commit.rb file.
Absence:
The Absence validator
checks whether a particular parameter is absent in a given parameters hash.
IntegerNoneAny:
The IntegerNoneAny validator
checks if the value of the given parameter is either an Integer, None, or Any.
It allows only either of these mentioned values to move forward in the request.
ArrayNoneAny:
The ArrayNoneAny validator
checks if the value of the given parameter is either an Array, None, or Any.
It allows only either of these mentioned values to move forward in the request.
EmailOrEmailList:
The EmailOrEmailList validator
checks if the value of a string or a list of strings contains only valid
email addresses. It allows only lists with all valid email addresses to move forward in the request.
Custom validators are a great way to validate parameters before sending them to platform for further processing. It saves some back-and-forth from the server to the platform if we identify invalid parameters at the beginning.
If you need to add a custom validator, it would be added to
it's own file in the validators directory.
Since we use Grape to add our API
we inherit from the Grape::Validations::Validators::Base class in our validator class.
Now, all you have to do is define the validate_param! method which takes
in two parameters: the params hash and the param name to validate.
The body of the method does the hard work of validating the parameter value and returns appropriate error messages to the caller method.
Lastly, we register the validator using the line below:
Grape::Validations.register_validator(<validator name as symbol>, ::API::Helpers::CustomValidators::<YourCustomValidatorClassName>)
Once you add the validator, make sure you add the rspecs for it into
it's own file in the validators directory.
The internal API is documented for internal use. Keep it up to date so we know what endpoints different components are making use of.
In order to avoid N+1 problems that are common when returning collections of records in an API endpoint, we should use eager loading.
A standard way to do this within the API is for models to implement a
scope called with_api_entity_associations that preloads the
associations and data returned in the API. An example of this scope can
be seen in
the Issue model.
In situations where the same model has multiple entities in the API
(for instance, UserBasic, User and UserPublic) you should use your
discretion with applying this scope. It may be that you optimize for the
most basic entity, with successive entities building upon that scope.
The with_api_entity_associations scope also
automatically preloads data
for Todo targets when returned in the to-dos API.
For more context and discussion about preloading see this merge request which introduced the scope.
When an API endpoint returns collections, always add a test to verify
that the API endpoint does not have an N+1 problem, now and in the future.
We can do this using ActiveRecord::QueryRecorder.
Example:
def make_api_request
get api('/foo', personal_access_token: pat)
end
it 'avoids N+1 queries', :request_store do
# Firstly, record how many PostgreSQL queries the endpoint will make
# when it returns a single record
create_record
control = ActiveRecord::QueryRecorder.new { make_api_request }
# Now create a second record and ensure that the API does not execute
# any more queries than before
create_record
expect { make_api_request }.not_to exceed_query_limit(control)
end
When writing tests for new API endpoints, consider using a schema fixture located in /spec/fixtures/api/schemas. You can expect a response to match a given schema:
expect(response).to match_response_schema('merge_requests')
Also see verifying N+1 performance in tests.
All client-facing changes must include a changelog entry. This does not include internal APIs.