docs/documentation/api.md
This document explains the system for documenting Zulip's REST API.
Zulip's API documentation is an essential resource both for users and for the developers of Zulip's mobile and terminal apps. Our vision is for the documentation to be sufficiently good that developers of Zulip's apps should never need to look at the server's implementation to answer questions about the API's semantics.
To achieve these goals, Zulip leverages the popular OpenAPI format as the data source to ensure that Zulip's API documentation is correct and remains so as Zulip's API evolves.
In particular, the top goal for this system is that all mistakes in verifiable content (i.e. not the English explanations) should cause the Zulip test suite to fail. This is incredibly important, because once you notice one error in API documentation, you no longer trust it to be correct, which ends up wasting the time of its users.
Since it's very difficult to not make little mistakes when writing any untested code, the only good solution to this is a way to test the documentation. We found dozens of errors in the process of adding the validation Zulip has today.
Our API documentation is defined by a few sets of files:
zerver/openapi/zulip.yaml.api_docs/api-doc-template.md, which renders the
OpenAPI description of the API endpoint. A handful of endpoints that
require special content, as well as pages that document general API
details rather than specific endpoints, live at api_docs/*.md.zerver/tests/test_openapi.py compares every endpoint's accepted
parameters in views code with those declared in zulip.yaml. And
the backend test suite checks
that every API response served during our extensive backend test
suite matches one the declared OpenAPI schema for that endpoint.zerver/openapi/python_examples.py; run via
tools/test-api). The generate_code_example macro will magically
read content from that test suite and render it as the code example.
This structure ensures that Zulip's API documentation is robust to a
wide range of possible typos and other bugs in the API
documentation.zerver/openapi/javascript_examples.js.zerver/openapi/curl_param_value_generators.py.api_docs/include/rest-endpoints.md) in the broader
/api left sidebar (api_docs/sidebar_index.md).This first section is focused on explaining how the API documentation system is put together; when actually documenting an endpoint, you'll want to also read the Step by step guide.
Let's use the existing documentation for one of our REST API endpoints to show how the documentation system works: POST /messages/render. We highly recommend looking at these resources while reading the above documentation page:
api_docs/api-doc-template.mdzerver/openapi/zulip.yaml, specifically the section with
operationId: render-messagezerver/openapi/python_examples.pyIf you look at the documentation for existing endpoints, you'll notice that a typical endpoint's documentation is divided into four sections:
The rest of this guide describes how each of these sections works.
The first line of api-doc-template.md generates a lot of key
information for our API endpoint documentation:
{generate_api_header(API_ENDPOINT_NAME)}
At the top of the endpoint documentation page is the title, and it
comes from the summary parameter in the OpenAPI data,
zerver/openapi/zulip.yaml.
The endpoint description in the OpenAPI data explains what the
endpoint does in clear English. It should include details on how to
use the endpoint correctly or what it's good or bad for, with links
to any alternative endpoints the user might want to consider.
The description should often contain a link to the documentation of
the relevant feature in the help center, and should
include Changes notes for all feature level updates documented
in the API changelog, see
api_docs/changelog.md, that reference the endpoint.
Endpoints that only administrators can use should be tagged with the
custom x-requires-administrator field in the OpenAPI definition.
All of this information is rendered via a Markdown preprocessor,
specifically the APIHeaderPreprocessor class defined in
zerver/openapi/markdown_extension.py.
We display usage examples in three languages: Python, JavaScript and curl; we may add more in the future. Every endpoint should have Python and curl documentation; JavaScript is optional as we don't consider that API library to be fully supported.
The examples are defined using a special Markdown extension, see
zerver/openapi/markdown_extension.py. Here's the Markdown file
block that uses this in api-doc-template.md:
{start_tabs}
{generate_code_example(python)|API_ENDPOINT_NAME|example}
{generate_code_example(javascript)|API_ENDPOINT_NAME|example}
{tab|curl}
{generate_code_example(curl)|API_ENDPOINT_NAME|example}
{end_tabs}
In some cases, one wants to configure specific parameters to be
included or excluded from the example curl requests for readability
reasons. One can do that using the x-curl-examples-parameters
parameter in the OpenAPI data.
For the Python examples, you'll write the example in
zerver/openapi/python_examples.py, and it'll be run and verified
automatically in Zulip's automated test suite. The code for our
example API endpoint looks like this:
@openapi_test_function('/messages/render:post')
def render_message(client: Client) -> None:
# {code_example|start}
# Render a message
request = {
'content': '**foo**'
}
result = client.render_message(request)
# {code_example|end}
validate_against_openapi_schema(result, '/messages/render', 'post', '200')
This is an actual Python function which will be run as part of the
tools/test-api test suite. The validate_against_openapi_schema
function will verify that the result of that request is as defined in
the examples in zerver/openapi/zulip.yaml.
To run as part of the test suite, the render_message function needs
to be called from test_messages (or one of the other functions at
the bottom of the file). The final function, test_the_api, is what
actually runs the tests. Tests with the openapi_test_function
decorator that are not called will fail tests, as will new endpoints
that are not covered by an openapi_test_function-decorated test.
You will still want to manually test the example using Zulip's Python API client by copy-pasting from the website; it's easy to make typos and other mistakes where variables are defined outside the tested block, and the tests are not foolproof.
The code that renders API documentation pages will extract the block
between the # {code_example|start} and # {code_example|end} comments,
and substitute it in place of
{generate_code_example(python)|/messages/render:post|example}. Note
that here the API_ENDPOINT_NAME has been filled in with our example
endpoint's information.
Additional Python imports can be added using the custom
x-python-examples-extra-imports field in the OpenAPI definition.
We have a separate Markdown extension to document the parameters that
an API endpoint supports. Implemented in
zerver/lib/markdown/api_arguments_table_generator.py, you can see
this in api-doc-template.md after the Parameters header:
{generate_api_arguments_table|zulip.yaml|API_ENDPOINT_NAME}
This generates the information from the endpoint's parameter definition in the OpenAPI data.
Additional content that you'd like to appear in the parameter
description area can be declared using the custom
x-parameter-description field in the OpenAPI definition.
Similar to the parameters section above, there is a separate Markdown
extension to document the endpoint's return values and generate the
example response(s) from the OpenAPI data. Implemented in
zerver/lib/markdown/api_return_values_table_generator.py, you can
see this in after the Response header in api-doc-template.md:
{generate_return_values_table|zulip.yaml|API_ENDPOINT_NAME}
To generate the example responses from the OpenAPI data, we again
use the special Markdown extension from the Usage examples
discussed above, except with the fixture argument instead of the
example argument:
{generate_code_example|API_ENDPOINT_NAME|fixture}
Additional content that you'd like to appear in the responses part of
the page can be added using the custom x-response-description field
in the OpenAPI definition.
This section offers a step-by-step process for adding documentation for a new API endpoint. It assumes you've read and understood the above.
Start by adding OpenAPI format
data to zerver/openapi/zulip.yaml for the endpoint. If you
copy-paste (which is helpful to get the indentation structure
right), be sure to update all the content that you copied to
correctly describe your endpoint!
In order to do this, you need to figure out how the endpoint in
question works by reading the code! To understand how arguments
are specified in Zulip backend endpoints, read our REST API
tutorial, paying special attention to the
details of typed_endpoint.
Once you understand that, the best way to determine the supported
arguments for an API endpoint is to find the corresponding URL
pattern in zproject/urls.py, look up the backend function for
that endpoint in zerver/views/, and inspect its keyword-only
arguments.
You can check your formatting using these helpful tools.
tools/check-openapi.ts will verify the syntax of zerver/openapi/zulip.yaml.tools/test-backend zerver/tests/test_openapi.py; this test compares
your documentation against the code and can find many common
mistakes in how arguments are declared.test-backend: The full Zulip backend test suite will fail if
any actual API responses generated by the tests don't match your
defined OpenAPI schema. Use test-backend --rerun for a fast
edit/refresh cycle when debugging.Add a function for the endpoint you'd like to document to
zerver/openapi/python_examples.py, decorated with
@openapi_test_function. render_message is a good example to
follow. There are generally two key pieces to your test: (1) doing
an API query and (2) verifying its result has the expected format
using validate_against_openapi_schema.
Make the desired API call inside the function. If our Python
bindings don't have a dedicated method for a specific API call,
you may either use client.call_endpoint or add a dedicated
function to the zulip PyPI
package.
Ultimately, the goal is for every endpoint to be documented the
latter way, but it's useful to be able to write working
documentation for an endpoint that isn't supported by
python-zulip-api yet.
Add the function to one of the test_* functions at the end of
zerver/openapi/python_examples.py; this will ensure your
function will be called when running test-api.
Capture the JSON response returned by the API call (the test
"fixture"). The easiest way to do this is add an appropriate print
statement (usually json.dumps(result, indent=4, sort_keys=True)),
and then run tools/test-api. You can also use
https://jsonformatter.curiousconcept.com/ to format the JSON
fixtures. Add the fixture to the example subsection of the
responses section for the endpoint in
zerver/openapi/zulip.yaml.
Run ./tools/test-api to make sure your new test function is being
run and the tests pass.
Now, inside the function, isolate the lines of code that call the API and could
be displayed as a code example. Wrap the relevant lines in
# {code_example|start} ... relevant lines go here ... # {code_example|end}
comments. The lines inside these comments are what will be displayed as the
code example on our /api page.
Finally, if the API docs page of the endpoint doesn't follow the
common API docs template in
api_docs/api-doc-template.md, then add its custom
Markdown file under api_docs/. However, it is a goal
to minimize the number of files that diverse from the common
template, so only do this if there's a good reason.
Add the endpoint to the index in
api_docs/include/rest-endpoints.md. The URL should
match the operationId for the endpoint, and the link text should
match the title of the endpoint from the OpenAPI summary field.
Test your endpoint, pretending to be a new user in a hurry, by
visiting it via the links on http://localhost:9991/api/ (the API
docs are rendered from the Markdown source files on page load, so
just reload to see an updated version as you edit). You should
make sure that copy-pasting the code in your examples works, and
post an example of the output in the pull request.
Run ./tools/create-api-changelog which will create a new empty
markdown file in api_docs/unmerged.d/ directory
(e.g., api_docs/unmerged.d/ZF-1f4a39.md). Open this
file and document the API changes, formatted as an unordered list
(* bullets). The raw content of this file will be merged into
api_docs/changelog.md when your commit is merged into the main
branch. Therefore, keep the formatting and line wrapping consistent
with the content in api_docs/changelog.md.
Add a **Changes** note in the description of all updates and
additions to the API documentation (zerver/openapi/zulip.yaml),
and mention the name of the file generated in the previous step
(without the .md extension) in place of the API feature level,
for example:
**Changes**: New in Zulip 11.0 (feature level ZF-1f4a39).
Proofread your new documentation in its rendered HTML, including
all links! Unmerged changelog entries are conveniently previewed on
/api/changelog.
From time to time, we might want to rename an article in the REST API documentation. This change will break incoming links, including links in published Zulip blog posts, links in other branches of the repository that haven't been rebased, and more importantly links from previous versions of Zulip.
To fix these broken links, you can easily add a URL redirect in:
zerver/lib/url_redirects.py.
For REST API documentation, you will either need to rename the file,
or you will need to update the endpoint's operationId in
zerver/openapi/zulip.yaml. Then, you need to add a new URLRedirect
to the API_DOCUMENTATION_REDIRECTS list in url_redirects.py:
API_DOCUMENTATION_REDIRECTS: List[URLRedirect] = [
# Add URL redirects for REST API documentation here:
URLRedirect("/api/delete-stream", "/api/archive-stream"),
...
You should still check for references to the old URL in your branch
and replace those with the new URL (e.g., git grep "/api/foo").
One exception to this are links with the old URL that were included
in the content of zulip_update_announcements, which can be found
in zerver/lib/zulip_update_announcements.py. It's preferable to
have the source code accurately reflect what was sent to users in
those Zulip update announcements, so these should
not be replaced with the new URL.
If you have the Zulip development environment set up, you can manually
test your changes by loading the old URL in your browser (e.g.,
http://localhost:9991/api/foo), and confirming that it redirects to
the new url (e.g., http://localhost:9991/api/bar`).
There is also an automated test in zerver/tests/test_urls.py that
checks all the URL redirects, which you can run from the command line:
./tools/test-backend zerver.tests.test_urls.URLRedirectTest
Given that our documentation is written in large part using the OpenAPI format, why maintain a custom Markdown system for displaying it? There's several major benefits to this system:
Using the standard OpenAPI format gives us flexibility, though; if we later choose to migrate to third-party tools, we don't need to redo the actual documentation work in order to migrate tools.
A common function used to validate and test Zulip's REST API is
validate_against_openapi_schema. It is used to verify that every
successful API response returned in the backend and documentation test
suites are a documented possibility in the API documentation.
Therefore, when you add a new feature or setting to Zulip, you will most
likely need to update the API documentation (zerver/openapi/zulip.yaml)
in order to pass existing tests that use this function. Additionally, if
you're writing documentation for a new or undocumented REST API endpoint,
you'll want to use this function to validate and test your changes in
zerver/openapi/python_examples.py.
Below are some examples to help you when debugging the schema validation
errors produced by validate_against_openapi_schema. Before reading
through the examples, we recommend reviewing the
OpenAPI configuration documentation if you're unfamiliar
with the format.
If you use Visual Studio Code, an OpenAPI extension can be very helpful in
navigating Zulip's large and detailed OpenAPI file; see
.vscode/extensions.json.
To start with a clear example, let's imagine that we are writing the documentation for the REST API endpoint for uploading a file, POST /api/v1/user_uploads.
There are no parameters for this endpoint, and only one return value
specific to this endpoint, uri, which is the URL of the uploaded file.
If we comment out that return value and example from the existing API
documentation in zerver/openapi/zulip.yaml, for example:
/user_uploads:
post:
operationId: upload-file
...
responses:
"200":
description: Success.
content:
application/json:
schema:
allOf:
- $ref: "#/components/schemas/JsonSuccessBase"
- additionalProperties: false
properties:
result: {}
msg: {}
# uri:
# type: string
# description: |
# The URI of the uploaded file.
example:
{
"msg": "",
"result": "success",
# "uri": "/user_uploads/1/4e/m2A3MSqFnWRLUf9SaPzQ0Up_/zulip.txt",
}
We will now get an error when we run the API documentation test suite
in the development environment (tools/test-api):
Running API tests...
2022-12-19 15:05:42.347 WARN [django.server] "POST /api/v1/users HTTP/1.1" 400 88
Waiting for children to stop...
Traceback (most recent call last):
File "tools/test-api", line 93, in <module>
test_the_api(client, nonadmin_client, owner_client)
File "/srv/zulip/zerver/openapi/python_examples.py", line 1636, in test_the_api
test_users(client, owner_client)
File "/srv/zulip/zerver/openapi/python_examples.py", line 1555, in test_users
upload_file(client)
File "/srv/zulip/zerver/openapi/python_examples.py", line 52, in _record_calls_wrapper
return test_func(*args, **kwargs)
File "/srv/zulip/zerver/openapi/python_examples.py", line 1284, in upload_file
validate_against_openapi_schema(result, "/user_uploads", "post", "200")
File "/srv/zulip/zerver/openapi/openapi.py", line 489, in validate_against_openapi_schema
raise SchemaError(message) from None
zerver.openapi.openapi.SchemaError: 1 response validation error(s) at post /api/v1/user_uploads (200):
ValidationError: Additional properties are not allowed ('uri' was unexpected)
Failed validating 'additionalProperties' in schema['allOf'][2]:
{'additionalProperties': False,
'example': {'msg': '',
'result': 'success',
'properties': {'msg': {}, 'result': {}}}
On instance:
{'msg': '',
'result': 'success',
'uri': '/user_uploads/2/85/XoqF0K7XEOLVGylgdpof80RB/img.jpg'}
We can see in the traceback that a SchemaError was raised in
validate_against_openapi_schema:
File "/srv/zulip/zerver/openapi/openapi.py", line 478, in validate_against_openapi_schema
raise SchemaError(message) from None
The next line in the output, lets us know how many errors were found and for what endpoint.
zerver.openapi.openapi.SchemaError: 1 response validation error(s) at post /api/v1/user_uploads (200):
As expected from commenting out the code above, there was one validation
error for the POST /api/v1/user_uploads endpoint. The next line gives
more information about that error.
ValidationError: Additional properties are not allowed ('uri' was unexpected)
We see that there was a uri value returned by the endpoint that hasn't
been documented. The next few lines of output, show us what return values
are documented (again due to our changes) for this endpoint.
Failed validating 'additionalProperties' in schema['allOf'][2]:
{'additionalProperties': False,
'example': {'msg': '',
'result': 'success',
'properties': {'msg': {}, 'result': {}}}
And finally, we see the test instance that did not match our current
documentation, which includes the uri return value.
On instance:
{'msg': '',
'result': 'success',
'uri': '/user_uploads/2/85/XoqF0K7XEOLVGylgdpof80RB/img.jpg'}
This is a useful example because the endpoint's documentation is short and straightforward, helping to easily identify the parts of the error output that are useful in debugging these errors when testing the API documentation.
Building on the new feature tutorial
example, if the realm setting for mandatory_topics was not documented
in the POST /api/v1/register endpoint, running tools/test-api in the
development environment would result in this error:
...
zerver.openapi.openapi.SchemaError: 1 response validation error(s) at post /api/v1/register (200):
ValidationError: Additional properties are not allowed ('realm_mandatory_topics' was unexpected)
Failed validating 'additionalProperties' in schema['allOf'][2]:
'OpenAPI schema omitted due to length of output.'
On instance:
'Error instance omitted due to length of output.'
Because this endpoint is very long and descriptive, we do not print the entire documentation schema (or test instance, in this case) to the console. Doing so would print thousands of lines of output that are not useful for debugging what is missing from the API documentation.
The key information for debugging this endpoint is in the line beginning
with ValidationError. There we can see that the documentation does not
include the new realm_mandatory_topics boolean that we added in the
example feature tutorial, and we can look at other similar realm settings
to add the documentation for that new feature.