ContextQMD
Back to Gitlabhq

Documentation testing

doc/development/documentation/testing/_index.md

18.11.224.0 KB
Original Source

GitLab documentation is stored in projects with code, and treated like code. To maintain standards and quality of documentation, we use processes similar to those used for code.

Merge requests containing changes to Markdown (.md) files run these CI/CD jobs:

A few files are generated from scripts. A CI/CD job fails when either the source code files or the documentation files are updated without following the correct process:

  • graphql-verify: Fails when doc/api/graphql/reference/_index.md is not updated with the update process.
  • docs-lint deprecations-and-removals: Fails when doc/update/deprecations.md is not updated with the update process.

For a full list of automated files, see Automated pages.

Tests in lint-doc.sh

The tests in /scripts/lint-doc.sh look for page content problems that Vale and markdownlint cannot test for. The docs-lint markdown job fails if any of these lint-doc.sh tests fail:

Mermaid chart linting

{{< history >}}

{{< /history >}}

Mermaid builds charts and diagrams from code.

The script (scripts/lint/check_mermaid.mjs) runs in the docs-lint mermaid job for all merge requests that contain changes to Markdown files. The script returns an error if any Markdown files return a Mermaid syntax error.

To help debug your Mermaid charts, use the Mermaid Live Editor.

To check for broken links, merge requests containing changes to Markdown (.md) files run these jobs in their pipelines:

These jobs check links, including anchor links, and report any problems. Any link that requires a network connection is skipped.

Tests for translated documentation

To ensure quality across all our translated content, we've implemented testing for our documentation in multiple languages. These tests mirror those used for the English version, but run on internationalized content in the /doc-locale/ or /docs-locale/ directories.

General translation linting

The docs-i18n-lint markdown job runs general linting tests on all translated documentation:

  • markdownlint: Checks Markdown structure and formatting.
  • Vale: Runs general documentation style rules using the gitlab_docs filter.

This job runs when any files in /doc-locale/ or /docs-locale/ are modified and provides baseline quality checks for all translated content.

ProjectEnglish DirTranslation DirLinting Jobs
GitLab/doc/doc-localedocs-i18n-lint markdown
docs-i18n-lint japanese-vale
GitLab Runner/docs/docs-localedocs:lint i18n markdown
Linux package/doc/doc-localedocs-lint-i18n markdown
docs-lint-i18n content
Charts/doc/doc-localecheck_docs_i18n_content
check_docs_i18n_markdown
Operator/doc/doc-localedocs-i18n-lint content
docs-i18n-lint markdown

Language-specific translation linting

For languages with specific style requirements, we provide dedicated CI/CD jobs that run language-specific Vale rules. Each language has its own job that only runs when files in that language are modified.

Japanese language support

The docs-i18n-lint japanese-vale job runs Japanese-specific Vale linting:

  • Script: scripts/i18n_lint_language_vale.sh
  • Configuration: doc-locale/ja-jp/.vale.ini
  • Vale filter: locale_rules
  • Triggers: Only runs when these files change:
    • doc-locale/ja-jp/**/*.md - Japanese documentation files
    • doc-locale/ja-jp/.vale.ini - Japanese Vale configuration
    • scripts/i18n_lint_language_vale.sh - Universal language linting script

The job uses environment variables to configure the universal script:

yaml
variables:
  LANGUAGE_CODE: "ja-jp"
  LANGUAGE_NAME: "Japanese"

Adding new languages

To add language-specific linting for additional languages:

  • Create language-specific Vale configuration at doc-locale/LANGUAGE_CODE/.vale.ini

  • Add a new CI job in .gitlab/ci/docs.gitlab-ci.yml:

    yaml
    docs-i18n-lint LANGUAGE-vale:
  extends:
    - .default-retry
    - .docs:rules:docs-i18n-lint-LANGUAGE
    - .docs-markdown-lint-image
  stage: lint
  needs: []
  variables:
    LANGUAGE_CODE: "LANGUAGE_CODE"     # e.g., "fr-fr", "de-de"
    LANGUAGE_NAME: "LANGUAGE_NAME"     # e.g., "French", "German"
  script:
    - source ./scripts/utils.sh
    - install_gitlab_gem
    - scripts/i18n_lint_language_vale.sh

  • Add triggering rules in .gitlab/ci/rules.gitlab-ci.yml:

    yaml
    .docs:rules:docs-i18n-lint-LANGUAGE:
  rules:
    - <<: *if-tech-docs-localization
      changes:
        - "doc-locale/LANGUAGE_CODE/**/*.md"
        - "doc-locale/LANGUAGE_CODE/.vale.ini"
        - "scripts/i18n_lint_language_vale.sh"

The universal script approach eliminates code duplication while maintaining language-specific customization through environment variables.

To check for broken links in translated documentation, merge requests containing changes to translated Markdown (.md) files run these jobs in their pipelines:

These jobs check for broken links, including anchor links. Any link that requires a network connection is skipped. If broken links are found, they are listed in the job log, but the jobs do not fail the pipeline. The jobs must be started manually, except for the docs:lint i18n markdown job in the Runner project.

Path verification of orphaned translation files

The docs-i18n-lint paths job fails if translated files in /doc-locale have no corresponding English source files. The job runs when:

  • Files in /doc-locale are modified
  • The path verification script changes

When orphaned translation files are detected, localization team members handle the necessary deletions. English fallback content provides coverage until new translations are available.

Install documentation linters

To help adhere to the documentation style guidelines, and improve the content added to documentation, install documentation linters and integrate them with your code editor. At a minimum, install markdownlint and Vale to match the checks run in build pipelines. Both tools can integrate with your code editor.

Running documentation tests locally

This has the advantage of:

  • Speeding up the feedback loop. You can know of any problems with the changes in your branch without waiting for a CI/CD pipeline to run.
  • Lowering costs. Running tests locally is cheaper than running tests on the cloud infrastructure GitLab uses.

It's important to:

  • Keep the tools up-to-date, and match the versions used in our CI/CD pipelines.
  • Run linters, documentation link tests, and UI link tests the same way they are run in CI/CD pipelines. It's important to use same configuration we use in CI/CD pipelines, which can be different than the default configuration of the tool.

Installation and configuration instructions are available for:

Run lint-doc.sh locally

Use a Rake task to run the lint-doc.sh tests locally.

Prerequisites:

  • You have either:
    • The required lint tools installed on your computer.
    • A working Docker or containerd installation, to use an image with these tools pre-installed.

  1. Go to your gitlab directory.

  2. Run:

    shell
    rake lint:markdown

To specify a single file or directory you would like to run lint checks for, run:

shell
MD_DOC_PATH=path/to/my_doc.md rake lint:markdown

The output should be similar to:

plaintext
=> Linting documents at path /path/to/gitlab as <user>...
=> Checking for cURL short options...
=> Checking for CHANGELOG.md duplicate entries...
=> Checking /path/to/gitlab/doc for executable permissions...
=> Checking for new README.md files...
=> Linting markdown style...
=> Linting prose...
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.
✔ Linting passed

Update linter configuration

Vale and markdownlint configurations are under source control in each project, so updates must be committed to each project individually.

The configuration in the gitlab project should be treated as the source of truth, and all updates should first be made there.

On a regular basis, the changes made in gitlab project to the Vale and markdownlint configuration should be synchronized to the other projects. In each of the supported projects:

  1. Create a new branch. Add docs- to the beginning or -docs to the end of the branch name. Some projects use this convention to limit the jobs that run.

  2. Copy the configuration files from the gitlab project. For example, in the root directory of the project, run:

    shell
    # Copy markdownlint configuration file
cp ../gitlab/.markdownlint-cli2.yaml .
# Remove existing Vale configuration in case some rules have been removed from the GitLab project
rm -r docs/.vale/gitlab
# Copy gitlab_base Vale configuration files for a project with documentation stored in 'docs' directory
cp -r ../gitlab/doc/.vale/gitlab_base docs/.vale

  3. If updating gitlab-runner, gitlab-omnibus, charts/gitlab, or gitlab-operator, also copy the gitlab-docs Vale configuration from the gitlab project. For example, in the root directory of the project, run:

    shell
    # Copy gitlab-docs Vale configuration files for a project with documentation stored in 'docs' directory
cp -r ../gitlab/doc/.vale/gitlab_docs docs/.vale

  4. Review the diff created for .markdownlint-cli2.yaml. For example, run:

    shell
    git diff .markdownlint-cli2.yaml

  5. Remove any changes that aren't required. For example, customRules is only used in the gitlab project.

  6. Review the diffs created for the Vale configuration. For example, run:

    shell
    git diff docs

  7. Remove unneeded changes to RelativeLinks.yml. This rule is specific to each project.

  8. Remove any .tmpl files. These files are only used in the gitlab project.

  9. Run markdownlint-cli2 to check for any violations of the new rules. For example:

    shell
    markdownlint-cli2 docs/**/*.md

  10. Run Vale to check for any violations of the new rules. For example:

    shell
    vale --minAlertLevel error docs

  11. Commit the changes to the new branch. Some projects require conventional commits so check the contributing information for the project before committing.

  12. Submit a merge request for review.

Update linting images

Lint tests run in CI/CD pipelines using images from the docs-gitlab-com container registry.

If a new version of a dependency is released (like a new version of Vale), we should update the images to use the newer version. Then, we can update the configuration files in each of our documentation projects to point to the new image.

To update the linting images:

  1. In docs-gitlab-com, open a merge request to update .gitlab-ci.yml to use the new tooling version. (Example MR)
  2. When merged, start a Build docker images pipeline (Manual) scheduled pipeline.
  3. Go the pipeline you started, and wait for the relevant test:image job to complete, for example test:image:docs-lint-markdown. If the job:
    • Passes, start the relevant image: job, for example, image:docs-lint-markdown.
    • Fails, review the test job log and start troubleshooting the issue. The image configuration likely needs some manual tweaks to work with the updated dependency.
  4. After the image: job passes, check the job's log for the name of the new image. (Example job output)
  5. Verify that the new image was added to the container registry.
  6. Open merge requests to update each of these configuration files to point to the new image. For jobs that use markdownlint, vale, or lychee:
  7. In each merge request:
    1. Include a small doc update to trigger the job that uses the image.
    2. Check the relevant job output to confirm the updated image was used for the test.
  8. Assign the merge requests to any technical writer to review and merge.

Configure pre-push hooks

Git pre-push hooks allow Git users to:

  • Run tests or other processes before pushing a branch.
  • Avoid pushing a branch if failures occur with these tests.

Lefthook is a Git hooks manager. It makes configuring, installing, and removing Git hooks simpler. Configuration for it is available in the lefthook.yml file for the gitlab project.

To set up Lefthook for documentation linting, see Pre-commit and pre-push static analysis with Lefthook.

To show Vale errors on commit or push, see Show Vale warnings on commit or push.

Disable linting on documentation

Some, but not all, linting can be disabled on documentation files:

Tool versions used in CI/CD pipelines

You should use linter versions that are the same as those used in our CI/CD pipelines for maximum compatibility with the linting rules we use.

To match the versions of markdownlint-cli2 and vale used in the GitLab projects, refer to:

Versions set in these two locations should be the same.

ToolVersionCommandAdditional information
markdownlint-cli2 (with yarn)Latestyarn global add markdownlint-cli2None.
markdownlint-cli2 (with mise)Latestmise use -g markdownlint-cli2Overridden by project-specific configuration.
markdownlint-cli2 (with yarn)Specificyarn global add [email protected]The @ indicates a specific version, and this example updates the tool to version 0.20.0.
markdownlint-cli2 (with mise)Specificmise installInstalls the version of markdownlint-cli2 set in the .tool-versions or mise.toml file in a project.
Vale (with mise)Specificmise installInstalls the version of Vale set in the .tool-versions or mise.toml file in a project.
Vale (with mise)Latestmise use -g valeOverridden by project-specific configuration.
Vale (other)SpecificNot applicable.Binaries can be directly downloaded.
Vale (with brew)Latestbrew update && brew upgrade valeThis command is for macOS only.

Supported projects

For the specifics of each test run in our CI/CD pipelines, see the configuration for those tests in the relevant projects:

We also run some documentation tests in these projects: