Documentation testing
GitLab documentation is stored in projects with code, and treated like code. To maintain standards and quality of documentation, we use processes similar to those used for code.
Merge requests containing changes to Markdown (.md
) files run these CI/CD jobs:
-
docs-lint markdown
: Tests the documentation content with Vale, the Markdown structure with markdownlint, and other tests inlint-docs.sh
. -
docs-lint links
: Checks the validity of internal links in the documentation suite. -
mermaidlint
: Checks for invalid Mermaid charts in the documentation. -
ui-docs-links lint
: Checks the validity of links from UI elements, such as files inapp/views
files.
Tests in lint-doc.sh
The tests in
/scripts/lint-doc.sh
look for page content problems that Vale and markdownlint cannot test for.
The docs-lint markdown
job fails if any of these lint-doc.sh
tests fail:
- Curl (
curl
) commands must use long-form options (--header
) instead of short options, like-h
. - Documentation pages must contain front matter indicating ownership of the page.
- Non-standard Unicode space characters (NBSP, NNBSP, ZWSP) must not be used in documentation, because they can cause irregularities in search indexing and grepping.
-
CHANGELOG.md
must not contain duplicate versions. - No files in the
doc/
directory may be executable. - Use
index.md
instead ofREADME.md
. - Directories and filenames must use underscores instead of dashes.
- Directories and filenames must be in lower case.
Tests in docs-lint links
Merge requests containing changes to Markdown (.md
) files run a docs-lint links
job, which runs two types of link checks. In both cases, links with destinations
that begin with http
or https
are considered external links, and skipped:
-
bundle exec nanoc check internal_links
: Tests links to internal pages. -
bundle exec nanoc check internal_anchors
: Tests links to topic title anchors on internal pages.
Failures from these tests are displayed at the end of the test results in the Issues found! area.
For example, failures in the internal_anchors
test follow this format:
[ ERROR ] internal_anchors - Broken anchor detected!
- source file `/tmp/gitlab-docs/public/ee/user/application_security/api_fuzzing/index.html`
- destination `/tmp/gitlab-docs/public/ee/development/code_review.html`
- link `../../../development/code_review.html#review-response-slo`
- anchor `#review-response-slo`
-
Source file: The full path to the file containing the error. To find the
file in the
gitlab
repository, replace/tmp/gitlab-docs/public/ee
withdoc
, and.html
with.md
. -
Destination: The full path to the file not found by the test. To find the
file in the
gitlab
repository, replace/tmp/gitlab-docs/public/ee
withdoc
, and.html
with.md
. - Link: The actual link the script attempted to find.
- Anchor: If present, the topic title anchor the script attempted to find.
Check for multiple instances of the same broken link on each page reporting an error. Even if a specific broken link appears multiple times on a page, the test reports it only once.
Tests in mermaidlint
- Introduced in GitLab 16.10.
Mermaid builds charts and diagrams from code.
The mermaidlint
job runs on merge requests that contain changes to Markdown files.
The script (scripts/lint/check_mermaid.mjs
) returns an error if any Markdown
files return a Mermaid syntax error.
To help debug your Mermaid charts, use the Mermaid Live Editor.
Tests in ui-docs-links lint
The ui-docs-links lint
job uses haml-lint
to test that all documentation links from
UI elements (app/views
files, for example) link to valid pages and anchors.
Install documentation linters
To help adhere to the documentation style guidelines, and improve the content added to documentation, install documentation linters and integrate them with your code editor. At a minimum, install markdownlint and Vale to match the checks run in build pipelines. Both tools can integrate with your code editor.
Run documentation tests locally
Similar to previewing your changes locally, you can also run documentation tests on your local computer. This has the advantage of:
- Speeding up the feedback loop. You can know of any problems with the changes in your branch without waiting for a CI/CD pipeline to run.
- Lowering costs. Running tests locally is cheaper than running tests on the cloud infrastructure GitLab uses.
It’s important to:
- Keep the tools up-to-date, and match the versions used in our CI/CD pipelines.
- Run linters, documentation link tests, and UI link tests the same way they are run in CI/CD pipelines. It’s important to use same configuration we use in CI/CD pipelines, which can be different than the default configuration of the tool.
Run Vale or markdownlint locally
Installation and configuration instructions for markdownlint and Vale are available.
Run lint-doc.sh
locally
Use a Rake task to run the lint-doc.sh
tests locally.
Prerequisites:
- You have either:
- The required lint tools installed on your computer.
- A working Docker or
containerd
installation, to use an image with these tools pre-installed.
- Go to your
gitlab
directory. -
Run:
rake lint:markdown
To specify a single file or directory you would like to run lint checks for, run:
MD_DOC_PATH=path/to/my_doc.md rake lint:markdown
The output should be similar to:
=> Linting documents at path /path/to/gitlab as <user>...
=> Checking for cURL short options...
=> Checking for CHANGELOG.md duplicate entries...
=> Checking /path/to/gitlab/doc for executable permissions...
=> Checking for new README.md files...
=> Linting markdown style...
=> Linting prose...
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.
✔ Linting passed
Run documentation link tests locally
To test links in the documentation locally:
- Go to the
gitlab-docs
directory. -
Run the following commands:
# Check for broken internal links bundle exec nanoc check internal_links # Check for broken external links (might take a lot of time to complete). # This test is allowed to fail, and is run only in the gitlab-docs project CI bundle exec nanoc check internal_anchors
Run UI link tests locally
To test documentation links in the GitLab UI locally:
- Open the
gitlab
directory in a terminal window. -
Run:
bundle exec haml-lint -i DocumentationLinks
If you receive an error the first time you run this test, run bundle install
, which
installs the dependencies for GitLab, and try again.
If you don’t want to install all of the dependencies to test the links, you can:
- Open the
gitlab
directory in a terminal window. -
Install
haml-lint
:gem install haml_lint
-
Run:
haml-lint -i DocumentationLinks
If you manually install haml-lint
with this process, it does not update automatically
and you should make sure your version matches the version used by GitLab.
Update linter configuration
Vale and markdownlint configurations are under source control in each project, so updates must be committed to each project individually.
The configuration in the gitlab
project should be treated as the source of truth,
and all updates should first be made there.
On a regular basis, the changes made in gitlab
project to the Vale and markdownlint configuration should be
synchronized to the other projects. In each of the supported projects:
- Create a new branch.
- Copy the configuration files from the
gitlab
project into this branch, overwriting the project’s old configuration. Make sure no project-specific changes from thegitlab
project are included. For example,RelativeLinks.yml
is hard coded for specific projects. - Create a merge request and submit it to a technical writer for review and merge.
Update linting images
Lint tests run in CI/CD pipelines using images from the
gitlab-docs
container registry.
If a new version of a dependency is released (like a new version of Ruby), we should update the images to use the newer version. Then, we can update the configuration files in each of our documentation projects to point to the new image.
To update the linting images:
- In
gitlab-docs
, open a merge request to update.gitlab-ci.yml
to use the new tooling version. (Example MR) - When merged, start a
Build docs.gitlab.com every hour
scheduled pipeline. - Go the pipeline you started, and manually run the relevant build-images job,
for example,
image:docs-lint-markdown
. - In the job output, get the name of the new image. (Example job output)
- Verify that the new image was added to the container registry.
- Open merge requests to update each of these configuration files to point to the new image.
In each merge request, include a small doc update to trigger the job that uses the image.
- https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml (Example MR)
- https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/.gitlab/ci/test.gitlab-ci.yml (Example MR)
- https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/gitlab-ci-config/gitlab-com.yml (Example MR)
- https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/.gitlab-ci.yml (Example MR)
- https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/.gitlab-ci.yml (Example MR)
- https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/.gitlab/ci/test.gitlab-ci.yml (Example MR)
- In each merge request, check the relevant job output to confirm the updated image was used for the test. (Example job output)
- Assign the merge requests to any technical writer to review and merge.
Configure pre-push hooks
Git pre-push hooks allow Git users to:
- Run tests or other processes before pushing a branch.
- Avoid pushing a branch if failures occur with these tests.
lefthook
is a Git hooks manager. It makes configuring,
installing, and removing Git hooks simpler. Configuration for it is available in the
lefthook.yml
file for the gitlab
project.
To set up lefthook
for documentation linting, see
Pre-push static analysis.
To show Vale errors on push, see Show Vale warnings on push.
Disable linting on documentation
Some, but not all, linting can be disabled on documentation files:
- Vale tests can be disabled for all or part of a file.
-
markdownlint
tests can be disabled for all or part of a file. - Documentation link tests cannot be disabled.
- UI link tests cannot be disabled.
Tool versions used in CI/CD pipelines
It’s preferable to use linter versions that are the same as those used in our CI/CD pipelines for maximum compatibility with the linting rules we use.
To match the versions of markdownlint-cli
(or markdownlint-cli2
) and vale
used in the GitLab projects, refer to:
- For projects managed with
asdf
, the.tool-versions
file in the project. For example, the.tool-versions
file in thegitlab
project. - The versions used (see
variables:
section) when building theimage:docs-lint-markdown
Docker image containing these tools for CI/CD.
Versions set in these two locations should be the same.
Tool | Version | Command | Additional information |
---|---|---|---|
markdownlint-cli
| Latest | yarn global add markdownlint-cli
| None. |
markdownlint-cli2
| Latest | yarn global add markdownlint-cli2
| None. |
markdownlint-cli
| Specific | yarn global add markdownlint-cli@0.35.0
| The @ indicates a specific version, and this example updates the tool to version 0.35.0 .
|
markdownlint-cli2
| Specific | yarn global add markdownlint-cli2@0.8.1
| The @ indicates a specific version, and this example updates the tool to version 0.8.1 .
|
Vale (using asdf )
| Specific | asdf install
| Installs the version of Vale set in .tool-versions file in a project.
|
Vale (other) | Specific | Not applicable. | Binaries can be directly downloaded. |
Vale (using brew )
| Latest | brew update && brew upgrade vale
| This command is for macOS only. |
Supported projects
For the specifics of each test run in our CI/CD pipelines, see the configuration for those tests in the relevant projects:
- https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml
- https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/.gitlab/ci/docs.gitlab-ci.yml
- https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/gitlab-ci-config/gitlab-com.yml
- https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/.gitlab-ci.yml
- https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/.gitlab-ci.yml
We also run some documentation tests in the:
- GitLab CLI project: https://gitlab.com/gitlab-org/cli/-/blob/main/.gitlab-ci.yml
- GitLab Development Kit project: https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/.gitlab/ci/test.gitlab-ci.yml.
- Gitaly project: https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab-ci.yml.
- GitLab Duo Plugin for JetBrains: https://gitlab.com/gitlab-org/editor-extensions/gitlab-jetbrains-plugin/-/blob/main/.gitlab-ci.yml
- GitLab VS Code Extension project: https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/.gitlab-ci.yml.
- GitLab Plugin for Neovim project: https://gitlab.com/gitlab-org/editor-extensions/gitlab.vim/-/blob/main/.gitlab-ci.yml.
- GitLab Language Server project: https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/blob/main/.gitlab-ci.yml.
- GitLab Extension for Visual Studio project: https://gitlab.com/gitlab-org/editor-extensions/gitlab-visual-studio-extension/-/blob/main/.gitlab-ci.yml.