Deploy a self-hosted large language model

When you deploy a self-hosted model, you can:

• Manage the end-to-end transmission of requests to enterprise-hosted large language model (LLM) backends for GitLab Duo features.
• Keep all of these requests within that enterprise network, ensuring no calls to external architecture.
• Isolate the GitLab instance, AI Gateway, and self-hosted model within their own environment, ensuring complete privacy and high security for using AI features, with no reliance on public services.

When you use self-hosted models, you:

• Can choose any GitLab-approved LLM.
• Can keep all data and request/response logs in your own domain.
• Can select specific GitLab Duo features for your users.
• Do not have to rely on the GitLab shared AI Gateway.

You can connect supported models to LLM features. Model-specific prompts and GitLab Duo feature support is provided by the GitLab Duo Self-Hosted Models feature. For more information about this offering, see subscriptions and the Blueprint.

Prerequisites

• You must be able to manage your own LLM infrastructure.
• You must have GitLab Enterprise Edition.

Deploy a self-hosted model

To deploy a self-hosted large language model:

1. Set up your self-hosted model infrastructure and connect it to your GitLab instance.
2. Configure your GitLab instance to access self-hosted models using instance and group settings.

Self-hosted models compared to the default GitLab AI vendor architecture