doc/administration/gitlab_duo_self_hosted/troubleshooting.md
{{< details >}}
{{< /details >}}
{{< history >}}
ai_custom_model. Disabled by default.ai_custom_model removed in GitLab 17.8.{{< /history >}}
Before you begin troubleshooting, you should:
gitlab-rails console.llm.log.For more information on troubleshooting GitLab Duo, see:
We provide two debugging scripts to help administrators verify their self-hosted model configuration.
Debug the GitLab to AI Gateway connection. From your GitLab instance, run the Rake task:
gitlab-rake "gitlab:duo:verify_self_hosted_setup[<username>]"
Optional: Include a <username> that has an assigned seat.
If you do not include a username parameter, the Rake task uses the root user.
Debug the AI Gateway setup. For your AI Gateway container:
Restart the AI Gateway container with authentication disabled by setting:
-e AIGW_AUTH__BYPASS_EXTERNAL=true
This setting is required for the troubleshooting command to run the System Exchange test. You must remove this setting after troubleshooting is complete.
From your AI Gateway container, run:
docker exec -it <ai-gateway-container> sh
poetry run troubleshoot [options]
The troubleshoot command supports the following options:
| Option | Default | Example | Description |
|---|---|---|---|
--endpoint | localhost:5052 | --endpoint=localhost:5052 | AI Gateway endpoint |
--model-family | - | --model-family=mistral | Model family to test. Possible values are mistral, mixtral, gpt, or claude_3 |
--model-endpoint | - | --model-endpoint=http://localhost:4000/v1 | Model endpoint. For models hosted on vLLM, add the /v1 suffix. |
--model-identifier | - | --model-identifier=custom_openai/Mixtral-8x7B-Instruct-v0.1 | Model identifier. |
--api-key | - | --api-key=your-api-key | Model API key. |
Examples:
For a claude_3 model running on AWS Bedrock:
poetry run troubleshoot \
--model-family=claude_3 \
--model-identifier=bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
For a mixtral model running on vLLM:
poetry run troubleshoot \
--model-family=mixtral \
--model-identifier=custom_openai/Mixtral-8x7B-Instruct-v0.1 \
--api-key=your-api-key \
--model-endpoint=http://<your-model-endpoint>/v1
After troubleshooting is complete, stop and restart the AI Gateway container without AIGW_AUTH__BYPASS_EXTERNAL=true.
[!warning] You must not bypass authentication in production.
Verify the output of the commands, and fix accordingly.
If both commands are successful, but GitLab Duo Code Suggestions is still not working, raise an issue on the issue tracker.
When you run a health check for GitLab Duo, you might get an error like a 401 response from the AI Gateway.
To resolve, first check if GitLab Duo features are functioning correctly. For example, send a message to GitLab Duo Chat.
If this does not work, the error might be because of a known issue with GitLab Duo health check. For more information, see issue 517097.
From the GitLab Rails console, verify that GitLab can make a request to the model by running:
model_name = "<your_model_name>"
model_endpoint = "<your_model_endpoint>"
model_api_key = "<your_model_api_key>"
body = {:prompt_components=>[{:type=>"prompt", :metadata=>{:source=>"GitLab EE", :version=>"17.3.0"}, :payload=>{:content=>[{:role=>:user, :content=>"Hello"}], :provider=>:litellm, :model=>model_name, :model_endpoint=>model_endpoint, :model_api_key=>model_api_key}}]}
ai_gateway_url = Ai::Setting.instance.ai_gateway_url # Verify that the AI Gateway URL is set in the database
client = Gitlab::Llm::AiGateway::Client.new(User.find_by_id(1), unit_primitive_name: :self_hosted_models)
client.complete(url: "#{ai_gateway_url}/v1/chat/agent", body: body)
This should return a response from the model in the format:
{"response"=> "<Model response>",
"metadata"=>
{"provider"=>"litellm",
"model"=>"<>",
"timestamp"=>1723448920}}
If that is not the case, this might means one of the following:
In the GitLab Rails console, check if a user can request Code Suggestions by running:
User.find_by_id("<user_id>").can?(:access_code_suggestions)
If this returns false, it means some configuration is missing, and the user
cannot access Code Suggestions.
This missing configuration might be because of either of the following:
Prerequisites:
To check if GitLab Duo was configured correctly:
To check that the AI Gateway URL is correct, run the following on the GitLab Rails console:
Ai::Setting.instance.ai_gateway_url == "<your-ai-gateway-instance-url>"
If the AI Gateway is not set up, configure your GitLab instance to access the AI Gateway.
To check that the URL for the Agent Platform service is correct, run the following on the GitLab Rails console:
Ai::Setting.instance.duo_agent_platform_service_url == "<your-duo-agent-platform-instance-url>"
The URL for the Agent Platform service is a TCP URL and cannot have the prefixes http:// or https://.
If the URL for the Agent Platform has not been set up, you must configure your GitLab instance to access the URL.
In the GitLab Rails console, verify that GitLab can make an HTTP request to AI Gateway by running:
HTTParty.get('<your-aigateway-endpoint>/monitoring/healthz', headers: { 'accept' => 'application/json' }).code
If the response is not 200, this means either of the following:
From the AI Gateway container, make an HTTP request to the AI Gateway API for a Code Suggestion. Replace:
<your_model_name> with the name of the model you are using. For example mistral or codegemma.<your_model_endpoint> with the endpoint where the model is hosted.docker exec -it <ai-gateway-container> sh
curl --request POST "http://localhost:5052/v1/chat/agent" \
--header 'accept: application/json' \
--header 'Content-Type: application/json' \
--data '{ "prompt_components": [ { "type": "string", "metadata": { "source": "string", "version": "string" }, "payload": { "content": "Hello", "provider": "litellm", "model": "<your_model_name>", "model_endpoint": "<your_model_endpoint>" } } ], "stream": false }'
If the request fails, the:
docker exec -it <ai-gateway-container> sh
curl '<your-aigateway-endpoint>/monitoring/healthz'
If the response is not 200, this means that AI Gateway is not installed correctly. To resolve, follow the documentation on how to install the AI Gateway.
To check that the AI Gateway environment variables are set up correctly, run the following in a console on the AI Gateway container:
docker exec -it <ai-gateway-container> sh
echo $AIGW_CUSTOM_MODELS__ENABLED # must be true
If the environment variables are not set up correctly, set them by creating a container.
Create a shell on the AI Gateway container and make a curl request to the model. If you find that the AI Gateway cannot make that request, this might be caused by the:
To resolve this, contact your network administrator.
The GitLab instance defined in AIGW_GITLAB_URL must be accessible from the AI Gateway container for request authentication.
If the instance is not reachable (for example, because of proxy configuration errors), requests can fail with errors, such as the following:
jose.exceptions.JWTError: Signature verification failed
gitlab_cloud_connector.providers.CompositeProvider.CriticalAuthError: No keys founds in JWKS; are OIDC providers up?
In this scenario, verify if AIGW_GITLAB_URL and $AIGW_GITLAB_API_URL are properly set to the container and accessible.
The following commands should be successful when run from the container:
poetry run troubleshoot
curl "$AIGW_GITLAB_API_URL/projects"
If not successful, verify your network configurations.
When finding the AI Gateway release,
you might get an error that states The requested image's platform (linux/amd64) does not match the detected host.
To work around this error, add --platform linux/amd64 to the docker run command:
docker run --platform linux/amd64 -e AIGW_GITLAB_URL=<your-gitlab-endpoint> <image>
If the LLM server is installed on the same instance as the AI Gateway container, it may not be accessible through the local host.
To resolve this:
--network host in the docker run command to enable local requests from the AI Gateway container.-e AIGW_FASTAPI__METRICS_PORT=8083 flag to address the port conflicts.docker run --network host -e AIGW_GITLAB_URL=<your-gitlab-endpoint> -e AIGW_FASTAPI__METRICS_PORT=8083 <image>
If you encounter a 404 error while using vLLM, follow these steps to resolve the issue:
Create a chat template file named chat_template.jinja with the following content:
{%- for message in messages %}
{%- if message["role"] == "user" %}
{{- "[INST] " + message["content"] + "[/INST]" }}
{%- elif message["role"] == "assistant" %}
{{- message["content"] }}
{%- elif message["role"] == "system" %}
{{- bos_token }}{{- message["content"] }}
{%- endif %}
{%- endfor %}
When running the vLLM command, ensure you specify the --served-model-name. For example:
vllm serve "mistralai/Mistral-7B-Instruct-v0.3" --port <port> --max-model-len 17776 --served-model-name mistral --chat-template chat_template.jinja
Check the vLLM server URL in the GitLab UI to make sure that URL includes the /v1 suffix. The correct format is:
http(s)://<your-host>:<your-port>/v1
If you are experiencing issues accessing Code Suggestions after setup, try the following steps:
In the Rails console, check and verify the license parameters:
sudo gitlab-rails console
user = User.find(id) # Replace id with the user provisioned with GitLab Duo Enterprise seat
Ability.allowed?(user, :access_code_suggestions) # Must return true
Check if the necessary features are enabled and available:
::Ai::FeatureSetting.exists?(feature: [:code_generations, :code_completions], provider: :self_hosted) # Should be true
When using GitLab Duo features with self-hosted models, you might encounter the following error:
I'm sorry, I couldn't respond in time. Please try again. Error code: A1000
This issue occurs when the request to your model might be taking longer than the configured timeout period.
Common causes include:
To resolve timeout errors:
To verify your GitLab Self-Managed setup, run the following command:
gitlab-rake gitlab:duo:verify_self_hosted_setup
If no logs are generated in the AI Gateway server, follow these steps to troubleshoot:
Ensure that AI logs are enabled.
Run the following commands to view the GitLab Rails logs for any errors:
sudo gitlab-ctl tail
sudo gitlab-ctl tail sidekiq
Look for keywords like "Error" or "Exception" in the logs to identify any underlying issues.
When attempting to initiate a GitLab Duo Chat inside the AI Gateway container, SSL certificate errors and key deserialization issues may occur.
The system might encounter issues loading the PEM file, resulting in errors like:
JWKError: Could not deserialize key data. The data may be in an incorrect format, the provided password may be incorrect, or it may be encrypted with an unsupported algorithm.
To resolve the SSL certificate error:
SSL_CERT_FILE=/path/to/ca-bundle.pemREQUESTS_CA_BUNDLE=/path/to/ca-bundle.pemIn the AIGW logs, the following error displays when the format of the model identifier is incorrect:
Invocation of model ID meta.llama3-3-70b-instruct-v1:0 with on-demand throughput isn\u2019t supported. Retry your request with the ID or ARN of an inference profile that contains this model
Ensure your model identifier has the format bedrock/<region>.<model-id>, where:
<region> is your AWS region (such as us)<model-id> is the full model identifier.For example: bedrock/us.meta.llama3-3-70b-instruct-v1:0. Update your model configuration to use the correct format.
If a feature is not working or a feature button (for example, /troubleshoot) is not visible:
Check if the feature's unit_primitive is listed in the self-hosted models unit primitives list in the gitlab-cloud-connector gem configuration.
If the feature is missing from this file, that could be the reason it's not accessible.
Optional. If the feature is not listed, you can verify this is the cause of the issue by setting the following in your GitLab instance:
CLOUD_CONNECTOR_SELF_SIGN_TOKENS=1
Then restart GitLab and check if the feature becomes accessible.
Important: After troubleshooting, restart GitLab without this flag set.
[!warning] Do not use
CLOUD_CONNECTOR_SELF_SIGN_TOKENS=1in production. Development environments should closely mirror production, with no hidden flags or internal-only workarounds.
To resolve this issue:
#g_custom_models Slack channel.This error can occur when you try to use agentic Chat in GitLab or your local environment.
You might also see the following in the logs of your IDE's GitLab Language Server:
2026-01-09T20:17:43:419 [error]: [WorkflowRailsService] Failed to fetch the workflow token
Error: Fetching direct_access from https://gitlab.example.com/api/v4/ai/duo_workflows/direct_access failed.
{"message":"400 Bad request - 14:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.x.x.x:50052: Ssl handshake failed (TSI_PROTOCOL_FAILURE): SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER: Invalid certificate verification context. debug_error_string:{UNKNOWN:Error received from peer {grpc_status:14, grpc_message:\"failed to connect to all addresses; last error: UNKNOWN: ipv4:172.x.x.x:50052: Ssl handshake failed (TSI_PROTOCOL_FAILURE): SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER: Invalid certificate verification context\"}}"}
2026-01-09T20:17:43:433 [error]: Max retries exceeded or non-retryable error: An error occurred while fetching an authentication token for this workflow.
2026-01-09T20:17:43:435 [error]: Workflow failed with status code "50": An error occurred while fetching an authentication token for this workflow.
This means that the language server could not communicate with the direct_access endpoint
to generate a JWT token due to the certificate issue.
If you do not use TLS to connect your self-hosted model with the Agent Platform, to resolve this issue, turn off TLS connection to the GitLab Duo Agent Platform service.