IBM WatsonX - Continue

watsonx, developed by IBM, offers a variety of pre-trained AI foundation models that can be used for natural language processing (NLP), computer vision, and speech recognition tasks.

Setup

Accessing watsonx models can be done either through watsonx SaaS on IBM Cloud or using a dedicated watsonx.ai Software instance.

watsonx.ai SaaS - IBM Cloud

To get started with watsonx SaaS, visit the registration page. If you do not have an existing IBM Cloud account, you can sign up for a free trial.

To authenticate to watsonx.ai SaaS with Continue, you will need to create a project and set up an API key. Then, in continue:

Set apiBase to your watsonx SaaS endpoint, e.g. https://us-south.ml.cloud.ibm.com for US South region.
Set projectId to your watsonx project ID.
Set apiKey to your watsonx API Key.

watsonx.ai Software

To authenticate to your watsonx.ai Software instance with Continue, you can use either username/password or ZenApiKey method:

Option 1 (Recommended): using ZenApiKey authentication:
- Set apiBase to your watsonx software endpoint, e.g. https://cpd-watsonx.apps.example.com.
- Set projectId to your watsonx project ID.
- Set apiKey to your watsonx Zen API Key. To generate it:
  1. Log in to the CPD web client.
  2. From the toolbar, click your avatar.
  3. Click Profile and settings.
  4. Click API key > Generate new key.
  5. Click Generate.
  6. Click Copy and save your key somewhere safe. You cannot recover this key if you lose it.
  7. Generate your ZenApiKey by running the following command in your preferred terminal: echo "<username>:<apikey>" | base64, replacing <username> with your CPD username and <apikey> with the API Key you just created.
Option 2: using username/password authentication:
- Set apiBase to your watsonx software endpoint, e.g. https://cpd-watsonx.apps.example.com.
- Set projectId to your watsonx project ID.
- Set API Key to your watsonx Username and Password using username:password as format.

Configuration

Add the following configuration:

<Tabs> <Tab title="YAML"> ```yaml title="config.yaml" name: My Config version: 0.0.1 schema: v1

models: - name: watsonx - Model Name provider: watsonx model: model ID apiBase: https://us-south.ml.cloud.ibm.com apiKey: API_KEY/ZENAPI_KEY/USERNAME:PASSWORD env: projectId: PROJECT_ID apiVersion: 2024-03-14

  </Tab>
  <Tab title="JSON">
  ```json title="config.json"
{
  	"models": [
  	  {
  	    "model": "model ID",
  	    "title": "watsonx - Model Name",
  	    "provider": "watsonx",
  	    "apiBase": "https://us-south.ml.cloud.ibm.com",
  	    "projectId": "PROJECT_ID",
  	    "apiKey": "API_KEY/ZENAPI_KEY/USERNAME:PASSWORD",
  	    "apiVersion": "2024-03-14"
  	  }
  	]
}
  ```
  </Tab>
</Tabs>

`apiVersion` is optional and defaults to the latest version.

If you are using a custom deployment endpoint, set `deploymentID` to the model's deployment ID. You can find it in the watsonx.ai Prompt Lab UI by selecting the corresponding model and opening the `</>` tab on the right, which will display the endpoint's URL containing the deployment ID.

<Tabs>
  <Tab title="YAML">
  ```yaml title="config.yaml"
  name: My Config
  version: 0.0.1
  schema: v1

  models:
    - name: watsonx - Model Name
      provider: watsonx
      model: model ID
      apiBase: watsonx endpoint e.g. https://us-south.ml.cloud.ibm.com
      apiKey: API_KEY/ZENAPI_KEY/USERNAME:PASSWORD
      env:
        apiVersion: 2024-03-14
        deploymentId: DEPLOYMENT_ID
  ```
  </Tab>
  <Tab title="JSON">
  ```json title="config.json"
  {
    "models": [
      {
        "model": "model ID",
        "title": "watsonx - Model Name",
        "provider": "watsonx",
        "apiBase": "watsonx endpoint e.g. https://us-south.ml.cloud.ibm.com",
        "apiKey": "API_KEY/ZENAPI_KEY/USERNAME:PASSWORD",
        "apiVersion": "2024-03-14",
        "deploymentId": "DEPLOYMENT_ID"
      }
    ]
  }
  ```
  </Tab>
</Tabs>

### Configuration Options

Make sure to specify a template name, such as `granite` or `llama3`, and to set the `contextLength` to the model's context window size.
You can also configure generation parameters, such as temperature, topP, topK, frequency penalty, and stop sequences:

<Tabs>
<Tab title="YAML">
```yaml title="config.yaml"
name: My Config
version: 0.0.1
schema: v1

models:
  - name: Granite Code 20b
    provider: watsonx
    model: ibm/granite-20b-code-instruct
    apiBase: watsonx endpoint e.g. https://us-south.ml.cloud.ibm.com
    apiKey: API_KEY/ZENAPI_KEY/USERNAME:PASSWORD
    template: granite
    defaultCompletionOptions:
      contextLength: 8000
      temperature: 0.1
      topP: 0.3
      topK: 20
      maxTokens: 2000
      frequencyPenalty: 1.1
      stop:
        - Question:
        - "\n\n\n"
    env:
      projectId: PROJECT_ID
      apiVersion: 2024-03-14

</Tab> <Tab title="JSON"> ```json title="config.json" { "models": [ { "model": "ibm/granite-20b-code-instruct", "title": "Granite Code 20b", "provider": "watsonx", "apiBase": "watsonx endpoint e.g. https://us-south.ml.cloud.ibm.com", "projectId": "PROJECT_ID", "apiKey": "API_KEY/ZENAPI_KEY/USERNAME:PASSWORD", "apiVersion": "2024-03-14", "template": "granite", "contextLength": 8000, "completionOptions": { "temperature": 0.1, "topP": 0.3, "topK": 20, "maxTokens": 2000, "frequencyPenalty": 1.1, "stop": ["Question:", "\n\n\n"] } } ] } ``` </Tab> </Tabs>

Tab Auto Complete Model

Granite models are recommended for tab auto complete. The configuration is similar to that of the chat models:

<Tabs> <Tab title="YAML"> ```yaml title="config.yaml" name: My Config version: 0.0.1 schema: v1

models: - name: Granite Code 8b provider: watsonx model: ibm/granite-8b-code-instruct apiBase: watsonx endpoint e.g. https://us-south.ml.cloud.ibm.com projectId: PROJECT_ID apiKey: API_KEY/ZENAPI_KEY/USERNAME:PASSWORD apiVersion: 2024-03-14 roles: - autocomplete

  </Tab>
  <Tab title="JSON">
```json title="config.json"
{
  "tabAutocompleteModel": {
    "model": "ibm/granite-8b-code-instruct",
    "title": "Granite Code 8b",
    "provider": "watsonx",
    "apiBase": "watsonx endpoint e.g. https://us-south.ml.cloud.ibm.com",
    "projectId": "PROJECT_ID",
    "apiKey": "API_KEY/ZENAPI_KEY/USERNAME:PASSWORD",
    "apiVersion": "2024-03-14",
    "contextLength": 4000
  }
}

</Tab>

</Tabs>

Embeddings Model

To view the list of available embeddings models, visit this page.

<Tabs> <Tab title="YAML"> ```yaml title="config.yaml" name: My Config version: 0.0.1 schema: v1

models: - name: Watsonx Embedder provider: watsonx model: ibm/slate-30m-english-rtrvr-v2 apiBase: https://us-south.ml.cloud.ibm.com projectId: PROJECT_ID apiKey: API_KEY/ZENAPI_KEY/USERNAME:PASSWORD apiVersion: 2024-03-14 roles: - embed

  </Tab>
  <Tab title="JSON">
```json title="config.json"
{
  "embeddingsProvider": {
    "provider": "watsonx",
    "model": "ibm/slate-30m-english-rtrvr-v2",
    "apiBase": "watsonx endpoint e.g. https://us-south.ml.cloud.ibm.com",
    "projectId": "PROJECT_ID",
    "apiKey": "API_KEY/ZENAPI_KEY/USERNAME:PASSWORD",
    "apiVersion": "2024-03-14"
  }
}

</Tab>

</Tabs>

Reranker

<Tabs> <Tab title="YAML"> ```yaml title="config.yaml" name: My Config version: 0.0.1 schema: v1

models: - name: Watsonx Reranker provider: watsonx model: cross-encoder/ms-marco-minilm-l-12-v2 apiBase: https://us-south.ml.cloud.ibm.com projectId: PROJECT_ID apiKey: API_KEY/ZENAPI_KEY/USERNAME:PASSWORD apiVersion: 2024-03-14

  </Tab>
  <Tab title="JSON">
```json title="config.json"
{
  "reranker": {
    "name": "watsonx",
    "params": {
      "model": "cross-encoder/ms-marco-minilm-l-12-v2",
      "apiBase": "watsonx endpoint e.g. https://us-south.ml.cloud.ibm.com",
      "projectId": "PROJECT_ID",
      "apiKey": "API_KEY/ZENAPI_KEY/USERNAME:PASSWORD",
      "apiVersion": "2024-03-14"
    }
  }
}

</Tab>

</Tabs>