Replicate (LLM) - Mindsdb

This handler was implemented using the replicate library that is provided by Replicate.

The required arguments to establish a connection are,

model_name: Model name which you want to access in MindsDB. e.g 'air-forever/kandinsky-2'
version: version hash/id which you want to use in MindsDB.
api_key: API key from Replicate Platform you can found here.
model_type: It should be set to 'LLM' while using Large language Model else it Optional

<Note>

Before you can use Replicate, it's essential to authenticate by setting your API token in an environment variable named REPLICATE_API_TOKEN. This token acts as a key to enable access to Replicate's features.

Using pip:

If you're working in a standard Python environment (using pip for package management), set your token as an environment variable by running the following command in your terminal:

On Linux, Mac:

export REPLICATE_API_TOKEN='YOUR_TOKEN'

On Windows:

set REPLICATE_API_TOKEN=YOUR_TOKEN

Using Docker:

For Docker users, the process slightly differs. You need to pass the environment variable directly to the Docker container when running it. Use this command:

docker run -e REPLICATE_API_TOKEN='YOUR_TOKEN' -p 47334:47334 -p 47335:47335 mindsdb/mindsdb

Again, replace 'YOUR_TOKEN' with your actual Replicate API token. </Note>

Usage

To use this handler and connect to a Replicate cluster in MindsDB, you need an account on Replicate. Make sure to create an account by following this link.

To establish the connection and create a model in MindsDB, use the following syntax:

sql

CREATE MODEL vicuna_13b
PREDICT output
USING
    engine = 'replicate',
    model_name= 'replicate/vicuna_13b',
    model_type='LLM',
    version ='6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b',
    api_key = 'r8_HEH............';

You can use the DESCRIBE PREDICTOR query to see the available parameters that you can specify to customize your predictions:

sql

DESCRIBE PREDICTOR mindsdb.vicuna_13b.features;

OUTPUT

sql

+--------------------+---------+---------+---------------------------------------------------------------------------------------------------------------------------------------+
| inputs             | type    | default | description                                                                                                                           |
+--------------------+---------+---------+---------------------------------------------------------------------------------------------------------------------------------------+
| seed               | integer | -1      | Seed for random number generator, for reproducibility                                                                                 |
| debug              | boolean | False   | provide debugging output in logs                                                                                                      |
| top_p              | number  | 1       | When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens                       |
| prompt             | string  | -       | Prompt to send to Llama.                                                                                                              |
| max_length         | integer | 500     | Maximum number of tokens to generate. A word is generally 2-3 tokens                                                                  |
| temperature        | number  | 0.75    | Adjusts randomness of outputs, greater than 1 is random and 0 is deterministic, 0.75 is a good starting value.                        |
| repetition_penalty | number  | 1       | Penalty for repeated words in generated text; 1 is no penalty, values greater than 1 discourage repetition, less than 1 encourage it. |
+--------------------+---------+---------+---------------------------------------------------------------------------------------------------------------------------------------+

Now, you can use the established connection to query your ML Model as follows:

sql

SELECT *
FROM vicuna_13b
WHERE prompt='Write a humourous  poem on Open Source'
USING
 max_length=200,
 temperature=0.75;

OUTPUT

sql

+--------------------------------------------------------------+----------------------------------------+
| output                                                       | prompt                                 |
+--------------------------------------------------------------+----------------------------------------+
| Opensource software, oh how we love thee                     | Write a humourous  poem on Open Source |  
| With bugs and glitches, oh so free                           |                                        |  
| You bring us laughter and joy each day                       |                                        |  
| And we'll never have to pay                                  |                                        |  
|                                                              |                                        |  
| The license is open, the code is there                       |                                        |  
| For all to see and share in cheer                            |                                        |  
| You bring us together, from far and wide                     |                                        |  
| To work on projects, side by side                            |                                        |  
|                                                              |                                        |  
| With open source, there's no end                             |                                        |  
| To the code we can bend                                      |                                        |  
| We can change it, mold it, make it our own                   |                                        |  
| And create something truly great, or really strange          |                                        |  
|                                                              |                                        |  
| So here's to open source, the future is bright               |                                        |  
| With code that's free, and with all of our might             |                                        |  
| We"ll make the future shine, with technology fine            |                                        |  
| And open source will always be our shining line              |                                        |  
+--------------------------------------------------------------+----------------------------------------+

Note: Replicate provides only a few free predictions, so choose your predictions wisely. Don't let the machines have all the fun, save some for yourself! 😉