Large Language Models

The spacy-llm package integrates Large Language Models (LLMs) into spaCy pipelines, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks, no training data required.

• Serializable llm component to integrate prompts into your pipeline
• Modular functions to define the task (prompting and parsing) and model (model to use)
• Support for hosted APIs and self-hosted open-source models
• Integration with LangChain
• Access to OpenAI API, including GPT-4 and various GPT-3 models
• Built-in support for various open-source models hosted on Hugging Face
• Usage examples for standard NLP tasks such as Named Entity Recognition and Text Classification
• Easy implementation of your own functions via the registry for custom prompting, parsing and model integrations

Motivation {id="motivation"}

Large Language Models (LLMs) feature powerful natural language understanding capabilities. With only a few (and sometimes no) examples, an LLM can be prompted to perform custom NLP tasks such as text categorization, named entity recognition, coreference resolution, information extraction and more.

Supervised learning is much worse than LLM prompting for prototyping, but for many tasks it's much better for production. A transformer model that runs comfortably on a single GPU is extremely powerful, and it's likely to be a better choice for any task for which you have a well-defined output. You train the model with anything from a few hundred to a few thousand labelled examples, and it will learn to do exactly that. Efficiency, reliability and control are all better with supervised learning, and accuracy will generally be higher than LLM prompting as well.

spacy-llm lets you have the best of both worlds. You can quickly initialize a pipeline with components powered by LLM prompts, and freely mix in components powered by other approaches. As your project progresses, you can look at replacing some or all of the LLM-powered components as you require.

Of course, there can be components in your system for which the power of an LLM is fully justified. If you want a system that can synthesize information from multiple documents in subtle ways and generate a nuanced summary for you, bigger is better. However, even if your production system needs an LLM for some of the task, that doesn't mean you need an LLM for all of it. Maybe you want to use a cheap text classification model to help you find the texts to summarize, or maybe you want to add a rule-based system to sanity check the output of the summary. These before-and-after tasks are much easier with a mature and well-thought-out library, which is exactly what spaCy provides.

Install {id="install"}

spacy-llm will be installed automatically in future spaCy versions. For now, you can run the following in the same virtual environment where you already have spacy installed.

⚠️ This package is still experimental and it is possible that changes made to the interface will be breaking in minor version updates.

python -m pip install spacy-llm

Usage {id="usage"}

The task and the model have to be supplied to the llm pipeline component using the config system. This package provides various built-in functionality, as detailed in the API documentation.

Example 1: Add a text classifier using a GPT-3 model from OpenAI {id="example-1"}

Create a new API key from openai.com or fetch an existing one, and ensure the keys are set as environmental variables. For more background information, see the OpenAI section.

Create a config file config.cfg containing at least the following (or see the full example here):

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["COMPLIMENT", "INSULT"]

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
config = {"temperature": 0.0}

Now run:

from spacy_llm.util import assemble

nlp = assemble("config.cfg")
doc = nlp("You look gorgeous!")
print(doc.cats)

Example 2: Add NER using an open-source model through Hugging Face {id="example-2"}

To run this example, ensure that you have a GPU enabled, and transformers, torch and CUDA installed. For more background information, see the DollyHF section.

Create a config file config.cfg containing at least the following (or see the full example here):

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v3"
labels = ["PERSON", "ORGANISATION", "LOCATION"]

[components.llm.model]
@llm_models = "spacy.Dolly.v1"
# For better performance, use dolly-v2-12b instead
name = "dolly-v2-3b"

Now run:

from spacy_llm.util import assemble

nlp = assemble("config.cfg")
doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
print([(ent.text, ent.label_) for ent in doc.ents])

Note that Hugging Face will download the "databricks/dolly-v2-3b" model the first time you use it. You can define the cached directory by setting the environmental variable HF_HOME. Also, you can upgrade the model to be "databricks/dolly-v2-12b" for better performance.

Example 3: Create the component directly in Python {id="example-3"}

The llm component behaves as any other component does, and there are task-specific components defined to help you hit the ground running with a reasonable built-in task implementation.

import spacy

nlp = spacy.blank("en")
llm_ner = nlp.add_pipe("llm_ner")
llm_ner.add_label("PERSON")
llm_ner.add_label("LOCATION")
nlp.initialize()
doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
print([(ent.text, ent.label_) for ent in doc.ents])

Note that for efficient usage of resources, typically you would use nlp.pipe(docs) with a batch, instead of calling nlp(doc) with a single document.

Example 4: Implement your own custom task {id="example-4"}

To write a task, you need to implement two functions: generate_prompts that takes a list of Doc objects and transforms them into a list of prompts, and parse_responses that transforms the LLM outputs into annotations on the Doc, e.g. entity spans, text categories and more.

To register your custom task, decorate a factory function using the spacy_llm.registry.llm_tasks decorator with a custom name that you can refer to in your config.

📖 For more details, see the usage example on writing your own task

from typing import Iterable, List
from spacy.tokens import Doc
from spacy_llm.registry import registry
from spacy_llm.util import split_labels


@registry.llm_tasks("my_namespace.MyTask.v1")
def make_my_task(labels: str, my_other_config_val: float) -> "MyTask":
    labels_list = split_labels(labels)
    return MyTask(labels=labels_list, my_other_config_val=my_other_config_val)


class MyTask:
    def __init__(self, labels: List[str], my_other_config_val: float):
        ...

    def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]:
        ...

    def parse_responses(
        self, docs: Iterable[Doc], responses: Iterable[str]
    ) -> Iterable[Doc]:
        ...

# config.cfg (excerpt)
[components.llm.task]
@llm_tasks = "my_namespace.MyTask.v1"
labels = LABEL1,LABEL2,LABEL3
my_other_config_val = 0.3

Logging {id="logging"}

spacy-llm has a built-in logger that can log the prompt sent to the LLM as well as its raw response. This logger uses the debug level and by default has a logging.NullHandler() configured.

In order to use this logger, you can setup a simple handler like this:

import logging
import spacy_llm


spacy_llm.logger.addHandler(logging.StreamHandler())
spacy_llm.logger.setLevel(logging.DEBUG)

NOTE: Any logging handler will work here so you probably want to use some sort of rotating FileHandler as the generated prompts can be quite long, especially for tasks with few-shot examples.

Then when using the pipeline you'll be able to view the prompt and response.

E.g. with the config and code from Example 1 above:

from spacy_llm.util import assemble


nlp = assemble("config.cfg")
doc = nlp("You look gorgeous!")
print(doc.cats)

You will see logging output similar to:

Generated prompt for doc: You look gorgeous!

You are an expert Text Classification system. Your task is to accept Text as input
and provide a category for the text based on the predefined labels.

Classify the text below to any of the following labels: COMPLIMENT, INSULT
The task is non-exclusive, so you can provide more than one label as long as
they're comma-delimited. For example: Label1, Label2, Label3.
Do not put any other text in your answer, only one or more of the provided labels with nothing before or after.
If the text cannot be classified into any of the provided labels, answer `==NONE==`.

Here is the text that needs classification


Text:
'''
You look gorgeous!
'''

Model response for doc: You look gorgeous!
COMPLIMENT

print(doc.cats) to standard output should look like:

{'COMPLIMENT': 1.0, 'INSULT': 0.0}

API {id="api"}

spacy-llm exposes an llm factory with configurable settings.

An llm component is defined by two main settings:

• A task, defining the prompt to send to the LLM as well as the functionality to parse the resulting response back into structured fields on the Doc objects.
• A model defining the model to use and how to connect to it. Note that spacy-llm supports both access to external APIs (such as OpenAI) as well as access to self-hosted open-source LLMs (such as using Dolly through Hugging Face).

Moreover, spacy-llm exposes a customizable caching functionality to avoid running the same document through an LLM service (be it local or through a REST API) more than once.

Finally, you can choose to save a stringified version of LLM prompts/responses within the Doc.user_data["llm_io"] attribute by setting save_io to True. Doc.user_data["llm_io"] is a dictionary containing one entry for every LLM component within the nlp pipeline. Each entry is itself a dictionary, with two keys: prompt and response.

A note on validate_types: by default, spacy-llm checks whether the signatures of the model and task callables are consistent with each other and emits a warning if they don't. validate_types can be set to False if you want to disable this behavior.

Tasks {id="tasks"}

A task defines an NLP problem or question, that will be sent to the LLM via a prompt. Further, the task defines how to parse the LLM's responses back into structured information. All tasks are registered in the llm_tasks registry.

Practically speaking, a task should adhere to the Protocol named LLMTask defined in ty.py. It needs to define a generate_prompts function and a parse_responses function.

Tasks may support prompt sharding (for more info see the API docs on sharding and non-sharding tasks). The function signatures for generate_prompts and parse_responses depend on whether they do.

For tasks not supporting sharding:

For tasks supporting sharding:

Moreover, the task may define an optional scorer method. It should accept an iterable of Example objects as input and return a score dictionary. If the scorer method is defined, spacy-llm will call it to evaluate the component.

Providing examples for few-shot prompts {id="few-shot-prompts"}

All built-in tasks support few-shot prompts, i. e. including examples in a prompt. Examples can be supplied in two ways: (1) as a separate file containing only examples or (2) by initializing llm with a get_examples() callback (like any other pipeline component).

(1) Few-shot example file

A file containing examples for few-shot prompting can be configured like this:

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = PERSON,ORGANISATION,LOCATION
[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "ner_examples.yml"

The supplied file has to conform to the format expected by the required task (see the task documentation further down).

(2) Initializing the llm component with a get_examples() callback

Alternatively, you can initialize your nlp pipeline by providing a get_examples callback for nlp.initialize and setting n_prompt_examples to a positive number to automatically fetch a few examples for few-shot learning. Set n_prompt_examples to -1 to use all examples as part of the few-shot learning prompt.

[initialize.components.llm]
n_prompt_examples = 3

Model {id="models"}

A model defines which LLM model to query, and how to query it. It can be a simple function taking a collection of prompts (consistent with the output type of task.generate_prompts()) and returning a collection of responses (consistent with the expected input of parse_responses). Generally speaking, it's a function of type Callable[[Iterable[Any]], Iterable[Any]], but specific implementations can have other signatures, like Callable[[Iterable[str]], Iterable[str]].

All built-in models are registered in llm_models. If no model is specified, the repo currently connects to the OpenAI API by default using REST, and accesses the "gpt-3.5-turbo" model.

Currently three different approaches to use LLMs are supported:

1. spacy-llms native REST interface. This is the default for all hosted models (e. g. OpenAI, Cohere, Anthropic, ...).
2. A HuggingFace integration that allows to run a limited set of HF models locally.
3. A LangChain integration that allows to run any model supported by LangChain (hosted or locally).

Approaches 1. and 2 are the default for hosted model and local models, respectively. Alternatively you can use LangChain to access hosted or local models by specifying one of the models registered with the langchain. prefix.

Third-party libraries like langchain focus on prompt management, integration of many different LLM APIs, and other related features such as conversational memory or agents. spacy-llm on the other hand emphasizes features we consider useful in the context of NLP pipelines utilizing LLMs to process documents (mostly) independent from each other. It makes sense that the feature sets of such third-party libraries and spacy-llm aren't identical - and users might want to take advantage of features not available in spacy-llm.

The advantage of implementing our own REST and HuggingFace integrations is that we can ensure a larger degree of stability and robustness, as we can guarantee backwards-compatibility and more smoothly integrated error handling.

If however there are features or APIs not natively covered by spacy-llm, it's trivial to utilize LangChain to cover this - and easy to customize the prompting mechanism, if so required.

Note that the chat models variants of Llama 2 are currently not supported. This is because they need a particular prompting setup and don't add any discernible benefits in the use case of spacy-llm (i. e. no interactive chat) compared to the completion model variants.

Cache {id="cache"}

Interacting with LLMs, either through an external API or a local instance, is costly. Since developing an NLP pipeline generally means a lot of exploration and prototyping, spacy-llm implements a built-in cache to avoid reprocessing the same documents at each run that keeps batches of documents stored on disk.

Various functions {id="various-functions"}