Lemmatizer

Component for assigning base forms to tokens using rules based on part-of-speech tags, or lookup tables. Different Language subclasses can implement their own lemmatizer components via language-specific factories. The default data used is provided by the spacy-lookups-data extension package.

For a trainable lemmatizer, see EditTreeLemmatizer.

As of v3.0, the Lemmatizer is a standalone pipeline component that can be added to your pipeline, and not a hidden part of the vocab that runs behind the scenes. This makes it easier to customize how lemmas should be assigned in your pipeline.

If the lemmatization mode is set to "rule", which requires coarse-grained POS (Token.pos) to be assigned, make sure a Tagger, Morphologizer or another component assigning POS is available in the pipeline and runs before the lemmatizer.

Assigned Attributes {id="assigned-attributes"}

Lemmas generated by rules or predicted will be saved to Token.lemma.

Config and implementation

The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the config argument on nlp.add_pipe or in your config.cfg for training. For examples of the lookups data format used by the lookup and rule-based lemmatizers, see spacy-lookups-data.

Example

pythonCopy

config = {"mode": "rule"}
nlp.add_pipe("lemmatizer", config=config)

Many languages specify a default lemmatizer mode other than lookup if a better lemmatizer is available. The lemmatizer modes rule and pos_lookup require token.pos from a previous pipeline component (see example pipeline configurations in the pretrained pipeline design details) or rely on third-party libraries (pymorphy3).

%%GITHUB_SPACY/spacy/pipeline/lemmatizer.py

Lemmatizer.__init__ {id="init",tag="method"}

Example

pythonCopy

# Construction via add_pipe with default model
lemmatizer = nlp.add_pipe("lemmatizer")

# Construction via add_pipe with custom settings
config = {"mode": "rule", "overwrite": True}
lemmatizer = nlp.add_pipe("lemmatizer", config=config)

Create a new pipeline instance. In your application, you would normally use a shortcut for this and instantiate the component using its string name and nlp.add_pipe.

Lemmatizer.__call__ {id="call",tag="method"}

Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the nlp object is called on a text and all pipeline components are applied to the Doc in order.

Example

pythonCopy

doc = nlp("This is a sentence.")
lemmatizer = nlp.add_pipe("lemmatizer")
# This usually happens under the hood
processed = lemmatizer(doc)

Lemmatizer.pipe {id="pipe",tag="method"}

Apply the pipe to a stream of documents. This usually happens under the hood when the nlp object is called on a text and all pipeline components are applied to the Doc in order.

Example

pythonCopy

lemmatizer = nlp.add_pipe("lemmatizer")
for doc in lemmatizer.pipe(docs, batch_size=50):
    pass

Lemmatizer.initialize {id="initialize",tag="method"}

Initialize the lemmatizer and load any data resources. This method is typically called by Language.initialize and lets you customize arguments it receives via the [initialize.components] block in the config. The loading only happens during initialization, typically before training. At runtime, all data is loaded from disk.

Example

pythonCopy

lemmatizer = nlp.add_pipe("lemmatizer")
lemmatizer.initialize(lookups=lookups)

iniCopy

### config.cfg
[initialize.components.lemmatizer]

[initialize.components.lemmatizer.lookups]
@misc = "load_my_lookups.v1"

Lemmatizer.lookup_lemmatize {id="lookup_lemmatize",tag="method"}

Lemmatize a token using a lookup-based approach. If no lemma is found, the original string is returned.

Lemmatizer.rule_lemmatize {id="rule_lemmatize",tag="method"}

Lemmatize a token using a rule-based approach. Typically relies on POS tags.

Lemmatizer.is_base_form {id="is_base_form",tag="method"}

Check whether we're dealing with an uninflected paradigm, so we can avoid lemmatization entirely.

Lemmatizer.get_lookups_config {id="get_lookups_config",tag="classmethod"}

Returns the lookups configuration settings for a given mode for use in Lemmatizer.load_lookups.

Lemmatizer.to_disk {id="to_disk",tag="method"}

Serialize the pipe to disk.

Example

pythonCopy

lemmatizer = nlp.add_pipe("lemmatizer")
lemmatizer.to_disk("/path/to/lemmatizer")

Lemmatizer.from_disk {id="from_disk",tag="method"}

Load the pipe from disk. Modifies the object in place and returns it.

Example

pythonCopy

lemmatizer = nlp.add_pipe("lemmatizer")
lemmatizer.from_disk("/path/to/lemmatizer")

Lemmatizer.to_bytes {id="to_bytes",tag="method"}

Example

pythonCopy

lemmatizer = nlp.add_pipe("lemmatizer")
lemmatizer_bytes = lemmatizer.to_bytes()

Serialize the pipe to a bytestring.

Lemmatizer.from_bytes {id="from_bytes",tag="method"}

Load the pipe from a bytestring. Modifies the object in place and returns it.

Example

pythonCopy

lemmatizer_bytes = lemmatizer.to_bytes()
lemmatizer = nlp.add_pipe("lemmatizer")
lemmatizer.from_bytes(lemmatizer_bytes)

Attributes {id="attributes"}

Serialization fields {id="serialization-fields"}

During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from serialization by passing in the string names via the exclude argument.

Example

pythonCopy

data = lemmatizer.to_disk("/path", exclude=["vocab"])