Sentencizer

A simple pipeline component to allow custom sentence boundary detection logic that doesn't require the dependency parse. By default, sentence segmentation is performed by the DependencyParser, so the Sentencizer lets you implement a simpler, rule-based strategy that doesn't require a statistical model to be loaded.

Assigned Attributes {id="assigned-attributes"}

Calculated values will be assigned to Token.is_sent_start. The resulting sentences can be accessed using Doc.sents.

Config and implementation {id="config"}

The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the config argument on nlp.add_pipe or in your config.cfg for training.

Example

pythonCopy

config = {"punct_chars": None}
nlp.add_pipe("sentencizer", config=config)

%%GITHUB_SPACY/spacy/pipeline/sentencizer.pyx

Sentencizer.__init__ {id="init",tag="method"}

Initialize the sentencizer.

Example

pythonCopy

# Construction via add_pipe
sentencizer = nlp.add_pipe("sentencizer")

# Construction from class
from spacy.pipeline import Sentencizer
sentencizer = Sentencizer()

['!', '.', '?', '։', '؟', '۔', '܀', '܁', '܂', '߹', '।', '॥', '၊', '။', '።',
 '፧', '፨', '᙮', '᜵', '᜶', '᠃', '᠉', '᥄', '᥅', '᪨', '᪩', '᪪', '᪫',
 '᭚', '᭛', '᭞', '᭟', '᰻', '᰼', '᱾', '᱿', '‼', '‽', '⁇', '⁈', '⁉',
 '⸮', '⸼', '꓿', '꘎', '꘏', '꛳', '꛷', '꡶', '꡷', '꣎', '꣏', '꤯', '꧈',
 '꧉', '꩝', '꩞', '꩟', '꫰', '꫱', '꯫', '﹒', '﹖', '﹗', '！', '．', '？',
 '𐩖', '𐩗', '𑁇', '𑁈', '𑂾', '𑂿', '𑃀', '𑃁', '𑅁', '𑅂', '𑅃', '𑇅',
 '𑇆', '𑇍', '𑇞', '𑇟', '𑈸', '𑈹', '𑈻', '𑈼', '𑊩', '𑑋', '𑑌', '𑗂',
 '𑗃', '𑗉', '𑗊', '𑗋', '𑗌', '𑗍', '𑗎', '𑗏', '𑗐', '𑗑', '𑗒', '𑗓',
 '𑗔', '𑗕', '𑗖', '𑗗', '𑙁', '𑙂', '𑜼', '𑜽', '𑜾', '𑩂', '𑩃', '𑪛',
 '𑪜', '𑱁', '𑱂', '𖩮', '𖩯', '𖫵', '𖬷', '𖬸', '𖭄', '𛲟', '𝪈', '｡', '。']

Sentencizer.__call__ {id="call",tag="method"}

Apply the sentencizer on a Doc. Typically, this happens automatically after the component has been added to the pipeline using nlp.add_pipe.

Example

pythonCopy

from spacy.lang.en import English

nlp = English()
nlp.add_pipe("sentencizer")
doc = nlp("This is a sentence. This is another sentence.")
assert len(list(doc.sents)) == 2

Sentencizer.pipe {id="pipe",tag="method"}

Apply the pipe to a stream of documents. This usually happens under the hood when the nlp object is called on a text and all pipeline components are applied to the Doc in order.

Example

pythonCopy

sentencizer = nlp.add_pipe("sentencizer")
for doc in sentencizer.pipe(docs, batch_size=50):
    pass

Sentencizer.to_disk {id="to_disk",tag="method"}

Save the sentencizer settings (punctuation characters) to a directory. Will create a file sentencizer.json. This also happens automatically when you save an nlp object with a sentencizer added to its pipeline.

Example

pythonCopy

config = {"punct_chars": [".", "?", "!", "。"]}
sentencizer = nlp.add_pipe("sentencizer", config=config)
sentencizer.to_disk("/path/to/sentencizer.json")

Sentencizer.from_disk {id="from_disk",tag="method"}

Load the sentencizer settings from a file. Expects a JSON file. This also happens automatically when you load an nlp object or model with a sentencizer added to its pipeline.

Example

pythonCopy

sentencizer = nlp.add_pipe("sentencizer")
sentencizer.from_disk("/path/to/sentencizer.json")

Sentencizer.to_bytes {id="to_bytes",tag="method"}

Serialize the sentencizer settings to a bytestring.

Example

pythonCopy

config = {"punct_chars": [".", "?", "!", "。"]}
sentencizer = nlp.add_pipe("sentencizer", config=config)
sentencizer_bytes = sentencizer.to_bytes()

Sentencizer.from_bytes {id="from_bytes",tag="method"}

Load the pipe from a bytestring. Modifies the object in place and returns it.

Example

pythonCopy

sentencizer_bytes = sentencizer.to_bytes()
sentencizer = nlp.add_pipe("sentencizer")
sentencizer.from_bytes(sentencizer_bytes)