API Reference

Full API documentation for TokenBoundaryDetector.

TokenBoundaryDetector

Constructor

TokenBoundaryDetector(
    language: str = 'en',
    min_sentence_length: int = 1,
    abbreviations: Set[str] = None,
    aggressive_abbreviations: bool = False,
    merge_short_sentences: bool = False,
    include_rules: List[int] = None,
    exclude_rules: List[int] = None,
    debug: bool = False
)

Parameters

Parameter	Type	Default	Description
language	str	'en'	Language code
min_sentence_length	int	1	Minimum sentence length
abbreviations	Set[str]	None	Custom abbreviations
aggressive_abbreviations	bool	False	Stricter abbreviation handling
merge_short_sentences	bool	False	Merge short sentences
include_rules	List[int]	None	Use specific rules
exclude_rules	List[int]	None	Exclude specific rules
debug	bool	False	Enable debug mode

Methods

split()

detector.split(
    text: str,
    return_spans: bool = False,
    return_metadata: bool = False
) -> Union[List[str], List[tuple], List[dict]]

set_abbreviations()

detector.set_abbreviations({'CEO', 'CFO', 'CTO'})

get_abbreviations()

abbrev = detector.get_abbreviations()

explain()

explanations = detector.explain(text)  # Requires debug=True

Exceptions

Exception	Description
PySETException	Base exception
ConfigurationError	Invalid configuration
RuleError	Rule evaluation error
LanguageError	Unsupported language