Rules Engine

PySET uses 85 priority-weighted rules for sentence boundary detection.

Rule Categories

1. Standard Terminals (Rules 1-15)

Common sentence endings: . ! ?

2. Ellipsis (Rules 16-20)

... and variants

3. Quotation Marks (Rules 21-30)

Smart quotes handling

4. Brackets (Rules 31-40)

Parentheticals, brackets

5. Numbers (Rules 41-50)

Abbreviations with periods

6. Context (Rules 51-70)

Word-based context rules

7. Special Cases (Rules 71-80)

URLs, emails, decimals, version numbers

8. Advanced (Rules 81-85)

Complex legal text

Priority System

Rules are evaluated in priority order (100 highest, 0 lowest). Early exit when:

• confidence >= 0.75 AND priority >= 85
• confidence >= 0.90 (BOUNDARY)

Adding Custom Rules

from pyset.rules import Rule, RuleContext

class MyCustomRule(Rule):
    priority = 90
    
    def evaluate(self, context: RuleContext) -> float:
        if context.prev_word() == "hello":
            return 1.0  # BOUNDARY
        return 0.0  # NOT_BOUNDARY