Benchmarks

Performance comparison between PySET and PySBD.

Performance Summary

Execution Time

Text Size	Words	PySET	PySBD	Speedup
Sentences	~5	0.05ms	0.10ms	2.0x
Paragraph	~104	0.60ms	1.37ms	2.3x
Article	~484	2.41ms	5.25ms	2.2x
Document	~1400	5.68ms	21.95ms	3.9x

Words per Second

Accuracy

Metric	PySET	PySBD
52 Languages	100%	100%
Legal Documents	Better	Over-splits

Key Findings

• PySET is 2-3x faster than PySBD across all text sizes
• Speedup increases with text size (up to 3.9x on documents)
• PySET is more accurate on legal documents (no false splits at commas)
• 100% accuracy on 52 languages

Optimizations

• Context Caching - ~5x speedup
• Early Exit Strategy - ~3x speedup
• Pre-compiled Patterns - One-time compilation

Run Your Own Benchmark

from pyset import TokenBoundaryDetector
import time

detector = TokenBoundaryDetector()
text = "Your text here. Add more sentences. More text."

start = time.perf_counter()
sentences = detector.split(text)
elapsed = time.perf_counter() - start

words_per_sec = len(text.split()) / elapsed
print(f"Time: {elapsed*1000:.2f}ms")
print(f"Rate: {words_per_sec:,.0f} words/sec")