Skip to content

Rule based tokenizer V1

Molly-Mitton Clementrequested to merge
clem/rule-based-tokenizer into dev

Uses spaCy language rules as a base and extends them with custom regex patterns and additional infix rules to support specific tokenization cases.

Merge request reports