KyTea Tokenizer

class konoha.word_tokenizers.kytea_tokenizer.KyTeaTokenizer(model_path: str | None = None)

tokenize(text: str) → List[Token]

Abstract method forkonoha.tokenization