Sudachi Tokenizer

class konoha.word_tokenizers.sudachi_tokenizer.SudachiTokenizer(mode: str)

tokenize(text: str) → List[Token]

Abstract method forkonoha.tokenization