Janome Tokenizer

class konoha.word_tokenizers.janome_tokenizer.JanomeTokenizer(user_dictionary_path: Optional[str] = None)

tokenize(text: str) → List[Token]

Abstract method forkonoha.tokenization