Word Level Tokenizer Interface

class konoha.word_tokenizer.WordTokenizer(tokenizer: str = 'MeCab', user_dictionary_path: Optional[str] = None, system_dictionary_path: Optional[str] = None, model_path: Optional[str] = None, mode: Optional[str] = None, dictionary_format: Optional[str] = None, endpoint: Optional[str] = None, ssl: Optional[bool] = None, port: Optional[int] = None)

batch_tokenize(texts: List[str]) → List[List[Token]]

Tokenize input texts

tokenize(text: str) → List[Token]

Tokenize input text