Word Level Tokenizer Interface

class konoha.word_tokenizer.WordTokenizer(tokenizer: str = 'MeCab', user_dictionary_path: str | None = None, system_dictionary_path: str | None = None, model_path: str | None = None, mode: str | None = None, dictionary_format: str | None = None, endpoint: str | None = None, ssl: bool | None = None, port: int | None = None)

batch_tokenize(texts: List[str]) → List[List[Token]]

Tokenize input texts

tokenize(text: str) → List[Token]

Tokenize input text