Base Word Tokenizer

class konoha.word_tokenizers.tokenizer.BaseTokenizer(name: str)

Base class for word levelkonoha.tokenizer

property name: str

Return name ofkonoha.tokenizer

abstract tokenize(text: str) → List[Token]

Abstract method forkonoha.tokenization