Whitespace Tokenizer

class konoha.word_tokenizers.whitespace_tokenizer.WhitespaceTokenizer

Simple rule-based word tokenizer.

tokenize(text: str) → List[Token]

Abstract method forkonoha.tokenization