Character Tokenizer
¶
class
konoha.word_tokenizers.character_tokenizer.
CharacterTokenizer
¶
tokenize
(
text
:
str
)
¶
Abstract method forkonoha.tokenization
konoha
Navigation
Quick Start
Installation
API Reference
Word Level Tokenizer Interface
Sentence Level Tokenizer Interface
Word Tokenizer Implementations
Token
Data classes
Server
Related Topics
Documentation overview
API Reference
Word Tokenizer Implementations
Previous:
Base Word Tokenizer
Next:
MeCab Tokenizer
Quick search