konoha
stable
Quick Start
Installation
API Reference
Word Level Tokenizer Interface
Sentence Level Tokenizer Interface
Word Tokenizer Implementations
Base Word Tokenizer
Character Tokenizer
CharacterTokenizer
MeCab Tokenizer
KyTea Tokenizer
Sentencepiece Tokenizer
Sudachi Tokenizer
Janome Tokenizer
nagisa Tokenizer
Whitespace Tokenizer
Token
Data classes
Server
konoha
Docs
»
API Reference
»
Word Tokenizer Implementations
»
Character Tokenizer
Edit on GitHub
Character Tokenizer
¶
class
konoha.word_tokenizers.character_tokenizer.
CharacterTokenizer
¶
tokenize
(
text
:
str
)
¶
Abstract method forkonoha.tokenization
Read the Docs
v: stable
Versions
latest
stable
Downloads
pdf
html
epub
On Read the Docs
Project Home
Builds
Free document hosting provided by
Read the Docs
.