Sentencepiece Tokenizer
¶
class
konoha.word_tokenizers.sentencepiece_tokenizer.
SentencepieceTokenizer
(
model_path
:
str
)
¶
tokenize
(
text
:
str
)
→
List
[
Token
]
¶
Abstract method forkonoha.tokenization
konoha
Navigation
Quick Start
Installation
API Reference
Word Level Tokenizer Interface
Sentence Level Tokenizer Interface
Word Tokenizer Implementations
Token
Data classes
Server
Related Topics
Documentation overview
API Reference
Word Tokenizer Implementations
Previous:
KyTea Tokenizer
Next:
Sudachi Tokenizer
Quick search