public abstract static class TokenizerBase.Builder extends Object
Modifier and Type | Field and Description |
---|---|
protected CharacterDefinitions |
characterDefinitions |
protected ConnectionCosts |
connectionCosts |
protected DoubleArrayTrie |
doubleArrayTrie |
protected InsertedDictionary |
insertedDictionary |
protected TokenizerBase.Mode |
mode |
protected int |
partOfSpeechFeature |
protected List<Integer> |
penalties |
protected int |
readingFeature |
protected ResourceResolver |
resolver |
protected boolean |
split |
protected TokenFactory |
tokenFactory |
protected TokenInfoDictionary |
tokenInfoDictionary |
protected int |
totalFeatures |
protected UnknownDictionary |
unknownDictionary |
protected UserDictionary |
userDictionary |
Constructor and Description |
---|
Builder() |
Modifier and Type | Method and Description |
---|---|
abstract <T extends TokenizerBase> |
build()
Creates a Tokenizer instance defined by this Builder
|
protected void |
loadDictionaries() |
TokenizerBase.Builder |
userDictionary(InputStream input)
Sets an optional user dictionary as an input stream
The inpuut stream provided is not closed by this method
|
TokenizerBase.Builder |
userDictionary(String filename)
Sets an optional user dictionary filename
|
protected DoubleArrayTrie doubleArrayTrie
protected ConnectionCosts connectionCosts
protected TokenInfoDictionary tokenInfoDictionary
protected UnknownDictionary unknownDictionary
protected CharacterDefinitions characterDefinitions
protected InsertedDictionary insertedDictionary
protected UserDictionary userDictionary
protected TokenizerBase.Mode mode
protected boolean split
protected int totalFeatures
protected int readingFeature
protected int partOfSpeechFeature
protected ResourceResolver resolver
protected TokenFactory tokenFactory
protected void loadDictionaries()
public abstract <T extends TokenizerBase> T build()
T
- token typepublic TokenizerBase.Builder userDictionary(InputStream input) throws IOException
The inpuut stream provided is not closed by this method
input
- user dictionary as an input streamIOException
- if an error occurs when reading the user dictionarypublic TokenizerBase.Builder userDictionary(String filename) throws IOException
filename
- user dictionary filenameIOException
- if an error occurs when reading the user dictionaryCopyright © 2020. All rights reserved.