TokenizerBase

java.lang.Object
- com.atilika.kuromoji.TokenizerBase

Direct Known Subclasses:: Tokenizer

public abstract class TokenizerBase
extends java.lang.Object

TokenizerBase main class

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class TokenizerBase.Builder
Abstract Builder shared by all tokenizers

static class TokenizerBase.Mode

Nested Classes
Modifier and Type	Class and Description
`static class`	`TokenizerBase.Builder` Abstract Builder shared by all tokenizers
`static class`	`TokenizerBase.Mode`

Field Summary

Fields
Modifier and Type Field and Description

protected java.util.EnumMap<ViterbiNode.Type,Dictionary> dictionaryMap

protected TokenFactory tokenFactory

Fields
Modifier and Type	Field and Description
`protected java.util.EnumMap<ViterbiNode.Type,Dictionary>`	`dictionaryMap`
`protected TokenFactory`	`tokenFactory`

Constructor Summary

Constructors
Constructor and Description

TokenizerBase()

Constructors
Constructor and Description
`TokenizerBase()`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`configure(TokenizerBase.Builder builder)`
`protected <T extends TokenBase> java.util.List<T>`	`createTokenList(java.lang.String text)` Tokenizes the provided text and returns a list of tokens with various feature information
`void`	`debugLattice(java.io.OutputStream outputStream, java.lang.String text)` Writes the Viterbi lattice for the provided text to an output stream
`void`	`debugTokenize(java.io.OutputStream outputStream, java.lang.String text)` Tokenizes the provided text and outputs the corresponding Viterbi lattice and the Viterbi path to the provided output stream
`java.util.List<? extends TokenBase>`	`tokenize(java.lang.String text)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - tokenFactory
```
protected TokenFactory tokenFactory
```
  - dictionaryMap
```
protected java.util.EnumMap<ViterbiNode.Type,Dictionary> dictionaryMap
```
- Constructor Detail
  - TokenizerBase
```
public TokenizerBase()
```
- Method Detail
  - configure
```
protected void configure(TokenizerBase.Builder builder)
```
  - tokenize
```
public java.util.List<? extends TokenBase> tokenize(java.lang.String text)
```
  - createTokenList
```
protected <T extends TokenBase> java.util.List<T> createTokenList(java.lang.String text)
```
    Tokenizes the provided text and returns a list of tokens with various feature information
    This method is thread safe
    
    Type Parameters:
    
    T - token type
    
    Parameters:
    
    text - text to tokenize
    
    Returns:
    
    list of Token, not null
  - debugTokenize
```
public void debugTokenize(java.io.OutputStream outputStream,
                          java.lang.String text)
                   throws java.io.IOException
```
    Tokenizes the provided text and outputs the corresponding Viterbi lattice and the Viterbi path to the provided output stream
    The output is written in DOT format.
    This method is not thread safe
    
    Parameters:
    
    outputStream - output stream to write to
    
    text - text to tokenize
    
    Throws:
    
    java.io.IOException - if an error occurs when writing the lattice and path
  - debugLattice
```
public void debugLattice(java.io.OutputStream outputStream,
                         java.lang.String text)
                  throws java.io.IOException
```
    Writes the Viterbi lattice for the provided text to an output stream
    The output is written in DOT format.
    This method is not thread safe
    
    Parameters:
    
    outputStream - output stream to write to
    
    text - text to create lattice for
    
    Throws:
    
    java.io.IOException - if an error occurs when writing the lattice

Class TokenizerBase

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

tokenFactory

dictionaryMap

Constructor Detail

TokenizerBase

Method Detail

configure

tokenize

createTokenList

debugTokenize

debugLattice