Field Detail

• tokenFactory

  protected TokenFactory tokenFactory

• dictionaryMap

  protected EnumMap<ViterbiNode.Type,Dictionary> dictionaryMap

Constructor Detail

• TokenizerBase

  public TokenizerBase()

Method Detail

• configure

  protected void configure(TokenizerBase.Builder builder)

• tokenize

  public List<? extends TokenBase> tokenize(String text)

• createTokenList

  protected <T extends TokenBase> List<T> createTokenList(String text)

  Tokenizes the provided text and returns a list of tokens with various feature information

  This method is thread safe

  Type Parameters:

  T - token type

  Parameters:

  text - text to tokenize

  Returns:

  list of Token, not null

• debugTokenize

  public void debugTokenize(OutputStream outputStream,
                            String text)
                     throws IOException

  Tokenizes the provided text and outputs the corresponding Viterbi lattice and the Viterbi path to the provided output stream

  The output is written in DOT format.

  This method is not thread safe

  Parameters:

  outputStream - output stream to write to

  text - text to tokenize

  Throws:

  IOException - if an error occurs when writing the lattice and path

• debugLattice

  public void debugLattice(OutputStream outputStream,
                           String text)
                    throws IOException

  Writes the Viterbi lattice for the provided text to an output stream

  The output is written in DOT format.

  This method is not thread safe

  Parameters:

  outputStream - output stream to write to

  text - text to create lattice for

  Throws:

  IOException - if an error occurs when writing the lattice