TokenBase (deeplearning4j 1.0.0-beta6 API)

java.lang.Object
- com.atilika.kuromoji.TokenBase

Direct Known Subclasses:

Token
```
public abstract class TokenBase
extends Object
```
Abstract token class with features shared by all tokens produced by all tokenizers

Constructor Summary

Constructors
Constructor and Description

TokenBase(int wordId, String surface, ViterbiNode.Type type, int position, Dictionary dictionary)

Constructors
Constructor and Description
`TokenBase(int wordId, String surface, ViterbiNode.Type type, int position, Dictionary dictionary)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`String`	`getAllFeatures()` Gets all features for this token as a comma-separated String
`String[]`	`getAllFeaturesArray()` Gets all features for this token as a String array
`protected String`	`getFeature(int feature)` Gets a numbered feature for this token
`int`	`getPosition()` Gets the position/start index where this token is found in the input text
`String`	`getSurface()` Gets the surface form of this token (表層形)
`boolean`	`isKnown()` Predicate indicating whether this token is known (contained in the standard dictionary)
`boolean`	`isUser()` Predicate indicating whether this token is included is from the user dictionary If a token is contained both in the user dictionary and standard dictionary, this method will return true
`String`	`toString()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Constructor Detail
  - TokenBase
```
public TokenBase(int wordId,
                 String surface,
                 ViterbiNode.Type type,
                 int position,
                 Dictionary dictionary)
```
- Method Detail
  - getSurface
```
public String getSurface()
```
    Gets the surface form of this token (表層形)
    
    Returns:
    
    surface form, not null
  - isKnown
```
public boolean isKnown()
```
    Predicate indicating whether this token is known (contained in the standard dictionary)
    
    Returns:
    
    true if the token is known, otherwise false
  - isUser
```
public boolean isUser()
```
    Predicate indicating whether this token is included is from the user dictionary
    If a token is contained both in the user dictionary and standard dictionary, this method will return true
    
    Returns:
    
    true if this token is in user dictionary. false if not.
  - getPosition
```
public int getPosition()
```
    Gets the position/start index where this token is found in the input text
    
    Returns:
    
    token position
  - getAllFeatures
```
public String getAllFeatures()
```
    Gets all features for this token as a comma-separated String
    
    Returns:
    
    token features, not null
  - getAllFeaturesArray
```
public String[] getAllFeaturesArray()
```
    Gets all features for this token as a String array
    
    Returns:
    
    token feature array, not null
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - getFeature
```
protected String getFeature(int feature)
```
    Gets a numbered feature for this token
    
    Parameters:
    
    feature - feature number
    
    Returns:
    
    token feature, not null

Class TokenBase

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

TokenBase

Method Detail

getSurface

isKnown

isUser

getPosition

getAllFeatures

getAllFeaturesArray

toString

getFeature