org.deeplearning4j.nlp.uima.tokenization.tokenizer.preprocessor

Class CustomStemmingPreprocessor

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizer.preprocessor.CommonPreprocessor

org.deeplearning4j.nlp.uima.tokenization.tokenizer.preprocessor.CustomStemmingPreprocessor

All Implemented Interfaces:

TokenPreProcess

public class CustomStemmingPreprocessor
extends CommonPreprocessor

This is StemmingPreprocessor compatible with different StemmingProcessors defined as lucene/tartarus SnowballProgram such as: RussianStemmer, DutchStemmer, FrenchStemmer etc.
Note that CommonPreprocessor#preProcess(String) is first applied (i.e. punctuation marks are removed and lower-cased), then the stemmer is applied.
This preprocessor is synchronized, thus thread-safe.

Author:

[email protected]

Constructor Summary

Constructors

Constructor and Description
CustomStemmingPreprocessor(@NonNull org.tartarus.snowball.SnowballProgram stemmer)

Method Summary

Modifier and Type	Method and Description
String	preProcess(String token) Pre process a token

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CustomStemmingPreprocessor

public CustomStemmingPreprocessor(@NonNull
                                  @NonNull org.tartarus.snowball.SnowballProgram stemmer)

Method Detail

preProcess

public String preProcess(String token)

Description copied from interface: TokenPreProcess

Pre process a token

Specified by:

preProcess in interface TokenPreProcess

Overrides:

preProcess in class CommonPreprocessor

Parameters:

token - the token to pre process

Returns:

the preprocessed token