org.datavec.api.transform.transform.nlp

Class TextToTermIndexSequenceTransform

java.lang.Object

org.datavec.api.transform.sequence.expansion.BaseSequenceExpansionTransform

org.datavec.api.transform.transform.nlp.TextToTermIndexSequenceTransform

All Implemented Interfaces:

Serializable, ColumnOp, Operation<Schema,Schema>, Transform

public class TextToTermIndexSequenceTransform
extends BaseSequenceExpansionTransform

Convert each text value in a sequence to a longer sequence of integer indices. For example, "zero one two" would be converted to [0, 1, 2]. Values in other columns will be duplicated.

Author:

Dave Kale

See Also:

Serialized Form

Field Summary

Fields inherited from class org.datavec.api.transform.sequence.expansion.BaseSequenceExpansionTransform

expandedColumnNames, inputSchema, requiredColumns

Constructor Summary

Constructors

Constructor and Description
TextToTermIndexSequenceTransform(String columnName, String newColumnName, List<String> vocabulary, String delimiter, boolean exceptionOnUnknown)
TextToTermIndexSequenceTransform(String columnName, String newColumnName, Map<String,Integer> wordIndexMap, String delimiter, boolean exceptionOnUnknown)

Method Summary

Modifier and Type	Method and Description
protected List<ColumnMetaData>	expandedColumnMetaDatas(List<ColumnMetaData> origColumnMeta, List<String> expandedColumnNames)
protected List<List<Writable>>	expandTimeStep(List<Writable> currentStepValues)

Methods inherited from class org.datavec.api.transform.sequence.expansion.BaseSequenceExpansionTransform

columnName, columnNames, map, map, mapSequence, mapSequence, outputColumnName, outputColumnNames, transform

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.datavec.api.transform.ColumnOp

getInputSchema, setInputSchema

Constructor Detail

TextToTermIndexSequenceTransform

public TextToTermIndexSequenceTransform(String columnName,
                                        String newColumnName,
                                        List<String> vocabulary,
                                        String delimiter,
                                        boolean exceptionOnUnknown)

Parameters:

columnName - Name of the text column

newColumnName - Name of the column after expansion

vocabulary - Vocabulary

delimiter - Delimiter

exceptionOnUnknown - If true: throw an exception on unknown characters. False: skip unknown characters.

TextToTermIndexSequenceTransform

public TextToTermIndexSequenceTransform(String columnName,
                                        String newColumnName,
                                        Map<String,Integer> wordIndexMap,
                                        String delimiter,
                                        boolean exceptionOnUnknown)

Parameters:

columnName - Name of the text column

newColumnName - Name of the column after expansion

wordIndexMap - Map from terms in vocabulary to indeces

delimiter - Delimiter

exceptionOnUnknown - If true: throw an exception on unknown characters. False: skip unknown characters.

Method Detail

expandedColumnMetaDatas

protected List<ColumnMetaData> expandedColumnMetaDatas(List<ColumnMetaData> origColumnMeta,
                                                       List<String> expandedColumnNames)

Specified by:

expandedColumnMetaDatas in class BaseSequenceExpansionTransform

expandTimeStep

protected List<List<Writable>> expandTimeStep(List<Writable> currentStepValues)

Specified by:

expandTimeStep in class BaseSequenceExpansionTransform