org.datavec.api.transform.transform.string

Class StringListToIndicesNDArrayTransform

java.lang.Object

org.datavec.api.transform.transform.BaseTransform

org.datavec.api.transform.transform.string.StringListToCountsNDArrayTransform

org.datavec.api.transform.transform.string.StringListToIndicesNDArrayTransform

All Implemented Interfaces:

Serializable, ColumnOp, Operation<Schema,Schema>, Transform

public class StringListToIndicesNDArrayTransform
extends StringListToCountsNDArrayTransform

Converts String column into a sparse bag-of-words (BOW) represented as an NDArray of indices. Appropriate for embeddings or as efficient storage before being expanded into a dense array.

Author:

dave@skymind.io

See Also:

Serialized Form

Field Summary

Fields inherited from class org.datavec.api.transform.transform.string.StringListToCountsNDArrayTransform

binary, columnIdx, columnName, delimiter, ignoreUnknown, map, newColumnName, vocabulary

Fields inherited from class org.datavec.api.transform.transform.BaseTransform

inputSchema

Constructor Summary

Constructors

Constructor and Description
StringListToIndicesNDArrayTransform(String columnName, List<String> vocabulary, String delimiter, boolean binary, boolean ignoreUnknown)
StringListToIndicesNDArrayTransform(String columnName, String newColumnName, List<String> vocabulary, String delimiter, boolean binary, boolean ignoreUnknown)

Method Summary

Modifier and Type	Method and Description
protected INDArray	makeBOWNDArray(Collection<Integer> indices)

Methods inherited from class org.datavec.api.transform.transform.string.StringListToCountsNDArrayTransform

columnName, columnNames, getIndices, map, map, mapSequence, outputColumnName, outputColumnNames, readVocabFromFile, setInputSchema, toString, transform

Methods inherited from class org.datavec.api.transform.transform.BaseTransform

getInputSchema, mapSequence

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

StringListToIndicesNDArrayTransform

public StringListToIndicesNDArrayTransform(String columnName,
                                           List<String> vocabulary,
                                           String delimiter,
                                           boolean binary,
                                           boolean ignoreUnknown)

Parameters:

columnName - The name of the column to convert

vocabulary - The possible tokens that may be present.

delimiter - The delimiter for the Strings to convert

ignoreUnknown - Whether to ignore unknown tokens

StringListToIndicesNDArrayTransform

public StringListToIndicesNDArrayTransform(String columnName,
                                           String newColumnName,
                                           List<String> vocabulary,
                                           String delimiter,
                                           boolean binary,
                                           boolean ignoreUnknown)

Method Detail

makeBOWNDArray

protected INDArray makeBOWNDArray(Collection<Integer> indices)

Overrides:

makeBOWNDArray in class StringListToCountsNDArrayTransform