org.deeplearning4j.models.word2vec.iterator

Class Word2VecDataSetIterator

java.lang.Object

org.deeplearning4j.models.word2vec.iterator.Word2VecDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<DataSet>, DataSetIterator

public class Word2VecDataSetIterator
extends Object
implements DataSetIterator

Iterates over a sentence with moving window to produce a data applyTransformToDestination for word windows based on a pretrained word2vec.

Author:

Adam Gibson

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Word2VecDataSetIterator(Word2Vec vec, LabelAwareSentenceIterator iter, List<String> labels) Initializes this iterator with homogenization and adding labels and a batch size of 10
Word2VecDataSetIterator(Word2Vec vec, LabelAwareSentenceIterator iter, List<String> labels, int batch) Initializes this iterator with homogenization and adding labels
Word2VecDataSetIterator(Word2Vec vec, LabelAwareSentenceIterator iter, List<String> labels, int batch, boolean homogenization, boolean addLabels) Allows for customization of all of the params of the iterator

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported() Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects?
int	batch() Batch size
List<String>	getLabels() Get dataset iterator class labels, if any.
boolean	hasNext() Returns true if the iteration has more elements.
int	inputColumns() Input columns for the dataset
DataSet	next() Returns the next element in the iteration.
DataSet	next(int num) Like the standard next method but allows a customizable number of examples returned
void	remove() Removes from the underlying collection the last element returned by this iterator (optional operation).
void	reset() Resets the iterator back to the beginning
boolean	resetSupported() Is resetting supported by this DataSetIterator?
void	setPreProcessor(DataSetPreProcessor preProcessor) Set a pre processor
int	totalOutcomes() The number of labels for the dataset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getPreProcessor

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

Word2VecDataSetIterator

public Word2VecDataSetIterator(Word2Vec vec,
                               LabelAwareSentenceIterator iter,
                               List<String> labels,
                               int batch,
                               boolean homogenization,
                               boolean addLabels)

Allows for customization of all of the params of the iterator

Parameters:

vec - the word2vec model to use

iter - the sentence iterator to use

labels - the possible labels

batch - the batch size

homogenization - whether to homogenize the sentences or not

addLabels - whether to add labels for windows

Word2VecDataSetIterator

public Word2VecDataSetIterator(Word2Vec vec,
                               LabelAwareSentenceIterator iter,
                               List<String> labels)

Initializes this iterator with homogenization and adding labels and a batch size of 10

Parameters:

vec - the vector model to use

iter - the sentence iterator to use

labels - the possible labels

Word2VecDataSetIterator

public Word2VecDataSetIterator(Word2Vec vec,
                               LabelAwareSentenceIterator iter,
                               List<String> labels,
                               int batch)

Initializes this iterator with homogenization and adding labels

Parameters:

vec - the vector model to use

iter - the sentence iterator to use

labels - the possible labels

batch - the batch size

Method Detail

next

public DataSet next(int num)

Like the standard next method but allows a customizable number of examples returned

Specified by:

next in interface DataSetIterator

Parameters:

num - the number of examples

Returns:

the next data applyTransformToDestination

inputColumns

public int inputColumns()

Description copied from interface: DataSetIterator

Input columns for the dataset

Specified by:

inputColumns in interface DataSetIterator

Returns:

totalOutcomes

public int totalOutcomes()

Description copied from interface: DataSetIterator

The number of labels for the dataset

Specified by:

totalOutcomes in interface DataSetIterator

Returns:

resetSupported

public boolean resetSupported()

Description copied from interface: DataSetIterator

Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't

Specified by:

resetSupported in interface DataSetIterator

Returns:

true if reset method is supported; false otherwise

asyncSupported

public boolean asyncSupported()

Description copied from interface: DataSetIterator

Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching. For example, it would not make sense to use asynchronous prefetching for the following types of iterators: (a) Iterators that store their full contents in memory already (b) Iterators that re-use features/labels arrays (as future next() calls will overwrite past contents) (c) Iterators that already implement some level of asynchronous prefetching (d) Iterators that may return different data depending on when the next() method is called

Specified by:

asyncSupported in interface DataSetIterator

Returns:

true if asynchronous prefetching from this iterator is OK; false if asynchronous prefetching should not be used with this iterator

reset

public void reset()

Description copied from interface: DataSetIterator

Resets the iterator back to the beginning

Specified by:

reset in interface DataSetIterator

batch

public int batch()

Description copied from interface: DataSetIterator

Batch size

Specified by:

batch in interface DataSetIterator

Returns:

setPreProcessor

public void setPreProcessor(DataSetPreProcessor preProcessor)

Description copied from interface: DataSetIterator

Set a pre processor

Specified by:

setPreProcessor in interface DataSetIterator

Parameters:

preProcessor - a pre processor to set

getLabels

public List<String> getLabels()

Description copied from interface: DataSetIterator

Get dataset iterator class labels, if any. Note that implementations are not required to implement this, and can simply return null

Specified by:

getLabels in interface DataSetIterator

hasNext

public boolean hasNext()

Returns true if the iteration has more elements. (In other words, returns true if next(int) would return an element rather than throwing an exception.)

Specified by:

hasNext in interface Iterator<DataSet>

Returns:

true if the iteration has more elements

next

public DataSet next()

Returns the next element in the iteration.

Specified by:

next in interface Iterator<DataSet>

Returns:

the next element in the iteration

remove

public void remove()

Removes from the underlying collection the last element returned by this iterator (optional operation). This method can be called only once per call to next(int). The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this method.

Specified by:

remove in interface Iterator<DataSet>

Throws:

UnsupportedOperationException - if the remove operation is not supported by this iterator

IllegalStateException - if the next method has not yet been called, or the remove method has already been called after the last call to the next method