RecordReaderDataSetIterator

java.lang.Object
- org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator

All Implemented Interfaces:: java.io.Serializable, java.util.Iterator<DataSet>, DataSetIterator

Direct Known Subclasses:: CifarDataSetIterator, LFWDataSetIterator, TinyImageNetDataSetIterator

public class RecordReaderDataSetIterator
extends java.lang.Object
implements DataSetIterator

Record reader dataset iterator. Takes a DataVec RecordReader as input, and handles the conversion to ND4J DataSet objects as well as producing minibatches from individual records.

Multiple constructors are available, though a RecordReaderDataSetIterator.Builder class is also available.

Example 1: Image classification, batch size 32, 10 classes

  RecordReader rr = new ImageRecordReader(28,28,3); //28x28 RGB images
  rr.initialize(new FileSplit(new File("/path/to/directory")));

  DataSetIterator iter = new RecordReaderDataSetIterator.Builder(rr, 32)
       //Label index (first arg): Always value 1 when using ImageRecordReader. For CSV etc: use index of the column
       //  that contains the label (should contain an integer value, 0 to nClasses-1 inclusive). Column indexes start
       // at 0. Number of classes (second arg): number of label classes (i.e., 10 for MNIST - 10 digits)
       .classification(1, nClasses)
       .preProcessor(new ImagePreProcessingScaler())      //For normalization of image values 0-255 to 0-1
       .build()

Example 2: Multi-output regression from CSV, batch size 128

  RecordReader rr = new CsvRecordReader(0, ','); //Skip 0 header lines, comma separated
  rr.initialize(new FileSplit(new File("/path/to/myCsv.txt")));

  DataSetIterator iter = new RecordReaderDataSetIterator.Builder(rr, 128)
       //Specify the columns that the regression labels/targets appear in. Note that all other columns will be
       // treated as features. Columns indexes start at 0
       .regression(labelColFrom, labelColTo)
       .build()

See Also:: Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class RecordReaderDataSetIterator.Builder
Builder class for RecordReaderDataSetIterator

Nested Classes
Modifier and Type	Class and Description
`static class`	`RecordReaderDataSetIterator.Builder` Builder class for RecordReaderDataSetIterator

Field Summary

Fields
Modifier and Type	Field and Description
`protected int`	`batchNum`
`protected int`	`batchSize`
`protected WritableConverter`	`converter`
`protected int`	`labelIndex`
`protected int`	`labelIndexTo`
`protected DataSet`	`last`
`protected int`	`maxNumBatches`
`protected int`	`numPossibleLabels`
`protected DataSetPreProcessor`	`preProcessor`
`protected RecordReader`	`recordReader`
`protected boolean`	`regression`
`protected java.util.Iterator<java.util.List<Writable>>`	`sequenceIter`
`protected boolean`	`useCurrent`

Constructor Summary

Constructors
Modifier	Constructor and Description
`protected`	`RecordReaderDataSetIterator(RecordReaderDataSetIterator.Builder b)`
	`RecordReaderDataSetIterator(RecordReader recordReader, int batchSize)` Constructor for classification, where: (a) the label index is assumed to be the very last Writable/column, and (b) the number of classes is inferred from RecordReader.getLabels() Note that if RecordReader.getLabels() returns null, no output labels will be produced
	`RecordReaderDataSetIterator(RecordReader recordReader, int batchSize, int labelIndex, int numPossibleLabels)` Main constructor for classification.
	`RecordReaderDataSetIterator(RecordReader recordReader, int batchSize, int labelIndexFrom, int labelIndexTo, boolean regression)` Main constructor for multi-label regression (i.e., regression with multiple outputs)
	`RecordReaderDataSetIterator(RecordReader recordReader, int batchSize, int labelIndex, int numPossibleLabels, int maxNumBatches)` Constructor for classification, where the maximum number of returned batches is limited to the specified value
	`RecordReaderDataSetIterator(RecordReader recordReader, WritableConverter converter, int batchSize, int labelIndexFrom, int labelIndexTo, int numPossibleLabels, int maxNumBatches, boolean regression)` Main constructor

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`boolean`	`asyncSupported()` Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching.
`int`	`batch()` Batch size
`java.util.List<java.lang.String>`	`getLabels()` Get dataset iterator class labels, if any.
`boolean`	`hasNext()`
`int`	`inputColumns()` Input columns for the dataset
`DataSet`	`loadFromMetaData(java.util.List<RecordMetaData> list)` Load a multiple examples to a DataSet, using the provided RecordMetaData instances.
`DataSet`	`loadFromMetaData(RecordMetaData recordMetaData)` Load a single example to a DataSet, using the provided RecordMetaData.
`DataSet`	`next()`
`DataSet`	`next(int num)` Like the standard next method but allows a customizable number of examples returned
`void`	`remove()`
`void`	`reset()` Resets the iterator back to the beginning
`boolean`	`resetSupported()` Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't
`void`	`setCollectMetaData(boolean collectMetaData)` When set to true: metadata for the current examples will be present in the returned DataSet.
`void`	`setPreProcessor(DataSetPreProcessor preProcessor)` Set a pre processor
`int`	`totalOutcomes()` The number of labels for the dataset

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator
getPreProcessor

Methods inherited from interface java.util.Iterator
forEachRemaining

- Field Detail
  - recordReader
```
protected RecordReader recordReader
```
  - converter
```
protected WritableConverter converter
```
  - batchSize
```
protected int batchSize
```
  - maxNumBatches
```
protected int maxNumBatches
```
  - batchNum
```
protected int batchNum
```
  - labelIndex
```
protected int labelIndex
```
  - labelIndexTo
```
protected int labelIndexTo
```
  - numPossibleLabels
```
protected int numPossibleLabels
```
  - sequenceIter
```
protected java.util.Iterator<java.util.List<Writable>> sequenceIter
```
  - last
```
protected DataSet last
```
  - useCurrent
```
protected boolean useCurrent
```
  - regression
```
protected boolean regression
```
  - preProcessor
```
protected DataSetPreProcessor preProcessor
```
- Constructor Detail
  - RecordReaderDataSetIterator
```
public RecordReaderDataSetIterator(RecordReader recordReader,
                                   int batchSize)
```
    Constructor for classification, where:
    (a) the label index is assumed to be the very last Writable/column, and
    (b) the number of classes is inferred from RecordReader.getLabels()
    Note that if RecordReader.getLabels() returns null, no output labels will be produced
    
    Parameters:
    
    recordReader - Record reader to use as the source of data
    
    batchSize - Minibatch size, for each call of .next()
  - RecordReaderDataSetIterator
```
public RecordReaderDataSetIterator(RecordReader recordReader,
                                   int batchSize,
                                   int labelIndex,
                                   int numPossibleLabels)
```
    Main constructor for classification. This will convert the input class index (at position labelIndex, with integer values 0 to numPossibleLabels-1 inclusive) to the appropriate one-hot output/labels representation.
    
    Parameters:
    
    recordReader - RecordReader: provides the source of the data
    
    batchSize - Batch size (number of examples) for the output DataSet objects
    
    labelIndex - Index of the label Writable (usually an IntWritable), as obtained by recordReader.next()
    
    numPossibleLabels - Number of classes (possible labels) for classification
  - RecordReaderDataSetIterator
```
public RecordReaderDataSetIterator(RecordReader recordReader,
                                   int batchSize,
                                   int labelIndex,
                                   int numPossibleLabels,
                                   int maxNumBatches)
```
    Constructor for classification, where the maximum number of returned batches is limited to the specified value
    
    Parameters:
    
    recordReader - the recordreader to use
    
    labelIndex - the index/column of the label (for classification)
    
    numPossibleLabels - the number of possible labels for classification. Not used if regression == true
    
    maxNumBatches - The maximum number of batches to return between resets. Set to -1 to return all available data
  - RecordReaderDataSetIterator
```
public RecordReaderDataSetIterator(RecordReader recordReader,
                                   int batchSize,
                                   int labelIndexFrom,
                                   int labelIndexTo,
                                   boolean regression)
```
    Main constructor for multi-label regression (i.e., regression with multiple outputs)
    
    Parameters:
    
    recordReader - RecordReader to get data from
    
    labelIndexFrom - Index of the first regression target
    
    labelIndexTo - Index of the last regression target, inclusive
    
    batchSize - Minibatch size
    
    regression - Require regression = true. Mainly included to avoid clashing with other constructors previously defined :/
  - RecordReaderDataSetIterator
```
public RecordReaderDataSetIterator(RecordReader recordReader,
                                   WritableConverter converter,
                                   int batchSize,
                                   int labelIndexFrom,
                                   int labelIndexTo,
                                   int numPossibleLabels,
                                   int maxNumBatches,
                                   boolean regression)
```
    Main constructor
    
    Parameters:
    
    recordReader - the recordreader to use
    
    converter - Converter. May be null.
    
    batchSize - Minibatch size - number of examples returned for each call of .next()
    
    labelIndexFrom - the index of the label (for classification), or the first index of the labels for multi-output regression
    
    labelIndexTo - only used if regression == true. The last index inclusive of the multi-output regression
    
    numPossibleLabels - the number of possible labels for classification. Not used if regression == true
    
    maxNumBatches - Maximum number of batches to return
    
    regression - if true: regression. If false: classification (assume labelIndexFrom is the class it belongs to)
  - RecordReaderDataSetIterator
```
protected RecordReaderDataSetIterator(RecordReaderDataSetIterator.Builder b)
```
- Method Detail
  - setCollectMetaData
```
public void setCollectMetaData(boolean collectMetaData)
```
    When set to true: metadata for the current examples will be present in the returned DataSet. Disabled by default.
    
    Parameters:
    
    collectMetaData - Whether to collect metadata or not
  - next
```
public DataSet next(int num)
```
    Description copied from interface: DataSetIterator
    
    Like the standard next method but allows a customizable number of examples returned
    
    Specified by:
    
    next in interface DataSetIterator
    
    Parameters:
    
    num - the number of examples
    
    Returns:
    
    the next data applyTransformToDestination
  - inputColumns
```
public int inputColumns()
```
    Description copied from interface: DataSetIterator
    
    Input columns for the dataset
    
    Specified by:
    
    inputColumns in interface DataSetIterator
    
    Returns:
  - totalOutcomes
```
public int totalOutcomes()
```
    Description copied from interface: DataSetIterator
    
    The number of labels for the dataset
    
    Specified by:
    
    totalOutcomes in interface DataSetIterator
    
    Returns:
  - resetSupported
```
public boolean resetSupported()
```
    Description copied from interface: DataSetIterator
    
    Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't
    
    Specified by:
    
    resetSupported in interface DataSetIterator
    
    Returns:
    
    true if reset method is supported; false otherwise
  - asyncSupported
```
public boolean asyncSupported()
```
    Description copied from interface: DataSetIterator
    
    Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching. For example, it would not make sense to use asynchronous prefetching for the following types of iterators: (a) Iterators that store their full contents in memory already (b) Iterators that re-use features/labels arrays (as future next() calls will overwrite past contents) (c) Iterators that already implement some level of asynchronous prefetching (d) Iterators that may return different data depending on when the next() method is called
    
    Specified by:
    
    asyncSupported in interface DataSetIterator
    
    Returns:
    
    true if asynchronous prefetching from this iterator is OK; false if asynchronous prefetching should not be used with this iterator
  - reset
```
public void reset()
```
    Description copied from interface: DataSetIterator
    
    Resets the iterator back to the beginning
    
    Specified by:
    
    reset in interface DataSetIterator
  - batch
```
public int batch()
```
    Description copied from interface: DataSetIterator
    
    Batch size
    
    Specified by:
    
    batch in interface DataSetIterator
    
    Returns:
  - setPreProcessor
```
public void setPreProcessor(DataSetPreProcessor preProcessor)
```
    Description copied from interface: DataSetIterator
    
    Set a pre processor
    
    Specified by:
    
    setPreProcessor in interface DataSetIterator
    
    Parameters:
    
    preProcessor - a pre processor to set
  - hasNext
```
public boolean hasNext()
```
    Specified by:
    
    hasNext in interface java.util.Iterator<DataSet>
  - next
```
public DataSet next()
```
    Specified by:
    
    next in interface java.util.Iterator<DataSet>
  - remove
```
public void remove()
```
    Specified by:
    
    remove in interface java.util.Iterator<DataSet>
  - getLabels
```
public java.util.List<java.lang.String> getLabels()
```
    Description copied from interface: DataSetIterator
    
    Get dataset iterator class labels, if any. Note that implementations are not required to implement this, and can simply return null
    
    Specified by:
    
    getLabels in interface DataSetIterator
  - loadFromMetaData
```
public DataSet loadFromMetaData(RecordMetaData recordMetaData)
                         throws java.io.IOException
```
    Load a single example to a DataSet, using the provided RecordMetaData. Note that it is more efficient to load multiple instances at once, using loadFromMetaData(List)
    
    Parameters:
    
    recordMetaData - RecordMetaData to load from. Should have been produced by the given record reader
    
    Returns:
    
    DataSet with the specified example
    
    Throws:
    
    java.io.IOException - If an error occurs during loading of the data
  - loadFromMetaData
```
public DataSet loadFromMetaData(java.util.List<RecordMetaData> list)
                         throws java.io.IOException
```
    Load a multiple examples to a DataSet, using the provided RecordMetaData instances.
    
    Parameters:
    
    list - List of RecordMetaData instances to load from. Should have been produced by the record reader provided to the RecordReaderDataSetIterator constructor
    
    Returns:
    
    DataSet with the specified examples
    
    Throws:
    
    java.io.IOException - If an error occurs during loading of the data

Class RecordReaderDataSetIterator

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Methods inherited from interface java.util.Iterator

Field Detail

recordReader

converter

batchSize

maxNumBatches

batchNum

labelIndex

labelIndexTo

numPossibleLabels

sequenceIter

last

useCurrent

regression

preProcessor

Constructor Detail

RecordReaderDataSetIterator

RecordReaderDataSetIterator

RecordReaderDataSetIterator

RecordReaderDataSetIterator

RecordReaderDataSetIterator

RecordReaderDataSetIterator

Method Detail

setCollectMetaData

next

inputColumns

totalOutcomes

resetSupported

asyncSupported

reset

batch

setPreProcessor

hasNext

next

remove

getLabels

loadFromMetaData

loadFromMetaData