org.datavec.api.records.reader.impl.transform

Class TransformProcessSequenceRecordReader

java.lang.Object

org.datavec.api.records.reader.impl.transform.TransformProcessSequenceRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader, SequenceRecordReader

Direct Known Subclasses:

LocalTransformProcessSequenceRecordReader

public class TransformProcessSequenceRecordReader
extends Object
implements SequenceRecordReader

This wraps a SequenceRecordReader with a TransformProcess which will allow every Record returned from the SequenceRecordReader to be transformed before being returned.

Author:

Adam Gibson

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected SequenceRecordReader	sequenceRecordReader
protected TransformProcess	transformProcess

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
TransformProcessSequenceRecordReader()

Method Summary

Modifier and Type	Method and Description
boolean	batchesSupported() This method returns true, if next(int) signature is supported by this RecordReader implementation.
void	close() Closes this stream and releases any system resources associated with it.
Configuration	getConf() Return the configuration used by this object.
List<String>	getLabels() List of label strings
List<RecordListener>	getListeners() Get the record listeners for this record reader.
boolean	hasNext() Whether there are anymore records
void	initialize(Configuration conf, InputSplit split) Called once at initialization.
void	initialize(InputSplit split) Called once at initialization.
List<Record>	loadFromMetaData(List<RecordMetaData> recordMetaDatas) Load multiple records from the given a list of RecordMetaData instances
Record	loadFromMetaData(RecordMetaData recordMetaData) Load a single record from the given RecordMetaData instance Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using loadFromMetaData(List)
List<SequenceRecord>	loadSequenceFromMetaData(List<RecordMetaData> recordMetaDatas) Load multiple sequence records from the given a list of RecordMetaData instances
SequenceRecord	loadSequenceFromMetaData(RecordMetaData recordMetaData) Load a single sequence record from the given RecordMetaData instance Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using loadSequenceFromMetaData(List)
List<Writable>	next() Get the next record
List<List<Writable>>	next(int num) This method will be used, if batchesSupported() returns true.
Record	nextRecord() Similar to next(), but returns a Record object, that may include metadata such as the source of the data
SequenceRecord	nextSequence() Similar to sequenceRecord(), but returns a Record object, that may include metadata such as the source of the data
List<Writable>	record(URI uri, DataInputStream dataInputStream) Load the record from the given DataInputStream Unlike next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream
void	reset() Reset record reader iterator
boolean	resetSupported()
List<List<Writable>>	sequenceRecord() Returns a sequence record.
List<List<Writable>>	sequenceRecord(URI uri, DataInputStream dataInputStream) Load a sequence record from the given DataInputStream Unlike next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream
void	setConf(Configuration conf) Set the configuration to be used by this object.
void	setListeners(Collection<RecordListener> listeners) Set the record listeners for this record reader.
void	setListeners(RecordListener... listeners) Set the record listeners for this record reader.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sequenceRecordReader

protected SequenceRecordReader sequenceRecordReader

transformProcess

protected TransformProcess transformProcess

Constructor Detail

TransformProcessSequenceRecordReader

public TransformProcessSequenceRecordReader()

Method Detail

setConf

public void setConf(Configuration conf)

Set the configuration to be used by this object.

Specified by:

setConf in interface Configurable

Parameters:

conf -

getConf

public Configuration getConf()

Return the configuration used by this object.

Specified by:

getConf in interface Configurable

sequenceRecord

public List<List<Writable>> sequenceRecord()

Returns a sequence record.

Specified by:

sequenceRecord in interface SequenceRecordReader

Returns:

a sequence of records

batchesSupported

public boolean batchesSupported()

Description copied from interface: RecordReader

This method returns true, if next(int) signature is supported by this RecordReader implementation.

Specified by:

batchesSupported in interface RecordReader

Returns:

next

public List<List<Writable>> next(int num)

Description copied from interface: RecordReader

This method will be used, if batchesSupported() returns true.

Specified by:

next in interface RecordReader

Returns:

sequenceRecord

public List<List<Writable>> sequenceRecord(URI uri,
                                           DataInputStream dataInputStream)
                                    throws IOException

Load a sequence record from the given DataInputStream Unlike next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Specified by:

sequenceRecord in interface SequenceRecordReader

Parameters:

uri -

dataInputStream -

Throws:

IOException - if error occurs during reading from the input stream

nextSequence

public SequenceRecord nextSequence()

Similar to sequenceRecord(), but returns a Record object, that may include metadata such as the source of the data

Specified by:

nextSequence in interface SequenceRecordReader

Returns:

next sequence record

loadSequenceFromMetaData

public SequenceRecord loadSequenceFromMetaData(RecordMetaData recordMetaData)
                                        throws IOException

Load a single sequence record from the given RecordMetaData instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using loadSequenceFromMetaData(List)

Specified by:

loadSequenceFromMetaData in interface SequenceRecordReader

Parameters:

recordMetaData - Metadata for the sequence record that we want to load from

Returns:

Single sequence record for the given RecordMetaData instance

Throws:

IOException - If I/O error occurs during loading

loadSequenceFromMetaData

public List<SequenceRecord> loadSequenceFromMetaData(List<RecordMetaData> recordMetaDatas)
                                              throws IOException

Load multiple sequence records from the given a list of RecordMetaData instances

Specified by:

loadSequenceFromMetaData in interface SequenceRecordReader

Parameters:

recordMetaDatas - Metadata for the records that we want to load from

Returns:

Multiple sequence record for the given RecordMetaData instances

Throws:

IOException - If I/O error occurs during loading

initialize

public void initialize(InputSplit split)
                throws IOException,
                       InterruptedException

Called once at initialization.

Specified by:

initialize in interface RecordReader

Parameters:

split - the split that defines the range of records to read

Throws:

IOException

InterruptedException

initialize

public void initialize(Configuration conf,
                       InputSplit split)
                throws IOException,
                       InterruptedException

Called once at initialization.

Specified by:

initialize in interface RecordReader

Parameters:

conf - a configuration for initialization

split - the split that defines the range of records to read

Throws:

IOException

InterruptedException

next

public List<Writable> next()

Get the next record

Specified by:

next in interface RecordReader

Returns:

hasNext

public boolean hasNext()

Whether there are anymore records

Specified by:

hasNext in interface RecordReader

Returns:

getLabels

public List<String> getLabels()

List of label strings

Specified by:

getLabels in interface RecordReader

Returns:

reset

public void reset()

Reset record reader iterator

Specified by:

reset in interface RecordReader

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface RecordReader

Returns:

True if the record reader can be reset, false otherwise. Note that some record readers cannot be reset - for example, if they are backed by a non-resettable input split (such as certain types of streams)

record

public List<Writable> record(URI uri,
                             DataInputStream dataInputStream)
                      throws IOException

Load the record from the given DataInputStream Unlike next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Specified by:

record in interface RecordReader

Parameters:

uri -

dataInputStream -

Throws:

IOException - if error occurs during reading from the input stream

nextRecord

public Record nextRecord()

Similar to next(), but returns a Record object, that may include metadata such as the source of the data

Specified by:

nextRecord in interface RecordReader

Returns:

next record

loadFromMetaData

public Record loadFromMetaData(RecordMetaData recordMetaData)
                        throws IOException

Load a single record from the given RecordMetaData instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using loadFromMetaData(List)

Specified by:

loadFromMetaData in interface RecordReader

Parameters:

recordMetaData - Metadata for the record that we want to load from

Returns:

Single record for the given RecordMetaData instance

Throws:

IOException - If I/O error occurs during loading

loadFromMetaData

public List<Record> loadFromMetaData(List<RecordMetaData> recordMetaDatas)
                              throws IOException

Load multiple records from the given a list of RecordMetaData instances

Specified by:

loadFromMetaData in interface RecordReader

Parameters:

recordMetaDatas - Metadata for the records that we want to load from

Returns:

Multiple records for the given RecordMetaData instances

Throws:

IOException - If I/O error occurs during loading

getListeners

public List<RecordListener> getListeners()

Get the record listeners for this record reader.

Specified by:

getListeners in interface RecordReader

setListeners

public void setListeners(RecordListener... listeners)

Set the record listeners for this record reader.

Specified by:

setListeners in interface RecordReader

Parameters:

listeners -

setListeners

public void setListeners(Collection<RecordListener> listeners)

Set the record listeners for this record reader.

Specified by:

setListeners in interface RecordReader

Parameters:

listeners -

close

public void close()
           throws IOException

Closes this stream and releases any system resources associated with it. If the stream is already closed then invoking this method has no effect.

As noted in AutoCloseable.close(), cases where the close may fail require careful attention. It is strongly advised to relinquish the underlying resources and to internally mark the Closeable as closed, prior to throwing the IOException.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException - if an I/O error occurs