RegexSequenceRecordReader

java.lang.Object
- org.datavec.api.records.reader.BaseRecordReader
- - org.datavec.api.records.reader.impl.FileRecordReader
  - - org.datavec.api.records.reader.impl.regex.RegexSequenceRecordReader

All Implemented Interfaces:

java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable, Configurable, RecordReader, SequenceRecordReader
```
public class RegexSequenceRecordReader
extends FileRecordReader
implements SequenceRecordReader
```
RegexSequenceRecordReader: Read an entire file (as a sequence), one line at a time and split each line into fields using a regex. Specifically, we are using Pattern and Matcher to do the splitting into groups Example: Data in format "2016-01-01 23:59:59.001 1 DEBUG First entry message!"
using regex String "(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}) (\\d+) ([A-Z]+) (.*)"
would be split into 4 Text writables: ["2016-01-01 23:59:59.001", "1", "DEBUG", "First entry message!"]
Note: RegexSequenceRecordReader supports multiple error handling modes, via RegexSequenceRecordReader.LineErrorHandling. Invalid lines that don't match the provided regex can result in an exception (FailOnInvalid), can be skipped silently (SkipInvalid), or skip invalid but log a warning (SkipInvalidWithWarning)

See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`RegexSequenceRecordReader.LineErrorHandling` Error handling mode: How should invalid lines (i.e., those that don't match the provided regex) be handled? FailOnInvalid: Throw an IllegalStateException when an invalid line is found SkipInvalid: Skip invalid lines (quietly, with no warning) SkipInvalidWithWarning: Skip invalid lines, but log a warning

Field Summary

Fields
Modifier and Type	Field and Description
`static java.nio.charset.Charset`	`DEFAULT_CHARSET`
`static RegexSequenceRecordReader.LineErrorHandling`	`DEFAULT_ERROR_HANDLING`
`static org.slf4j.Logger`	`LOG`
`static java.lang.String`	`SKIP_NUM_LINES`

Fields inherited from class org.datavec.api.records.reader.impl.FileRecordReader
appendLabel, conf, currentFile, inputSplit, iter, labels, locationsIterator

Fields inherited from class org.datavec.api.records.reader.BaseRecordReader
listeners

Fields inherited from interface org.datavec.api.records.reader.RecordReader
APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors
Constructor and Description
`RegexSequenceRecordReader(java.lang.String regex, int skipNumLines)`
`RegexSequenceRecordReader(java.lang.String regex, int skipNumLines, java.nio.charset.Charset encoding, RegexSequenceRecordReader.LineErrorHandling errorHandling)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`initialize(Configuration conf, InputSplit split)` Called once at initialization.
`java.util.List<SequenceRecord>`	`loadSequenceFromMetaData(java.util.List<RecordMetaData> recordMetaDatas)` Load multiple sequence records from the given a list of `RecordMetaData` instances
`SequenceRecord`	`loadSequenceFromMetaData(RecordMetaData recordMetaData)` Load a single sequence record from the given `RecordMetaData` instance Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using `SequenceRecordReader.loadSequenceFromMetaData(List)`
`SequenceRecord`	`nextSequence()` Similar to `SequenceRecordReader.sequenceRecord()`, but returns a `Record` object, that may include metadata such as the source of the data
`void`	`reset()` Reset record reader iterator
`java.util.List<java.util.List<Writable>>`	`sequenceRecord()` Returns a sequence record.
`java.util.List<java.util.List<Writable>>`	`sequenceRecord(java.net.URI uri, java.io.DataInputStream dataInputStream)` Load a sequence record from the given DataInputStream Unlike `RecordReader.next()` the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Methods inherited from class org.datavec.api.records.reader.impl.FileRecordReader
advanceToNextLocation, close, doInitialize, getConf, getCurrentLabel, getLabels, hasNext, initialize, loadFromMetaData, loadFromMetaData, next, next, nextFile, nextRecord, record, resetSupported, setConf, setLabels

Methods inherited from class org.datavec.api.records.reader.BaseRecordReader
batchesSupported, getListeners, invokeListeners, setListeners, setListeners

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.datavec.api.records.reader.RecordReader
batchesSupported, getLabels, getListeners, hasNext, initialize, loadFromMetaData, loadFromMetaData, next, next, nextRecord, record, resetSupported, setListeners, setListeners

Methods inherited from interface java.io.Closeable
close

Methods inherited from interface org.datavec.api.conf.Configurable
getConf, setConf

- Field Detail
  - SKIP_NUM_LINES
```
public static final java.lang.String SKIP_NUM_LINES
```
  - DEFAULT_CHARSET
```
public static final java.nio.charset.Charset DEFAULT_CHARSET
```
  - DEFAULT_ERROR_HANDLING
```
public static final RegexSequenceRecordReader.LineErrorHandling DEFAULT_ERROR_HANDLING
```
  - LOG
```
public static final org.slf4j.Logger LOG
```
- Constructor Detail
  - RegexSequenceRecordReader
```
public RegexSequenceRecordReader(java.lang.String regex,
                                 int skipNumLines)
```
  - RegexSequenceRecordReader
```
public RegexSequenceRecordReader(java.lang.String regex,
                                 int skipNumLines,
                                 java.nio.charset.Charset encoding,
                                 RegexSequenceRecordReader.LineErrorHandling errorHandling)
```
- Method Detail
  - initialize
```
public void initialize(Configuration conf,
                       InputSplit split)
                throws java.io.IOException,
                       java.lang.InterruptedException
```
    Description copied from interface: RecordReader
    
    Called once at initialization.
    
    Specified by:
    
    initialize in interface RecordReader
    
    Overrides:
    
    initialize in class FileRecordReader
    
    Parameters:
    
    conf - a configuration for initialization
    
    split - the split that defines the range of records to read
    
    Throws:
    
    java.io.IOException
    
    java.lang.InterruptedException
  - sequenceRecord
```
public java.util.List<java.util.List<Writable>> sequenceRecord()
```
    Description copied from interface: SequenceRecordReader
    
    Returns a sequence record.
    
    Specified by:
    
    sequenceRecord in interface SequenceRecordReader
    
    Returns:
    
    a sequence of records
  - sequenceRecord
```
public java.util.List<java.util.List<Writable>> sequenceRecord(java.net.URI uri,
                                                               java.io.DataInputStream dataInputStream)
                                                        throws java.io.IOException
```
    Description copied from interface: SequenceRecordReader
    
    Load a sequence record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream
    
    Specified by:
    
    sequenceRecord in interface SequenceRecordReader
    
    Throws:
    
    java.io.IOException - if error occurs during reading from the input stream
  - reset
```
public void reset()
```
    Description copied from interface: RecordReader
    
    Reset record reader iterator
    
    Specified by:
    
    reset in interface RecordReader
    
    Overrides:
    
    reset in class FileRecordReader
  - nextSequence
```
public SequenceRecord nextSequence()
```
    Description copied from interface: SequenceRecordReader
    
    Similar to SequenceRecordReader.sequenceRecord(), but returns a Record object, that may include metadata such as the source of the data
    
    Specified by:
    
    nextSequence in interface SequenceRecordReader
    
    Returns:
    
    next sequence record
  - loadSequenceFromMetaData
```
public SequenceRecord loadSequenceFromMetaData(RecordMetaData recordMetaData)
                                        throws java.io.IOException
```
    Description copied from interface: SequenceRecordReader
    
    Load a single sequence record from the given RecordMetaData instance
    Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using SequenceRecordReader.loadSequenceFromMetaData(List)
    
    Specified by:
    
    loadSequenceFromMetaData in interface SequenceRecordReader
    
    Parameters:
    
    recordMetaData - Metadata for the sequence record that we want to load from
    
    Returns:
    
    Single sequence record for the given RecordMetaData instance
    
    Throws:
    
    java.io.IOException - If I/O error occurs during loading
  - loadSequenceFromMetaData
```
public java.util.List<SequenceRecord> loadSequenceFromMetaData(java.util.List<RecordMetaData> recordMetaDatas)
                                                        throws java.io.IOException
```
    Description copied from interface: SequenceRecordReader
    
    Load multiple sequence records from the given a list of RecordMetaData instances
    
    Specified by:
    
    loadSequenceFromMetaData in interface SequenceRecordReader
    
    Parameters:
    
    recordMetaDatas - Metadata for the records that we want to load from
    
    Returns:
    
    Multiple sequence record for the given RecordMetaData instances
    
    Throws:
    
    java.io.IOException - If I/O error occurs during loading

Class RegexSequenceRecordReader

Nested Class Summary

Field Summary

Fields inherited from class org.datavec.api.records.reader.impl.FileRecordReader

Fields inherited from class org.datavec.api.records.reader.BaseRecordReader

Fields inherited from interface org.datavec.api.records.reader.RecordReader

Constructor Summary

Method Summary

Methods inherited from class org.datavec.api.records.reader.impl.FileRecordReader

Methods inherited from class org.datavec.api.records.reader.BaseRecordReader

Methods inherited from class java.lang.Object

Methods inherited from interface org.datavec.api.records.reader.RecordReader

Methods inherited from interface java.io.Closeable

Methods inherited from interface org.datavec.api.conf.Configurable

Field Detail

SKIP_NUM_LINES

DEFAULT_CHARSET

DEFAULT_ERROR_HANDLING

LOG

Constructor Detail

RegexSequenceRecordReader

RegexSequenceRecordReader

Method Detail

initialize

sequenceRecord

sequenceRecord

reset

nextSequence

loadSequenceFromMetaData

loadSequenceFromMetaData