public class MapFileRecordWriter extends AbstractMapFileWriter<List<Writable>> implements RecordWriter
MapFileRecordReaderMapFileRecordReaderconvertTextTo, counter, DEFAULT_FILENAME_PATTERN, DEFAULT_INDEX_INTERVAL, DEFAULT_MAP_FILE_SPLIT_SIZE, filenamePattern, hadoopConfiguration, indexInterval, isClosed, KEY_CLASS, MAP_FILE_INDEX_INTERVAL_KEY, mapFileSplitSize, opts, outputDir, outputFiles, writersAPPEND| Constructor and Description |
|---|
MapFileRecordWriter(File outputDir)
Constructor for all default values.
|
MapFileRecordWriter(@NonNull File outputDir,
int mapFileSplitSize)
Constructor for most default values.
|
MapFileRecordWriter(@NonNull File outputDir,
int mapFileSplitSize,
WritableType convertTextTo) |
MapFileRecordWriter(@NonNull File outputDir,
int mapFileSplitSize,
WritableType convertTextTo,
org.apache.hadoop.conf.Configuration hadoopConfiguration) |
MapFileRecordWriter(@NonNull File outputDir,
int mapFileSplitSize,
WritableType convertTextTo,
int indexInterval,
org.apache.hadoop.conf.Configuration hadoopConfiguration) |
MapFileRecordWriter(@NonNull File outputDir,
int mapFileSplitSize,
WritableType convertTextTo,
int indexInterval,
String filenamePattern,
org.apache.hadoop.conf.Configuration hadoopConfiguration) |
MapFileRecordWriter(@NonNull File outputDir,
WritableType convertTextTo) |
| Modifier and Type | Method and Description |
|---|---|
protected org.apache.hadoop.io.Writable |
getHadoopWritable(List<Writable> input) |
protected Class<? extends org.apache.hadoop.io.Writable> |
getValueClass() |
void |
initialize(Configuration configuration,
InputSplit split,
Partitioner partitioner)
Initialize the record reader with the given configuration
and
InputSplit |
void |
initialize(InputSplit inputSplit,
Partitioner partitioner)
Initialize a record writer with the given input split
|
boolean |
supportsBatch()
Returns true if this record writer
supports efficient batch writing using
RecordWriter.writeBatch(List) |
PartitionMetaData |
writeBatch(List<List<Writable>> batch)
Write a batch of records
|
close, convertTextWritables, getConf, setConf, writeclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitclose, writegetConf, setConfpublic MapFileRecordWriter(File outputDir)
outputDir - Output directory for the map file(s)public MapFileRecordWriter(@NonNull
@NonNull File outputDir,
int mapFileSplitSize)
outputDir - Output directory for the map file(s)mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0,
multiple map files will be used: each will contain a maximum of mapFileSplitSize
examples. This can be used to avoid having a single multi gigabyte map file, which may
be undesirable in some cases (transfer across the network, for example).public MapFileRecordWriter(@NonNull
@NonNull File outputDir,
WritableType convertTextTo)
outputDir - Output directory for the map file(s)convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances
will be converted to this type. This is useful, when would rather store numerical values
even if the original record reader produces strings/text.public MapFileRecordWriter(@NonNull
@NonNull File outputDir,
int mapFileSplitSize,
WritableType convertTextTo)
outputDir - Output directory for the map file(s)mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0,
multiple map files will be used: each will contain a maximum of mapFileSplitSize
examples. This can be used to avoid having a single multi gigabyte map file, which may
be undesirable in some cases (transfer across the network, for example).convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances
will be converted to this type. This is useful, when would rather store numerical values
even if the original record reader produces strings/text.public MapFileRecordWriter(@NonNull
@NonNull File outputDir,
int mapFileSplitSize,
WritableType convertTextTo,
org.apache.hadoop.conf.Configuration hadoopConfiguration)
outputDir - Output directory for the map file(s)mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0,
multiple map files will be used: each will contain a maximum of mapFileSplitSize
examples. This can be used to avoid having a single multi gigabyte map file, which may
be undesirable in some cases (transfer across the network, for example).convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances
will be converted to this type. This is useful, when would rather store numerical values
even if the original record reader produces strings/text.hadoopConfiguration - Hadoop configuration.public MapFileRecordWriter(@NonNull
@NonNull File outputDir,
int mapFileSplitSize,
WritableType convertTextTo,
int indexInterval,
org.apache.hadoop.conf.Configuration hadoopConfiguration)
outputDir - Output directory for the map file(s)mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0,
multiple map files will be used: each will contain a maximum of mapFileSplitSize
examples. This can be used to avoid having a single multi gigabyte map file, which may
be undesirable in some cases (transfer across the network, for example).convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances
will be converted to this type. This is useful, when would rather store numerical values
even if the original record reader produces strings/text.indexInterval - Index interval for the Map file. Defaults to 1, which is suitable for most caseshadoopConfiguration - Hadoop configuration.public MapFileRecordWriter(@NonNull
@NonNull File outputDir,
int mapFileSplitSize,
WritableType convertTextTo,
int indexInterval,
String filenamePattern,
org.apache.hadoop.conf.Configuration hadoopConfiguration)
outputDir - Output directory for the map file(s)mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0,
multiple map files will be used: each will contain a maximum of mapFileSplitSize
examples. This can be used to avoid having a single multi gigabyte map file, which may
be undesirable in some cases (transfer across the network, for example).convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances
will be converted to this type. This is useful, when would rather store numerical values
even if the original record reader produces strings/text.indexInterval - Index interval for the Map file. Defaults to 1, which is suitable for most casesfilenamePattern - The naming pattern for the map files. Used with String.format(pattern, int)hadoopConfiguration - Hadoop configuration.protected Class<? extends org.apache.hadoop.io.Writable> getValueClass()
getValueClass in class AbstractMapFileWriter<List<Writable>>protected org.apache.hadoop.io.Writable getHadoopWritable(List<Writable> input)
getHadoopWritable in class AbstractMapFileWriter<List<Writable>>public boolean supportsBatch()
RecordWriterRecordWriter.writeBatch(List)supportsBatch in interface RecordWriterpublic void initialize(InputSplit inputSplit, Partitioner partitioner) throws Exception
RecordWriterinitialize in interface RecordWriterinputSplit - the input split to initialize withExceptionpublic void initialize(Configuration configuration, InputSplit split, Partitioner partitioner) throws Exception
RecordWriterInputSplitinitialize in interface RecordWriterconfiguration - the configuration to iniailize withsplit - the split to useExceptionpublic PartitionMetaData writeBatch(List<List<Writable>> batch) throws IOException
RecordWriterwriteBatch in interface RecordWriterbatch - the batch to writeIOExceptionCopyright © 2020. All rights reserved.