Field Detail

• MAP_FILE_INDEX_INTERVAL_KEY

  public static final String MAP_FILE_INDEX_INTERVAL_KEY

  Configuration key for the map file interval. This is defined in MapFile.Writer.INDEX_INTERVAL but unfortunately that field is private, hence cannot be referenced here.

  See Also:

  Constant Field Values

• DEFAULT_MAP_FILE_INTERVAL

  public static final int DEFAULT_MAP_FILE_INTERVAL

  By default, a map file's index stores only a fraction of the keys. This is good, in that it reduces memory requirements (all keys are loaded into memory); however, it has a cost in terms of time taken for look up. Instead of using the default interval of 128, Will use a default interval of 1: given that the keys are LongWritable objects, the marginal increase in space is more than outweighed by the increased performance for use cases such as MapFileRecordReader and MapFileSequenceRecordReader

  See Also:

  Constant Field Values

Method Detail

• saveSequenceFile

  public static void saveSequenceFile(String path,
                                      org.apache.spark.api.java.JavaRDD<List<Writable>> rdd)

  Save a JavaRDD<List<Writable>> to a Hadoop SequenceFile. Each record is given a unique (but noncontiguous) LongWritable key, and values are stored as RecordWritable instances.

  Use restoreSequenceFile(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the sequence file

  rdd - RDD to save

  See Also:

  saveSequenceFileSequences(String, JavaRDD), saveMapFile(String, JavaRDD)

• saveSequenceFile

  public static void saveSequenceFile(String path,
                                      org.apache.spark.api.java.JavaRDD<List<Writable>> rdd,
                                      Integer maxOutputFiles)

  Save a JavaRDD<List<Writable>> to a Hadoop SequenceFile. Each record is given a unique (but noncontiguous) LongWritable key, and values are stored as RecordWritable instances.

  Use restoreSequenceFile(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the sequence file

  rdd - RDD to save

  maxOutputFiles - Nullable. If non-null: first coalesce the RDD to the specified size (number of partitions) to limit the maximum number of output sequence files

  See Also:

  saveSequenceFileSequences(String, JavaRDD), saveMapFile(String, JavaRDD)

• restoreSequenceFile

  public static org.apache.spark.api.java.JavaRDD<List<Writable>> restoreSequenceFile(String path,
                                                                                      org.apache.spark.api.java.JavaSparkContext sc)

  Restore a JavaRDD<List<Writable>> previously saved with saveSequenceFile(String, JavaRDD)

  Parameters:

  path - Path of the sequence file

  sc - Spark context

  Returns:

  The restored RDD

• saveSequenceFileSequences

  public static void saveSequenceFileSequences(String path,
                                               org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd)

  Save a JavaRDD<List<List<Writable>>> to a Hadoop SequenceFile. Each record is given a unique (but noncontiguous) LongWritable key, and values are stored as SequenceRecordWritable instances.

  Use restoreSequenceFileSequences(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the sequence file

  rdd - RDD to save

  See Also:

  saveSequenceFile(String, JavaRDD), saveMapFileSequences(String, JavaRDD)

• saveSequenceFileSequences

  public static void saveSequenceFileSequences(String path,
                                               org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd,
                                               Integer maxOutputFiles)

  Save a JavaRDD<List<List<Writable>>> to a Hadoop SequenceFile. Each record is given a unique (but noncontiguous) LongWritable key, and values are stored as SequenceRecordWritable instances.

  Use restoreSequenceFileSequences(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the sequence file

  rdd - RDD to save

  maxOutputFiles - Nullable. If non-null: first coalesce the RDD to the specified size (number of partitions) to limit the maximum number of output sequence files

  See Also:

  saveSequenceFile(String, JavaRDD), saveMapFileSequences(String, JavaRDD)

• restoreSequenceFileSequences

  public static org.apache.spark.api.java.JavaRDD<List<List<Writable>>> restoreSequenceFileSequences(String path,
                                                                                                     org.apache.spark.api.java.JavaSparkContext sc)

  Restore a JavaRDD<List<List<Writable>> previously saved with saveSequenceFileSequences(String, JavaRDD)

  Parameters:

  path - Path of the sequence file

  sc - Spark context

  Returns:

  The restored RDD

• saveMapFile

  public static void saveMapFile(String path,
                                 org.apache.spark.api.java.JavaRDD<List<Writable>> rdd)

  Save a JavaRDD<List<Writable>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as RecordWritable instances.
  Note 1: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileRecordReader
  Note 2: This use a MapFile interval of DEFAULT_MAP_FILE_INTERVAL, which is usually suitable for use cases such as MapFileRecordReader. Use saveMapFile(String, JavaRDD, int, Integer) or saveMapFile(String, JavaRDD, Configuration, Integer) to customize this.
  

  Use restoreMapFile(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the MapFile

  rdd - RDD to save

  See Also:

  saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

• saveMapFile

  public static void saveMapFile(String path,
                                 org.apache.spark.api.java.JavaRDD<List<Writable>> rdd,
                                 int interval,
                                 Integer maxOutputFiles)

  Save a JavaRDD<List<Writable>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as RecordWritable instances.
  Note: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileRecordReader

  Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the MapFile

  rdd - RDD to save

  interval - The map file index interval to use. Smaller values may result in the faster look up, at the expense of more memory/disk use. However, usually the increase is relatively minor, due to keys being stored as LongWritable objects

  maxOutputFiles - Nullable. If non-null: first coalesce the RDD to the specified size (number of partitions) to limit the maximum number of output map files

  See Also:

  saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

• saveMapFile

  public static void saveMapFile(String path,
                                 org.apache.spark.api.java.JavaRDD<List<Writable>> rdd,
                                 org.apache.hadoop.conf.Configuration c,
                                 Integer maxOutputFiles)

  Save a JavaRDD<List<Writable>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as RecordWritable instances.
  Note: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileRecordReader

  Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the MapFile

  rdd - RDD to save

  c - Configuration object, used to customise options for the map file

  maxOutputFiles - Nullable. If non-null: first coalesce the RDD to the specified size (number of partitions) to limit the maximum number of output map files

  See Also:

  saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

• restoreMapFile

  public static org.apache.spark.api.java.JavaPairRDD<Long,List<Writable>> restoreMapFile(String path,
                                                                                          org.apache.spark.api.java.JavaSparkContext sc)

  Restore a JavaPairRDD<Long,List<Writable>> previously saved with saveMapFile(String, JavaRDD)}
  Note that if the keys are not required, simply use restoreMapFile(...).values()

  Parameters:

  path - Path of the MapFile

  sc - Spark context

  Returns:

  The restored RDD, with their unique indices as the key

• saveMapFileSequences

  public static void saveMapFileSequences(String path,
                                          org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd)

  Save a JavaRDD<List<List<Writable>>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as SequenceRecordWritable instances.
  Note 1: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileSequenceRecordReader
  Note 2: This use a MapFile interval of DEFAULT_MAP_FILE_INTERVAL, which is usually suitable for use cases such as MapFileSequenceRecordReader. Use saveMapFileSequences(String, JavaRDD, int, Integer) or saveMapFileSequences(String, JavaRDD, Configuration, Integer) to customize this.
  

  Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the MapFile

  rdd - RDD to save

  See Also:

  saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

• saveMapFileSequences

  public static void saveMapFileSequences(String path,
                                          org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd,
                                          int interval,
                                          Integer maxOutputFiles)

  Save a JavaRDD<List<List<Writable>>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as SequenceRecordWritable instances.
  Note: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileSequenceRecordReader
  

  Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the MapFile

  rdd - RDD to save

  interval - The map file index interval to use. Smaller values may result in the faster look up, at the expense of more memory/disk use. However, usually the increase is relatively minor, due to keys being stored as LongWritable objects

  See Also:

  saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

• saveMapFileSequences

  public static void saveMapFileSequences(String path,
                                          org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd,
                                          org.apache.hadoop.conf.Configuration c,
                                          Integer maxOutputFiles)

  Save a JavaRDD<List<List<Writable>>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as SequenceRecordWritable instances.
  Note: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileSequenceRecordReader
  

  Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

  Parameters:

  path - Path to save the MapFile

  rdd - RDD to save

  c - Configuration object, used to customise options for the map file

  See Also:

  saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

• restoreMapFileSequences

  public static org.apache.spark.api.java.JavaPairRDD<Long,List<List<Writable>>> restoreMapFileSequences(String path,
                                                                                                         org.apache.spark.api.java.JavaSparkContext sc)

  Restore a JavaPairRDD<Long,List<List<Writable>>> previously saved with saveMapFile(String, JavaRDD)}
  Note that if the keys are not required, simply use restoreMapFileSequences(...).values()

  Parameters:

  path - Path of the MapFile

  sc - Spark context

  Returns:

  The restored RDD, with their unique indices as the key