org.datavec.spark.storage

Class SparkStorageUtils

java.lang.Object

org.datavec.spark.storage.SparkStorageUtils

public class SparkStorageUtils
extends Object

Utility methods for saving and restoring Writable objects from Spark RDD is to Hadoop formats

Author:

Alex Black

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_MAP_FILE_INTERVAL By default, a map file's index stores only a fraction of the keys.
static String	MAP_FILE_INDEX_INTERVAL_KEY Configuration key for the map file interval.

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.api.java.JavaPairRDD<Long,List<Writable>>	restoreMapFile(String path, org.apache.spark.api.java.JavaSparkContext sc) Restore a JavaPairRDD<Long,List<Writable>> previously saved with saveMapFile(String, JavaRDD)} Note that if the keys are not required, simply use restoreMapFile(...).values()
static org.apache.spark.api.java.JavaPairRDD<Long,List<List<Writable>>>	restoreMapFileSequences(String path, org.apache.spark.api.java.JavaSparkContext sc) Restore a JavaPairRDD<Long,List<List<Writable>>> previously saved with saveMapFile(String, JavaRDD)} Note that if the keys are not required, simply use restoreMapFileSequences(...).values()
static org.apache.spark.api.java.JavaRDD<List<Writable>>	restoreSequenceFile(String path, org.apache.spark.api.java.JavaSparkContext sc) Restore a JavaRDD<List<Writable>> previously saved with saveSequenceFile(String, JavaRDD)
static org.apache.spark.api.java.JavaRDD<List<List<Writable>>>	restoreSequenceFileSequences(String path, org.apache.spark.api.java.JavaSparkContext sc) Restore a JavaRDD<List<List<Writable>> previously saved with saveSequenceFileSequences(String, JavaRDD)
static void	saveMapFile(String path, org.apache.spark.api.java.JavaRDD<List<Writable>> rdd) Save a JavaRDD<List<Writable>> to a Hadoop MapFile.
static void	saveMapFile(String path, org.apache.spark.api.java.JavaRDD<List<Writable>> rdd, org.apache.hadoop.conf.Configuration c, Integer maxOutputFiles) Save a JavaRDD<List<Writable>> to a Hadoop MapFile.
static void	saveMapFile(String path, org.apache.spark.api.java.JavaRDD<List<Writable>> rdd, int interval, Integer maxOutputFiles) Save a JavaRDD<List<Writable>> to a Hadoop MapFile.
static void	saveMapFileSequences(String path, org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd) Save a JavaRDD<List<List<Writable>>> to a Hadoop MapFile.
static void	saveMapFileSequences(String path, org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd, org.apache.hadoop.conf.Configuration c, Integer maxOutputFiles) Save a JavaRDD<List<List<Writable>>> to a Hadoop MapFile.
static void	saveMapFileSequences(String path, org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd, int interval, Integer maxOutputFiles) Save a JavaRDD<List<List<Writable>>> to a Hadoop MapFile.
static void	saveSequenceFile(String path, org.apache.spark.api.java.JavaRDD<List<Writable>> rdd) Save a JavaRDD<List<Writable>> to a Hadoop SequenceFile.
static void	saveSequenceFile(String path, org.apache.spark.api.java.JavaRDD<List<Writable>> rdd, Integer maxOutputFiles) Save a JavaRDD<List<Writable>> to a Hadoop SequenceFile.
static void	saveSequenceFileSequences(String path, org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd) Save a JavaRDD<List<List<Writable>>> to a Hadoop SequenceFile.
static void	saveSequenceFileSequences(String path, org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd, Integer maxOutputFiles) Save a JavaRDD<List<List<Writable>>> to a Hadoop SequenceFile.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAP_FILE_INDEX_INTERVAL_KEY

public static final String MAP_FILE_INDEX_INTERVAL_KEY

Configuration key for the map file interval. This is defined in MapFile.Writer.INDEX_INTERVAL but unfortunately that field is private, hence cannot be referenced here.

See Also:

Constant Field Values

DEFAULT_MAP_FILE_INTERVAL

public static final int DEFAULT_MAP_FILE_INTERVAL

By default, a map file's index stores only a fraction of the keys. This is good, in that it reduces memory requirements (all keys are loaded into memory); however, it has a cost in terms of time taken for look up. Instead of using the default interval of 128, Will use a default interval of 1: given that the keys are LongWritable objects, the marginal increase in space is more than outweighed by the increased performance for use cases such as MapFileRecordReader and MapFileSequenceRecordReader

See Also:

Constant Field Values

Method Detail

saveSequenceFile

public static void saveSequenceFile(String path,
                                    org.apache.spark.api.java.JavaRDD<List<Writable>> rdd)

Save a JavaRDD<List<Writable>> to a Hadoop SequenceFile. Each record is given a unique (but noncontiguous) LongWritable key, and values are stored as RecordWritable instances.

Use restoreSequenceFile(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the sequence file

rdd - RDD to save

See Also:

saveSequenceFileSequences(String, JavaRDD), saveMapFile(String, JavaRDD)

saveSequenceFile

public static void saveSequenceFile(String path,
                                    org.apache.spark.api.java.JavaRDD<List<Writable>> rdd,
                                    Integer maxOutputFiles)

Save a JavaRDD<List<Writable>> to a Hadoop SequenceFile. Each record is given a unique (but noncontiguous) LongWritable key, and values are stored as RecordWritable instances.

Use restoreSequenceFile(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the sequence file

rdd - RDD to save

maxOutputFiles - Nullable. If non-null: first coalesce the RDD to the specified size (number of partitions) to limit the maximum number of output sequence files

See Also:

saveSequenceFileSequences(String, JavaRDD), saveMapFile(String, JavaRDD)

restoreSequenceFile

public static org.apache.spark.api.java.JavaRDD<List<Writable>> restoreSequenceFile(String path,
                                                                                    org.apache.spark.api.java.JavaSparkContext sc)

Restore a JavaRDD<List<Writable>> previously saved with saveSequenceFile(String, JavaRDD)

Parameters:

path - Path of the sequence file

sc - Spark context

Returns:

The restored RDD

saveSequenceFileSequences

public static void saveSequenceFileSequences(String path,
                                             org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd)

Save a JavaRDD<List<List<Writable>>> to a Hadoop SequenceFile. Each record is given a unique (but noncontiguous) LongWritable key, and values are stored as SequenceRecordWritable instances.

Use restoreSequenceFileSequences(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the sequence file

rdd - RDD to save

See Also:

saveSequenceFile(String, JavaRDD), saveMapFileSequences(String, JavaRDD)

saveSequenceFileSequences

public static void saveSequenceFileSequences(String path,
                                             org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd,
                                             Integer maxOutputFiles)

Save a JavaRDD<List<List<Writable>>> to a Hadoop SequenceFile. Each record is given a unique (but noncontiguous) LongWritable key, and values are stored as SequenceRecordWritable instances.

Use restoreSequenceFileSequences(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the sequence file

rdd - RDD to save

maxOutputFiles - Nullable. If non-null: first coalesce the RDD to the specified size (number of partitions) to limit the maximum number of output sequence files

See Also:

saveSequenceFile(String, JavaRDD), saveMapFileSequences(String, JavaRDD)

restoreSequenceFileSequences

public static org.apache.spark.api.java.JavaRDD<List<List<Writable>>> restoreSequenceFileSequences(String path,
                                                                                                   org.apache.spark.api.java.JavaSparkContext sc)

Restore a JavaRDD<List<List<Writable>> previously saved with saveSequenceFileSequences(String, JavaRDD)

Parameters:

path - Path of the sequence file

sc - Spark context

Returns:

The restored RDD

saveMapFile

public static void saveMapFile(String path,
                               org.apache.spark.api.java.JavaRDD<List<Writable>> rdd)

Save a JavaRDD<List<Writable>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as RecordWritable instances.
Note 1: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileRecordReader
Note 2: This use a MapFile interval of DEFAULT_MAP_FILE_INTERVAL, which is usually suitable for use cases such as MapFileRecordReader. Use saveMapFile(String, JavaRDD, int, Integer) or saveMapFile(String, JavaRDD, Configuration, Integer) to customize this.

Use restoreMapFile(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the MapFile

rdd - RDD to save

See Also:

saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

saveMapFile

public static void saveMapFile(String path,
                               org.apache.spark.api.java.JavaRDD<List<Writable>> rdd,
                               int interval,
                               Integer maxOutputFiles)

Save a JavaRDD<List<Writable>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as RecordWritable instances.
Note: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileRecordReader

Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the MapFile

rdd - RDD to save

interval - The map file index interval to use. Smaller values may result in the faster look up, at the expense of more memory/disk use. However, usually the increase is relatively minor, due to keys being stored as LongWritable objects

maxOutputFiles - Nullable. If non-null: first coalesce the RDD to the specified size (number of partitions) to limit the maximum number of output map files

See Also:

saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

saveMapFile

public static void saveMapFile(String path,
                               org.apache.spark.api.java.JavaRDD<List<Writable>> rdd,
                               org.apache.hadoop.conf.Configuration c,
                               Integer maxOutputFiles)

Save a JavaRDD<List<Writable>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as RecordWritable instances.
Note: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileRecordReader

Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the MapFile

rdd - RDD to save

c - Configuration object, used to customise options for the map file

maxOutputFiles - Nullable. If non-null: first coalesce the RDD to the specified size (number of partitions) to limit the maximum number of output map files

See Also:

saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

restoreMapFile

public static org.apache.spark.api.java.JavaPairRDD<Long,List<Writable>> restoreMapFile(String path,
                                                                                        org.apache.spark.api.java.JavaSparkContext sc)

Restore a JavaPairRDD<Long,List<Writable>> previously saved with saveMapFile(String, JavaRDD)}
Note that if the keys are not required, simply use restoreMapFile(...).values()

Parameters:

path - Path of the MapFile

sc - Spark context

Returns:

The restored RDD, with their unique indices as the key

saveMapFileSequences

public static void saveMapFileSequences(String path,
                                        org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd)

Save a JavaRDD<List<List<Writable>>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as SequenceRecordWritable instances.
Note 1: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileSequenceRecordReader
Note 2: This use a MapFile interval of DEFAULT_MAP_FILE_INTERVAL, which is usually suitable for use cases such as MapFileSequenceRecordReader. Use saveMapFileSequences(String, JavaRDD, int, Integer) or saveMapFileSequences(String, JavaRDD, Configuration, Integer) to customize this.

Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the MapFile

rdd - RDD to save

See Also:

saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

saveMapFileSequences

public static void saveMapFileSequences(String path,
                                        org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd,
                                        int interval,
                                        Integer maxOutputFiles)

Save a JavaRDD<List<List<Writable>>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as SequenceRecordWritable instances.
Note: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileSequenceRecordReader

Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the MapFile

rdd - RDD to save

interval - The map file index interval to use. Smaller values may result in the faster look up, at the expense of more memory/disk use. However, usually the increase is relatively minor, due to keys being stored as LongWritable objects

See Also:

saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

saveMapFileSequences

public static void saveMapFileSequences(String path,
                                        org.apache.spark.api.java.JavaRDD<List<List<Writable>>> rdd,
                                        org.apache.hadoop.conf.Configuration c,
                                        Integer maxOutputFiles)

Save a JavaRDD<List<List<Writable>>> to a Hadoop MapFile. Each record is given a unique and contiguous LongWritable key, and values are stored as SequenceRecordWritable instances.
Note: If contiguous keys are not required, using a sequence file instead is preferable from a performance point of view. Contiguous keys are often only required for non-Spark use cases, such as with MapFileSequenceRecordReader

Use restoreMapFileSequences(String, JavaSparkContext) to restore values saved with this method.

Parameters:

path - Path to save the MapFile

rdd - RDD to save

c - Configuration object, used to customise options for the map file

See Also:

saveMapFileSequences(String, JavaRDD), saveSequenceFile(String, JavaRDD)

restoreMapFileSequences

public static org.apache.spark.api.java.JavaPairRDD<Long,List<List<Writable>>> restoreMapFileSequences(String path,
                                                                                                       org.apache.spark.api.java.JavaSparkContext sc)

Restore a JavaPairRDD<Long,List<List<Writable>>> previously saved with saveMapFile(String, JavaRDD)}
Note that if the keys are not required, simply use restoreMapFileSequences(...).values()

Parameters:

path - Path of the MapFile

sc - Spark context

Returns:

The restored RDD, with their unique indices as the key