SparkUtils (deeplearning4j 1.0.0-beta6 API)

java.lang.Object
- org.datavec.spark.transform.utils.SparkUtils

```
public class SparkUtils
extends Object
```
Created by Alex on 7/03/2016.

Constructor Summary

Constructors
Constructor and Description

SparkUtils()

Constructors
Constructor and Description
`SparkUtils()`

Method Summary

All Methods Static Methods Concrete Methods
Modifier and Type	Method and Description
`static Class<? extends org.apache.hadoop.io.compress.CompressionCodec>`	`getCompressionCodeClass(String compressionCodecClass)`
`static <T> T`	`readObjectFromFile(String path, Class<T> type, org.apache.hadoop.conf.Configuration hadoopConfig)` Read an object from HDFS (or local) using default Java object serialization
`static <T> T`	`readObjectFromFile(String path, Class<T> type, org.apache.spark.api.java.JavaSparkContext sc)` Read an object from HDFS (or local) using default Java object serialization
`static <T> T`	`readObjectFromFile(String path, Class<T> type, org.apache.spark.SparkContext sc)` Read an object from HDFS (or local) using default Java object serialization
`static String`	`readStringFromFile(String path, org.apache.hadoop.conf.Configuration hadoopConfig)` Read a UTF-8 format String from HDFS (or local)
`static String`	`readStringFromFile(String path, org.apache.spark.api.java.JavaSparkContext sc)` Read a UTF-8 format String from HDFS (or local)
`static String`	`readStringFromFile(String path, org.apache.spark.SparkContext sc)` Read a UTF-8 format String from HDFS (or local)
`static void`	`registerKryoClasses(org.apache.spark.SparkConf conf)` Register the DataVec writable classes for Kryo
`static <T> List<org.apache.spark.api.java.JavaRDD<T>>`	`splitData(SplitStrategy splitStrategy, org.apache.spark.api.java.JavaRDD<T> data, long seed)`
`static void`	`writeAnalysisHTMLToFile(String outputPath, DataAnalysis dataAnalysis, org.apache.spark.api.java.JavaSparkContext sc)` Write a DataAnalysis to HDFS (or locally) as a HTML file
`static void`	`writeObjectToFile(String path, Object toWrite, org.apache.hadoop.conf.Configuration hadoopConfig)` Write an object to HDFS (or local) using default Java object serialization
`static void`	`writeObjectToFile(String path, Object toWrite, org.apache.spark.api.java.JavaSparkContext sc)` Write an object to HDFS (or local) using default Java object serialization
`static void`	`writeObjectToFile(String path, Object toWrite, org.apache.spark.SparkContext sc)` Write an object to HDFS (or local) using default Java object serialization
`static void`	`writeSchema(String outputPath, Schema schema, org.apache.spark.api.java.JavaSparkContext sc)` Write a schema to a HDFS (or, local) file in a human-readable format
`static void`	`writeStringToFile(String path, String toWrite, org.apache.hadoop.conf.Configuration hadoopConfig)` Write a String to a file (on HDFS or local) in UTF-8 format
`static void`	`writeStringToFile(String path, String toWrite, org.apache.spark.api.java.JavaSparkContext sc)` Write a String to a file (on HDFS or local) in UTF-8 format
`static void`	`writeStringToFile(String path, String toWrite, org.apache.spark.SparkContext sc)` Write a String to a file (on HDFS or local) in UTF-8 format
`static void`	`writeWritablesToFile(String outputPath, String delim, List<List<Writable>> writables, org.apache.spark.api.java.JavaSparkContext sc)` Wlite a set of writables (or, sequence) to HDFS (or, locally).

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail
- SparkUtils
```
public SparkUtils()
```

Method Detail

splitData

public static <T> List<org.apache.spark.api.java.JavaRDD<T>> splitData(SplitStrategy splitStrategy,
                                                                       org.apache.spark.api.java.JavaRDD<T> data,
                                                                       long seed)

writeStringToFile

public static void writeStringToFile(String path,
                                     String toWrite,
                                     org.apache.spark.api.java.JavaSparkContext sc)
                              throws IOException

Write a String to a file (on HDFS or local) in UTF-8 format

Parameters:: path - Path to write to; toWrite - String to write; sc - Spark context
Throws:: IOException

writeStringToFile

public static void writeStringToFile(String path,
                                     String toWrite,
                                     org.apache.spark.SparkContext sc)
                              throws IOException

Write a String to a file (on HDFS or local) in UTF-8 format

Parameters:: path - Path to write to; toWrite - String to write; sc - Spark context
Throws:: IOException

writeStringToFile

public static void writeStringToFile(String path,
                                     String toWrite,
                                     org.apache.hadoop.conf.Configuration hadoopConfig)
                              throws IOException

Write a String to a file (on HDFS or local) in UTF-8 format

Parameters:: path - Path to write to; toWrite - String to write; hadoopConfig - Hadoop configuration, for example from SparkContext.hadoopConfiguration()
Throws:: IOException

readStringFromFile

public static String readStringFromFile(String path,
                                        org.apache.spark.api.java.JavaSparkContext sc)
                                 throws IOException

Read a UTF-8 format String from HDFS (or local)

Parameters:: path - Path to write the string; sc - Spark context
Throws:: IOException

readStringFromFile

public static String readStringFromFile(String path,
                                        org.apache.spark.SparkContext sc)
                                 throws IOException

Read a UTF-8 format String from HDFS (or local)

Parameters:: path - Path to write the string; sc - Spark context
Throws:: IOException

readStringFromFile
```
public static String readStringFromFile(String path,
                                        org.apache.hadoop.conf.Configuration hadoopConfig)
                                 throws IOException
```
Read a UTF-8 format String from HDFS (or local)

Parameters:

path - Path to write the string

hadoopConfig - Hadoop configuration, for example from SparkContext.hadoopConfiguration()

Throws:

IOException

writeObjectToFile

public static void writeObjectToFile(String path,
                                     Object toWrite,
                                     org.apache.spark.api.java.JavaSparkContext sc)
                              throws IOException

Write an object to HDFS (or local) using default Java object serialization

Parameters:: path - Path to write the object to; toWrite - Object to write; sc - Spark context
Throws:: IOException

writeObjectToFile

public static void writeObjectToFile(String path,
                                     Object toWrite,
                                     org.apache.spark.SparkContext sc)
                              throws IOException

Write an object to HDFS (or local) using default Java object serialization

Parameters:: path - Path to write the object to; toWrite - Object to write; sc - Spark context
Throws:: IOException

writeObjectToFile

public static void writeObjectToFile(String path,
                                     Object toWrite,
                                     org.apache.hadoop.conf.Configuration hadoopConfig)
                              throws IOException

Write an object to HDFS (or local) using default Java object serialization

Parameters:: path - Path to write the object to; toWrite - Object to write; hadoopConfig - Hadoop configuration, for example from SparkContext.hadoopConfiguration()
Throws:: IOException

readObjectFromFile

public static <T> T readObjectFromFile(String path,
                                       Class<T> type,
                                       org.apache.spark.api.java.JavaSparkContext sc)
                                throws IOException

Read an object from HDFS (or local) using default Java object serialization

Type Parameters:: T - Type of the object to read
Parameters:: path - File to read; type - Class of the object to read; sc - Spark context
Throws:: IOException

readObjectFromFile

public static <T> T readObjectFromFile(String path,
                                       Class<T> type,
                                       org.apache.spark.SparkContext sc)
                                throws IOException

Read an object from HDFS (or local) using default Java object serialization

Type Parameters:: T - Type of the object to read
Parameters:: path - File to read; type - Class of the object to read; sc - Spark context
Throws:: IOException

readObjectFromFile
```
public static <T> T readObjectFromFile(String path,
                                       Class<T> type,
                                       org.apache.hadoop.conf.Configuration hadoopConfig)
                                throws IOException
```
Read an object from HDFS (or local) using default Java object serialization

Type Parameters:

T - Type of the object to read

Parameters:

path - File to read

type - Class of the object to read

hadoopConfig - Hadoop configuration, for example from SparkContext.hadoopConfiguration()

Throws:

IOException

writeSchema

public static void writeSchema(String outputPath,
                               Schema schema,
                               org.apache.spark.api.java.JavaSparkContext sc)
                        throws IOException

Write a schema to a HDFS (or, local) file in a human-readable format

Parameters:: outputPath - Output path to write to; schema - Schema to write; sc - Spark context
Throws:: IOException

writeAnalysisHTMLToFile

public static void writeAnalysisHTMLToFile(String outputPath,
                                           DataAnalysis dataAnalysis,
                                           org.apache.spark.api.java.JavaSparkContext sc)

Write a DataAnalysis to HDFS (or locally) as a HTML file

Parameters:: outputPath - Output path; dataAnalysis - Analysis to generate HTML file for; sc - Spark context

writeWritablesToFile

public static void writeWritablesToFile(String outputPath,
                                        String delim,
                                        List<List<Writable>> writables,
                                        org.apache.spark.api.java.JavaSparkContext sc)
                                 throws IOException

Wlite a set of writables (or, sequence) to HDFS (or, locally).

Parameters:: outputPath - Path to write the outptu; delim - Delimiter; writables - data to write; sc - Spark context
Throws:: IOException

registerKryoClasses

public static void registerKryoClasses(org.apache.spark.SparkConf conf)

getCompressionCodeClass

public static Class<? extends org.apache.hadoop.io.compress.CompressionCodec> getCompressionCodeClass(String compressionCodecClass)

Class SparkUtils

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

SparkUtils

Method Detail

splitData

writeStringToFile

writeStringToFile

writeStringToFile

readStringFromFile

readStringFromFile

readStringFromFile

writeObjectToFile

writeObjectToFile

writeObjectToFile

readObjectFromFile

readObjectFromFile

readObjectFromFile

writeSchema

writeAnalysisHTMLToFile

writeWritablesToFile

registerKryoClasses

getCompressionCodeClass