SparkComputationGraph (deeplearning4j 1.0.0-beta7 API)

java.lang.Object
- org.deeplearning4j.spark.impl.SparkListenable
- - org.deeplearning4j.spark.impl.graph.SparkComputationGraph

```
public class SparkComputationGraph
extends SparkListenable
```
Main class for training ComputationGraph networks using Spark. Also used for performing distributed evaluation and inference on these networks

Author:

Alex Black

Field Summary

Fields
Modifier and Type Field and Description

static int DEFAULT_EVAL_SCORE_BATCH_SIZE

static int DEFAULT_EVAL_WORKERS

static int DEFAULT_ROC_THRESHOLD_STEPS
- Fields inherited from class org.deeplearning4j.spark.impl.SparkListenable
  trainingMaster

Fields
Modifier and Type	Field and Description
`static int`	`DEFAULT_EVAL_SCORE_BATCH_SIZE`
`static int`	`DEFAULT_EVAL_WORKERS`
`static int`	`DEFAULT_ROC_THRESHOLD_STEPS`

Constructor Summary

Constructors
Constructor and Description
`SparkComputationGraph(org.apache.spark.api.java.JavaSparkContext sparkContext, ComputationGraphConfiguration conf, TrainingMaster trainingMaster)`
`SparkComputationGraph(org.apache.spark.api.java.JavaSparkContext javaSparkContext, ComputationGraph network, TrainingMaster trainingMaster)`
`SparkComputationGraph(org.apache.spark.SparkContext sparkContext, ComputationGraphConfiguration conf, TrainingMaster trainingMaster)`
`SparkComputationGraph(org.apache.spark.SparkContext sparkContext, ComputationGraph network, TrainingMaster trainingMaster)` Instantiate a ComputationGraph instance with the given context, network and training master.

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`double`	`calculateScore(org.apache.spark.api.java.JavaRDD<DataSet> data, boolean average)` Calculate the score for all examples in the provided `JavaRDD<DataSet>`, either by summing or averaging over the entire data set.
`double`	`calculateScore(org.apache.spark.api.java.JavaRDD<DataSet> data, boolean average, int minibatchSize)` Calculate the score for all examples in the provided `JavaRDD<DataSet>`, either by summing or averaging over the entire data set.
`double`	`calculateScoreMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> data, boolean average)` Calculate the score for all examples in the provided `JavaRDD<MultiDataSet>`, either by summing or averaging over the entire data set.
`double`	`calculateScoreMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> data, boolean average, int minibatchSize)` Calculate the score for all examples in the provided `JavaRDD<MultiDataSet>`, either by summing or averaging over the entire data set
`<T extends IEvaluation> T[]`	`doEvaluation(org.apache.spark.api.java.JavaRDD<DataSet> data, int evalNumWorkers, int evalBatchSize, T... emptyEvaluations)` Perform distributed evaluation on a single output ComputationGraph form DataSet objects using Spark.
`<T extends IEvaluation> T[]`	`doEvaluation(org.apache.spark.api.java.JavaRDD<DataSet> data, int evalBatchSize, T... emptyEvaluations)` Perform distributed evaluation on a single output ComputationGraph form DataSet objects using Spark.
`<T extends IEvaluation> T`	`doEvaluation(org.apache.spark.api.java.JavaRDD<DataSet> data, T emptyEvaluation, int evalBatchSize)` Perform distributed evaluation of any type of `IEvaluation`.
`IEvaluation[]`	`doEvaluation(org.apache.spark.api.java.JavaRDD<String> data, DataSetLoader loader, IEvaluation... emptyEvaluations)` Perform evaluation on serialized DataSet objects on disk, (potentially in any format), that are loaded using an `DataSetLoader`.
`IEvaluation[]`	`doEvaluation(org.apache.spark.api.java.JavaRDD<String> data, int evalNumWorkers, int evalBatchSize, DataSetLoader loader, IEvaluation... emptyEvaluations)` Perform evaluation on serialized DataSet objects on disk, (potentially in any format), that are loaded using an `DataSetLoader`.
`protected IEvaluation[]`	`doEvaluation(org.apache.spark.api.java.JavaRDD<String> data, int evalNumWorkers, int evalBatchSize, DataSetLoader loader, MultiDataSetLoader mdsLoader, IEvaluation... emptyEvaluations)`
`IEvaluation[]`	`doEvaluation(org.apache.spark.api.java.JavaRDD<String> data, int evalNumWorkers, int evalBatchSize, MultiDataSetLoader loader, IEvaluation... emptyEvaluations)` Perform evaluation on serialized MultiDataSet objects on disk, (potentially in any format), that are loaded using an `MultiDataSetLoader`
`IEvaluation[]`	`doEvaluation(org.apache.spark.api.java.JavaRDD<String> data, MultiDataSetLoader loader, IEvaluation... emptyEvaluations)` Perform evaluation on serialized MultiDataSet objects on disk, (potentially in any format), that are loaded using an `MultiDataSetLoader`.
`<T extends IEvaluation> T[]`	`doEvaluationMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data, int evalNumWorkers, int evalBatchSize, T... emptyEvaluations)`
`<T extends IEvaluation> T[]`	`doEvaluationMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data, int evalBatchSize, T... emptyEvaluations)` Perform distributed evaluation on a single output ComputationGraph form MultiDataSet objects using Spark.
`<T extends Evaluation> T`	`evaluate(org.apache.spark.api.java.JavaRDD<DataSet> data)` Evaluate the network (classification performance) in a distributed manner on the provided data
`<T extends Evaluation> T`	`evaluate(org.apache.spark.api.java.JavaRDD<DataSet> data, List<String> labelsList)` Evaluate the network (classification performance) in a distributed manner, using default batch size and a provided list of labels
`<T extends Evaluation> T`	`evaluate(org.apache.spark.api.java.JavaRDD<DataSet> data, List<String> labelsList, int evalBatchSize)` Evaluate the network (classification performance) in a distributed manner, using specified batch size and a provided list of labels
`<T extends Evaluation> T`	`evaluate(org.apache.spark.rdd.RDD<DataSet> data)` `RDD<DataSet>` overload of `evaluate(JavaRDD)`
`<T extends Evaluation> T`	`evaluate(org.apache.spark.rdd.RDD<DataSet> data, List<String> labelsList)` `RDD<DataSet>` overload of `evaluate(JavaRDD, List)`
`Evaluation`	`evaluate(String path, DataSetLoader loader)` Evaluate the single-output network on a directory containing a set of DataSet objects to be loaded with a `DataSetLoader`.
`Evaluation`	`evaluate(String path, MultiDataSetLoader loader)` Evaluate the single-output network on a directory containing a set of MultiDataSet objects to be loaded with a `MultiDataSetLoader`.
`<T extends Evaluation> T`	`evaluateMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data)` Evaluate the network (classification performance) in a distributed manner on the provided data
`<T extends Evaluation> T`	`evaluateMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data, int minibatchSize)` Evaluate the network (classification performance) in a distributed manner on the provided data
`<T extends RegressionEvaluation> T`	`evaluateRegression(org.apache.spark.api.java.JavaRDD<DataSet> data)` Evaluate the network (regression performance) in a distributed manner on the provided data
`<T extends RegressionEvaluation> T`	`evaluateRegression(org.apache.spark.api.java.JavaRDD<DataSet> data, int minibatchSize)` Evaluate the network (regression performance) in a distributed manner on the provided data
`<T extends RegressionEvaluation> T`	`evaluateRegressionMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data)` Evaluate the network (regression performance) in a distributed manner on the provided data
`<T extends RegressionEvaluation> T`	`evaluateRegressionMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data, int minibatchSize)` Evaluate the network (regression performance) in a distributed manner on the provided data
`<T extends ROC> T`	`evaluateROC(org.apache.spark.api.java.JavaRDD<DataSet> data)` Perform ROC analysis/evaluation on the given DataSet in a distributed manner, using the default number of threshold steps (`DEFAULT_ROC_THRESHOLD_STEPS`) and the default minibatch size (`DEFAULT_EVAL_SCORE_BATCH_SIZE`)
`<T extends ROC> T`	`evaluateROC(org.apache.spark.api.java.JavaRDD<DataSet> data, int thresholdSteps, int evaluationMinibatchSize)` Perform ROC analysis/evaluation on the given DataSet in a distributed manner
`ROC`	`evaluateROCMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data)` Perform ROC analysis/evaluation on the given DataSet in a distributed manner, using the default number of threshold steps (`DEFAULT_ROC_THRESHOLD_STEPS`) and the default minibatch size (`DEFAULT_EVAL_SCORE_BATCH_SIZE`)
`<T extends ROC> T`	`evaluateROCMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data, int rocThresholdNumSteps, int minibatchSize)` Perform ROC analysis/evaluation on the given DataSet in a distributed manner, using the specified number of steps and minibatch size
`<T extends ROCMultiClass> T`	`evaluateROCMultiClass(org.apache.spark.api.java.JavaRDD<DataSet> data)` Perform ROC analysis/evaluation (for the multi-class case, using `ROCMultiClass` on the given DataSet in a distributed manner
`<T extends ROCMultiClass> T`	`evaluateROCMultiClass(org.apache.spark.api.java.JavaRDD<DataSet> data, int thresholdSteps, int evaluationMinibatchSize)` Perform ROC analysis/evaluation (for the multi-class case, using `ROCMultiClass` on the given DataSet in a distributed manner
`<K> org.apache.spark.api.java.JavaPairRDD<K,INDArray[]>`	`feedForwardWithKey(org.apache.spark.api.java.JavaPairRDD<K,INDArray[]> featuresData, int batchSize)` Feed-forward the specified data, with the given keys. i.e., get the network output/predictions for the specified data
`<K> org.apache.spark.api.java.JavaPairRDD<K,INDArray>`	`feedForwardWithKeySingle(org.apache.spark.api.java.JavaPairRDD<K,INDArray> featuresData, int batchSize)` Feed-forward the specified data, with the given keys. i.e., get the network output/predictions for the specified data
`ComputationGraph`	`fit(org.apache.spark.api.java.JavaRDD<DataSet> rdd)` Fit the ComputationGraph with the given data set
`ComputationGraph`	`fit(org.apache.spark.rdd.RDD<DataSet> rdd)` Fit the ComputationGraph with the given data set
`ComputationGraph`	`fit(String path)` Fit the SparkComputationGraph network using a directory of serialized DataSet objects The assumption here is that the directory contains a number of `DataSet` objects, each serialized using `DataSet.save(OutputStream)`
`ComputationGraph`	`fit(String path, int minPartitions)` Deprecated. Use `fit(String)`
`ComputationGraph`	`fitMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> rdd)` Fit the ComputationGraph with the given data set
`ComputationGraph`	`fitMultiDataSet(org.apache.spark.rdd.RDD<MultiDataSet> rdd)` Fit the ComputationGraph with the given data set
`ComputationGraph`	`fitMultiDataSet(String path)` Fit the SparkComputationGraph network using a directory of serialized MultiDataSet objects The assumption here is that the directory contains a number of serialized `MultiDataSet` objects
`ComputationGraph`	`fitMultiDataSet(String path, int minPartitions)` Deprecated. use `fitMultiDataSet(String)`
`ComputationGraph`	`fitPaths(org.apache.spark.api.java.JavaRDD<String> paths)` Fit the network using a list of paths for serialized DataSet objects.
`ComputationGraph`	`fitPaths(org.apache.spark.api.java.JavaRDD<String> paths, DataSetLoader loader)`
`ComputationGraph`	`fitPaths(org.apache.spark.api.java.JavaRDD<String> paths, MultiDataSetLoader loader)`
`ComputationGraph`	`fitPathsMultiDataSet(org.apache.spark.api.java.JavaRDD<String> paths)` Fit the network using a list of paths for serialized MultiDataSet objects.
`int`	`getDefaultEvaluationWorkers()` Returns the currently set default number of evaluation workers/threads.
`ComputationGraph`	`getNetwork()`
`double`	`getScore()` Gets the last (average) minibatch score from calling fit.
`org.apache.spark.api.java.JavaSparkContext`	`getSparkContext()`
`SparkTrainingStats`	`getSparkTrainingStats()`
`TrainingMaster`	`getTrainingMaster()`
`<K> org.apache.spark.api.java.JavaPairRDD<K,Double>`	`scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,DataSet> data, boolean includeRegularizationTerms)` DataSet version of `scoreExamples(JavaPairRDD, boolean)`
`<K> org.apache.spark.api.java.JavaPairRDD<K,Double>`	`scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,DataSet> data, boolean includeRegularizationTerms, int batchSize)` DataSet version of `scoreExamples(JavaPairRDD, boolean, int)`
`org.apache.spark.api.java.JavaDoubleRDD`	`scoreExamples(org.apache.spark.api.java.JavaRDD<DataSet> data, boolean includeRegularizationTerms)` DataSet version of `scoreExamples(JavaRDD, boolean)`
`org.apache.spark.api.java.JavaDoubleRDD`	`scoreExamples(org.apache.spark.api.java.JavaRDD<DataSet> data, boolean includeRegularizationTerms, int batchSize)` DataSet version of `scoreExamples(JavaPairRDD, boolean, int)`
`<K> org.apache.spark.api.java.JavaPairRDD<K,Double>`	`scoreExamplesMultiDataSet(org.apache.spark.api.java.JavaPairRDD<K,MultiDataSet> data, boolean includeRegularizationTerms)` Score the examples individually, using the default batch size `DEFAULT_EVAL_SCORE_BATCH_SIZE`.
`<K> org.apache.spark.api.java.JavaPairRDD<K,Double>`	`scoreExamplesMultiDataSet(org.apache.spark.api.java.JavaPairRDD<K,MultiDataSet> data, boolean includeRegularizationTerms, int batchSize)` Score the examples individually, using a specified batch size.
`org.apache.spark.api.java.JavaDoubleRDD`	`scoreExamplesMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> data, boolean includeRegularizationTerms)` Score the examples individually, using the default batch size `DEFAULT_EVAL_SCORE_BATCH_SIZE`.
`org.apache.spark.api.java.JavaDoubleRDD`	`scoreExamplesMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> data, boolean includeRegularizationTerms, int batchSize)` Score the examples individually, using a specified batch size.
`void`	`setCollectTrainingStats(boolean collectTrainingStats)`
`void`	`setDefaultEvaluationWorkers(int workers)` Set the default number of evaluation workers/threads.
`void`	`setNetwork(ComputationGraph network)`
`void`	`setScore(double lastScore)`

Methods inherited from class org.deeplearning4j.spark.impl.SparkListenable
setListeners, setListeners, setListeners, setListeners

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - DEFAULT_ROC_THRESHOLD_STEPS
```
public static final int DEFAULT_ROC_THRESHOLD_STEPS
```
    See Also:
    
    Constant Field Values
  - DEFAULT_EVAL_SCORE_BATCH_SIZE
```
public static final int DEFAULT_EVAL_SCORE_BATCH_SIZE
```
    See Also:
    
    Constant Field Values
  - DEFAULT_EVAL_WORKERS
```
public static final int DEFAULT_EVAL_WORKERS
```
    See Also:
    
    Constant Field Values
- Constructor Detail
  - SparkComputationGraph
```
public SparkComputationGraph(org.apache.spark.SparkContext sparkContext,
                             ComputationGraph network,
                             TrainingMaster trainingMaster)
```
    Instantiate a ComputationGraph instance with the given context, network and training master.
    
    Parameters:
    
    sparkContext - the spark context to use
    
    network - the network to use
    
    trainingMaster - Required for training. May be null if the SparkComputationGraph is only to be used for evaluation or inference
  - SparkComputationGraph
```
public SparkComputationGraph(org.apache.spark.api.java.JavaSparkContext javaSparkContext,
                             ComputationGraph network,
                             TrainingMaster trainingMaster)
```
  - SparkComputationGraph
```
public SparkComputationGraph(org.apache.spark.SparkContext sparkContext,
                             ComputationGraphConfiguration conf,
                             TrainingMaster trainingMaster)
```
  - SparkComputationGraph
```
public SparkComputationGraph(org.apache.spark.api.java.JavaSparkContext sparkContext,
                             ComputationGraphConfiguration conf,
                             TrainingMaster trainingMaster)
```
- Method Detail
  - getSparkContext
```
public org.apache.spark.api.java.JavaSparkContext getSparkContext()
```
  - setCollectTrainingStats
```
public void setCollectTrainingStats(boolean collectTrainingStats)
```
  - getSparkTrainingStats
```
public SparkTrainingStats getSparkTrainingStats()
```
  - getNetwork
```
public ComputationGraph getNetwork()
```
    Returns:
    
    The trained ComputationGraph
  - getTrainingMaster
```
public TrainingMaster getTrainingMaster()
```
    Returns:
    
    The TrainingMaster for this network
  - setNetwork
```
public void setNetwork(ComputationGraph network)
```
    Parameters:
    
    network - The network to be used for any subsequent training, inference and evaluation steps
  - getDefaultEvaluationWorkers
```
public int getDefaultEvaluationWorkers()
```
    Returns the currently set default number of evaluation workers/threads. Note that when the number of workers is provided explicitly in an evaluation method, the default value is not used.
    In many cases, we may want this to be smaller than the number of Spark threads, to reduce memory requirements. For example, with 32 Spark threads and a large network, we don't want to spin up 32 instances of the network to perform evaluation. Better (for memory requirements, and reduced cache thrashing) to use say 4 workers.
    If it is not set explicitly, DEFAULT_EVAL_WORKERS will be used
    
    Returns:
    
    Default number of evaluation workers (threads).
  - setDefaultEvaluationWorkers
```
public void setDefaultEvaluationWorkers(int workers)
```
    Set the default number of evaluation workers/threads. Note that when the number of workers is provided explicitly in an evaluation method, the default value is not used.
    In many cases, we may want this to be smaller than the number of Spark threads, to reduce memory requirements. For example, with 32 Spark threads and a large network, we don't want to spin up 32 instances of the network to perform evaluation. Better (for memory requirements, and reduced cache thrashing) to use say 4 workers.
    If it is not set explicitly, DEFAULT_EVAL_WORKERS will be used
  - fit
```
public ComputationGraph fit(org.apache.spark.rdd.RDD<DataSet> rdd)
```
    Fit the ComputationGraph with the given data set
    
    Parameters:
    
    rdd - Data to train on
    
    Returns:
    
    Trained network
  - fit
```
public ComputationGraph fit(org.apache.spark.api.java.JavaRDD<DataSet> rdd)
```
    Fit the ComputationGraph with the given data set
    
    Parameters:
    
    rdd - Data to train on
    
    Returns:
    
    Trained network
  - fit
```
public ComputationGraph fit(String path)
```
    Fit the SparkComputationGraph network using a directory of serialized DataSet objects The assumption here is that the directory contains a number of DataSet objects, each serialized using DataSet.save(OutputStream)
    
    Parameters:
    
    path - Path to the directory containing the serialized DataSet objcets
    
    Returns:
    
    The MultiLayerNetwork after training
  - fit
```
@Deprecated
public ComputationGraph fit(String path,
                                         int minPartitions)
```
    Deprecated. Use fit(String)
  - fitPaths
```
public ComputationGraph fitPaths(org.apache.spark.api.java.JavaRDD<String> paths)
```
    Fit the network using a list of paths for serialized DataSet objects.
    
    Parameters:
    
    paths - List of paths
    
    Returns:
    
    trained network
  - fitPaths
```
public ComputationGraph fitPaths(org.apache.spark.api.java.JavaRDD<String> paths,
                                 DataSetLoader loader)
```
  - fitMultiDataSet
```
public ComputationGraph fitMultiDataSet(org.apache.spark.rdd.RDD<MultiDataSet> rdd)
```
    Fit the ComputationGraph with the given data set
    
    Parameters:
    
    rdd - Data to train on
    
    Returns:
    
    Trained network
  - fitMultiDataSet
```
public ComputationGraph fitMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> rdd)
```
    Fit the ComputationGraph with the given data set
    
    Parameters:
    
    rdd - Data to train on
    
    Returns:
    
    Trained network
  - fitMultiDataSet
```
public ComputationGraph fitMultiDataSet(String path)
```
    Fit the SparkComputationGraph network using a directory of serialized MultiDataSet objects The assumption here is that the directory contains a number of serialized MultiDataSet objects
    
    Parameters:
    
    path - Path to the directory containing the serialized MultiDataSet objcets
    
    Returns:
    
    The MultiLayerNetwork after training
  - fitPathsMultiDataSet
```
public ComputationGraph fitPathsMultiDataSet(org.apache.spark.api.java.JavaRDD<String> paths)
```
    Fit the network using a list of paths for serialized MultiDataSet objects.
    
    Parameters:
    
    paths - List of paths
    
    Returns:
    
    trained network
  - fitPaths
```
public ComputationGraph fitPaths(org.apache.spark.api.java.JavaRDD<String> paths,
                                 MultiDataSetLoader loader)
```
  - fitMultiDataSet
```
@Deprecated
public ComputationGraph fitMultiDataSet(String path,
                                                     int minPartitions)
```
    Deprecated. use fitMultiDataSet(String)
  - getScore
```
public double getScore()
```
    Gets the last (average) minibatch score from calling fit. This is the average score across all executors for the last minibatch executed in each worker
  - setScore
```
public void setScore(double lastScore)
```
  - calculateScore
```
public double calculateScore(org.apache.spark.api.java.JavaRDD<DataSet> data,
                             boolean average)
```
    Calculate the score for all examples in the provided JavaRDD<DataSet>, either by summing or averaging over the entire data set. To calculate a score for each example individually, use scoreExamples(JavaPairRDD, boolean) or one of the similar methods. Uses default minibatch size in each worker, DEFAULT_EVAL_SCORE_BATCH_SIZE
    
    Parameters:
    
    data - Data to score
    
    average - Whether to sum the scores, or average them
  - calculateScore
```
public double calculateScore(org.apache.spark.api.java.JavaRDD<DataSet> data,
                             boolean average,
                             int minibatchSize)
```
    Calculate the score for all examples in the provided JavaRDD<DataSet>, either by summing or averaging over the entire data set. To calculate a score for each example individually, use scoreExamples(JavaPairRDD, boolean) or one of the similar methods
    
    Parameters:
    
    data - Data to score
    
    average - Whether to sum the scores, or average them
    
    minibatchSize - The number of examples to use in each minibatch when scoring. If more examples are in a partition than this, multiple scoring operations will be done (to avoid using too much memory by doing the whole partition in one go)
  - calculateScoreMultiDataSet
```
public double calculateScoreMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> data,
                                         boolean average)
```
    Calculate the score for all examples in the provided JavaRDD<MultiDataSet>, either by summing or averaging over the entire data set. Uses default minibatch size in each worker, DEFAULT_EVAL_SCORE_BATCH_SIZE
    
    Parameters:
    
    data - Data to score
    
    average - Whether to sum the scores, or average them
  - calculateScoreMultiDataSet
```
public double calculateScoreMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> data,
                                         boolean average,
                                         int minibatchSize)
```
    Calculate the score for all examples in the provided JavaRDD<MultiDataSet>, either by summing or averaging over the entire data set. *
    
    Parameters:
    
    data - Data to score
    
    average - Whether to sum the scores, or average them
    
    minibatchSize - The number of examples to use in each minibatch when scoring. If more examples are in a partition than this, multiple scoring operations will be done (to avoid using too much memory by doing the whole partition in one go)
  - scoreExamples
```
public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                                             boolean includeRegularizationTerms)
```
    DataSet version of scoreExamples(JavaRDD, boolean)
  - scoreExamples
```
public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                                             boolean includeRegularizationTerms,
                                                             int batchSize)
```
    DataSet version of scoreExamples(JavaPairRDD, boolean, int)
  - scoreExamples
```
public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,DataSet> data,
                                                                         boolean includeRegularizationTerms)
```
    DataSet version of scoreExamples(JavaPairRDD, boolean)
  - scoreExamples
```
public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,DataSet> data,
                                                                         boolean includeRegularizationTerms,
                                                                         int batchSize)
```
    DataSet version of scoreExamples(JavaPairRDD, boolean, int)
  - scoreExamplesMultiDataSet
```
public org.apache.spark.api.java.JavaDoubleRDD scoreExamplesMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> data,
                                                                         boolean includeRegularizationTerms)
```
    Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately. If scoring is needed for specific examples use either scoreExamples(JavaPairRDD, boolean) or scoreExamples(JavaPairRDD, boolean, int) which can have a key for each example.
    
    Parameters:
    
    data - Data to score
    
    includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)
    
    Returns:
    
    A JavaDoubleRDD containing the scores of each example
    
    See Also:
    
    ComputationGraph.scoreExamples(MultiDataSet, boolean)
  - scoreExamplesMultiDataSet
```
public org.apache.spark.api.java.JavaDoubleRDD scoreExamplesMultiDataSet(org.apache.spark.api.java.JavaRDD<MultiDataSet> data,
                                                                         boolean includeRegularizationTerms,
                                                                         int batchSize)
```
    Score the examples individually, using a specified batch size. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately. If scoring is needed for specific examples use either scoreExamples(JavaPairRDD, boolean) or scoreExamples(JavaPairRDD, boolean, int) which can have a key for each example.
    
    Parameters:
    
    data - Data to score
    
    includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)
    
    batchSize - Batch size to use when doing scoring
    
    Returns:
    
    A JavaDoubleRDD containing the scores of each example
    
    See Also:
    
    ComputationGraph.scoreExamples(MultiDataSet, boolean)
  - scoreExamplesMultiDataSet
```
public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamplesMultiDataSet(org.apache.spark.api.java.JavaPairRDD<K,MultiDataSet> data,
                                                                                     boolean includeRegularizationTerms)
```
    Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately
    Note: The provided JavaPairRDD has a key that is associated with each example and returned score.
    Note: The DataSet objects passed in must have exactly one example in them (otherwise: can't have a 1:1 association between keys and data sets to score)
    
    Type Parameters:
    
    K - Key type
    
    Parameters:
    
    data - Data to score
    
    includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)
    
    Returns:
    
    A JavaPairRDD<K,Double> containing the scores of each example
    
    See Also:
    
    MultiLayerNetwork.scoreExamples(DataSet, boolean)
  - feedForwardWithKeySingle
```
public <K> org.apache.spark.api.java.JavaPairRDD<K,INDArray> feedForwardWithKeySingle(org.apache.spark.api.java.JavaPairRDD<K,INDArray> featuresData,
                                                                                      int batchSize)
```
    Feed-forward the specified data, with the given keys. i.e., get the network output/predictions for the specified data
    
    Type Parameters:
    
    K - Type of data for key - may be anything
    
    Parameters:
    
    featuresData - Features data to feed through the network
    
    batchSize - Batch size to use when doing feed forward operations
    
    Returns:
    
    Network output given the input, by key
  - feedForwardWithKey
```
public <K> org.apache.spark.api.java.JavaPairRDD<K,INDArray[]> feedForwardWithKey(org.apache.spark.api.java.JavaPairRDD<K,INDArray[]> featuresData,
                                                                                  int batchSize)
```
    Feed-forward the specified data, with the given keys. i.e., get the network output/predictions for the specified data
    
    Type Parameters:
    
    K - Type of data for key - may be anything
    
    Parameters:
    
    featuresData - Features data to feed through the network
    
    batchSize - Batch size to use when doing feed forward operations
    
    Returns:
    
    Network output given the input, by key
  - scoreExamplesMultiDataSet
```
public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamplesMultiDataSet(org.apache.spark.api.java.JavaPairRDD<K,MultiDataSet> data,
                                                                                     boolean includeRegularizationTerms,
                                                                                     int batchSize)
```
    Score the examples individually, using a specified batch size. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately
    Note: The provided JavaPairRDD has a key that is associated with each example and returned score.
    Note: The DataSet objects passed in must have exactly one example in them (otherwise: can't have a 1:1 association between keys and data sets to score)
    
    Type Parameters:
    
    K - Key type
    
    Parameters:
    
    data - Data to score
    
    includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)
    
    Returns:
    
    A JavaPairRDD<K,Double> containing the scores of each example
    
    See Also:
    
    MultiLayerNetwork.scoreExamples(DataSet, boolean)
  - evaluate
```
public Evaluation evaluate(String path,
                           DataSetLoader loader)
```
    Evaluate the single-output network on a directory containing a set of DataSet objects to be loaded with a DataSetLoader. Uses default batch size of DEFAULT_EVAL_SCORE_BATCH_SIZE
    
    Parameters:
    
    path - Path/URI to the directory containing the datasets to load
    
    Returns:
    
    Evaluation
  - evaluate
```
public Evaluation evaluate(String path,
                           MultiDataSetLoader loader)
```
    Evaluate the single-output network on a directory containing a set of MultiDataSet objects to be loaded with a MultiDataSetLoader. Uses default batch size of DEFAULT_EVAL_SCORE_BATCH_SIZE
    
    Parameters:
    
    path - Path/URI to the directory containing the datasets to load
    
    Returns:
    
    Evaluation
  - evaluate
```
public <T extends Evaluation> T evaluate(org.apache.spark.rdd.RDD<DataSet> data)
```
    RDD<DataSet> overload of evaluate(JavaRDD)
  - evaluate
```
public <T extends Evaluation> T evaluate(org.apache.spark.api.java.JavaRDD<DataSet> data)
```
    Evaluate the network (classification performance) in a distributed manner on the provided data
    
    Parameters:
    
    data - Data to evaluate on
    
    Returns:
    
    Evaluation object; results of evaluation on all examples in the data set
  - evaluate
```
public <T extends Evaluation> T evaluate(org.apache.spark.rdd.RDD<DataSet> data,
                                         List<String> labelsList)
```
    RDD<DataSet> overload of evaluate(JavaRDD, List)
  - evaluateRegression
```
public <T extends RegressionEvaluation> T evaluateRegression(org.apache.spark.api.java.JavaRDD<DataSet> data)
```
    Evaluate the network (regression performance) in a distributed manner on the provided data
    
    Parameters:
    
    data - Data to evaluate
    
    Returns:
    
    RegressionEvaluation instance with regression performance
  - evaluateRegression
```
public <T extends RegressionEvaluation> T evaluateRegression(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                                             int minibatchSize)
```
    Evaluate the network (regression performance) in a distributed manner on the provided data
    
    Parameters:
    
    data - Data to evaluate
    
    minibatchSize - Minibatch size to use when doing performing evaluation
    
    Returns:
    
    RegressionEvaluation instance with regression performance
  - evaluate
```
public <T extends Evaluation> T evaluate(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                         List<String> labelsList)
```
    Evaluate the network (classification performance) in a distributed manner, using default batch size and a provided list of labels
    
    Parameters:
    
    data - Data to evaluate on
    
    labelsList - List of labels used for evaluation
    
    Returns:
    
    Evaluation object; results of evaluation on all examples in the data set
  - evaluateROC
```
public <T extends ROC> T evaluateROC(org.apache.spark.api.java.JavaRDD<DataSet> data)
```
    Perform ROC analysis/evaluation on the given DataSet in a distributed manner, using the default number of threshold steps (DEFAULT_ROC_THRESHOLD_STEPS) and the default minibatch size (DEFAULT_EVAL_SCORE_BATCH_SIZE)
    
    Parameters:
    
    data - Test set data (to evaluate on)
    
    Returns:
    
    ROC for the entire data set
  - evaluateROC
```
public <T extends ROC> T evaluateROC(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                     int thresholdSteps,
                                     int evaluationMinibatchSize)
```
    Perform ROC analysis/evaluation on the given DataSet in a distributed manner
    
    Parameters:
    
    data - Test set data (to evaluate on)
    
    thresholdSteps - Number of threshold steps for ROC - see ROC
    
    evaluationMinibatchSize - Minibatch size to use when performing ROC evaluation
    
    Returns:
    
    ROC for the entire data set
  - evaluateROCMultiClass
```
public <T extends ROCMultiClass> T evaluateROCMultiClass(org.apache.spark.api.java.JavaRDD<DataSet> data)
```
    Perform ROC analysis/evaluation (for the multi-class case, using ROCMultiClass on the given DataSet in a distributed manner
    
    Parameters:
    
    data - Test set data (to evaluate on)
    
    Returns:
    
    ROC for the entire data set
  - evaluateROCMultiClass
```
public <T extends ROCMultiClass> T evaluateROCMultiClass(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                                         int thresholdSteps,
                                                         int evaluationMinibatchSize)
```
    Perform ROC analysis/evaluation (for the multi-class case, using ROCMultiClass on the given DataSet in a distributed manner
    
    Parameters:
    
    data - Test set data (to evaluate on)
    
    thresholdSteps - Number of threshold steps for ROC - see ROC
    
    evaluationMinibatchSize - Minibatch size to use when performing ROC evaluation
    
    Returns:
    
    ROCMultiClass for the entire data set
  - evaluate
```
public <T extends Evaluation> T evaluate(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                         List<String> labelsList,
                                         int evalBatchSize)
```
    Evaluate the network (classification performance) in a distributed manner, using specified batch size and a provided list of labels
    
    Parameters:
    
    data - Data to evaluate on
    
    labelsList - List of labels used for evaluation
    
    evalBatchSize - Batch size to use when conducting evaluations
    
    Returns:
    
    Evaluation object; results of evaluation on all examples in the data set
  - evaluateMDS
```
public <T extends Evaluation> T evaluateMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data)
```
    Evaluate the network (classification performance) in a distributed manner on the provided data
  - evaluateMDS
```
public <T extends Evaluation> T evaluateMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data,
                                            int minibatchSize)
```
    Evaluate the network (classification performance) in a distributed manner on the provided data
  - evaluateRegressionMDS
```
public <T extends RegressionEvaluation> T evaluateRegressionMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data)
```
    Evaluate the network (regression performance) in a distributed manner on the provided data
    
    Parameters:
    
    data - Data to evaluate
    
    Returns:
    
    RegressionEvaluation instance with regression performance
  - evaluateRegressionMDS
```
public <T extends RegressionEvaluation> T evaluateRegressionMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data,
                                                                int minibatchSize)
```
    Evaluate the network (regression performance) in a distributed manner on the provided data
    
    Parameters:
    
    data - Data to evaluate
    
    minibatchSize - Minibatch size to use when doing performing evaluation
    
    Returns:
    
    RegressionEvaluation instance with regression performance
  - evaluateROCMDS
```
public ROC evaluateROCMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data)
```
    Perform ROC analysis/evaluation on the given DataSet in a distributed manner, using the default number of threshold steps (DEFAULT_ROC_THRESHOLD_STEPS) and the default minibatch size (DEFAULT_EVAL_SCORE_BATCH_SIZE)
    
    Parameters:
    
    data - Test set data (to evaluate on)
    
    Returns:
    
    ROC for the entire data set
  - evaluateROCMDS
```
public <T extends ROC> T evaluateROCMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data,
                                        int rocThresholdNumSteps,
                                        int minibatchSize)
```
    Perform ROC analysis/evaluation on the given DataSet in a distributed manner, using the specified number of steps and minibatch size
    
    Parameters:
    
    data - Test set data (to evaluate on)
    
    rocThresholdNumSteps - See ROC for details
    
    minibatchSize - Minibatch size for evaluation
    
    Returns:
    
    ROC for the entire data set
  - doEvaluation
```
public <T extends IEvaluation> T doEvaluation(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                              T emptyEvaluation,
                                              int evalBatchSize)
```
    Perform distributed evaluation of any type of IEvaluation. For example, Evaluation, RegressionEvaluation, ROC, ROCMultiClass etc.
    
    Type Parameters:
    
    T - Type of evaluation instance to return
    
    Parameters:
    
    data - Data to evaluate on
    
    emptyEvaluation - Empty evaluation instance. This is the starting point (serialized/duplicated, then merged)
    
    evalBatchSize - Evaluation batch size
    
    Returns:
    
    IEvaluation instance
  - doEvaluation
```
public <T extends IEvaluation> T[] doEvaluation(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                                int evalBatchSize,
                                                T... emptyEvaluations)
```
    Perform distributed evaluation on a single output ComputationGraph form DataSet objects using Spark. Can be used to perform multiple evaluations on this single output (for example, Evaluation and ROC) at the same time.
    Note that the default number of worker threads getDefaultEvaluationWorkers() will be used
    
    Parameters:
    
    data - Data to evaluatie
    
    evalBatchSize - Minibatch size for evaluation
    
    emptyEvaluations - Evaluations to perform
    
    Returns:
    
    Evaluations
  - doEvaluation
```
public <T extends IEvaluation> T[] doEvaluation(org.apache.spark.api.java.JavaRDD<DataSet> data,
                                                int evalNumWorkers,
                                                int evalBatchSize,
                                                T... emptyEvaluations)
```
    Perform distributed evaluation on a single output ComputationGraph form DataSet objects using Spark. Can be used to perform multiple evaluations on this single output (for example, Evaluation and ROC) at the same time.
    
    Parameters:
    
    data - Data to evaluatie
    
    evalNumWorkers - Number of worker threads (per machine) to use for evaluation. May want tis to be less than the number of Spark threads per machine/JVM to reduce memory requirements
    
    evalBatchSize - Minibatch size for evaluation
    
    emptyEvaluations - Evaluations to perform
    
    Returns:
    
    Evaluations
  - doEvaluationMDS
```
public <T extends IEvaluation> T[] doEvaluationMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data,
                                                   int evalBatchSize,
                                                   T... emptyEvaluations)
```
    Perform distributed evaluation on a single output ComputationGraph form MultiDataSet objects using Spark. Can be used to perform multiple evaluations on this single output (for example, Evaluation and ROC) at the same time.
    
    Parameters:
    
    data - Data to evaluatie
    
    evalBatchSize - Minibatch size for evaluation
    
    emptyEvaluations - Evaluations to perform
    
    Returns:
    
    Evaluations
  - doEvaluationMDS
```
public <T extends IEvaluation> T[] doEvaluationMDS(org.apache.spark.api.java.JavaRDD<MultiDataSet> data,
                                                   int evalNumWorkers,
                                                   int evalBatchSize,
                                                   T... emptyEvaluations)
```
  - doEvaluation
```
public IEvaluation[] doEvaluation(org.apache.spark.api.java.JavaRDD<String> data,
                                  DataSetLoader loader,
                                  IEvaluation... emptyEvaluations)
```
    Perform evaluation on serialized DataSet objects on disk, (potentially in any format), that are loaded using an DataSetLoader.
    Uses the default number of workers (model replicas per JVM) of DEFAULT_EVAL_WORKERS with the default minibatch size of DEFAULT_EVAL_SCORE_BATCH_SIZE
    
    Parameters:
    
    data - List of paths to the data (that can be loaded as / converted to DataSets)
    
    loader - Used to load DataSets from their paths
    
    emptyEvaluations - Evaluations to perform
    
    Returns:
    
    Evaluation
  - doEvaluation
```
public IEvaluation[] doEvaluation(org.apache.spark.api.java.JavaRDD<String> data,
                                  int evalNumWorkers,
                                  int evalBatchSize,
                                  DataSetLoader loader,
                                  IEvaluation... emptyEvaluations)
```
    Perform evaluation on serialized DataSet objects on disk, (potentially in any format), that are loaded using an DataSetLoader.
    
    Parameters:
    
    data - List of paths to the data (that can be loaded as / converted to DataSets)
    
    evalNumWorkers - Number of workers to perform evaluation with. To reduce memory requirements and cache thrashing, it is common to set this to a lower value than the number of spark threads per JVM/executor
    
    evalBatchSize - Batch size to use when performing evaluation
    
    loader - Used to load DataSets from their paths
    
    emptyEvaluations - Evaluations to perform
    
    Returns:
    
    Evaluation
  - doEvaluation
```
public IEvaluation[] doEvaluation(org.apache.spark.api.java.JavaRDD<String> data,
                                  MultiDataSetLoader loader,
                                  IEvaluation... emptyEvaluations)
```
    Perform evaluation on serialized MultiDataSet objects on disk, (potentially in any format), that are loaded using an MultiDataSetLoader.
    Uses the default number of workers (model replicas per JVM) of DEFAULT_EVAL_WORKERS with the default minibatch size of DEFAULT_EVAL_SCORE_BATCH_SIZE
    
    Parameters:
    
    data - List of paths to the data (that can be loaded as / converted to DataSets)
    
    loader - Used to load MultiDataSets from their paths
    
    emptyEvaluations - Evaluations to perform
    
    Returns:
    
    Evaluation
  - doEvaluation
```
public IEvaluation[] doEvaluation(org.apache.spark.api.java.JavaRDD<String> data,
                                  int evalNumWorkers,
                                  int evalBatchSize,
                                  MultiDataSetLoader loader,
                                  IEvaluation... emptyEvaluations)
```
    Perform evaluation on serialized MultiDataSet objects on disk, (potentially in any format), that are loaded using an MultiDataSetLoader
    
    Parameters:
    
    data - List of paths to the data (that can be loaded as / converted to DataSets)
    
    evalNumWorkers - Number of workers to perform evaluation with. To reduce memory requirements and cache thrashing, it is common to set this to a lower value than the number of spark threads per JVM/executor
    
    evalBatchSize - Batch size to use when performing evaluation
    
    loader - Used to load MultiDataSets from their paths
    
    emptyEvaluations - Evaluations to perform
    
    Returns:
    
    Evaluation
  - doEvaluation
```
protected IEvaluation[] doEvaluation(org.apache.spark.api.java.JavaRDD<String> data,
                                     int evalNumWorkers,
                                     int evalBatchSize,
                                     DataSetLoader loader,
                                     MultiDataSetLoader mdsLoader,
                                     IEvaluation... emptyEvaluations)
```

Class SparkComputationGraph

Field Summary

Fields inherited from class org.deeplearning4j.spark.impl.SparkListenable

Constructor Summary

Method Summary

Methods inherited from class org.deeplearning4j.spark.impl.SparkListenable

Methods inherited from class java.lang.Object

Field Detail

DEFAULT_ROC_THRESHOLD_STEPS

DEFAULT_EVAL_SCORE_BATCH_SIZE

DEFAULT_EVAL_WORKERS

Constructor Detail

SparkComputationGraph

SparkComputationGraph

SparkComputationGraph

SparkComputationGraph

Method Detail

getSparkContext

setCollectTrainingStats

getSparkTrainingStats

getNetwork

getTrainingMaster

setNetwork

getDefaultEvaluationWorkers

setDefaultEvaluationWorkers

fit

fit

fit

fit

fitPaths

fitPaths

fitMultiDataSet

fitMultiDataSet

fitMultiDataSet

fitPathsMultiDataSet

fitPaths

fitMultiDataSet

getScore

setScore

calculateScore

calculateScore

calculateScoreMultiDataSet

calculateScoreMultiDataSet

scoreExamples

scoreExamples

scoreExamples

scoreExamples

scoreExamplesMultiDataSet

scoreExamplesMultiDataSet

scoreExamplesMultiDataSet

feedForwardWithKeySingle

feedForwardWithKey

scoreExamplesMultiDataSet

evaluate

evaluate

evaluate

evaluate

evaluate

evaluateRegression

evaluateRegression

evaluate

evaluateROC

evaluateROC

evaluateROCMultiClass

evaluateROCMultiClass

evaluate

evaluateMDS

evaluateMDS

evaluateRegressionMDS

evaluateRegressionMDS

evaluateROCMDS

evaluateROCMDS

doEvaluation

doEvaluation

doEvaluation

doEvaluationMDS

doEvaluationMDS

doEvaluation

doEvaluation

doEvaluation