Modifier and Type | Method and Description |
---|---|
static org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> |
BroadcastHadoopConfigHolder.get(org.apache.spark.api.java.JavaSparkContext sc) |
Constructor and Description |
---|
ExecuteWorkerPathFlatMap(TrainingWorker<R> worker,
DataSetLoader dataSetLoader,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> hadoopConfig) |
ExecuteWorkerPathMDSFlatMap(TrainingWorker<R> worker,
MultiDataSetLoader loader,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> hadoopConfig) |
Constructor and Description |
---|
BatchAndExportDataSetsFunction(int minibatchSize,
String exportBaseDirectory,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
BatchAndExportMultiDataSetsFunction(int minibatchSize,
String exportBaseDirectory,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
DataSetExportFunction(URI outputDir,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
MultiDataSetExportFunction(URI outputDir,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
PathToDataSetFunction(org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
PathToMultiDataSetFunction(org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
Constructor and Description |
---|
RemoteFileSourceFactory(org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
Constructor and Description |
---|
StringToDataSetExportFunction(URI outputDir,
RecordReader recordReader,
int batchSize,
boolean regression,
int labelIndex,
int numPossibleLabels,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
Modifier and Type | Field and Description |
---|---|
protected org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> |
IEvaluateMDSPathsFlatMapFunction.conf |
Constructor and Description |
---|
IEvaluateMDSPathsFlatMapFunction(org.apache.spark.broadcast.Broadcast<String> json,
org.apache.spark.broadcast.Broadcast<byte[]> params,
int evalNumWorkers,
int evalBatchSize,
DataSetLoader dsLoader,
MultiDataSetLoader mdsLoader,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration,
IEvaluation[] evaluations) |
Modifier and Type | Field and Description |
---|---|
protected org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> |
BaseTrainingMaster.broadcastHadoopConfig |
Constructor and Description |
---|
PathSparkDataSetIterator(Collection<String> dataSetStreams,
DataSetLoader dataSetLoader,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> hadoopConfig) |
PathSparkDataSetIterator(Iterator<String> iter,
DataSetLoader dataSetLoader,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> hadoopConfig) |
PathSparkMultiDataSetIterator(Collection<String> dataSetStreams,
MultiDataSetLoader loader,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> hadoopConfig) |
PathSparkMultiDataSetIterator(Iterator<String> iter,
MultiDataSetLoader loader,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> hadoopConfig) |
Modifier and Type | Field and Description |
---|---|
protected org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> |
SharedFlatMapPaths.hadoopConfig |
protected org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> |
SharedFlatMapPathsMDS.hadoopConfig |
Constructor and Description |
---|
SharedFlatMapPaths(TrainingWorker<R> worker,
DataSetLoader loader,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> hadoopConfig) |
SharedFlatMapPathsMDS(TrainingWorker<R> worker,
MultiDataSetLoader loader,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> hadoopConfig) |
Constructor and Description |
---|
ValidateDataSetFn(boolean deleteInvalid,
int[] featuresShape,
int[] labelsShape,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
ValidateMultiDataSetFn(boolean deleteInvalid,
int numFeatures,
int numLabels,
List<int[]> featuresShape,
List<int[]> labelsShape,
org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> configuration) |
Copyright © 2020. All rights reserved.