public interface SparkTrainingStats extends Serializable
TrainingMaster
implementations.
The idea is that for debugging purposes, we want to collect a number of statistics related to the training. However, these statistics will vary, depending on which the type of training we are doing. Specifically, both the keys (number/names of stats) and their actual values (types/classes) can vary.
The interface here operates essentially as a Map<String,Object>
. Note however that SparkTrainingStats instances
may be nested: for example a ParameterAveragingTrainingMasterStats
may have a
CommonSparkTrainingStats
instance which may in turn have a ParameterAveragingTrainingWorkerStats
instance.
Modifier and Type | Field and Description |
---|---|
static String |
DEFAULT_PRINT_FORMAT
Default formatter used for
statsAsString() |
static int |
PRINT_INDENT
Default indentation for
statsAsString() |
Modifier and Type | Method and Description |
---|---|
void |
addOtherTrainingStats(SparkTrainingStats other)
Combine the two training stats instances.
|
boolean |
defaultIncludeInPlots(String key)
When plotting statistics, we don't necessarily want to plot everything.
|
void |
exportStatFiles(String outputPath,
org.apache.spark.SparkContext sc)
Export the stats as a collection of files.
|
Set<String> |
getKeySet() |
SparkTrainingStats |
getNestedTrainingStats()
Return the nested training stats - if any.
|
String |
getShortNameForKey(String key)
Return a short (display) name for the given key.
|
List<EventStats> |
getValue(String key)
Get the statistic value for this key
|
String |
statsAsString()
Get a String representation of the stats.
|
static final int PRINT_INDENT
statsAsString()
static final String DEFAULT_PRINT_FORMAT
statsAsString()
Set<String> getKeySet()
getValue(String)
List<EventStats> getValue(String key)
key
- Key to get the value forString getShortNameForKey(String key)
key
- Keyboolean defaultIncludeInPlots(String key)
key
- Key to check for default plotting behaviourvoid addOtherTrainingStats(SparkTrainingStats other)
other
- Other training stats to returnSparkTrainingStats getNestedTrainingStats()
String statsAsString()
getValue(String)
or export full data via exportStatFiles(String, SparkContext)
void exportStatFiles(String outputPath, org.apache.spark.SparkContext sc) throws IOException
outputPath
- Base directory to write files toIOException
Copyright © 2020. All rights reserved.