SparkTrainingStats

All Superinterfaces:

java.io.Serializable

All Known Implementing Classes:

CommonSparkTrainingStats, ParameterAveragingTrainingMasterStats, ParameterAveragingTrainingWorkerStats
```
public interface SparkTrainingStats
extends java.io.Serializable
```
SparkTrainingStats is an interface that is used for accessing training statistics, for multiple TrainingMaster implementations.
The idea is that for debugging purposes, we want to collect a number of statistics related to the training. However, these statistics will vary, depending on which the type of training we are doing. Specifically, both the keys (number/names of stats) and their actual values (types/classes) can vary.
The interface here operates essentially as a Map<String,Object>. Note however that SparkTrainingStats instances may be nested: for example a ParameterAveragingTrainingMasterStats may have a CommonSparkTrainingStats instance which may in turn have a ParameterAveragingTrainingWorkerStats instance.

Field Summary

Fields
Modifier and Type	Field and Description
`static java.lang.String`	`DEFAULT_PRINT_FORMAT` Default formatter used for `statsAsString()`
`static int`	`PRINT_INDENT` Default indentation for `statsAsString()`

Method Summary

All Methods Instance Methods Abstract Methods
Modifier and Type	Method and Description
`void`	`addOtherTrainingStats(SparkTrainingStats other)` Combine the two training stats instances.
`boolean`	`defaultIncludeInPlots(java.lang.String key)` When plotting statistics, we don't necessarily want to plot everything.
`void`	`exportStatFiles(java.lang.String outputPath, org.apache.spark.SparkContext sc)` Export the stats as a collection of files.
`java.util.Set<java.lang.String>`	`getKeySet()`
`SparkTrainingStats`	`getNestedTrainingStats()` Return the nested training stats - if any.
`java.lang.String`	`getShortNameForKey(java.lang.String key)` Return a short (display) name for the given key.
`java.util.List<EventStats>`	`getValue(java.lang.String key)` Get the statistic value for this key
`java.lang.String`	`statsAsString()` Get a String representation of the stats.

- Field Detail
  - PRINT_INDENT
```
static final int PRINT_INDENT
```
    Default indentation for statsAsString()
    
    See Also:
    
    Constant Field Values
  - DEFAULT_PRINT_FORMAT
```
static final java.lang.String DEFAULT_PRINT_FORMAT
```
    Default formatter used for statsAsString()
    
    See Also:
    
    Constant Field Values
- Method Detail
  - getKeySet
```
java.util.Set<java.lang.String> getKeySet()
```
    Returns:
    
    Set of keys that can be used with getValue(String)
  - getValue
```
java.util.List<EventStats> getValue(java.lang.String key)
```
    Get the statistic value for this key
    
    Parameters:
    
    key - Key to get the value for
    
    Returns:
    
    Statistic for this key, or an exception if key is invalid
  - getShortNameForKey
```
java.lang.String getShortNameForKey(java.lang.String key)
```
    Return a short (display) name for the given key.
    
    Parameters:
    
    key - Key
    
    Returns:
    
    Short/display name for key
  - defaultIncludeInPlots
```
boolean defaultIncludeInPlots(java.lang.String key)
```
    When plotting statistics, we don't necessarily want to plot everything. For example, some statistics/measurements are made up multiple smaller components; it does not always make sense to plot both the larger stat, and the components that make it up
    
    Parameters:
    
    key - Key to check for default plotting behaviour
    
    Returns:
    
    Whether the specified key should be included in plots by default or not
  - addOtherTrainingStats
```
void addOtherTrainingStats(SparkTrainingStats other)
```
    Combine the two training stats instances. Usually, the two objects must be of the same type
    
    Parameters:
    
    other - Other training stats to return
  - getNestedTrainingStats
```
SparkTrainingStats getNestedTrainingStats()
```
    Return the nested training stats - if any.
    
    Returns:
    
    The nested stats, if present/applicable, or null otherwise
  - statsAsString
```
java.lang.String statsAsString()
```
    Get a String representation of the stats. This functionality is implemented as a separate method (as opposed to toString()) as the resulting String can be very large.
    NOTE: The String representation typically includes only duration information. To get full statistics (including machine IDs, etc) use getValue(String) or export full data via exportStatFiles(String, SparkContext)
    
    Returns:
    
    A String representation of the training statistics
  - exportStatFiles
```
void exportStatFiles(java.lang.String outputPath,
                     org.apache.spark.SparkContext sc)
              throws java.io.IOException
```
    Export the stats as a collection of files. Stats are comma-delimited (CSV) with 1 header line
    
    Parameters:
    
    outputPath - Base directory to write files to
    
    Throws:
    
    java.io.IOException

Interface SparkTrainingStats

Field Summary

Method Summary

Field Detail

PRINT_INDENT

DEFAULT_PRINT_FORMAT

Method Detail

getKeySet

getValue

getShortNameForKey

defaultIncludeInPlots

addOtherTrainingStats

getNestedTrainingStats

statsAsString

exportStatFiles