public class BroadcastHadoopConfigHolder extends Object
Broadcast<SerializableHadoopConfig>for re-use across multiple places. The idea is that we often need spark configuration available for reading from (for example) HDFS directly, but Hadoop's Configuration class is not serializable (hence using
SerializableHadoopConfig; we also don't want to have multiple copies of this in memory on each worker (Hadoop Configuration is immutable).
|Modifier and Type||Method and Description|
public static org.apache.spark.broadcast.Broadcast<SerializableHadoopConfig> get(org.apache.spark.api.java.JavaSparkContext sc)
Copyright © 2020. All rights reserved.