Spark对多HDFS集群Namenode HA的支持

本文主要是介绍Spark对多HDFS集群Namenode HA的支持，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

具体的配置需要参考core-site.xml和hdfs-site.xml

val sc = new SparkContext()
// 多个HDFS的相同配置
sc.hadoopConfiguration.setStrings("fs.defaultFS", "hdfs://cluster1", "hdfs://cluster2");
sc.hadoopConfiguration.setStrings("dfs.nameservices", "cluster1", "cluster2");
// cluster1的配置
sc.hadoopConfiguration.set("dfs.ha.namenodes.cluster1", "nn1,nn2");
sc.hadoopConfiguration.set("dfs.namenode.rpc-address.cluster1.nn1", "namenode001:8020");
sc.hadoopConfiguration.set("dfs.namenode.rpc-address.cluster1.nn2", "namenode002:8020");
sc.hadoopConfiguration.set("dfs.client.failover.proxy.provider.cluster1", "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
// cluster2的配置
sc.hadoopConfiguration.set("dfs.ha.namenodes.cluster2", "nn3,nn4");
sc.hadoopConfiguration.set("dfs.namenode.rpc-address.cluster2.nn3", "namenode003:8020");
sc.hadoopConfiguration.set("dfs.namenode.rpc-address.cluster2.nn4", "namenode004:8020");
sc.hadoopConfiguration.set("dfs.client.failover.proxy.provider.cluster2", "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");

这篇关于Spark对多HDFS集群Namenode HA的支持的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---