【SparkAPI JAVA版】JavaPairRDD——cartesian（三）

本文主要是介绍【SparkAPI JAVA版】JavaPairRDD——cartesian（三），希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

JavaPairRDD的cartesian方法讲解

官方文档说明

Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in `this` and b is in `other`.

中文含义

该函数返回的是Pair类型的RDD，计算结果是当前RDD和other RDD中每个元素进行笛卡儿计算的结果。最后返回的是CartesianRDD。

方法原型

//scala
def cartesian[U: ClassTag](other: RDD[U]): RDD[(T, U)]
//java
public static <U> JavaPairRDD<T,U> cartesian(JavaRDDLike<U,?> other)

实例

public class Cartesian {public static void main(String[] args) {System.setProperty("hadoop.home.dir","F:\\hadoop-2.7.1");SparkConf conf = new SparkConf().setMaster("local").setAppName("TestSpark");JavaSparkContext sc = new JavaSparkContext(conf);//JavaPairRDD cartesian示例JavaPairRDD<Integer,Integer> javaPairRDD1 = sc.parallelizePairs(Lists.newArrayList(new Tuple2<Integer, Integer>(1,11),new Tuple2<Integer, Integer>(2,22),new Tuple2<Integer, Integer>(3,33)));JavaPairRDD<Integer,Integer> javaPairRDD2 = sc.parallelizePairs(Lists.newArrayList(new Tuple2<Integer, Integer>(7,71),new Tuple2<Integer, Integer>(8,82),new Tuple2<Integer, Integer>(9,93)));JavaPairRDD<Tuple2<Integer, Integer>, Tuple2<Integer, Integer>> javaPairRDD3 = javaPairRDD1.cartesian(javaPairRDD2);javaPairRDD3.foreach(new VoidFunction<Tuple2<Tuple2<Integer, Integer>, Tuple2<Integer, Integer>>>() {public void call(Tuple2<Tuple2<Integer, Integer>, Tuple2<Integer, Integer>> tuple2Tuple2Tuple2) throws Exception {System.out.println(tuple2Tuple2Tuple2);/*System.out.print("key:"+tuple2Tuple2Tuple2._1+",");System.out.println("value:"+tuple2Tuple2Tuple2._2);*/}});//JavaRDD cartesian示例JavaRDD<Integer> javaRDD1 = sc.parallelize(Lists.newArrayList(1,2,3));JavaRDD<Integer> javaRDD2 = sc.parallelize(Lists.newArrayList(7,8,9));JavaPairRDD<Integer,Integer> javaPairRDD = javaRDD1.cartesian(javaRDD2);javaPairRDD.foreach(new VoidFunction<Tuple2<Integer, Integer>>() {public void call(Tuple2<Integer, Integer> integerIntegerTuple2) throws Exception {System.out.println(integerIntegerTuple2);}});}
}

结果

//JavaPairRDD cartesian示例
((1,11),(7,71))
((1,11),(8,82))
((1,11),(9,93))
((2,22),(7,71))
((2,22),(8,82))
((2,22),(9,93))
((3,33),(7,71))
((3,33),(8,82))
((3,33),(9,93))
19/03/03 23:37:52 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 751 bytes result sent to driver
//JavaRDD cartesian示例
(1,7)
(1,8)
(1,9)
(2,7)
(2,8)
(2,9)
(3,7)
(3,8)
(3,9)

注意：计算笛卡尔积很消耗内存，不要在大量数据的时候随便使用。

这篇关于【SparkAPI JAVA版】JavaPairRDD——cartesian（三）的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---