2.2.3 hadoop体系之离线计算-mapreduce分布式计算-MapReduce分区

本文主要是介绍2.2.3 hadoop体系之离线计算-mapreduce分布式计算-MapReduce分区，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

1.复习MapReduce的8个步骤

2.MapReduce中的分区

2.1 新需求：需要分别reduce

3.IDEA实现分区-JAVA

3.1 实现自定义Partitioner-MyPartitioner.java

3.2 设置主函数：定义分区+设置分区数目

3.3 打成jar包

4.运行并且查看结果

1.复习MapReduce的8个步骤

2.MapReduce中的分区

在MapReduce中, 通过我们指定分区, 会将同一个分区的数据发送到同一个 Reduce 当中进行处理。例如: 为了数据的统计, 可以把一批类似的数据发送到同一个 Reduce 当中, 在同一个Reduce 当中统计相同类型的数据, 就可以实现类似的数据分区和统计等其实就是相同类型的数据, 有共性的数据, 送到一起去处理。

Reduce 当中默认的分区只有一个。包括我们上次写的wordcount案例，都是一个分区一个reduce。

2.1 新需求：需要分别reduce

3.IDEA实现分区-JAVA

3.1 实现自定义Partitioner-MyPartitioner.java

主要的逻辑就在这里, 这也是这个案例的意义, 通过 Partitioner 将数据分发给不同的 Reducer

package com.ucas.mapredece;import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Partitioner;/*** @author GONG* @version 1.0* @date 2020/10/9 15:44*/
public class MyPartitioner extends Partitioner<Text, LongWritable> {/*Text:表示K2LongWritable:表示V2i：表示reduce的个数*/@Overridepublic int getPartition(Text text, LongWritable longWritable, int i) {//如果单词长度大于等于5，进入第一分区，否则进入第二分区if (text.toString().length() >= 5) {return 0;} else {return 1;}}
}

3.2 设置主函数：定义分区+设置分区数目

package com.ucas.mapredece;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.hadoop.conf.Configured;public class JobMain extends Configured implements Tool {@Overridepublic int run(String[] args) throws Exception {Job job = Job.getInstance(super.getConf(), JobMain.class.getSimpleName());//打包到集群上面运行时候，必须要添加以下配置，指定程序的main函数job.setJarByClass(JobMain.class);//第一步：读取输入文件解析成key，value对job.setInputFormatClass(TextInputFormat.class);TextInputFormat.addInputPath(job, new Path("hdfs://192.168.0.101:8020/wordcount"));//第二步：设置我们的mapper类job.setMapperClass(WordCountMapper.class);//设置我们map阶段完成之后的输出类型job.setMapOutputKeyClass(Text.class);job.setMapOutputValueClass(LongWritable.class);//第三步，第四步，第五步，第六步//第三步：设置分区规则job.setPartitionerClass(MyPartitioner.class);//第七步：设置我们的reduce类job.setReducerClass(WordCountReducer.class);//设置我们reduce阶段完成之后的输出类型job.setOutputKeyClass(Text.class);job.setOutputValueClass(LongWritable.class);//两个分区用到两个reduce，设置reduce个数job.setNumReduceTasks(2);//第八步：设置输出类以及输出路径job.setOutputFormatClass(TextOutputFormat.class);TextOutputFormat.setOutputPath(job, new Path("hdfs://192.168.0.101:8020/wordcount_out"));//上面那个路径时不允许存在的，会帮我们自动创建这个文件夹boolean b = job.waitForCompletion(true);return b ? 0 : 1;}/*** 程序main函数的入口类** @param args* @throws Exception*/public static void main(String[] args) throws Exception {Configuration configuration = new Configuration();Tool tool = new JobMain();int run = ToolRunner.run(configuration, tool, args);System.exit(run);}
}

3.3 打成jar包

先clean一下以前的东西，再双击packet打包

4.运行并且查看结果

将jar发送到node01中的 /export/software

进入：cd /export/software

运行命令：hadoop jar day03_mapreduce_wordcount-1.0-SNAPSHOT.jar com.ucas.mapredece.JobMain

[root@node01 software]# hadoop jar day03_mapreduce_wordcount-1.0-SNAPSHOT.jar com.ucas.mapredece.JobMain
2020-10-10 00:08:46,441 INFO client.RMProxy: Connecting to ResourceManager at node01/192.168.0.101:8032
2020-10-10 00:08:47,468 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1602247634978_0002
2020-10-10 00:08:47,823 INFO input.FileInputFormat: Total input files to process : 1
2020-10-10 00:08:47,990 INFO mapreduce.JobSubmitter: number of splits:1
2020-10-10 00:08:48,052 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2020-10-10 00:08:48,342 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1602247634978_0002
2020-10-10 00:08:48,345 INFO mapreduce.JobSubmitter: Executing with tokens: []
2020-10-10 00:08:48,636 INFO conf.Configuration: resource-types.xml not found
2020-10-10 00:08:48,636 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2020-10-10 00:08:48,757 INFO impl.YarnClientImpl: Submitted application application_1602247634978_0002
2020-10-10 00:08:48,835 INFO mapreduce.Job: The url to track the job: http://node01:8088/proxy/application_1602247634978_0002/
2020-10-10 00:08:48,836 INFO mapreduce.Job: Running job: job_1602247634978_0002
2020-10-10 00:09:00,140 INFO mapreduce.Job: Job job_1602247634978_0002 running in uber mode : false
2020-10-10 00:09:00,156 INFO mapreduce.Job:  map 0% reduce 0%
2020-10-10 00:09:10,638 INFO mapreduce.Job:  map 100% reduce 0%
2020-10-10 00:09:18,756 INFO mapreduce.Job:  map 100% reduce 50%
2020-10-10 00:09:19,772 INFO mapreduce.Job:  map 100% reduce 100%
2020-10-10 00:09:25,853 INFO mapreduce.Job: Job job_1602247634978_0002 completed successfully
2020-10-10 00:09:26,115 INFO mapreduce.Job: Counters: 53File System CountersFILE: Number of bytes read=203FILE: Number of bytes written=647888FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=185HDFS: Number of bytes written=70HDFS: Number of read operations=13HDFS: Number of large read operations=0HDFS: Number of write operations=4Job Counters Launched map tasks=1Launched reduce tasks=2Data-local map tasks=1Total time spent by all maps in occupied slots (ms)=7940Total time spent by all reduces in occupied slots (ms)=11782Total time spent by all map tasks (ms)=7940Total time spent by all reduce tasks (ms)=11782Total vcore-milliseconds taken by all map tasks=7940Total vcore-milliseconds taken by all reduce tasks=11782Total megabyte-milliseconds taken by all map tasks=8130560Total megabyte-milliseconds taken by all reduce tasks=12064768Map-Reduce FrameworkMap input records=4Map output records=12Map output bytes=167Map output materialized bytes=203Input split bytes=114Combine input records=0Combine output records=0Reduce input groups=9Reduce shuffle bytes=203Reduce input records=12Reduce output records=9Spilled Records=24Shuffled Maps =2Failed Shuffles=0Merged Map outputs=2GC time elapsed (ms)=279CPU time spent (ms)=2730Physical memory (bytes) snapshot=604041216Virtual memory (bytes) snapshot=7283023872Total committed heap usage (bytes)=318500864Peak Map Physical memory (bytes)=364597248Peak Map Virtual memory (bytes)=2409140224Peak Reduce Physical memory (bytes)=122572800Peak Reduce Virtual memory (bytes)=2436947968Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format Counters Bytes Read=71File Output Format Counters Bytes Written=70
[root@node01 software]#