本文主要是介绍MapReduce词频统计，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
1.1 文件准备
创建本地目录和创建两个文本文件，在两个文件中输入单词，用于统计词频。
cd /usr/local/hadoop
mkdir WordFile
cd WordFile
touch wordfile1.txt
touch wordfile2.txt
1.2 创建一个HDFS目录，在本地上不可见,并将本地文本文件上传到HDFS目录。通过如下命令创建。
cd /usr/local/hadoop
./bin/hdfs dfs -mkdir wordfileinput
./bin/hdfs dfs -put ./WordFile/wordfile1.txt wordfileinput
./bin/hdfs dfs -put ./WordFile/wordfile2.txt wordfileinput1.3 保证HDFS目录不存在output,我们执行如下命令，每次运行词频统计都要删除output输出文件,/user/hadoop/是HDFS的用户目录，不是本地目录。
./bin/hdfs dfs -rm -r /user/hadoop/output
1.4 Eclips编写代码
创建Java project ,项目名称为MapReduceWordCount,右键项目名，导入相关Jar包。
1.5 点击Add External Jars,进入目录/usr/local/hadoop/share/hadoop,导入如下包。
- “/usr/local/hadoop/share/hadoop/common”目录下的hadoop-common-3.1.3.jar和haoop-nfs-3.1.3.jar；
- “/usr/local/hadoop/share/hadoop/common/lib”目录下的所有JAR包；
- “/usr/local/hadoop/share/hadoop/mapreduce”目录下的所有JAR包，但是，不包括jdiff、lib、lib-examples和sources目录;
- “/usr/local/hadoop/share/hadoop/mapreduce/lib”目录下的所有JAR包。
1.6 创建类WordCount.java
import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class WordCount {public WordCount() {}public static void main(String[] args) throws Exception {Configuration conf = new Configuration();String[] otherArgs = (new GenericOptionsParser(conf, args)).getRemainingArgs();if(otherArgs.length < 2) {System.err.println("Usage: wordcount <in> [<in>...] <out>");System.exit(2);}Job job = Job.getInstance(conf, "word count");job.setJarByClass(WordCount.class);job.setMapperClass(WordCount.TokenizerMapper.class);job.setCombinerClass(WordCount.IntSumReducer.class);job.setReducerClass(WordCount.IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class); for(int i = 0; i < otherArgs.length - 1; ++i) {FileInputFormat.addInputPath(job, new Path(otherArgs[i]));}FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1]));System.exit(job.waitForCompletion(true)?0:1);}public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {private static final IntWritable one = new IntWritable(1);private Text word = new Text();public TokenizerMapper() {}public void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer(value.toString()); while(itr.hasMoreTokens()) {this.word.set(itr.nextToken());context.write(this.word, one);}}}
public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {private IntWritable result = new IntWritable();public IntSumReducer() {}public void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {int sum = 0;IntWritable val;for(Iterator i$ = values.iterator(); i$.hasNext(); sum += val.get()) {val = (IntWritable)i$.next();}this.result.set(sum);context.write(key, this.result);}}
}
1.7 编译打包程序
将程序打包到 /usr/local/hadoop/myapp目录下，
cd /usr/local/hadoop
mkdir myapp
- Run As 运行程序；
- 右键工程名－>Export－>Java－>Runnable JAR file
- “Launch configuration”用于设置生成的JAR包被部署启动时运行的主类，需要在下拉列表中选择刚才配置的类“WordCount-MapReduceWordCount”。在“Export destination”中需要设置JAR包要输出保存到哪个目录即其名称。点击finish,中间会出现一些信息，一直点击Ｏｋ即可。
1.8 运行程序
启动hadoop
cd /usr/local/hadoop
./sbin/start-dfs.sh
./bin/hadoop jar ./myapp/WordCount.jar wordfileinput output
1.9 查看结果
cd /usr/local/hadoop
./bin/hdfs dfs -cat output/*
1.20 查看HDFS 文件系统
进入/usr/local/hadoop/bin 目录，执行相关命令。
./hadoop fs -ls
1.21 源文档
http://dblab.xmu.edu.cn/blog/2481-2/#more-2481
这篇关于MapReduce词频统计的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
