配置Hadoop2.x的HDFS、MapReduce来运行WordCount程序

本文主要是介绍配置Hadoop2.x的HDFS、MapReduce来运行WordCount程序，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

主机	HDFS	MapReduce
node1	NameNode	ResourceManager
node2	SecondaryNameNode & DataNode	NodeManager
node3	DataNode	NodeManager
node4	DataNode	NodeManager

1.配置hadoop-env.sh

export JAVA_HOME=/csh/link/jdk

2.配置core-site.xml

<property><name>fs.defaultFS</name><value>hdfs://node1:9000</value>
</property>
<property><name>hadoop.tmp.dir</name><value>/csh/hadoop/hadoop2.7.2/tmp</value>
</property>

3.配置hdfs-site.xml

<property><name>dfs.namenode.http-address</name><value>node1:50070</value>
</property>
<property><name>dfs.namenode.secondary.http-address</name><value>node2:50090</value>
</property>
<property><name>dfs.namenode.name.dir</name><value>/csh/hadoop/hadoop2.7.2/name</value>
</property>
<property><name>dfs.datanode.data.dir</name><value>/csh/hadoop/hadoop2.7.2/data</value>
</property>
<property><name>dfs.replication</name><value>3</value>
</property>

4.配置mapred-site.xml

<property><name>mapreduce.framework.name</name><value>yarn</value>
</property>

5.配置yarn-site.xml

<property><name>yarn.resourcemanager.hostname</name><value>node1</value>
</property>
<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value>
</property>
<property><name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

6.配置masters

node2

7.配置slaves

node2
node3
node4

8.启动Hadoop

bin/hadoop namenode -format
sbin/start-dfs.sh
sbin/start-yarn.sh

9.运行WordCount程序

//创建文件wc.txt
echo "I love Java I love Hadoop I love BigData Good Good Study, Day Day Up" > wc.txt
//创建HDFS中的文件
hdfs dfs -mkdir -p /input/wordcount/
//将wc.txt上传到HDFS中
hdfs dfs -put wc.txt /input/wordcount
//运行WordCount程序
hadoop jar /csh/software/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /input/wordcount/ /output/wordcount/

10.结果

[root@node1 sbin]# hadoop jar /csh/software/hadoop-2.7.2/share/hadoop/mapreduce/hadoapreduce-examples-2.7.2.jar wordcount /input/wordcount/ /output/wordcount/
16/03/24 19:26:48 INFO client.RMProxy: Connecting to ResourceManager at node1/192.161.11:8032
16/03/24 19:26:56 INFO input.FileInputFormat: Total input paths to process : 1
16/03/24 19:26:56 INFO mapreduce.JobSubmitter: number of splits:1
16/03/24 19:26:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_145887175_0001
16/03/24 19:26:59 INFO impl.YarnClientImpl: Submitted application application_145887175_0001
16/03/24 19:27:00 INFO mapreduce.Job: The url to track the job: http://node1:8088/prapplication_1458872237175_0001/
16/03/24 19:27:00 INFO mapreduce.Job: Running job: job_1458872237175_0001
16/03/24 19:28:13 INFO mapreduce.Job: Job job_1458872237175_0001 running in uber modfalse
16/03/24 19:28:13 INFO mapreduce.Job:  map 0% reduce 0%
16/03/24 19:30:07 INFO mapreduce.Job:  map 100% reduce 0%
16/03/24 19:31:13 INFO mapreduce.Job:  map 100% reduce 33%
16/03/24 19:31:16 INFO mapreduce.Job:  map 100% reduce 100%
16/03/24 19:31:23 INFO mapreduce.Job: Job job_1458872237175_0001 completed successfu
16/03/24 19:31:24 INFO mapreduce.Job: Counters: 49File System CountersFILE: Number of bytes read=106FILE: Number of bytes written=235387FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=174HDFS: Number of bytes written=64HDFS: Number of read operations=6HDFS: Number of large read operations=0HDFS: Number of write operations=2Job Counters Launched map tasks=1Launched reduce tasks=1Data-local map tasks=1Total time spent by all maps in occupied slots (ms)=116501Total time spent by all reduces in occupied slots (ms)=53945Total time spent by all map tasks (ms)=116501Total time spent by all reduce tasks (ms)=53945Total vcore-milliseconds taken by all map tasks=116501Total vcore-milliseconds taken by all reduce tasks=53945Total megabyte-milliseconds taken by all map tasks=119297024Total megabyte-milliseconds taken by all reduce tasks=55239680Map-Reduce FrameworkMap input records=4Map output records=15Map output bytes=129Map output materialized bytes=106Input split bytes=105Combine input records=15Combine output records=9Reduce input groups=9Reduce shuffle bytes=106Reduce input records=9Reduce output records=9Spilled Records=18Shuffled Maps =1Failed Shuffles=0Merged Map outputs=1GC time elapsed (ms)=1468CPU time spent (ms)=6780Physical memory (bytes) snapshot=230531072Virtual memory (bytes) snapshot=4152713216Total committed heap usage (bytes)=134795264Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format Counters Bytes Read=69File Output Format Counters Bytes Written=64
[root@node1 sbin]# hdfs dfs -cat /output/wordcount/*
BigData 1
Day 2
Good    2
Hadoop  1
I   3
Java    1
Study,  1
Up  1
love    3