本文主要是介绍Hadoop企业开发案例调优场景,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!
需求
(1)需求:从1G数据中,统计每个单词出现次数。服务器3台,每台配置4G内存,4核CPU,4线程。
(2)需求分析:
1G / 128m = 8个MapTask;1个ReduceTask;1个mrAppMaster
平均每个节点运行10个 / 3台 ≈ 3个任务(4 3 3)
HDFS参数调优
(1)修改:hadoop-env.sh
export HDFS_NAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS -Xmx1024m"export HDFS_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS -Xmx1024m"
(2)修改hdfs-site.xml
<!-- NameNode有一个工作线程池,默认值是10 -->
<property><name>dfs.namenode.handler.count</name><value>21</value>
</property>
(3)修改core-site.xml
<!-- 配置垃圾回收时间为60分钟 -->
<property><name>fs.trash.interval</name><value>60</value>
</property>
(4)分发配置
[lytfly@hadoop102 hadoop]$ xsync hadoop-env.sh hdfs-site.xml core-site.xml
MapReduce参数调优
(1)修改mapred-site.xml
<!-- 环形缓冲区大小,默认100m -->
<property><name>mapreduce.task.io.sort.mb</name><value>100</value>
</property><!-- 环形缓冲区溢写阈值,默认0.8 -->
<property><name>mapreduce.map.sort.spill.percent</name><value>0.80</value>
</property><!-- merge合并次数,默认10个 -->
<property><name>mapreduce.task.io.sort.factor</name><value>10</value>
</property><!-- maptask内存,默认1g; maptask堆内存大小默认和该值大小一致mapreduce.map.java.opts -->
<property><name>mapreduce.map.memory.mb</name><value>-1</value><description>The amount of memory to request from the scheduler for each map task. If this is not specified or is non-positive, it is inferred from mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio. If java-opts are also not specified, we set it to 1024.</description>
</property><!-- matask的CPU核数,默认1个 -->
<property><name>mapreduce.map.cpu.vcores</name><value>1</value>
</property><!-- matask异常重试次数,默认4次 -->
<property><name>mapreduce.map.maxattempts</name><value>4</value>
</property><!-- 每个Reduce去Map中拉取数据的并行数。默认值是5 -->
<property><name>mapreduce.reduce.shuffle.parallelcopies</name><value>5</value>
</property><!-- Buffer大小占Reduce可用内存的比例,默认值0.7 -->
<property><name>mapreduce.reduce.shuffle.input.buffer.percent</name><value>0.70</value>
</property><!-- Buffer中的数据达到多少比例开始写入磁盘,默认值0.66。 -->
<property><name>mapreduce.reduce.shuffle.merge.percent</name><value>0.66</value>
</property><!-- reducetask内存,默认1g;reducetask堆内存大小默认和该值大小一致mapreduce.reduce.java.opts -->
<property><name>mapreduce.reduce.memory.mb</name><value>-1</value><description>The amount of memory to request from the scheduler for each reduce task. If this is not specified or is non-positive, it is inferredfrom mapreduce.reduce.java.opts and mapreduce.job.heap.memory-mb.ratio.If java-opts are also not specified, we set it to 1024.</description>
</property><!-- reducetask的CPU核数,默认1个 -->
<property><name>mapreduce.reduce.cpu.vcores</name><value>2</value>
</property><!-- reducetask失败重试次数,默认4次 -->
<property><name>mapreduce.reduce.maxattempts</name><value>4</value>
</property><!-- 当MapTask完成的比例达到该值后才会为ReduceTask申请资源。默认是0.05 -->
<property><name>mapreduce.job.reduce.slowstart.completedmaps</name><value>0.05</value>
</property><!-- 如果程序在规定的默认10分钟内没有读到数据,将强制超时退出 -->
<property><name>mapreduce.task.timeout</name><value>600000</value>
</property>
(2)分发配置
[lytfly@hadoop102 hadoop]$ xsync mapred-site.xml
Yarn参数调优
(1)修改yarn-site.xml配置参数
<!-- 选择调度器,默认容量 -->
<property><name>yarn.resourcemanager.scheduler.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property><!-- ResourceManager处理调度器请求的线程数量,默认50;如果提交的任务数大于50,可以增加该值,但是不能超过3台 * 4线程 = 12线程(去除其他应用程序实际不能超过8) -->
<property><name>yarn.resourcemanager.scheduler.client.thread-count</name><value>8</value>
</property><!-- 是否让yarn自动检测硬件进行配置,默认是false,如果该节点有很多其他应用程序,建议手动配置。如果该节点没有其他应用程序,可以采用自动 -->
<property><name>yarn.nodemanager.resource.detect-hardware-capabilities</name><value>false</value>
</property><!-- 是否将虚拟核数当作CPU核数,默认是false,采用物理CPU核数 -->
<property><name>yarn.nodemanager.resource.count-logical-processors-as-cores</name><value>false</value>
</property><!-- 虚拟核数和物理核数乘数,默认是1.0 -->
<property><name>yarn.nodemanager.resource.pcores-vcores-multiplier</name><value>1.0</value>
</property><!-- NodeManager使用内存数,默认8G,修改为4G内存 -->
<property><name>yarn.nodemanager.resource.memory-mb</name><value>4096</value>
</property><!-- nodemanager的CPU核数,不按照硬件环境自动设定时默认是8个,修改为4个 -->
<property><name>yarn.nodemanager.resource.cpu-vcores</name><value>4</value>
</property><!-- 容器最小内存,默认1G -->
<property><name>yarn.scheduler.minimum-allocation-mb</name><value>1024</value>
</property><!-- 容器最大内存,默认8G,修改为2G -->
<property><name>yarn.scheduler.maximum-allocation-mb</name><value>2048</value>
</property><!-- 容器最小CPU核数,默认1个 -->
<property><name>yarn.scheduler.minimum-allocation-vcores</name><value>1</value>
</property><!-- 容器最大CPU核数,默认4个,修改为2个 -->
<property><name>yarn.scheduler.maximum-allocation-vcores</name><value>2</value>
</property><!-- 虚拟内存检查,默认打开,修改为关闭 -->
<property><name>yarn.nodemanager.vmem-check-enabled</name><value>false</value>
</property><!-- 虚拟内存和物理内存设置比例,默认2.1 -->
<property><name>yarn.nodemanager.vmem-pmem-ratio</name><value>2.1</value>
</property>
(2)分发配置
[lytfly@hadoop102 hadoop]$ xsync yarn-site.xml
执行程序
(1)重启集群
[lytfly@hadoop102 hadoop-3.1.4]$ sbin/stop-yarn.sh
[lytfly@hadoop103 hadoop-3.1.4]$ sbin/start-yarn.sh
(2)执行WordCount程序
[lytfly@hadoop102 hadoop-3.1.4]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar wordcount /input /output
(3)观察Yarn任务执行页面
http://hadoop103:8088/cluster/apps
这篇关于Hadoop企业开发案例调优场景的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!