本文主要是介绍solve hadoop streaming memory exceed，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
出现内存不够，指定mapper和reducer tasks数大点，就能避免这个问题，如 指定
-jobconf mapred.map.tasks=500 -jobconf mapred.reduce.tasks=500
#!/bin/bash
/usr/bin/hadoop fs -rmr /user/hmthdfs/upload/out
/usr/bin/hadoop fs -lsr /user/hmthdfs/upload
/usr/bin/hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u4.jar -input /user/hmthdfs/upload/allinone -output /user/hmthdfs/upload/out -file /search/hadoop/upload/mapper.php -file /search/hadoop/upload/reducer.php -mapper mapper.php -reducer reducer.php -jobconf mapred.map.tasks=500 -jobconf mapred.reduce.tasks=500
/usr/bin/hadoop fs -lsr /user/hmthdfs/upload/out
这篇关于solve hadoop streaming memory exceed的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
