MapReducer 取前五的案列Top n

本文主要是介绍MapReducer 取前五的案列Top n，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

1.业务需求,统计单词个数取前五Top 5

2.数据

Chief Justice Roberts, Vice President Harris, Speaker Pelosi, Leader Schumer, Leader McConnell, Vice President Pence, my distinguished guests, [and] my fellow Americans.
This is America’s day.
This is democracy’s day.
A day of history and hope.
Of renewal and resolve.
Through a crucible for the ages America has been tested anew and America has risen to the challenge.
Today, we celebrate the triumph not of a candidate, but of a cause, the cause of democracy.
The people…the will of the people has been heard and the will of the people has been heeded.
We have learned again that democracy is precious.
Democracy is fragile.
And at this hour, my friends, democracy has prevailed.
So now, on this hallowed ground where just a few days ago, violence sought to shake the Capitol’s very foundation, we come together as one nation, under God, indivisible, to carry out the peaceful transfer of power as we have for more than two centuries.
As we look ahead in our uniquely American way – restless, bold, optimistic – and set our sights on the nation we know we can be and we must be.
I thank my predecessors of both parties for their presence here today.
I thank them from the bottom of my heart.
And I know… and I know the resilience of our Constitution and the strength…the strength of our nation.
As does President Carter, who I spoke with last night but who cannot be with us today, but whom we salute for his lifetime in [of] service.
I have just taken the sacred oath each of those patriots have taken – the oath first sworn by George Washington.
But the American story depends not on any one of us, not on some of us, but on all of us.
On “We the People” who seek a more perfect Union.
This is a great nation. We are good people.
And over the centuries through storm and strife, in peace and in war, we have come so far. But we still have far to go.
We’ll press forward with speed and urgency, for we have much to do in this winter of peril and significant possibilities.
Much to repair.
Much to restore.
Much to heal.
Much to build.
And much to gain.
Few people in our nation’s history have been more challenged or found a time more challenging or difficult than the time we’re in now.
A once-in-a-century virus that silently stalks the country.
It’s taken as many lives in one year as America lost in all of World War II.
Millions of jobs have been lost.
Hundreds of thousands of businesses closed.
A cry for racial justice some 400 years in the making moves us. The dream of justice for all will be deferred no longer.
A cry for survival comes from the planet itself. A cry that can’t be any more desperate or any more clear.
And now, a rise in political extremism, white supremacy, domestic terrorism that we must confront and we will defeat.
To overcome these challenges – to restore the soul and secure the future of America – requires more than words.
It requires the most elusive of all things in a democracy:
Unity.
Unity.
In another January in Washington, on New Year’s Day 1863, Abraham Lincoln signed the Emancipation Proclamation.
When he put pen to paper, the President said, and I quote, “If my name ever goes down into history, it’ll be for this act and my whole soul is in it.”
My whole soul is in it.
Today, on this January day, my whole soul is in this:
Bringing America together.
Uniting our people.
Uniting our nation.
And I ask every American to join me in this cause.
Uniting to fight the foes we face:
Anger, resentment, and hatred.
Extremism, lawlessness, violence.
Disease, joblessness, and hopelessness.
With unity, we can do great things. Important things.
We can right wrongs.
We can put people to work in good jobs.
We can teach our children in safe schools.
We can overcome the deadly virus.
We can reward…reward work, and rebuild the middle class, and make health care secure for all.
We can deliver racial justice.
And we can make America, once again, the leading force for good in the world.
I know speaking of unity can sound to some like a foolish fantasy these days.
I know the forces that divide us are deep and they are real.
But I also know they are not new.
Our history has been a constant struggle between the American ideal that we are all created equal and the harsh, ugly reality that racism, nativism, fear, and demonization have long torn us apart.
The battle is perennial.
And victory is never assured.
Through [the] Civil War, the Great Depression, World War, 9/11, through struggle, sacrifice, and setbacks, our “better angels” have always prevailed.
In each of these moments, enough of us, enough of us have come together to carry all of us forward.
And, we can do that now.
History, faith, and reason show the way, the way of unity.
We can see each other not as adversaries but as neighbors.
We can treat each other with dignity and respect.
We can join forces, stop the shouting, and lower the temperature.
For without unity, there is no peace, only bitterness and fury.
No progress, only exhausting outrage.
No nation, only a state of chaos.
This is our historic moment of crisis and challenge, and unity is the path forward.
And, we must meet this moment as the United States of America.
If we do that, I guarantee you, we will not fail.
We have never, ever, ever failed in America when we have acted together.
And so today, at this time, [and] in this place, let’s start afresh.
All of us.
Let us begin to listen to one another again.
Hear one another.
See one another.
Show respect to one another.
Politics doesn’t have to be a raging fire destroying everything in its path.
Every disagreement doesn’t have to be a cause for total war.
And we must reject the culture in which facts themselves are manipulated and even manufactured.
My fellow Americans, we have to be different than this.
America has to be better than this.
And, I believe America is so much better than this.
Just look around.
Here we stand, in the shadow of the Capitol dome, as was mentioned earlier, completed amid the Civil War, when the Union itself was literally hanging in the balance.
Yet we endured. We prevailed.
Here we stand, looking out on the Great Mall where Dr. King spoke of his dream.
Here we stand, where 108 years ago at another inaugural, thousands of protestors tried to block brave women marching for the right to vote.
And today, we mark the swearing-in of the first woman in American history elected to national office – Vice President Kamala Harris.
Don’t tell me things can’t change.
Here we stand across the Potomac from Arlington Cemetery, where heroes who gave the last full measure of devotion rest in eternal peace.
And here we stand, just days after a riotous mob thought they could use violence to silence the will of the people, to stop the work of our democracy, to drive us from this sacred ground.
That did not happen.
It will never happen.
Not today.
Not tomorrow.
Not ever.
Not ever.
To all those who supported our campaign, I’m humbled by the faith you have placed in us.
To all those who did not support us, let me say this: Hear me out as we move forward. Take a measure of me and my heart.
And if you still disagree, so be it.
That’s democracy. That’s America. The right to dissent peaceably, within the guardrails of our Republic, is perhaps this nation’s greatest strength.
Yet hear me clearly: Disagreement must not lead to disunion.
And I pledge this to you: I will be a President for all Americans. All Americans.
And I promise you: I will fight as hard for those who did not support me as for those who did.
Many centuries ago, Saint Augustine, a saint of my church, wrote that a people was a multitude defined by the common objects of their love – defined by the common objects of their love.
What are the common objects we as Americans love that define us as Americans?
I think we know.
Opportunity.
Security.
Liberty.
Dignity.
Respect.
Honor.
And, yes, the truth.
Recent weeks and months have taught us a painful lesson.
There is truth and there are lies.
Lies told for power and for profit.
And each of us has a duty and responsibility, as citizens, as Americans, and especially as leaders – leaders who have pledged to honor our Constitution and protect our nation – to defend the truth and defeat the lies.
Look, I understand that many fellow Americans view the future with fear and trepidation.
I understand they worry about their jobs. I understand, like my dad, they lay in bed staring at night, staring at the ceiling, wondering, “Can I keep my health care? Can I pay my mortgage?” Thinking about their families, about what comes next.
I promise you: I get it.
But the answer is not to turn inward, to retreat into competing factions, distrusting those who don’t look like…look like you, or worship the way you do, or don’t get their news from the same sources you do.
We must end this uncivil war that pits red against blue, rural versus urban, conservative versus liberal.
We can do this if we open our souls instead of hardening our hearts.
If we show a little tolerance and humility.
And if we’re willing to stand in the other person’s shoes, as my mom would say, “just for a moment, stand in their shoes.”
Because here is the thing about life: There is no accounting for what fate will deal you.
Some days when you need a hand.
There are other days when we’re called to lend a hand.
That’s how it has to be. That’s what we do for one another.
And, if we are this way, our country will be stronger, more prosperous, more ready for the future. And we can still disagree.
My fellow Americans, in the work ahead of us, we’re going to need each other.
We need all our strength to persevere through this dark winter.
We are entering what may be the toughest and deadliest period of the virus.
We must set aside politics and finally face this pandemic as one nation. One nation.
And I promise you this: as the Bible says weeping may endure for a night but joy cometh in the morning.
We will get through this, together. Together.
Look, folks, all my colleagues I served with in the House and the Senate up here, we all understand the world is watching, watching all of us today.
So here is my message to those beyond our borders: America has been tested and we have come out stronger for it.
We will repair our alliances and engage with the world once again.
Not to meet yesterday’s challenges, but today’s and tomorrow’s challenges.
And we will lead not merely by the example of our power but by the power of our example.
We’ll be a strong and trusted partner for peace, progress, and security.
Look, you all know we’ve been through so much in this nation.
And [in] my first act as President, I’d like to ask you to join me in a moment of silent prayer to remember all those who we lost in this past year to the pandemic.
Those 400,000 fellow Americans – moms, dads, husbands, wives, sons, daughters, friends, neighbors, and co-workers.
We’ll honor them by becoming the people and the nation we know we can and should be.
So I ask you, let’s say a silent prayer for those who’ve lost their lives and those left behind, and for our country.
Amen.
Folks, this is a time of testing.
We face an attack on our democracy and on truth.
A raging virus.
Growing inequity.
The sting of systemic racism.
A climate in crisis.
America’s role in the world.
Any one of these would be enough to challenge us in profound ways.
But the fact is we face them all at once, presenting this nation with one of the gravest of responsibilities we’ve had.
Now we’re going to be tested. Are we going to step up, all of us?
It is a time for boldness, for there is so much to do.
And, this is certain.
I promise you: we will be judged, you and I, by how we resolve these cascading crises of our era.
We will rise to the occasion – here’s the question.
Will we master this rare and difficult hour?
Will we meet our obligations and pass along a new and better world to our children?
I believe we must. I’m sure you do as well. I believe we will.
And when we do, we’ll write the next great chapter in the history of the United States of America, the American story, a story that might sound something like a song that means a lot to me. It’s called “American Anthem” and there’s one verse that stands out at least for me, and it goes like this:
“The work and prayers
Of centuries have brought us to this day
What shall be our legacy?
What will our children say?
…
Let me know in my heart
When my days are through
America
America
I gave my best to you.”
Let’s add. Let us add our own work and prayers to the unfolding story of our great nation.
If we do this, then when our days are through, our children and our children’s children will say of us: “They gave their best. They did their duty. They healed a broken land.”
My fellow Americans, I close today where I began, with a sacred oath.
Before God and all of you, I give you my word.
I will always level with you.
I will defend the Constitution.
I’ll defend our democracy.
I’ll defend America.
And I’ll give all, all of you, keep everything I do in your service thinking not of power, but of possibilities.
Not of personal interest, but [of] the public good.
And together, we shall write an American story of hope, not fear.
Of unity, not division.
Of light, not darkness.
A story of decency and dignity.
Love and healing.
Greatness and goodness.
May this be the story that guides us.
The story that inspires us.
And the story that tells ages yet to come that we answered the call of history.
We met the moment.
Democracy and hope, truth and justice, did not die on our watch but thrived.
That America secured liberty at home and stood once again as a beacon to the world.
That is what we owe our forebearers, one another, and generations to follow.
So, with purpose and resolve, we turn to those tasks of our time.
Sustained by faith.
Driven by conviction.
And, devoted to one another and the country we love with all our hearts.
May God bless America and may God protect our troops.
Thank you, America.

3.创建maven项目

4.需要先对上面数据进行统计,统计个数,就是写一个wordcount

4.导入依赖

<dependencies><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>3.1.3</version></dependency><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.12</version></dependency><dependency><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId><version>1.7.30</version></dependency></dependencies><build><plugins><plugin><artifactId>maven-compiler-plugin</artifactId><version>3.6.1</version><configuration><source>1.8</source><target>1.8</target></configuration></plugin><plugin><artifactId>maven-assembly-plugin</artifactId><configuration><descriptorRefs><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs></configuration><executions><execution><id>make-assembly</id><phase>package</phase><goals><goal>single</goal></goals></execution></executions></plugin></plugins></build>

4.1,mapper

package com.lhh;import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;import java.io.IOException;/*** @program: lhh* @description:* @author: 华仔* @create: 2021-04-16 09:29*/
public class KSMapper extends Mapper<LongWritable, Text,Text, IntWritable> {@Overrideprotected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {//读取一行String stxt = value.toString();//替换标点符号为 \\s  \\s正则表达式,表示一个或多个空格String svalue = stxt.replaceAll("\\W"," ");//根据 \\s拆分String[] split = svalue.split("\\s+");for (String s : split) {  //循环写出context.write(new Text(s),new IntWritable(1));}}
}

4.2reducer

package com.lhh;import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;import java.io.IOException;/*** @program: lhh* @description:* @author: 华仔* @create: 2021-04-16 09:29*/
public class KSReduce extends Reducer<Text, IntWritable, Text, IntWritable> {@Overrideprotected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {//声明变量int num=0; for (IntWritable value : values) {num += value.get(); //累加}//写出数据context.write(key,new IntWritable(num));}
}

4.3job

package com.lhh;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import java.io.IOException;/*** @program: lhh* @description:* @author: 华仔* @create: 2021-04-16 09:29*/
public class KSJob {public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {//获取jobConfiguration conf = new Configuration();Job job = Job.getInstance(conf);job.setJarByClass(KSJob.class);//设置map reducejob.setMapperClass(KSMapper.class);job.setReducerClass(KSReduce.class);//设置map输出类job.setMapOutputKeyClass(Text.class);job.setMapOutputValueClass(IntWritable.class);//设置最终输出类job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);//设置输入和输出路径FileInputFormat.setInputPaths(job, new Path("D:/test/in/ceshi.txt"));FileOutputFormat.setOutputPath(job, new Path("D:/test/out/ceshiout1"));//等待任务完成boolean b = job.waitForCompletion(true);if(b){System.out.println("程序完成");}else{System.out.println("程序异常");}//关闭程序System.exit(b ? 0 : 1);}
}

注意:这里是将上面的结果作为这一次的数据

5.导入依赖

<dependencies><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>3.1.3</version></dependency><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.12</version></dependency><dependency><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId><version>1.7.30</version></dependency></dependencies><build><plugins><plugin><artifactId>maven-compiler-plugin</artifactId><version>3.6.1</version><configuration><source>1.8</source><target>1.8</target></configuration></plugin><plugin><artifactId>maven-assembly-plugin</artifactId><configuration><descriptorRefs><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs></configuration><executions><execution><id>make-assembly</id><phase>package</phase><goals><goal>single</goal></goals></execution></executions></plugin></plugins></build>

5.1.创建Bean

package lhh;import org.apache.hadoop.io.WritableComparable;import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;/*** @program: lhh* @description:* @author: 华仔* @create: 2021-03-09 14:14*/
public class KSBean implements WritableComparable<KSBean> {//创建变量并且set get and toStringprivate String word;private int num;private int sort;public KSBean() {}public KSBean(String word, int num, int sort) {this.word = word;this.num = num;this.sort = sort;}@Overridepublic String toString() {return word + '\t' + num +'\t'+ sort;}//实行比较器,指定排序规则public int compareTo(KSBean o) {//降序排序,升序排序就换个位置if(o.getNum()>this.getNum()){ //如果相等就返回1,不相同就返回-1return 1;}else{return -1;}}//实现序列化public void write(DataOutput dataOutput) throws IOException {dataOutput.writeUTF(word);dataOutput.writeInt(num);}//反序列化public void readFields(DataInput dataInput) throws IOException {this.word = dataInput.readUTF(); //读取字符串this.num = dataInput.readInt();//读取数字}public String getWord() {return word;}public void setWord(String word) {this.word = word;}public int getNum() {return num;}public void setNum(int num) {this.num = num;}public int getSort() {return sort;}public void setSort(int sort) {this.sort = sort;}
}

5.2.创建Mapper

package lhh;import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;import java.io.IOException;/*** @program: lhh* @description:* @author: 华仔* @create: 2021-04-16 09:29*/
public class KSMapper extends Mapper<LongWritable, Text,KSBean, Text> {@Overrideprotected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {//读取一行String stxt = value.toString();String[] split = stxt.split("\t");String str1=split[0]; //取值String str2=split[1];KSBean ksBean = new KSBean(); //ksBean.setWord(str1);ksBean.setNum(Integer.parseInt(str2));context.write(ksBean,new Text(ksBean.getWord()));}}

5…3reducer 序号不跳序 1 2 3 4 5 ,意思就如果有数字一样的,还是直接排下去,方式一

package lhh;import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;import java.io.IOException;
import java.util.ArrayList;
import java.util.List;/*** @program: lhh* @description:* @author: 华仔* @create: 2021-04-16 09:29*/
public class KSReducebutiaoxu extends Reducer<KSBean, Text, NullWritable, KSBean> {int sum = 0; //正常的排名List<KSBean> list = new ArrayList<>();KSBean ksBean = null;int rank = 0; //计算上一次的排名int count = 0; //计算上一次数量@Overrideprotected void reduce(KSBean key, Iterable<Text> values, Context context) throws IOException, InterruptedException {for (Text value : values) {//当记录和上一次相等时if (key.getNum() == count) {ksBean = new KSBean(key.getWord(), key.getNum(), rank);list.add(ksBean);} else {if (rank >= 5) {break;} else {rank++;count = key.getNum();ksBean = new KSBean(key.getWord(), key.getNum(), rank);list.add(ksBean);}}}}//只执行一次@Overrideprotected void cleanup(Context context) throws IOException, InterruptedException {for (KSBean bean : list) {context.write(NullWritable.get(),bean);}}}

5.4reducer 跳序 1 2 2 4 5 意思就如果有数字一样的,就会跳过序号,方式二

package lhh;import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;import java.io.IOException;
import java.util.ArrayList;
import java.util.List;/*** @program: lhh* @description:* @author: 华仔* @create: 2021-04-18 16:19*/
public class KSReducetiaoxu extends Reducer<KSBean, Text, NullWritable, KSBean>{int sum = 0; //正常的排名List<KSBean> list = new ArrayList<>();KSBean ksBean = null;int rank = 0; //计算上一次的排名int count = 0; //计算上一次数量@Overrideprotected void reduce(KSBean key, Iterable<Text> values, Context context) throws IOException, InterruptedException {for (Text value : values) {//当记录和上一次相等时if (key.getNum() == count) {ksBean = new KSBean(key.getWord(), key.getNum(), rank);list.add(ksBean);sum++;} else {if (sum >= 5) {break;} else {sum++;rank = sum;count = key.getNum();ksBean = new KSBean(key.getWord(), key.getNum(), rank);list.add(ksBean);}}}}@Overrideprotected void cleanup(Reducer.Context context) throws IOException, InterruptedException {for (KSBean bean : list) {context.write(NullWritable.get(),bean);}}}

5.5job

package lhh;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import java.io.IOException;/*** @program: lhh* @description:* @author: 华仔* @create: 2021-04-16 09:29*/
public class KSJob {public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {//获取jobConfiguration conf = new Configuration();Job job = Job.getInstance(conf);job.setJarByClass(KSJob.class);//设置map reducejob.setMapperClass(KSMapper.class);job.setReducerClass(KSReducebutiaoxu.class);//设置map输出类job.setMapOutputKeyClass(KSBean.class);job.setMapOutputValueClass(Text.class);//设置最终输出类job.setOutputKeyClass(NullWritable.class);job.setOutputValueClass(KSBean.class);//设置输入和输出路径FileInputFormat.setInputPaths(job, new Path("D:/test/out/test/abc.txt"));FileOutputFormat.setOutputPath(job, new Path("D:/test/out/abc6"));//等待任务完成boolean b = job.waitForCompletion(true);if(b){System.out.println("程序完成");}else{System.out.println("程序异常");}//关闭程序System.exit(b ? 0 : 1);}
}