一、storm基础概念

本文主要是介绍一、storm基础概念，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

1、什么是storm

Storm是一个分布式的、高容错的实时计算系统。
Storm对于实时计算的的意义相当于Hadoop对于批处理的意义。Hadoop为我们提供了Map和Reduce原语，使我们对数据进行批处理变的非常的简单和优美。同样，Storm也对数据的实时计算提供了简单Spout和Bolt原语。
Storm适用的场景：
(1)、流数据处理：Storm可以用来用来处理源源不断的消息，并将处理之后的结果保存到持久化介质中。
(2)、分布式RPC：由于Storm的处理组件都是分布式的，而且处理延迟都极低，所以可以Storm可以做为一个通用的分布式RPC框架使用。

2、入门级程序wordcount

topology:

SentenceSpout.java

public class SentenceSpout extends BaseRichSpout {private SpoutOutputCollector collector;private String[] sentences = {"my dog has fleas","i like cold beverages","the dog ate my homework","don't have a cow man","i don't think i like fleas"};private int index = 0;public void declareOutputFields(OutputFieldsDeclarer declarer) {declarer.declare(new Fields("sentence"));}public void open(Map config, TopologyContextcontext, SpoutOutputCollector collector) {this.collector = collector;}public void nextTuple() {this.collector.emit(new Values(sentences[index]));index++;if (index >= sentences.length) {index = 0;}Utils.sleep(1);}
}

SplitSentenceBolt.java

public class SplitSentenceBolt extends BaseRichBolt {private OutputCollector collector;public void prepare(Map config, TopologyContextcontext, OutputCollector collector) {this.collector = collector;}public void execute(Tuple tuple) {String sentence = tuple.getStringByField("sentence");String[] words = sentence.split(" ");for(String word : words){this.collector.emit(new Values(word));}}public void declareOutputFields(OutputFieldsDeclarer declarer) {declarer.declare(new Fields("word"));}
}

WordCountBolt.java

public class WordCountBolt extends BaseRichBolt {private OutputCollector collector;private HashMap<String, Long> counts = null;public void prepare(Map config, TopologyContextcontext, OutputCollector collector) {this.collector = collector;this.counts = new HashMap<String, Long>();}public void execute(Tuple tuple) {String word = tuple.getStringByField("word");Long count = this.counts.get(word);if(count == null){count = 0L;}count++;this.counts.put(word, count);this.collector.emit(new Values(word, count));}public void declareOutputFields(OutputFieldsDeclarer declarer) {declarer.declare(new Fields("word", "count"));}
}

ReportBolt.java

public class ReportBolt extends BaseRichBolt {private HashMap<String, Long> counts = null;public void prepare(Map config, TopologyContext context, OutputCollector collector) {this.counts = new HashMap<String, Long>();}public void execute(Tuple tuple) {String word = tuple.getStringByField("word");Long count = tuple.getLongByField("count");this.counts.put(word, count);}public void declareOutputFields(OutputFieldsDeclarer declarer) {// this bolt does not emit anything}
public void cleanup() {System.out.println("--- FINAL COUNTS ---");List<String> keys = new ArrayList<String>();keys.addAll(this.counts.keySet());Collections.sort(keys);for (String key : keys) {System.out.println(key + " : " + this.counts.get(key));}System.out.println("--------------");}}

WordCountTopology.java

public class WordCountTopology {private static final String SENTENCE_SPOUT_ID = "sentence-spout";private static final String SPLIT_BOLT_ID = "split-bolt";private static final String COUNT_BOLT_ID = "count-bolt";private static final String REPORT_BOLT_ID = "report-bolt";private static final String TOPOLOGY_NAME = "word-count-topology";public static void main(String[] args) throwsException {SentenceSpout spout = new SentenceSpout();SplitSentenceBolt splitBolt = newSplitSentenceBolt();WordCountBolt countBolt = new WordCountBolt();ReportBolt reportBolt = new ReportBolt();TopologyBuilder builder = new TopologyBuilder();builder.setSpout(SENTENCE_SPOUT_ID, spout);// SentenceSpout --> SplitSentenceBoltbuilder.setBolt(SPLIT_BOLT_ID, splitBolt).shuffleGrouping(SENTENCE_SPOUT_ID);// SplitSentenceBolt --> WordCountBoltbuilder.setBolt(COUNT_BOLT_ID, countBolt).fieldsGrouping(SPLIT_BOLT_ID, new Fields("word"));// WordCountBolt --> ReportBoltbuilder.setBolt(REPORT_BOLT_ID, reportBolt).globalGrouping(COUNT_BOLT_ID);Config config = new Config();LocalCluster cluster = new LocalCluster();cluster.submitTopology(TOPOLOGY_NAME, config,builder.createTopology());Utils.sleep(10000);cluster.killTopology(TOPOLOGY_NAME);cluster.shutdown();}
}

3、storm集群结构

4、Topology

一个实时计算应用程序的逻辑在storm里面被封装到topology对象里面，我把它叫做计算拓补. Storm里面的topology相当于Hadoop里面的一个MapReduce Job, 它们的关键区别是：一个MapReduce Job最终总是会结束的，然而一个storm的topoloy会一直运行 — 除非你显式的杀死它。一个Topology是Spouts和Bolts组成的图状结构，而链接Spouts和Bolts的则是Stream groupings。

5、spout

喷口（Spout）是拓扑的流的来源，是一个拓扑中产生源数据流的组件。通常情况下，Spout会从外部数据源（例如Kestrel队列或Twitter API）中读取数据，然后转换为拓扑内部的源数据。Spout可以是可靠的，也可以是不可靠的。如果Storm处理元组失败，可靠的Spout能够重新发射，而不可靠的Spout就尽快忘记发出的元组。Spout是一个主动的角色，其接口中有个nextTuple()函数，Storm框架会不停地调用此函数，用户只要在其中生成源数据即可。

Spout的其他主要方法是ack()和fail()。当Storm检测到一个元组从Spout发出时，ack()和fail()会被调用，要么成功完成通过拓扑，要么未能完成。ack()和fail()仅被可靠的Spout调用。

6、bolt

所有的消息处理逻辑被封装在bolts里面。 Bolts可以做很多事情：过滤，聚合，查询数据库等等。
Bolts的主要方法是execute, 它以一个tuple作为输入，Bolts使用OutputCollector来发射tuple, Bolts必须要为它处理的每一个tuple调用OutputCollector的ack方法，以通知storm这个tuple被处理完成了。– 从而我们通知这个tuple的发射者Spouts。一般的流程是： Bolts处理一个输入tuple, 发射0个或者多个tuple, 然后调用ack通知storm自己已经处理过这个tuple了。storm提供了一个IBasicBolt会自动调用ack