本文主要是介绍Flume java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
之前我们的架构方式采用的是spark+hbase+oozie解析存储及调用算法模式，最近突然出现一个需求，会有很多小文件上传，而且要求达到伪实时处理，也就是秒级别，spark很显然不适合解析了，哪怕是几十行的文件， spark也基本是分钟级别。
我想过2个方案来处理，一个是使用纯JAVA来解析文件，另外一个就是使用flume来解析并直接存储到HBASE。
下载最新版本Flume1.8，通过spoolDir方式，配置文件如下：
a1.sources = r1
a1.sinks = k1
a1.channels = c1a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /data/flume/r1/data
a1.sources.r1.batchSize = 100
a1.sources.r1.channels = c1a1.channels.c1.type=file
a1.channels.c1.write-timeout=10
a1.channels.c1.keep-alive=10
a1.channels.c1.checkpointDir=/data/flume/c1/checkpoint
a1.channels.c1.dataDirs=/data/flume/c1/data
a1.channels.c1.maxFileSize= 268435456#a1.sinks.k1.type = logger
a1.sinks.k1.type = hbase
a1.sinks.k1.table = flume
a1.sinks.k1.columnFamily = cf
#a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
a1.sinks.k1.batchSize = 100
a1.sinks.k1.serializer.regex = (.*?)\\|\\|(.*?)\\|\\|(.*?)\\|\\|(.*?)\\|\\|(.*)
a1.sinks.k1.serializer.colNames = ROW_KEY,cnc_rdspmeter[0],cnc_rdsvmeter,cnc_statinfo[3],ext_toolno
a1.sinks.k1.serializer.regexIgnoreCase = true
a1.sinks.k1.serializer.depositHeaders = true
a1.sinks.hbaseSink.zookeeperQuorum = datanode01-ucloud.isesol.com:2181
a1.sinks.k1.channel = c1bin/flume-ng agent -n a1 -c conf -f conf/flume-conf.properties 在消费文件的时候错误如下：
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)Lorg/apache/hadoop/hbase/client/Put;at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:380)at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:375)at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:375)at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:345)at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)at java.lang.Thread.run(Thread.java:748)
^CAttempting to shutdown background worker.setWriteWal在之前版本存在，但是1.0之后应该就没有了，我不知道为什么Flume的开发者在最新的1.8仍然在使用这个方法，很无奈，查询了一下网上，基本没什么解决方案，于是打开源代码，看看究竟怎么回事。
因为我使用的是type是hbase，因此找到hbaseSink.java, 通过find查找哪里有setWriteWAL, 发现有3个地方存在,
public Void run() throws Exception {for (Row r : actions) {if (r instanceof Put) {// ((Put) r).setWriteToWAL(enableWal);}// Newer versions of HBase - Increment implements Row.if (r instanceof Increment) {// ((Increment) r).setWriteToWAL(enableWal);}}table.batch(actions);return null;} public Void run() throws Exception {List<Increment> processedIncrements;if (batchIncrements) {processedIncrements = coalesceIncrements(incs);} else {processedIncrements = incs;}// Only used for unit testing.if (debugIncrCallback != null) {debugIncrCallback.onAfterCoalesce(processedIncrements);}for (final Increment i : processedIncrements) {// i.setWriteToWAL(enableWal);table.increment(i);}return null;}});上面3个被我注视掉的地方，就是setWriteWAL, 这个东西实际无所谓，因此我很暴力的直接注释，然后再重新打一个包进行替换，官方名字叫：flume-ng-hbase-sink-1.8.0.jar。重新启动Flume,查看结果：
hbase(main):001:0> scan 'flume'
ROW COLUMN+CELL 1529992556110-SzjikLv1LH-0 column=cf:ROW_KEY, timestamp=1529992556407, value=cnc_exeprgname:418 1529992556110-SzjikLv1LH-0 column=cf:cnc_rdspmeter[0], timestamp=1529992556407, value=cnc_rdspmeter[0]:0 1529992556110-SzjikLv1LH-0 column=cf:cnc_rdsvmeter, timestamp=1529992556407, value=cnc_rdsvmeter:6,7,92,0 1529992556110-SzjikLv1LH-0 column=cf:cnc_statinfo[3], timestamp=1529992556407, value=cnc_statinfo[3]:3 1529992556110-SzjikLv1LH-0 column=cf:ext_toolno, timestamp=1529992556407, value=ext_toolno:30 1529992556125-SzjikLv1LH-1 column=cf:ROW_KEY, timestamp=1529992556407, value=cnc_exeprgname:418 1529992556125-SzjikLv1LH-1 column=cf:cnc_rdspmeter[0], timestamp=1529992556407, value=cnc_rdspmeter[0]:0 1529992556125-SzjikLv1LH-1 column=cf:cnc_rdsvmeter, timestamp=1529992556407, value=cnc_rdsvmeter:6,7,93,0 1529992556125-SzjikLv1LH-1 column=cf:cnc_statinfo[3], timestamp=1529992556407, value=cnc_statinfo[3]:3 1529992556125-SzjikLv1LH-1 column=cf:ext_toolno, timestamp=1529992556407, value=ext_toolno:30 1529992556126-SzjikLv1LH-2 column=cf:ROW_KEY, timestamp=1529992556407, value=cnc_exeprgname:418 1529992556126-SzjikLv1LH-2 column=cf:cnc_rdspmeter[0], timestamp=1529992556407, value=cnc_rdspmeter[0]:0 1529992556126-SzjikLv1LH-2 column=cf:cnc_rdsvmeter, timestamp=1529992556407, value=cnc_rdsvmeter:5,10,93,0 1529992556126-SzjikLv1LH-2 column=cf:cnc_statinfo[3], timestamp=1529992556407, value=cnc_statinfo[3]:3 1529992556126-SzjikLv1LH-2 column=cf:ext_toolno, timestamp=1529992556407, value=ext_toolno:30 1529992556127-SzjikLv1LH-3 column=cf:ROW_KEY, timestamp=1529992556407, value=cnc_exeprgname:418 1529992556127-SzjikLv1LH-3 column=cf:cnc_rdspmeter[0], timestamp=1529992556407, value=cnc_rdspmeter[0]:0 1529992556127-SzjikLv1LH-3 column=cf:cnc_rdsvmeter, timestamp=1529992556407, value=cnc_rdsvmeter:7,8,93,0 1529992556127-SzjikLv1LH-3 column=cf:cnc_statinfo[3], timestamp=1529992556407, value=cnc_statinfo[3]:3 1529992556127-SzjikLv1LH-3 column=cf:ext_toolno, timestamp=1529992556407, value=ext_toolno:30 1529992556128-SzjikLv1LH-4 column=cf:ROW_KEY, timestamp=1529992556407, value=cnc_exeprgname:418 1529992556128-SzjikLv1LH-4 column=cf:cnc_rdspmeter[0], timestamp=1529992556407, value=cnc_rdspmeter[0]:0 1529992556128-SzjikLv1LH-4 column=cf:cnc_rdsvmeter, timestamp=1529992556407, value=cnc_r这篇关于Flume java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
