storm统计服务开启zookeeper、kafka 、Storm（sasl认证）

2024-01-18 16:36
文章标签 服务 统计 认证 开启 zookeeper kafka storm sasl

本文主要是介绍storm统计服务开启zookeeper、kafka 、Storm（sasl认证），希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

部署storm统计服务开启zookeeper、kafka 、Storm（sasl认证）

  • 当前测试验证结果：
    • 单独配置zookeeper 支持acl 设置用户和密码，在storm不修改代码情况下和kafka支持
    • 当kafka 开启ACL时，storm 和ccod模块不清楚配置用户和密码。
      • 使用python脚本连接kafka用户和密码是能成功发送消息。
  • 当前部署环境服务版本
服务IP模块版本信息
10.130.41.42zookeeperzookeeper-3.6.3
10.130.41.43kafkakafka_2.11-2.3.1
10.130.41.44stormapache-storm-1.2.4
zookeeper部署
  • 部署mongodb_1服务器的zookeeper
  • 传安装部署包和配置文件
[root@mongodb_1 ~]# su - storm
[storm@mongodb_1 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be  storm_node1.tar.gz
[storm@mongodb_1 ~]$ tar xvf storm_node1.tar.gz 
[storm@mongodb_1 ~]$ cd  storm_node1
[storm@mongodb_1 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_1 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • 修改zookeeper配置文件,然后在启动
[storm@mongodb_1 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_1 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_1:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_1 conf]$ vim  zk_server_jaas.conf
Server {org.apache.kafka.common.security.plain.PlainLoginModule requiredusername="zk_cluster"password="zk_cluster_passwd"user_admin="Admin@123";
};#拷贝kafka acl lib
[storm@mongodb_1 zookeeper-3.6.3]$  cd /home/storm/zookeeper-3.6.3
[storm@mongodb_1 zookeeper-3.6.3]$  mkdir zk_sasl_dependencies
[storm@mongodb_1 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_1 libs]$ cp kafka-clients-2.3.1.jar  lz4-java-1.6.0.jar  slf4j-api-1.7.26.jar  snappy-java-1.1.7.3.jar  ~/zookeeper-3.6.3/lib/
[storm@mongodb_1 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_1 ~]$ echo "1" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_1 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_1 bin]$ vim zkEnv.sh  #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
doCLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_1 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_1 bin]$ ./zkServer.sh  status 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
  • 部署mongodb_2服务器的zookeeper
  • 上传安装部署包和配置文件
[root@mongodb_2 ~]# su - storm
[storm@mongodb_2 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be storm_node2.tar.gz
[storm@mongodb_2 ~]$ tar xvf storm_node2.tar.gz 
[storm@mongodb_2 ~]$ cd  storm_node2
[storm@mongodb_2 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_2 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • 修改zookeeper配置文件,然后在启动
[storm@mongodb_2 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_2 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_2:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_2 conf]$ vim  zk_server_jaas.conf
Server {org.apache.kafka.common.security.plain.PlainLoginModule requiredusername="zk_cluster"password="zk_cluster_passwd"user_admin="Admin@123";
};#拷贝kafka acl lib
[storm@mongodb_2 zookeeper-3.6.3]$  cd /home/storm/zookeeper-3.6.3
[storm@mongodb_2 zookeeper-3.6.3]$  mkdir zk_sasl_dependencies
[storm@mongodb_2 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_2 libs]$ cp kafka-clients-2.3.1.jar  lz4-java-1.6.0.jar  slf4j-api-1.7.26.jar  snappy-java-1.1.7.3.jar  ~/zookeeper-3.6.3/lib/
[storm@mongodb_2 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_2 ~]$ echo "1" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_2 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_2 bin]$ vim zkEnv.sh  #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
doCLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_2 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_2 bin]$ ./zkServer.sh  status 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
  • 部署mongodb_3服务器的zookeeper
  • 上传安装部署包和配置文件
[root@mongodb_3 ~]# su - storm
[storm@mongodb_3 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be storm_node3.tar.gz
[storm@mongodb_3 ~]$ tar xvf storm_node3.tar.gz 
[storm@mongodb_3 ~]$ cd  storm_node3
[storm@mongodb_3 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_3 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • 修改zookeeper配置文件,然后在启动
[storm@mongodb_3 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_3 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_3:2182:3181
server.2:mongodb_3:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_3 conf]$ vim  zk_server_jaas.conf
Server {org.apache.zookeeper.server.auth.DigestLoginModule requiredadmin="Admin@123";
};#拷贝kafka acl lib
[storm@mongodb_3 zookeeper-3.6.3]$  cd /home/storm/zookeeper-3.6.3
[storm@mongodb_3 zookeeper-3.6.3]$  mkdir zk_sasl_dependencies
[storm@mongodb_3 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_3 libs]$ cp kafka-clients-2.3.1.jar  lz4-java-1.6.0.jar  slf4j-api-1.7.26.jar  snappy-java-1.1.7.3.jar  ~/zookeeper-3.6.3/lib/
[storm@mongodb_3 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_3 ~]$ echo "1" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_3 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_3 bin]$ vim zkEnv.sh  #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
doCLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_3 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_3 bin]$ ./zkServer.sh  status 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
  • zookeeper设置用户密码，并设置连接zookeeper白名单。（如已经开启sals认证，则选acl认证跳过）
    • 当前新部署平台，zookeeper未做下面操作addauth
[storm@mongodb_2 bin]$ ./zkCli.sh   -server mongodb_2:2182,mongodb_1:2182,mongodb_3:2182[zk: localhost:2181(CONNECTED) 0] addauth digest admin:123456  #admin为用户名, 123456为密码。
[zk: localhost:2181(CONNECTED) 1] setAcl / auth:admin:123456:cdrwa  
[zk: localhost:2181(CONNECTED) 2] getAcl /
'digest,'admin:0uek/hZ/V9fgiM35b0Z2226acMQ=
: cdrwa[storm@mongodb_2 bin]$ ./zkCli.sh  #登录连接成功后每次连接需要输入   addauth digest admin:123456
[zk: localhost:2181(CONNECTED) 0] ls /
Insufficient permission : /
[zk: localhost:2181(CONNECTED) 1]  addauth digest admin:123456
[zk: localhost:2181(CONNECTED) 2] ls /
[test, zookeeper][zk: localhost:2181(CONNECTED) 8] setAcl / ip:10.130.41.42:cdrwa,ip:10.130.41.43:cdrwa,ip:10.130.41.44:cdrwa,ip:10.130.41.86:cdrwa,ip:10.130.41.119:cdrwa,ip:127.0.0.1:cdrwa,auth:kafka:cdrwa  #设置连接zookeeper白名单
初始化storm配置
  • 需要三个随便找一个zookeeper进行登录
[storm@mongodb_1 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_1 bin]$ ./zkCli.sh 
[zk: localhost:2181(CONNECTED) 2]   addauth digest admin:123456
create  /storm config
create  /storm/config 1
create /umg_cloud cluster
create /umg_cloud/cluster 1
create /umg_cloud/cfg  1 
create /umg_cloud/cfg/notify  1
create  /storm/config/ENT_CALL_RECORD_LIST 0103290030,0103290035,0103290023,0103290021,0103290022,0103290031
create  /storm/config/KAFKA_BROKER_LIST mongodb_1:9092,mongodb_2:9092,mongodb_3:9092
create  /storm/config/KAFKA_ZOOKEEPER_CONNECT mongodb_1:2181,mongodb_2:2181,mongodb_3:2181
create  /storm/config/MAX_SPOUT_PENDING 100
create  /storm/config/MONGO_ADMIN admin
create  /storm/config/MONGO_BLOCKSIZE 100
create  /storm/config/MONGO_HOST mongodb_1:30000;mongodb_2:30000;mongodb_3:30000
create  /storm/config/MONGO_NAME ""
create  /storm/config/MONGO_PASSWORD ""
create  /storm/config/MONGO_POOLSIZE 100
create  /storm/config/MONGO_SWITCH 0
create  /storm/config/OFF_ON_DTS 0
create  /storm/config/REDIS_HOST mongodb_3:27381
create  /storm/config/REDIS_PASSWORD  ""
create  /storm/config/SEND_MESSAGE 0
create  /storm/config/TOPO_COMPUBOLT_NUM 4
create  /storm/config/TOPO_MONGOBOLT_NUM 16
create  /storm/config/TOPO_SPOUT_NUM 3
create  /storm/config/TOPO_WORK_NUM 2
create  /storm/config/ENTID_MESSAGE  1
create /storm/config/MESSAGE_TOPIC  1
kafka部署
  • 部署mongodb_1服务器的kafka
时[storm@mongodb_1 ~]$ cd ~/kafka_2.11-2.3.1/config/
[storm@mongodb_1 config]$ cat server.properties
broker.id=1
listeners=SASL_PLAINTEXT://10.130.41.42:9092
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:kafkacluster
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/storm//kafka_2.11-2.3.1/logs
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=mongodb_1:2181,mongodb_2:2181,mongodb_3:2181
compression.type=producer
zookeeper.set.acl=true
zookeeper.defaultAcl=root;123456:rwcd
zookeeper.connection.timeout.ms=6000
zookeeper.digest.enabled=true
group.initial.rebalance.delay.ms=0[storm@mongodb_1 config]$ more  sasl.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Admin@123";[storm@mongodb_1 config]$ cat kafka_server_jaas.conf
KafkaServer {org.apache.kafka.common.security.plain.PlainLoginModule requiredusername="kafkacluster"password="Admin@123"user_kafkacluster="Admin@123"user_admin="Admin@123"
};Client{org.apache.kafka.common.security.plain.PlainLoginModule required  username="admin"  password="Admin@123";  
};[storm@mongodb_1 ~]$mkdir /home/storm/kafka_2.11-2.3.1/log -p 
[storm@mongodb_1 ~]$cd  /home/storm/kafka_2.11-2.3.1/bin
[storm@mongodb_1 bin]$ vim   kafka-server-start.sh #把创建的文件 kafka_server_jaas.conf 加载到启动环境中
export KAFKA_HEAP_OPTS=$KAFKA_HEAP_OPTS" -Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf"[storm@mongodb_1 ~]$ ./kafka-server-start.sh -daemon config/server.properties
  • 部署mongodb_2服务器的kafka
[storm@mongodb_2 ~]$ cd ~/kafka_2.11-2.3.1/config/
[storm@mongodb_2 config]$ cat server.properties
broker.id=1
listeners=SASL_PLAINTEXT://10.130.41.42:9092
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:kafkacluster
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/storm//kafka_2.11-2.3.1/logs
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=mongodb_2:2181,mongodb_2:2181,mongodb_3:2181
compression.type=producer
zookeeper.set.acl=true
zookeeper.defaultAcl=root;123456:rwcd
zookeeper.connection.timeout.ms=6000
zookeeper.digest.enabled=true
group.initial.rebalance.delay.ms=0[storm@mongodb_2 config]$ more  sasl.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Admin@123";[storm@mongodb_2 config]$ cat kafka_server_jaas.conf
KafkaServer {org.apache.kafka.common.security.plain.PlainLoginModule requiredusername="kafkacluster"password="Admin@123"user_kafkacluster="Admin@123"user_admin="Admin@123"
};Client{org.apache.kafka.common.security.plain.PlainLoginModule required  username="admin"  password="Admin@123";  
};[storm@mongodb_2 ~]$mkdir /home/storm/kafka_2.11-2.3.1/log -p 
[storm@mongodb_2 ~]$cd  /home/storm/kafka_2.11-2.3.1/bin
[storm@mongodb_2 bin]$ vim   kafka-server-start.sh #把创建的文件 kafka_server_jaas.conf 加载到启动环境中
export KAFKA_HEAP_OPTS=$KAFKA_HEAP_OPTS" -Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf"[storm@mongodb_2 ~]$ ./kafka-server-start.sh -daemon config/server.properties
  • 部署mongodb_3服务器的kafka
[storm@mongodb_3 ~]$ cd ~/kafka_2.11-2.3.1/config/
[storm@mongodb_3 config]$ cat server.properties
broker.id=1
listeners=SASL_PLAINTEXT://10.130.41.42:9092
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:kafkacluster
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/storm//kafka_2.11-2.3.1/logs
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=mongodb_3:2181,mongodb_2:2181,mongodb_3:2181
compression.type=producer
zookeeper.set.acl=true
zookeeper.defaultAcl=root;123456:rwcd
zookeeper.connection.timeout.ms=6000
zookeeper.digest.enabled=true
group.initial.rebalance.delay.ms=0[storm@mongodb_3 config]$ more  sasl.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Admin@123";[storm@mongodb_3 config]$ cat kafka_server_jaas.conf
KafkaServer {org.apache.kafka.common.security.plain.PlainLoginModule requiredusername="kafkacluster"password="Admin@123"user_kafkacluster="Admin@123"user_admin="Admin@123"
};Client{org.apache.kafka.common.security.plain.PlainLoginModule required  username="admin"  password="Admin@123";  
};[storm@mongodb_3 ~]$mkdir /home/storm/kafka_2.11-2.3.1/log -p 
[storm@mongodb_3 ~]$cd  /home/storm/kafka_2.11-2.3.1/bin
[storm@mongodb_3 bin]$ vim   kafka-server-start.sh #把创建的文件 kafka_server_jaas.conf 加载到启动环境中
export KAFKA_HEAP_OPTS=$KAFKA_HEAP_OPTS" -Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf"[storm@mongodb_3 ~]$ ./kafka-server-start.sh -daemon config/server.properties
  • 初始化创建topics,在其中一台kafka上面操作即可
[storm@mongodb_1 kafka_2.11-2.3.1]$  cd kafka_2.11-2.3.1/[storm@mongodb_1 kafka_2.11-2.3.1]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic ent_record_fastdfs_url
  • 给对应topics配置acls权限
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181 --add --allow-principal User:admin --producer --topic ent_record_fastdfs_url
  • 查看那些topic设置alcs
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181 --list 
Current ACLs for resource `Topic:LITERAL:ent_record_fastdfs_url`: User:admin has Allow permission for operations: Describe from hosts: *User:admin has Allow permission for operations: Create from hosts: *User:admin has Allow permission for operations: Write from hosts: *
storm部署

  • 当前版本不支持连接kafka支持 ACL， 需要修改storm jar包，待研发布支持版本。

  • 下面配置文件配置已支持zookeeper连接设置ACL

  • 部署mongodb_1服务器的storm

[storm@mongodb_1 ~]$ cd ~/apache-storm-1.2.4/conf/
[storm@mongodb_1 conf]$ vim storm.yaml 
storm.zookeeper.factory.class: "org.apache.storm.security.auth.SASLAuthenticationProvider"
storm.zookeeper.auth.provider.1: "org.apache.storm.security.auth.digest.DigestSaslServerCallbackHandler"
storm.zookeeper.auth.provider.2: "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"
storm.zookeeper.superACL: "sasl:admin"
storm.zookeeper.servers:- "10.130.41.42"- "10.130.41.43"- "10.130.41.44"
nimbus.seeds: ["10.130.41.42","10.130.41.43","10.130.41.44"]
ui.port: 8888
storm.local.dir: "/home/storm/apache-storm-1.2.4/data"
supervisor.slots.ports:- 6700- 6701- 6702- 6703- 6704- 6705- 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.user: admin
storm.zookeeper.auth.password: Admin@123
storm.zookeeper.auth.digest.1: admin=Admin@123
java.security.auth.login.config: /home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf
worker.childopts: '-Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf'
worker.childopts: '-Djava.security.auth.login.config=/home/storm/zookeeper-3.6.3/conf/jaas.conf'
[storm@mongodb_1 ~]$  mkdir -p /home/storm/apache-storm-1.2.4/data
[storm@mongodb_1 ~]$  cd  /home/storm/apache-storm-1.2.4/bin
[storm@mongodb_1 bin]$  nohup ./storm nimbus &
[storm@mongodb_1 bin]$  nohup ./storm ui &
[storm@mongodb_1 bin]$ nohup ./storm supervisor &[storm@mongodb_1 bin]$ jps  #查看是否正常运行
1808 worker
1792 LogWriter
1907 worker
29171 Supervisor
1891 LogWriter
20328 core
25224 Kafka
31867 ZooKeeperMain
2444 Jps
18957 QuorumPeerMain
32334 nimbus
  • 部署mongodb_2服务器的storm
[storm@mongodb_2 ~]$ cd ~/apache-storm-1.2.4/conf/
[storm@mongodb_2 conf]$ vim storm.yaml 
storm.zookeeper.factory.class: "org.apache.storm.security.auth.SASLAuthenticationProvider"
storm.zookeeper.auth.provider.1: "org.apache.storm.security.auth.digest.DigestSaslServerCallbackHandler"
storm.zookeeper.auth.provider.2: "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"
storm.zookeeper.superACL: "sasl:admin"
storm.zookeeper.servers:- "10.130.41.42"- "10.130.41.43"- "10.130.41.44"
nimbus.seeds: ["10.130.41.42","10.130.41.43","10.130.41.44"]
ui.port: 8888
storm.local.dir: "/home/storm/apache-storm-1.2.4/data"
supervisor.slots.ports:- 6700- 6701- 6702- 6703- 6704- 6705- 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.user: admin
storm.zookeeper.auth.password: Admin@123
storm.zookeeper.auth.digest.1: admin=Admin@123
java.security.auth.login.config: /home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf
worker.childopts: '-Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf'
worker.childopts: '-Djava.security.auth.login.config=/home/storm/zookeeper-3.6.3/conf/jaas.conf'
[storm@mongodb_2 ~]$  mkdir -p /home/storm/apache-storm-1.2.4/data
[storm@mongodb_2 ~]$  cd  /home/storm/apache-storm-1.2.4/bin
[storm@mongodb_2 bin]$  nohup ./storm nimbus &
[storm@mongodb_2 bin]$  nohup ./storm ui &
[storm@mongodb_2 bin]$ nohup ./storm supervisor &[storm@mongodb_2 bin]$ jps  #查看是否正常运行
1808 worker
1792 LogWriter
1907 worker
29171 Supervisor
1891 LogWriter
20328 core
25224 Kafka
31867 ZooKeeperMain
2444 Jps
18957 QuorumPeerMain
32334 nimbus
  • 部署mongodb_3服务器的storm
[storm@mongodb_3 ~]$ cd ~/apache-storm-1.2.4/conf/
[storm@mongodb_3 conf]$ vim storm.yaml 
storm.zookeeper.factory.class: "org.apache.storm.security.auth.SASLAuthenticationProvider"
storm.zookeeper.auth.provider.1: "org.apache.storm.security.auth.digest.DigestSaslServerCallbackHandler"
storm.zookeeper.auth.provider.2: "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"
storm.zookeeper.superACL: "sasl:admin"
storm.zookeeper.servers:- "10.130.41.42"- "10.130.41.43"- "10.130.41.44"
nimbus.seeds: ["10.130.41.42","10.130.41.43","10.130.41.44"]
ui.port: 8888
storm.local.dir: "/home/storm/apache-storm-1.2.4/data"
supervisor.slots.ports:- 6700- 6701- 6702- 6703- 6704- 6705- 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.user: admin
storm.zookeeper.auth.password: Admin@123
storm.zookeeper.auth.digest.1: admin=Admin@123
java.security.auth.login.config: /home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf
worker.childopts: '-Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf'
worker.childopts: '-Djava.security.auth.login.config=/home/storm/zookeeper-3.6.3/conf/jaas.conf'
[storm@mongodb_3 ~]$  mkdir -p /home/storm/apache-storm-1.2.4/data
[storm@mongodb_3 ~]$  cd  /home/storm/apache-storm-1.2.4/bin
[storm@mongodb_3 bin]$  nohup ./storm nimbus &
[storm@mongodb_3 bin]$  nohup ./storm ui &
[storm@mongodb_3 bin]$ nohup ./storm supervisor &[storm@mongodb_3 bin]$ jps  #查看是否正常运行
1808 worker
1792 LogWriter
1907 worker
29171 Supervisor
1891 LogWriter
20328 core
25224 Kafka
31867 ZooKeeperMain
2444 Jps
18957 QuorumPeerMain
32334 nimbus
  • storm的主节点nimbus在第一台上启动拓扑图
  • 启动拓扑图时需要指定stormStatics.jar 包，如果使用新storm安装包没有，请自行找测试部要stormStatics.jar包
[storm@mongodb_1 bin]$ cd /home/storm/apache-storm-1.2.4/bin[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.FastdfsUrlTopology FastdfsUrlTopology mongodb_1,mongodb_2,mongodb_3 2181 ent_record_fastdfs_url /brokers
  • 使用storm-ui查看刚启动拓扑是否正常
  • 访问第一台的服务器ip端口是8888

本文由mdnice多平台发布

这篇关于storm统计服务开启zookeeper、kafka 、Storm（sasl认证）的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！


http://www.chinasem.cn/article/619606。

相关文章

浅析Spring Security认证过程

浅析Spring Security认证过程

类图 为了方便理解Spring Security认证流程，特意画了如下的类图，包含相关的核心认证类 概述 核心验证器 AuthenticationManager 该对象提供了认证方法的入口，接收一个Authentiaton对象作为参数; public interface AuthenticationManager {Authentication authenticate(Authenti
阅读更多...
Zookeeper安装和配置说明

Zookeeper安装和配置说明

一、Zookeeper的搭建方式 Zookeeper安装方式有三种，单机模式和集群模式以及伪集群模式。 ■ 单机模式：Zookeeper只运行在一台服务器上，适合测试环境； ■ 伪集群模式：就是在一台物理机上运行多个Zookeeper 实例； ■ 集群模式：Zookeeper运行于一个集群上，适合生产环境，这个计算机集群被称为一个“集合体”（ensemble） Zookeeper通过复制来实现
阅读更多...
hadoop开启回收站配置

hadoop开启回收站配置

开启回收站功能，可以将删除的文件在不超时的情况下，恢复原数据，起到防止误删除、备份等作用。 开启回收站功能参数说明 （1）默认值fs.trash.interval = 0，0表示禁用回收站；其他值表示设置文件的存活时间。 （2）默认值fs.trash.checkpoint.interval = 0，检查回收站的间隔时间。如果该值为0，则该值设置和fs.trash.interval的参数值相等。
阅读更多...
hdu1496（用hash思想统计数目）

hdu1496（用hash思想统计数目）

作为一个刚学hash的孩子，感觉这道题目很不错，灵活的运用的数组的下标。 解题步骤：如果用常规方法解，那么时间复杂度为O（n^4）,肯定会超时，然后参考了网上的解题方法，将等式分成两个部分，a*x1^2+b*x2^2和c*x3^2+d*x4^2， 各自作为数组的下标，如果两部分相加为0，则满足等式； 代码如下： #include<iostream>#include<algorithm
阅读更多...
【区块链 + 人才服务】可信教育区块链治理系统 | FISCO BCOS应用案例

【区块链 + 人才服务】可信教育区块链治理系统 | FISCO BCOS应用案例

伴随着区块链技术的不断完善，其在教育信息化中的应用也在持续发展。利用区块链数据共识、不可篡改的特性， 将与教育相关的数据要素在区块链上进行存证确权，在确保数据可信的前提下，促进教育的公平、透明、开放，为教育教学质量提升赋能，实现教育数据的安全共享、高等教育体系的智慧治理。 可信教育区块链治理系统的顶层治理架构由教育部、高校、企业、学生等多方角色共同参与建设、维护，支撑教育资源共享、教学质量评估、
阅读更多...
搭建Kafka+zookeeper集群调度

搭建Kafka+zookeeper集群调度

前言 硬件环境 172.18.0.5        kafkazk1        Kafka+zookeeper                Kafka Broker集群 172.18.0.6        kafkazk2        Kafka+zookeeper                Kafka Broker集群 172.18.0.7        kafkazk3
阅读更多...
【区块链 + 人才服务】区块链集成开发平台 | FISCO BCOS应用案例

【区块链 + 人才服务】区块链集成开发平台 | FISCO BCOS应用案例

随着区块链技术的快速发展，越来越多的企业开始将其应用于实际业务中。然而，区块链技术的专业性使得其集成开发成为一项挑战。针对此，广东中创智慧科技有限公司基于国产开源联盟链 FISCO BCOS 推出了区块链集成开发平台。该平台基于区块链技术，提供一套全面的区块链开发工具和开发环境，支持开发者快速开发和部署区块链应用。此外，该平台还可以提供一套全面的区块链开发教程和文档，帮助开发者快速上手区块链开发。
阅读更多...
flume系列之：查看flume系统日志、查看统计flume日志类型、查看flume日志

flume系列之：查看flume系统日志、查看统计flume日志类型、查看flume日志

遍历指定目录下多个文件查找指定内容 服务器系统日志会记录flume相关日志 cat /var/log/messages |grep -i oom 查找系统日志中关于flume的指定日志 import osdef search_string_in_files(directory, search_string):count = 0
阅读更多...
hdu4267区间统计

hdu4267区间统计

题意：给一些数，有两种操作，一种是在[a,b] 区间内，对（i - a）% k == 0 的加value,另一种操作是询问某个位置的值。 import java.io.BufferedInputStream;import java.io.BufferedReader;import java.io.IOException;import java.io.InputStream;import
阅读更多...
hdu4417区间统计

hdu4417区间统计

给你一个数列{An}，然后有m次查询，每次查询一段区间 [l,r] <= h 的值的个数。 import java.io.BufferedInputStream;import java.io.BufferedReader;import java.io.IOException;import java.io.InputStream;import java.io.InputStreamRead
阅读更多...

Copyright China编程 23002807@qq.com

沪ICP备2023033072号-2