本文主要是介绍Kafka 基本操作之节点退役，增加副本，数据迁移期间限制带宽使用，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
目录
一. 前言
二. 节点退役（Decommissioning brokers）
三. 增加副本（Increasing replication factor）
四. 数据迁移期间限制带宽使用（Limiting Bandwidth Usage during Data Migration）
一. 前言
Kafka 节点退役是指将一个节点永久地从集群中移除的过程。节点退役通常发生在节点故障、硬件损坏或集群扩容等情况下。节点退役的原理同样基于 Kafka 的分布式设计和复制机制。当一个节点需要退役时，需要将该节点上的所有分区的 Leader 副本迁移到其他节点上，并将该节点上的所有副本从集群中移除。
二. 节点退役（Decommissioning brokers）
原文引用：The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning brokers yet. As such, the admin has to come up with a reassignment plan to move the replica for all partitions hosted on the broker to be decommissioned, to the rest of the brokers. This can be relatively tedious as the reassignment needs to ensure that all the replicas are not moved from the decommissioned broker to only one other broker. To make this process effortless, we plan to add tooling support for decommissioning brokers in the future.
分区重新分配工具还不能自动为退役的 Broker 生成重新分配计划。因此，管理员必须制定一个重新分配计划，将 Broker 上托管的所有分区的副本移到其他 Broker 上，以使其退役。这可能会比较繁琐，因为重新分配需要确保所有副本不会从退役的 Broker 移动到仅一个其他 Broker。为了使这一过程变得轻松，我们计划在未来为退役 Broker 添加工具支持。
三. 增加副本（Increasing replication factor）
原文引用：Increasing the replication factor of an existing partition is easy. Just specify the extra replicas in the custom reassignment json file and use it with the --execute option to increase the replication factor of the specified partitions.
增加现有分区的复制因子很容易。只需在自定义重新分配 json 文件中指定额外的副本，并将其与 --execute 选项一起使用，即可增加指定分区的复制因子。
原文引用：For instance, the following example increases the replication factor of partition 0 of topic foo from 1 to 3. Before increasing the replication factor, the partition's only replica existed on broker 5. As part of increasing the replication factor, we will add more replicas on brokers 6 and 7.
The first step is to hand craft the custom reassignment plan in a json file:
例如，以下示例将 Topic foo 的分区0的复制因子从1增加到3。在增加复制因子之前，分区的唯一副本存在于 Broker 5上。作为增加复制因子的一部分，我们将在 Broker 6和7上添加更多副本。
第一步是在 json 文件中手工制作自定义重新分配计划：
> cat increase-replication-factor.json{"version":1,"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}原文引用：Then, use the json file with the --execute option to start the reassignment process:
然后，使用带有 --execute 选项的 json 文件来启动重新分配过程：
> bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file increase-replication-factor.json --executeCurrent partition replica assignment{"version":1,"partitions":[{"topic":"foo","partition":0,"replicas":[5]}]}Save this to use as the --reassignment-json-file option during rollbackSuccessfully started partition reassignment for foo-0原文引用：The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same increase-replication-factor.json (used with the --execute option) should be used with the --verify option:
--verify 选项可以与该工具一起使用，以检查分区重新分配的状态。请注意，相同的 increase-replication-factor.json（与 --execute 选项一起使用）应与 --verify 选项一起使用：
> bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file increase-replication-factor.json --verifyStatus of partition reassignment:Reassignment of partition [foo,0] is completed原文引用：You can also verify the increase in replication factor with the kafka-topics tool:
您还可以使用 Kafka-topics 工具验证复制因子的增加：
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic foo --describeTopic:foo PartitionCount:1 ReplicationFactor:3 Configs:Topic: foo Partition: 0 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7四. 数据迁移期间限制带宽使用（Limiting Bandwidth Usage during Data Migration）
原文引用：Kafka lets you apply a throttle to replication traffic, setting an upper bound on the bandwidth used to move replicas from machine to machine. This is useful when rebalancing a cluster, bootstrapping a new broker or adding or removing brokers, as it limits the impact these data-intensive operations will have on users.
Kafka 允许您对复制流量进行限制，限制了将副本从一台机器移动到另一台机器的带宽上限。这在重新平衡集群、引导新 Broker、添加或删除 Broker 时非常有用，因为它限制了这些数据密集型操作对用户的影响。
原文引用：There are two interfaces that can be used to engage a throttle. The simplest, and safest, is to apply a throttle when invoking the kafka-reassign-partitions.sh, but kafka-configs.sh can also be used to view and alter the throttle values directly.
So for example, if you were to execute a rebalance, with the below command, it would move partitions at no more than 50MB/s.
有两个接口可用于实现限制。最简单、最安全的方法是在调用 kafka-reallocate-partitions.sh 时应用限制，但 kafka-configs.sh 也可以用于直接查看和修改限制值。
例如，如果使用以下命令执行重新平衡，它将以不超过 50MB/s 的速度移动分区。
$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file bigger-cluster.json --throttle 50000000原文引用：When you execute this script you will see the throttle engage:
当您执行此脚本时，您将看到这个限制：
The inter-broker throttle limit was set to 50000000 B/sSuccessfully started partition reassignment for foo1-0原文引用：Should you wish to alter the throttle, during a rebalance, say to increase the throughput so it completes quicker, you can do this by re-running the execute command with the --additional option passing the same reassignment-json-file:
如果您希望在重新平衡过程中修改限制，比如说增加吞吐量，使其更快地完成，您可以通过使用 --additional 选项重新运行 execute 命令来实现这一点，该选项传递相同的 reassignment-json-file 文件：
$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --additional --execute --reassignment-json-file bigger-cluster.json --throttle 700000000The inter-broker throttle limit was set to 700000000 B/s原文引用：Once the rebalance completes the administrator can check the status of the rebalance using the --verify option. If the rebalance has completed, the throttle will be removed via the --verify command. It is important that administrators remove the throttle in a timely manner once rebalancing completes by running the command with the --verify option. Failure to do so could cause regular replication traffic to be throttled.
When the --verify option is executed, and the reassignment has completed, the script will confirm that the throttle was removed:
重新平衡完成后，管理员可以使用 --verify 选项检查重新平衡的状态。如果重新平衡已经完成，将通过 --verify 命令移除限制。重要的是，管理员在重新平衡完成后，通过运行带有 --verify 选项的命令，及时删除限制。否则，可能会导致常规复制流量被抑制。
当 --verify 选项被执行，并且重新分配已经完成时，脚本将确认限制已被删除：
> bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --verify --reassignment-json-file bigger-cluster.jsonStatus of partition reassignment:Reassignment of partition [my-topic,1] is completedReassignment of partition [my-topic,0] is completedClearing broker-level throttles on brokers 1,2,3Clearing topic-level throttles on topic my-topic原文引用：The administrator can also validate the assigned configs using the kafka-configs.sh. There are two pairs of throttle configuration used to manage the throttling process. First pair refers to the throttle value itself. This is configured, at a broker level, using the dynamic properties:
管理员还可以使用 kafka-configs.sh 验证分配的配置。有两对限制配置用于管理节流过程。第一对是指限制值本身。这是在 Broker 级别使用动态属性配置的：
leader.replication.throttled.rate
follower.replication.throttled.rate原文引用：Then there is the configuration pair of enumerated sets of throttled replicas:
然后是限流副本的枚举集合配置对：
leader.replication.throttled.replicas
follower.replication.throttled.replicas原文引用：Which are configured per topic.
All four config values are automatically assigned by kafka-reassign-partitions.sh (discussed below).
To view the throttle limit configuration:
按每个 Topic 配置。
所有4个配置值都由 kafka-remove-partitions.sh 自动分配（如下所述）。
要查看限流配置：
> bin/kafka-configs.sh --describe --bootstrap-server localhost:9092 --entity-type brokersConfigs for brokers '2' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000Configs for brokers '1' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000原文引用：This shows the throttle applied to both leader and follower side of the replication protocol. By default both sides are assigned the same throttled throughput value.
To view the list of throttled replicas:
这显示了应用于复制协议的 Leader 和 Follower 的限制。默认情况下，为双方分配相同的限流吞吐量值。
要查看限流副本的列表，请执行以下操作：
> bin/kafka-configs.sh --describe --bootstrap-server localhost:9092 --entity-type topicsConfigs for topic 'my-topic' are leader.replication.throttled.replicas=1:102,0:101,follower.replication.throttled.replicas=1:101,0:102原文引用：Here we see the leader throttle is applied to partition 1 on broker 102 and partition 0 on broker 101. Likewise the follower throttle is applied to partition 1 on broker 101 and partition 0 on broker 102.
在这里，我们看到 Leader 限流被应用于 Broker 102上的分区1和 Broker 101上的分区0。类似地，Follower 限流被应用于 Broker 101上的分区1和 Broker 102上的分区0。
原文引用：By default kafka-reassign-partitions.sh will apply the leader throttle to all replicas that exist before the rebalance, any one of which might be leader. It will apply the follower throttle to all move destinations. So if there is a partition with replicas on brokers 101,102, being reassigned to 102,103, a leader throttle, for that partition, would be applied to 101,102 and a follower throttle would be applied to 103 only.
默认情况下，kafka-reallocate-partitions.sh 将对重新平衡之前存在的所有副本应用 Leader 限流，其中任何一个副本都可能是 Leader。它将对所有移动目的地应用 Follower 限制。因此，如果代理101,102上有一个具有副本的分区被重新分配给102,103，则该分区的 Leader 限流值将应用于101,102，而 Follower 限流值将仅应用于103。
原文引用：If required, you can also use the --alter switch on kafka-configs.sh to alter the throttle configurations manually.
如果需要，还可以使用 kafka-configs.sh 上的 --alter 开关手动更改限流配置。
节流复制的安全使用（Safe usage of throttled replication）
原文引用：
Some care should be taken when using throttled replication. In particular:
(1) Throttle Removal:
The throttle should be removed in a timely manner once reassignment completes (by running kafka-reassign-partitions.sh --verify).(2) Ensuring Progress:
If the throttle is set too low, in comparison to the incoming write rate, it is possible for replication to not make progress. This occurs when:
使用限流复制时应格外小心。特别是：
（1） 限制移除：
重新分配完成后，应及时移除限制（通过运行 kafka-remove-partitions.sh --verify 移除）。
（2） 确保进度：
如果与传入写入速率相比，限制值设置得太低，则复制可能无法进行。这种情况发生在：
max(BytesInPerSec) > throttle原文引用：Where BytesInPerSec is the metric that monitors the write throughput of producers into each broker.
The administrator can monitor whether replication is making progress, during the rebalance, using the metric:
其中，BytesInPerSec 是监视生产者对每个 Broker 写入吞吐量的度量。
在重新平衡期间，管理员可以使用以下度量来监视复制是否正在进行：
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+)原文引用：The lag should constantly decrease during replication. If the metric does not decrease the administrator should increase the throttle throughput as described above.
在复制过程中，滞后应不断减少。如果 lag 度量没有减少，则管理员应如上所述增加限流吞吐量。
这篇关于Kafka 基本操作之节点退役，增加副本，数据迁移期间限制带宽使用的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
