kafka高可用失败问题:3broker,单杀一个broker就不能消费的问题探讨

发表于

测试集群有三个broker

broker0:9099
broker1:9091
broker2:9092

出现问题

1.当单独杀掉9091 或者 单独杀掉9092,集群可以重新平衡,并很快恢复正常工作.

2.当单独杀掉9099时,消费者就无法消费了(生产者可以正常工作)

对consumer源码进行了跟踪,分析了下流程
发现问题阻塞在 consumer选取coordinator时

1
2
ensureCoordinatorReady()
lookupCoordinator()

流程大概说就是

  1. consumer会从集群中选取一个broker作为coordinator,
  2. 然后group中的consumer会向coordinator发请求申请成为consumergroup中的leader
  3. 最后有1个consumer会成为consumerLeader ,其他consumer成为follower
  4. consumerLeader做分区分配任务,同步给coordinator
  5. consumerFollower从coordinator同步分区分配数据

问题出现在第1步

只有consumer将9099作为coordinator,并向9099broker发送申请时,会获得成功反馈

1
ClientResponse(receivedTimeMs=1515898295592, disconnected=false, request=ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@7bc58891, request=RequestSend(header={api_key=10,api_version=0,correlation_id=30285,client_id=consumer-1}, body={group_id=testGroup}), createdTimeMs=1515898285668, sendTimeMs=1515898295578), **responseBody={error_code=0,coordinator={node_id=0,host=192.168.0.108,port=9099}})**

当consumer依次尝试将9091,9092作为coordinator并依次发送申请时,都会失败
并得到如下反馈

1
2
ClientResponse(receivedTimeMs=1515897561105, disconnected=false, request=ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@488f3dd1, request=RequestSend(header={api_key=10,api_version=0,correlation_id=30281,client_id=consumer-1}, body={group_id=testGroup}), createdTimeMs=1515897558800, sendTimeMs=1515897561104),
responseBody=**{error_code=15,coordinator={node_id=-1,host=,port=-1}})**

为啥会是这样的呢

1
responseBody=**{error_code=15,coordinator={node_id=-1,host=,port=-1}})**

9099这个broker与众不同之处在哪
拿着这个错误翻墙找了一波google第一条就找到了
https://stackoverflow.com/questions/42362911/kafka-high-level-consumer-error-code-15

As it turned out, the all partitions of the consumer_offsets topic were located on dead nodes (nodes that I turned off and that will never come back). I solved the issue by shutting the cluster down, deleting the consumer_offsets topic from Zookeeper and then starting the cluster again.

然后上zk看了下__consumer_offsets 这个topic的分区副本情况,果然50个分区全在broker0上,我也是醉了

1
2
3
4
5
6
7
8
> bin/kafka-topics.sh --describe --zookeeper localhost:2182 --topic __consumer_offsets
Topic:__consumer_offsets    PartitionCount:50    ReplicationFactor:1    Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets    Partition: 0    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 1    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 2    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 3    Leader: 0
......

然后zk上删掉 __consumer_offsets目录, 重启集群每一个broker,此时zookeeper上__consumer_offsets还并没有生成,要开启消费者之后才会生成.

然后再观察__consumer_offsets,分区已经均匀分布在三个broker上面了

1
2
3
4
5
6
7
8
> bin/kafka-topics.sh --zookeeper localhost:2182 --describe --topic __consumer_offsets
Topic:__consumer_offsets    PartitionCount:50    ReplicationFactor:1    Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets    Partition: 0    Leader: 2    Replicas: 2    Isr: 2
    Topic: __consumer_offsets    Partition: 1    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 2    Leader: 1    Replicas: 1    Isr: 1
    Topic: __consumer_offsets    Partition: 3    Leader: 2    Replicas: 2    Isr: 2
    .........

以为问题出在这里
就是在集群还未充分启动完全的情况下,客户端就开始了消费者进程进行消费,导致用于内部偏移量收集的 topic:__consumer_offsets 只分布到了已启动的broker,当这个broker挂掉,集群便不可用了,所以当__consumer_offsets的所有partition均匀分布到多个broker,按道理就应该可以实现高可用了

于是,再重试一次,问题还是出现了……….简直蛋都碎了
高可用依旧失败,发现整个集群能否正常运行依赖于首个启动的broker

无奈,回头又看了下配置文件,有关于 __consumer_offsets的配置

1
2
3
4
5
6
############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

确保高可用需要改成3,修改成3之后,终于实现了高可用,3个broker ,只要存活任意1个broker就可以正常工作

总结

要保证__consumer_offsets的所有分区可用,offsets.topic.replication.factor至少配置为3
要保证集群所有的broker启动后,客户端再进行消费,否则会导致__consumer_offsets 主题的分区副本不能均匀分布到每个broker,这样一旦某个副本所在broker全挂掉,就不能消费了?
未解之谜???
offsets.topic.replication.factor为1的时候,不管__consumer_offsets分区怎么分布,反正只要首个启动的broker存活,集群就能工作……

更多思考

1.offsets.topic.replication.factor为1的时候,不管__consumer_offsets分区怎么分布

反正只要首个启动的broker存活,集群就能工作……

答: 这个是因为消费者消费offsets topics的副本如果只存在leader里存了一份，当刚好是这个leader的broker挂掉时，即使其他broker竞选成功为leader,他那里也没有消费者的offsets信息。
一个消费者组消费消息的offset只会存在__consumer_offsets 里的一个分区上面，所以如果副本只有一份，那就只会在特定的broker上

2.但是很奇怪的时，我把offsets.topic.replication.factor已经设置为2和3了，但是broker1挂掉时消费者还是无法消费，其他broker2或broker3挂掉没有关系

另外奇怪的是，offsets.topic.replication.factor设置为3后，查看__consumer_offsets的每个分区的副本数，还是只有一个，我是否要删掉__consumer_offsets再重新创建呢？
回答: 要在zk里面把__consumer_offsets节点和下面的所有节点都删掉,再重新启动集群做消费

参考

kafka高可用失败问题:3broker,单杀一个broker就不能消费的问题探讨