kafka upgrade issue from 0.10.1.1 to 0.11.0.2: offsets.topic.replication.factor

kafka version 0.10.1.1 to 0.11.0.2 upgrade

zk는 consume이 잘 되는데, kf 기반 consume은 잘 안 됨..

kafka consume is not working while zk consume works well.

현상:

zookeeper working

bin/kafka-console-consumer.sh --zookeeper $(hostname -f):2181 --topic test

kafka coordinator not working

bin/kafka-console-consumer.sh --bootstrap-server $(hostname -f):9092 --topic test

logs/server.log

[2018-03-20 21:37:18,827] ERROR [KafkaApi-15176] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

solved.

0.10버전까지 min(# of brokers, offsets.topic.replication.factor) 값을 취해서 썼는데, config 값 그대로 쓰고 에러를 내도록 변경되었다.

The config "offsets.topic.replication.factor(default 3)" was introduced from v0.9. And this config set the value of min(# of brokers, offsets.topic.replication.factor) until v0.10.1.

And it changes not to take min and just take config as set. check the notable changes in 0.11.0.0.

* The offsets.topic.replication.factor broker config is now enforced upon auto topic creation. Internal auto topic creation will fail with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets this replication factor requirement.

just append as followed, and restart. and then read from kafka works well!

offsets.topic.replication.factor=1

default kafka config 설정에는 위 설정이 포함되어 있었다..

아래 링크를 좀 더 꼼꼼히 봐야겠다.

ref: https://kafka.apache.org/documentation/#upgrade_1100_notable

+ 눈여겨서 볼만한 changes..

unclean.leader.election.enable 가 default로 true에서 false로 바뀐다. 

true면 data loss가 생길 수 있지만 availability 가 증가하고 

false면 data loss가 없는 대신, no live in-sync replicas면 에러가 나면서 partition을 쓸 수없는 상태가 된다. 즉, data에 대한 durability가 증가한다.

If unclean.leader.election.enable is set to true (enabled), an out-of-sync replica will be elected as leader when there is no live in-sync replica (ISR). This preserves the availability of the partition, but there is a chance of data loss.

If unclean.leader.election.enable is set to false and there are no live in-sync replicas, Kafka returns an error and the partition will be unavailable.

ref: 바닐라 버전은 아니지만 참조했다.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_kafka-component-guide/content/kafka-broker-settings.html