消费kafka数据出错Found a message larger than the maximum fetch size

具体错误如下: [code]ERROR [2017-01-05 02:59:26,534] com.elasticsearch.river.kafka.KafkaRiver: Unexpected Error Occurred ! kafka.common.MessageSizeTooLargeException: Found a message larger than the maximum fetch size of this consumer on topic mobileTopic partition 0 at fetch offset 2563424. Increase the fetch size, or decrease the maximum message size the broker will allow. ! at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:91) ~[kafka-consumer-for-hdfs.jar:na] ! at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33) ~[kafka-consumer-for-hdfs.jar:na] ! at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66) ~[kafka-consumer-for-hdfs.jar:na] ! at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58) ~[kafka-consumer-for-hdfs.jar:na] ! at com.elasticsearch.river.kafka.KafkaRiver$KafkaRiverFetcher.run(KafkaRiver.java:203) ~[original-kafka-consumer-for-hdfs.jar:1.1.5] ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51] ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] ! at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51][/code]
已邀请:

空心菜 - 心向阳光,茁壮成长

赞同来自: Ansible

这个就是你kafka中有一条消息大于你能获取的大小,要不增加客户端获取消息大小,增大:
fetch.message.max.bytes
的值,要不就是在kafka broker那控制只允许多大的数据进入。

要回复问题请先登录注册