Kafka Consumer is not reading the topic data through Java2019 Community Moderator ElectionKafka consumer not returning any eventsHow do I read / convert an InputStream into a String in Java?is it possible to exclude a topic from a broker in kafkakafka consumer to dynamically detect topics addedkafka custom consumer to read incoming recordsHow to get the relationship of consumer group and topicsConsumer not receiving message in Apache KafkaKafka: Consumer can't read records from topicKafka Consumer seektoBeginningKafka consumer to unzip stream of gz files and readRe-use of existing kafka consumer that has been idle for long duration
When were linguistics departments first established
Counter-example to the existence of left Bousfield localization of combinatorial model category
Can the druid cantrip Thorn Whip really defeat a water weird this easily?
What is the definition of "Natural Selection"?
Is a lawful good "antagonist" effective?
Make a transparent 448*448 image
Need some help with my first LaTeX drawing…
Is having access to past exams cheating and, if yes, could it be proven just by a good grade?
Time dilation for a moving electronic clock
How can I discourage/prevent PCs from using door choke-points?
Deleting missing values from a dataset
Is going from continuous data to categorical always wrong?
The three point beverage
Does splitting a potentially monolithic application into several smaller ones help prevent bugs?
Should we release the security issues we found in our product as CVE or we can just update those on weekly release notes?
What to do when during a meeting client people start to fight (even physically) with each others?
Plywood subfloor won't screw down in a trailer home
Want to switch to tankless, but can I use my existing wiring?
Replacing Windows 7 security updates with anti-virus?
Word for a person who has no opinion about whether god exists
What is the difference between "shut" and "close"?
My story is written in English, but is set in my home country. What language should I use for the dialogue?
What exactly is the purpose of connection links straped between the rocket and the launch pad
Why don't MCU characters ever seem to have language issues?
Kafka Consumer is not reading the topic data through Java
2019 Community Moderator ElectionKafka consumer not returning any eventsHow do I read / convert an InputStream into a String in Java?is it possible to exclude a topic from a broker in kafkakafka consumer to dynamically detect topics addedkafka custom consumer to read incoming recordsHow to get the relationship of consumer group and topicsConsumer not receiving message in Apache KafkaKafka: Consumer can't read records from topicKafka Consumer seektoBeginningKafka consumer to unzip stream of gz files and readRe-use of existing kafka consumer that has been idle for long duration
I am sending input JSON data to Kafka topic. I can see the same JSON data in kafka consumer by using below command.
.binwindowskafka-console-consumer.bat --bootstrap-server localhost:9092 --topic mytopic --from-beginning
Case1:
But when I am trying to read the records from kafka consumer (java) not getting any records in java console. I have tried the examples given in this link https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Case2:
If I send any message from the producer (command window) the same getting in the consumer(command window). and able to see the same records in the java console. This scenario is working.
If I submit the data to the topic through java program. Then the same JSON data appearing in consumer(command window). But not coming in java console.Case1 is not working.
Case2 is working. Please let me know any configuration need to be done?
java apache-kafka kafka-consumer-api
edited Mar 7 at 16:03
iehrlich
3,22242640
asked Mar 7 at 10:59
Praveen KumarPraveen Kumar
114
add a comment |
I am sending input JSON data to Kafka topic. I can see the same JSON data in kafka consumer by using below command.
.binwindowskafka-console-consumer.bat --bootstrap-server localhost:9092 --topic mytopic --from-beginning
Case1:
But when I am trying to read the records from kafka consumer (java) not getting any records in java console. I have tried the examples given in this link https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Case2:
If I send any message from the producer (command window) the same getting in the consumer(command window). and able to see the same records in the java console. This scenario is working.
If I submit the data to the topic through java program. Then the same JSON data appearing in consumer(command window). But not coming in java console.Case1 is not working.
Case2 is working. Please let me know any configuration need to be done?
java apache-kafka kafka-consumer-api
edited Mar 7 at 16:03
iehrlich
3,22242640
asked Mar 7 at 10:59
Praveen KumarPraveen Kumar
114
so you start consuming from java and then send new data to the topic and no messages appear (case 1)?
– tgrez
Mar 7 at 11:14
Are you running case2 using only one JVM?
– Lyubomir Papazov
Mar 7 at 11:19
Possible duplicate of Kafka consumer not returning any events
– wardziniak
Mar 7 at 11:23
1
How did you write your producer? Share us your code.
– JR ibkr
Mar 7 at 15:39
Start both console-consumer and custom-consumer. Then run your java producer. Also, make sure that your producer sends messages to Kafka. Do you see messages on both consumers ?
– JR ibkr
Mar 7 at 15:46
add a comment |
I am sending input JSON data to Kafka topic. I can see the same JSON data in kafka consumer by using below command.
.binwindowskafka-console-consumer.bat --bootstrap-server localhost:9092 --topic mytopic --from-beginning
Case1:
But when I am trying to read the records from kafka consumer (java) not getting any records in java console. I have tried the examples given in this link https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Case2:
If I send any message from the producer (command window) the same getting in the consumer(command window). and able to see the same records in the java console. This scenario is working.
If I submit the data to the topic through java program. Then the same JSON data appearing in consumer(command window). But not coming in java console.Case1 is not working.
Case2 is working. Please let me know any configuration need to be done?
java apache-kafka kafka-consumer-api
edited Mar 7 at 16:03
iehrlich
3,22242640
asked Mar 7 at 10:59
Praveen KumarPraveen Kumar
114
I am sending input JSON data to Kafka topic. I can see the same JSON data in kafka consumer by using below command.
.binwindowskafka-console-consumer.bat --bootstrap-server localhost:9092 --topic mytopic --from-beginning
Case1:
But when I am trying to read the records from kafka consumer (java) not getting any records in java console. I have tried the examples given in this link https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Case2:
If I send any message from the producer (command window) the same getting in the consumer(command window). and able to see the same records in the java console. This scenario is working.
If I submit the data to the topic through java program. Then the same JSON data appearing in consumer(command window). But not coming in java console.Case1 is not working.
Case2 is working. Please let me know any configuration need to be done?
java apache-kafka kafka-consumer-api
java apache-kafka kafka-consumer-api
edited Mar 7 at 16:03
iehrlich
3,22242640
asked Mar 7 at 10:59
Praveen KumarPraveen Kumar
114
edited Mar 7 at 16:03
iehrlich
3,22242640
asked Mar 7 at 10:59
Praveen KumarPraveen Kumar
114
edited Mar 7 at 16:03
iehrlich
3,22242640
edited Mar 7 at 16:03
iehrlich
3,22242640
edited Mar 7 at 16:03
iehrlich
3,22242640
3,22242640
asked Mar 7 at 10:59
Praveen KumarPraveen Kumar
114
asked Mar 7 at 10:59
Praveen KumarPraveen Kumar
114
asked Mar 7 at 10:59
Praveen KumarPraveen Kumar
114
114
so you start consuming from java and then send new data to the topic and no messages appear (case 1)?
– tgrez
Mar 7 at 11:14
Are you running case2 using only one JVM?
– Lyubomir Papazov
Mar 7 at 11:19
Possible duplicate of Kafka consumer not returning any events
– wardziniak
Mar 7 at 11:23
1
How did you write your producer? Share us your code.
– JR ibkr
Mar 7 at 15:39
Start both console-consumer and custom-consumer. Then run your java producer. Also, make sure that your producer sends messages to Kafka. Do you see messages on both consumers ?
– JR ibkr
Mar 7 at 15:46
add a comment |
so you start consuming from java and then send new data to the topic and no messages appear (case 1)?
– tgrez
Mar 7 at 11:14
Are you running case2 using only one JVM?
– Lyubomir Papazov
Mar 7 at 11:19
Possible duplicate of Kafka consumer not returning any events
– wardziniak
Mar 7 at 11:23
1
How did you write your producer? Share us your code.
– JR ibkr
Mar 7 at 15:39
Start both console-consumer and custom-consumer. Then run your java producer. Also, make sure that your producer sends messages to Kafka. Do you see messages on both consumers ?
– JR ibkr
Mar 7 at 15:46
so you start consuming from java and then send new data to the topic and no messages appear (case 1)?
– tgrez
Mar 7 at 11:14
so you start consuming from java and then send new data to the topic and no messages appear (case 1)?
– tgrez
Mar 7 at 11:14
Are you running case2 using only one JVM?
– Lyubomir Papazov
Mar 7 at 11:19
Are you running case2 using only one JVM?
– Lyubomir Papazov
Mar 7 at 11:19
Possible duplicate of Kafka consumer not returning any events
– wardziniak
Mar 7 at 11:23
Possible duplicate of Kafka consumer not returning any events
– wardziniak
Mar 7 at 11:23
1
1
How did you write your producer? Share us your code.
– JR ibkr
Mar 7 at 15:39
How did you write your producer? Share us your code.
– JR ibkr
Mar 7 at 15:39
Start both console-consumer and custom-consumer. Then run your java producer. Also, make sure that your producer sends messages to Kafka. Do you see messages on both consumers ?
– JR ibkr
Mar 7 at 15:46
Start both console-consumer and custom-consumer. Then run your java producer. Also, make sure that your producer sends messages to Kafka. Do you see messages on both consumers ?
– JR ibkr
Mar 7 at 15:46
add a comment |
3 Answers
3
active
oldest
votes
You need to set the ConsumerConfig.AUTO_OFFSET_RESET_CONFIG in order to read from the beginning.
kafkaConsumerProperties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
And also make sure that you are not running different-different consumer processes using the same consumer group ID, as the data from one partition might be read from one process and another will not see anything.
answered Mar 7 at 12:13
Nishu TayalNishu Tayal
12.8k73483
add a comment |
Assuming producer & consumer code is correct.
- Stop all consumers.
- Reset your consumer group
$KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server
localhost:9092 --group group_name --topic topic_name --reset-offsets
--to-earliest --execute
- Now start your consumer
This should fix your issue.
Here are some Kafka consumer properties:
bootstrap.servers: 'localhost:9092'
group.id: 'group_id'
auto.offset.reset: 'earliest'
key.deserializer: 'org.apache.kafka.common.serialization.*' //Replace * with class
value.deserializer: 'org.apache.kafka.common.serialization.*'
Thanks
answered Mar 7 at 13:26
Ashish BhosleAshish Bhosle
330110
add a comment |
You need to make sure you are not sending empty lines at the end and only consuming the latest - use auto.offset.reset: 'earliest' like above, or properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
And then, consumer.seekToBeginning(consumer.assignment()); to make sure.
answered Mar 8 at 7:23
Wisani SalaniWisani Salani
413
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55042190%2fkafka-consumer-is-not-reading-the-topic-data-through-java%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
You need to set the ConsumerConfig.AUTO_OFFSET_RESET_CONFIG in order to read from the beginning.
kafkaConsumerProperties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
And also make sure that you are not running different-different consumer processes using the same consumer group ID, as the data from one partition might be read from one process and another will not see anything.
answered Mar 7 at 12:13
Nishu TayalNishu Tayal
12.8k73483
add a comment |
You need to set the ConsumerConfig.AUTO_OFFSET_RESET_CONFIG in order to read from the beginning.
kafkaConsumerProperties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
And also make sure that you are not running different-different consumer processes using the same consumer group ID, as the data from one partition might be read from one process and another will not see anything.
answered Mar 7 at 12:13
Nishu TayalNishu Tayal
12.8k73483
add a comment |
You need to set the ConsumerConfig.AUTO_OFFSET_RESET_CONFIG in order to read from the beginning.
kafkaConsumerProperties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
And also make sure that you are not running different-different consumer processes using the same consumer group ID, as the data from one partition might be read from one process and another will not see anything.
answered Mar 7 at 12:13
Nishu TayalNishu Tayal
12.8k73483
You need to set the ConsumerConfig.AUTO_OFFSET_RESET_CONFIG in order to read from the beginning.
kafkaConsumerProperties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
And also make sure that you are not running different-different consumer processes using the same consumer group ID, as the data from one partition might be read from one process and another will not see anything.
answered Mar 7 at 12:13
Nishu TayalNishu Tayal
12.8k73483
answered Mar 7 at 12:13
Nishu TayalNishu Tayal
12.8k73483
answered Mar 7 at 12:13
Nishu TayalNishu Tayal
12.8k73483
answered Mar 7 at 12:13
Nishu TayalNishu Tayal
12.8k73483
12.8k73483
add a comment |
add a comment |
Assuming producer & consumer code is correct.
- Stop all consumers.
- Reset your consumer group
$KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server
localhost:9092 --group group_name --topic topic_name --reset-offsets
--to-earliest --execute
- Now start your consumer
This should fix your issue.
Here are some Kafka consumer properties:
bootstrap.servers: 'localhost:9092'
group.id: 'group_id'
auto.offset.reset: 'earliest'
key.deserializer: 'org.apache.kafka.common.serialization.*' //Replace * with class
value.deserializer: 'org.apache.kafka.common.serialization.*'
Thanks
answered Mar 7 at 13:26
Ashish BhosleAshish Bhosle
330110
add a comment |
Assuming producer & consumer code is correct.
- Stop all consumers.
- Reset your consumer group
$KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server
localhost:9092 --group group_name --topic topic_name --reset-offsets
--to-earliest --execute
- Now start your consumer
This should fix your issue.
Here are some Kafka consumer properties:
bootstrap.servers: 'localhost:9092'
group.id: 'group_id'
auto.offset.reset: 'earliest'
key.deserializer: 'org.apache.kafka.common.serialization.*' //Replace * with class
value.deserializer: 'org.apache.kafka.common.serialization.*'
Thanks
answered Mar 7 at 13:26
Ashish BhosleAshish Bhosle
330110
add a comment |
Assuming producer & consumer code is correct.
- Stop all consumers.
- Reset your consumer group
$KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server
localhost:9092 --group group_name --topic topic_name --reset-offsets
--to-earliest --execute
- Now start your consumer
This should fix your issue.
Here are some Kafka consumer properties:
bootstrap.servers: 'localhost:9092'
group.id: 'group_id'
auto.offset.reset: 'earliest'
key.deserializer: 'org.apache.kafka.common.serialization.*' //Replace * with class
value.deserializer: 'org.apache.kafka.common.serialization.*'
Thanks
answered Mar 7 at 13:26
Ashish BhosleAshish Bhosle
330110
Assuming producer & consumer code is correct.
- Stop all consumers.
- Reset your consumer group
$KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server
localhost:9092 --group group_name --topic topic_name --reset-offsets
--to-earliest --execute
- Now start your consumer
This should fix your issue.
Here are some Kafka consumer properties:
bootstrap.servers: 'localhost:9092'
group.id: 'group_id'
auto.offset.reset: 'earliest'
key.deserializer: 'org.apache.kafka.common.serialization.*' //Replace * with class
value.deserializer: 'org.apache.kafka.common.serialization.*'
Thanks
answered Mar 7 at 13:26
Ashish BhosleAshish Bhosle
330110
edited Mar 8 at 6:09
answered Mar 7 at 13:26
Ashish BhosleAshish Bhosle
330110
answered Mar 7 at 13:26
Ashish BhosleAshish Bhosle
330110
answered Mar 7 at 13:26
Ashish BhosleAshish Bhosle
330110
330110
add a comment |
add a comment |
You need to make sure you are not sending empty lines at the end and only consuming the latest - use auto.offset.reset: 'earliest' like above, or properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
And then, consumer.seekToBeginning(consumer.assignment()); to make sure.
answered Mar 8 at 7:23
Wisani SalaniWisani Salani
413
add a comment |
You need to make sure you are not sending empty lines at the end and only consuming the latest - use auto.offset.reset: 'earliest' like above, or properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
And then, consumer.seekToBeginning(consumer.assignment()); to make sure.
answered Mar 8 at 7:23
Wisani SalaniWisani Salani
413
add a comment |
You need to make sure you are not sending empty lines at the end and only consuming the latest - use auto.offset.reset: 'earliest' like above, or properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
And then, consumer.seekToBeginning(consumer.assignment()); to make sure.
answered Mar 8 at 7:23
Wisani SalaniWisani Salani
413
You need to make sure you are not sending empty lines at the end and only consuming the latest - use auto.offset.reset: 'earliest' like above, or properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
And then, consumer.seekToBeginning(consumer.assignment()); to make sure.
answered Mar 8 at 7:23
Wisani SalaniWisani Salani
413
answered Mar 8 at 7:23
Wisani SalaniWisani Salani
413
answered Mar 8 at 7:23
Wisani SalaniWisani Salani
413
answered Mar 8 at 7:23
Wisani SalaniWisani Salani
413
413
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55042190%2fkafka-consumer-is-not-reading-the-topic-data-through-java%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
so you start consuming from java and then send new data to the topic and no messages appear (case 1)?
– tgrez
Mar 7 at 11:14
Are you running case2 using only one JVM?
– Lyubomir Papazov
Mar 7 at 11:19
Possible duplicate of Kafka consumer not returning any events
– wardziniak
Mar 7 at 11:23
1
How did you write your producer? Share us your code.
– JR ibkr
Mar 7 at 15:39
Start both console-consumer and custom-consumer. Then run your java producer. Also, make sure that your producer sends messages to Kafka. Do you see messages on both consumers ?
– JR ibkr
Mar 7 at 15:46