Hortonworks.com
  • Explore
    • All Tags
    • All Questions
    • All Articles
    • All Ideas
    • All Repos
    • All SKB
    • All Users
    • All Badges
    • Leaderboard
  • Create
    • Ask a question
    • Create Article
    • Post Idea
    • Add Repo
  • Tracks
    • All Tracks
    • Community Help
    • Cloud & Operations
    • CyberSecurity
    • Data Ingestion & Streaming
    • Data Processing
    • Data Science & Advanced Analytics
    • Design & Architecture
    • Governance & Lifecycle
    • Hadoop Core
    • Sandbox & Learning
    • Security
    • Solutions
  • Login
HCC Hortonworks Community Connection
  • Home /
  • Data Ingestion & Streaming /
avatar image

How to setup Kafka broker for producer-consumer in a single node.

Question by Dinesh Das May 15, 2017 at 02:58 PM KafkaSandbox

Looking for a help in setting up kafka broker on single node. I am using HDP 2.5.0

Producer Node:

[root@sandbox ~]# /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 10.74.58.106:6667 --topic test
Hi
[2017-05-15 11:47:24,559] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
[2017-05-15 11:48:24,564] ERROR Error when sending message to topic test with key: null, value: 9 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.


Conusmer Node:

[root@sandbox ~]# /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
{metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-18753, security.protocol=PLAINTEXT}
{metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-18753, security.protocol=PLAINTEXT}
{metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-18753, security.protocol=PLAINTEXT}
{metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-18753, security.protocol=PLAINTEXT}
{metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-18753, security.protocol=PLAINTEXT}
{metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-18753, security.protocol=PLAINTEXT}


I also have changed the url from localhost to IP but same issue with different error.

[root@sandbox ~]# /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper 10.74.58.106:2181 --topic test --from-beginning
[2017-05-15 11:46:28,215] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)


Commands used /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 10.74.58.106:6667 --topic test

/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper 10.74.58.106:2181 --topic test --from-beginning

Comment
Timothy Spann

People who voted for this

1 Show 0
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

3 Replies

· Add your reply
  • Sort: 
  • Votes
  • Created
  • Oldest
avatar image

Answer by Steve Qin · Aug 10, 2018 at 01:09 PM

Check the correct the IP with netstat -na | grep 6667 and replace localhost with the correct one shown in the list

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by Samant Thakur · Jan 14, 2018 at 07:30 PM

@Dinesh Das, I am also experiencing the same issue. Could you please let me know how did you resolve this issue?

Thanks a lot !

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by Balint Molnar · May 16, 2017 at 08:00 AM

Can you please check if your topic is created properly with the following command:

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper 10.74.58.106:2181 --list
Comment

People who voted for this

0 Show 3 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image Dinesh Das · May 16, 2017 at 08:48 AM 0
Share

@Balint Molnar

Yes topic has been created properly

[root@sandbox ~]# /usr/hdp/current/kafka-broker/bin/kafka-topics.sh  --list --zookeeper localhost:2181
ATLAS_ENTITIES
ATLAS_HOOK
my-topic-01
pa1
pa1_dinesh
test
test_topic - marked for deletion




avatar image Balint Molnar Dinesh Das · May 16, 2017 at 09:19 AM 0
Share

Hmm, My first guess is the kafka-broker is not listening on port 6667. You can check it using the ambari, or with

zkCli.sh -server localhost:2181

get /brokers/ids/<your broker id>

My second guess is, try with localhost:6667 instead of using an ip.

avatar image Balint Molnar Dinesh Das · May 16, 2017 at 09:39 AM 0
Share

If this does not help either: try recreating the test topic:

kafka-topics.sh --zookeeper localhost:2181 --create --topic mytest --partitions 1 --replication-factor 1

Your answer

Hint: You can notify a user about this post by typing @username

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

61
Followers

Answers Answer & comments

This website uses cookies for analytics, personalisation and advertising. To learn more or change your cookie settings, please read our Cookie Policy. By continuing to browse, you agree to our use of cookies.

HCC Guidelines | HCC FAQs | HCC Privacy Policy | Privacy Policy | Terms of Service

© 2011-2019 Hortonworks Inc. All Rights Reserved.

Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Login
  • Create
  • Ask a question
  • Create Article
  • Post Idea
  • Add Repo
  • Create SupportKB
  • Tracks
  • Community Help
  • Cloud & Operations
  • CyberSecurity
  • Data Ingestion & Streaming
  • Data Processing
  • Data Science & Advanced Analytics
  • Design & Architecture
  • Governance & Lifecycle
  • Hadoop Core
  • Sandbox & Learning
  • Security
  • Solutions
  • Explore
  • All Tags
  • All Questions
  • All Articles
  • All Ideas
  • All Repos
  • All SKB
  • All Users
  • Leaderboard
  • All Badges