Hortonworks.com
  • Explore
    • All Tags
    • All Questions
    • All Repos
    • All SKB
    • All Articles
    • All Ideas
    • All Repos
    • All SKB
    • All Articles
    • All Ideas
    • All Users
    • All Badges
    • Leaderboard
  • Create
    • Ask a question
    • Add Repo
    • Create Article
    • Post Idea
    • Add Repo
    • Create Article
    • Post Idea
  • Tracks
    • All Tracks
    • Community Help
    • Cloud & Operations
    • CyberSecurity
    • Data Ingestion & Streaming
    • Data Processing
    • Data Science & Advanced Analytics
    • Design & Architecture
    • Governance & Lifecycle
    • Hadoop Core
    • Sandbox & Learning
    • Security
    • Solutions
  • Login
HCC Hortonworks Community Connection
  • Home /
  • Data Ingestion & Streaming /
avatar image

Can we bind spark2 history server to private IP?

Question by Kuldeep Kulkarni Mar 17 at 12:47 AM Sparkspark-history-server

I have multi-homing network, I want to bind Spark2 history server to private interface instead of 0.0.0.0

I tried changing spark.history.bind.host to have private IP and also tried adding SPARK_LOCAL_IP to spark env by referring https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-history-server.html however in vain.

e.g.

Instead of 0.0.0.0 I want private IP to be there.

Any pointers would be appreciated!

ye.jpg (14.0 kB)
Comment

People who voted for this

0 Show 0
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

3 Replies

· Add your reply
  • Sort: 
  • Votes
  • Created
  • Oldest
avatar image

Answer by Aditya Sirna · Mar 17 at 04:16 AM

@Kuldeep Kulkarni,

I'm not sure if there is a regression in HDP 2.6. I tried setting SPARK_LOCAL_IP in one of my HDP 2.5.5 version and this seems to work. I have exported SPARK_LOCAL_IP in under Advanced spark2-env and started spark2 History server. Below is the output

18/03/17 04:06:17 INFO log: Logging initialized @4468ms
18/03/17 04:06:18 INFO Server: jetty-9.2.z-SNAPSHOT
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@71154f21{/,null,AVAILABLE}
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@15f193b8{/json,null,AVAILABLE}
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2516fc68{/api,null,AVAILABLE}
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@304a9d7b{/static,null,AVAILABLE}
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6bfdb014{/history,null,AVAILABLE}
18/03/17 04:06:18 INFO ServerConnector: Started ServerConnector@21ae6e73{HTTP/1.1}{10.10.2.21:18081}
18/03/17 04:06:18 INFO Server: Started @4722ms
18/03/17 04:06:18 INFO Utils: Successfully started service on port 18081.
18/03/17 04:06:18 INFO HistoryServer: Bound HistoryServer to 10.10.2.21, and started at http://10.10.2.21:18081
^C
[root@vnode-21 ~]# netstat -tupln | grep 18081
tcp6       0      0 10.10.2.21:18081        :::*                    LISTEN      37481/java

Tried the same on HDP 2.6.3 and this is the output. It says started on 0.0.0.0:18081 and also Bound HistoryServer to 172.31.252.173

18/03/17 03:54:30 INFO AbstractConnector: Started ServerConnector@1d269ed7{HTTP/1.1,[http/1.1]}{0.0.0.0:18081}
18/03/17 03:54:30 INFO Utils: Successfully started service on port 18081.
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3163987e{/,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4e6d7365{/json,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@d4602a{/api,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@11a82d0f{/static,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@60e949e1{/history,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO HistoryServer: Bound HistoryServer to 172.31.252.173, and started at http://172.31.252.173:18081

Guessing that there may be some issue with Jetty

https://github.com/hortonworks/spark2/blob/5a07aca4d464e96d75ea17bf6768e24b829872ec/core/src/main/scala/org/apache/spark/ui/WebUI.scala#L129

https://github.com/hortonworks/spark2/blob/5a07aca4d464e96d75ea17bf6768e24b829872ec/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L351

.

-Aditya

Comment
Kuldeep Kulkarni

People who voted for this

1 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Sandeep Nemuri · Mar 17 at 05:13 AM

@Kuldeep Kulkarni, As @Aditya Sirna mentioned setting SPARK_LOCAL_IP gets SHS to bind to the mentioned ip, However if you are setting this value in spark-env through ambari it will affect all your spark jobs. As SPARK_LOCAL_IP is a driver ip all drivers of your jobs using this spark-env will try to bind to mentioned ip.

Instead set export SPARK_LOCAL_IP=<ip> in /usr/hdp/current/spark-historyserver/sbin/start-history-server.sh so that only SHS will pick the ip.

Comment
Kuldeep Kulkarni

People who voted for this

1 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Kuldeep Kulkarni · Mar 17 at 05:48 PM

@Sandeep Nemuri / @Aditya Sirna - Thanks for your help. I forgot to mention that I had tried adding it in the script as well and I could see that variable is being picked up in /proc/<pid-of-S2HS> but it still shows 0.0.0.0 in netstat.

Let me go through the additional notes given by @Aditya Sirna

Thanks for your help guys.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Your answer

Hint: You can notify a user about this post by typing @username

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

81
Followers
follow question

Answers Answer & comments

HCC Guidelines | HCC FAQs | HCC Privacy Policy

Hortonworks - Develops, Distributes and Supports Open Enterprise Hadoop.

© 2011-2017 Hortonworks Inc. All Rights Reserved.
Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.
Privacy Policy | Terms of Service

HCC Guidelines | HCC FAQs | HCC Privacy Policy | Privacy Policy | Terms of Service

© 2011-2018 Hortonworks Inc. All Rights Reserved.

Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Login
  • Create
  • Ask a question
  • Add Repo
  • Create SupportKB
  • Create Article
  • Post Idea
  • Add Repo
  • Create SupportKB
  • Create Article
  • Post Idea
  • Tracks
  • Community Help
  • Cloud & Operations
  • CyberSecurity
  • Data Ingestion & Streaming
  • Data Processing
  • Data Science & Advanced Analytics
  • Design & Architecture
  • Governance & Lifecycle
  • Hadoop Core
  • Sandbox & Learning
  • Security
  • Solutions
  • Explore
  • All Tags
  • All Questions
  • All Repos
  • All SKB
  • All Articles
  • All Ideas
  • All Repos
  • All SKB
  • All Articles
  • All Ideas
  • All Users
  • Leaderboard
  • All Badges