I have multi-homing network, I want to bind Spark2 history server to private interface instead of 0.0.0.0
I tried changing spark.history.bind.host to have private IP and also tried adding SPARK_LOCAL_IP to spark env by referring https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-history-server.html however in vain.
e.g.
Instead of 0.0.0.0 I want private IP to be there.
Any pointers would be appreciated!
Answer by Aditya Sirna · Mar 17 at 04:16 AM
I'm not sure if there is a regression in HDP 2.6. I tried setting SPARK_LOCAL_IP in one of my HDP 2.5.5 version and this seems to work. I have exported SPARK_LOCAL_IP in under Advanced spark2-env and started spark2 History server. Below is the output
18/03/17 04:06:17 INFO log: Logging initialized @4468ms 18/03/17 04:06:18 INFO Server: jetty-9.2.z-SNAPSHOT 18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@71154f21{/,null,AVAILABLE} 18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@15f193b8{/json,null,AVAILABLE} 18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2516fc68{/api,null,AVAILABLE} 18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@304a9d7b{/static,null,AVAILABLE} 18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6bfdb014{/history,null,AVAILABLE} 18/03/17 04:06:18 INFO ServerConnector: Started ServerConnector@21ae6e73{HTTP/1.1}{10.10.2.21:18081} 18/03/17 04:06:18 INFO Server: Started @4722ms 18/03/17 04:06:18 INFO Utils: Successfully started service on port 18081. 18/03/17 04:06:18 INFO HistoryServer: Bound HistoryServer to 10.10.2.21, and started at http://10.10.2.21:18081 ^C [root@vnode-21 ~]# netstat -tupln | grep 18081 tcp6 0 0 10.10.2.21:18081 :::* LISTEN 37481/java
Tried the same on HDP 2.6.3 and this is the output. It says started on 0.0.0.0:18081 and also Bound HistoryServer to 172.31.252.173
18/03/17 03:54:30 INFO AbstractConnector: Started ServerConnector@1d269ed7{HTTP/1.1,[http/1.1]}{0.0.0.0:18081} 18/03/17 03:54:30 INFO Utils: Successfully started service on port 18081. 18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3163987e{/,null,AVAILABLE,@Spark} 18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4e6d7365{/json,null,AVAILABLE,@Spark} 18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@d4602a{/api,null,AVAILABLE,@Spark} 18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@11a82d0f{/static,null,AVAILABLE,@Spark} 18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@60e949e1{/history,null,AVAILABLE,@Spark} 18/03/17 03:54:30 INFO HistoryServer: Bound HistoryServer to 172.31.252.173, and started at http://172.31.252.173:18081
Guessing that there may be some issue with Jetty
.
-Aditya
Answer by Sandeep Nemuri · Mar 17 at 05:13 AM
@Kuldeep Kulkarni, As @Aditya Sirna mentioned setting SPARK_LOCAL_IP gets SHS to bind to the mentioned ip, However if you are setting this value in spark-env through ambari it will affect all your spark jobs. As SPARK_LOCAL_IP is a driver ip all drivers of your jobs using this spark-env will try to bind to mentioned ip.
Instead set export SPARK_LOCAL_IP=<ip> in /usr/hdp/current/spark-historyserver/sbin/start-history-server.sh so that only SHS will pick the ip.
Answer by Kuldeep Kulkarni · Mar 17 at 05:48 PM
@Sandeep Nemuri / @Aditya Sirna - Thanks for your help. I forgot to mention that I had tried adding it in the script as well and I could see that variable is being picked up in /proc/<pid-of-S2HS> but it still shows 0.0.0.0 in netstat.
Let me go through the additional notes given by @Aditya Sirna
Thanks for your help guys.
HCC Guidelines | HCC FAQs | HCC Privacy Policy
© 2011-2017 Hortonworks Inc. All Rights Reserved.
Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.
Privacy Policy |
Terms of Service
HCC Guidelines | HCC FAQs | HCC Privacy Policy | Privacy Policy | Terms of Service
© 2011-2018 Hortonworks Inc. All Rights Reserved.
Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.