Can we bind spark2 history server to private IP?

Question

Can we bind spark2 history server to private IP?

Question by Kuldeep Kulkarni Mar 17, 2018 at 12:47 AM Spark spark-history-server

I have multi-homing network, I want to bind Spark2 history server to private interface instead of 0.0.0.0

I tried changing spark.history.bind.host to have private IP and also tried adding SPARK_LOCAL_IP to spark env by referring https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-history-server.html however in vain.

e.g.

Instead of 0.0.0.0 I want private IP to be there.

Any pointers would be appreciated!

ye.jpg (14.0 kB)

0

10 |6000 characters needed characters left characters exceeded

Attachments: Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Answer 1 · 2018-03-17T05:13:44Z

Answer by Sandeep Nemuri · Mar 17, 2018 at 05:13 AM

@Kuldeep Kulkarni, As @Aditya Sirna mentioned setting SPARK_LOCAL_IP gets SHS to bind to the mentioned ip, However if you are setting this value in spark-env through ambari it will affect all your spark jobs. As SPARK_LOCAL_IP is a driver ip all drivers of your jobs using this spark-env will try to bind to mentioned ip.

Instead set export SPARK_LOCAL_IP=<ip> in /usr/hdp/current/spark-historyserver/sbin/start-history-server.sh so that only SHS will pick the ip.

1 · Share

10 |6000 characters needed characters left characters exceeded

Attachments: Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Answer 2 · 2018-03-17T04:16:04Z

Answer by Aditya Sirna · Mar 17, 2018 at 04:16 AM

@Kuldeep Kulkarni,

I'm not sure if there is a regression in HDP 2.6. I tried setting SPARK_LOCAL_IP in one of my HDP 2.5.5 version and this seems to work. I have exported SPARK_LOCAL_IP in under Advanced spark2-env and started spark2 History server. Below is the output

18/03/17 04:06:17 INFO log: Logging initialized @4468ms
18/03/17 04:06:18 INFO Server: jetty-9.2.z-SNAPSHOT
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@71154f21{/,null,AVAILABLE}
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@15f193b8{/json,null,AVAILABLE}
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2516fc68{/api,null,AVAILABLE}
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@304a9d7b{/static,null,AVAILABLE}
18/03/17 04:06:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6bfdb014{/history,null,AVAILABLE}
18/03/17 04:06:18 INFO ServerConnector: Started ServerConnector@21ae6e73{HTTP/1.1}{10.10.2.21:18081}
18/03/17 04:06:18 INFO Server: Started @4722ms
18/03/17 04:06:18 INFO Utils: Successfully started service on port 18081.
18/03/17 04:06:18 INFO HistoryServer: Bound HistoryServer to 10.10.2.21, and started at http://10.10.2.21:18081
^C
[root@vnode-21 ~]# netstat -tupln | grep 18081
tcp6       0      0 10.10.2.21:18081        :::*                    LISTEN      37481/java

Tried the same on HDP 2.6.3 and this is the output. It says started on 0.0.0.0:18081 and also Bound HistoryServer to 172.31.252.173

18/03/17 03:54:30 INFO AbstractConnector: Started ServerConnector@1d269ed7{HTTP/1.1,[http/1.1]}{0.0.0.0:18081}
18/03/17 03:54:30 INFO Utils: Successfully started service on port 18081.
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3163987e{/,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4e6d7365{/json,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@d4602a{/api,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@11a82d0f{/static,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@60e949e1{/history,null,AVAILABLE,@Spark}
18/03/17 03:54:30 INFO HistoryServer: Bound HistoryServer to 172.31.252.173, and started at http://172.31.252.173:18081

Guessing that there may be some issue with Jetty

https://github.com/hortonworks/spark2/blob/5a07aca4d464e96d75ea17bf6768e24b829872ec/core/src/main/scala/org/apache/spark/ui/WebUI.scala#L129

https://github.com/hortonworks/spark2/blob/5a07aca4d464e96d75ea17bf6768e24b829872ec/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L351

.

-Aditya

1 · Share

10 |6000 characters needed characters left characters exceeded

Attachments: Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Answer 3 · 2018-03-17T17:48:06Z

Answer by Kuldeep Kulkarni · Mar 17, 2018 at 05:48 PM

@Sandeep Nemuri / @Aditya Sirna - Thanks for your help. I forgot to mention that I had tried adding it in the script as well and I could see that variable is being picked up in /proc/<pid-of-S2HS> but it still shows 0.0.0.0 in netstat.

Let me go through the additional notes given by @Aditya Sirna

Thanks for your help guys.

0 · Share

10 |6000 characters needed characters left characters exceeded

Attachments: Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Can we bind spark2 history server to private IP?

3 Replies

Your answer