Error:"java.net.SocketException: Connection reset and org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)" when running PySpark using RDD and data frames to load images

SupportKB

Problem Description:
When running PySpark using RDD and data frames to load images, the following error is displayed in the Spark Executor logs: 

17/09/07 10:37:19 ERROR Executor: Exception in task 1.0 in stage 6.0 (TID 118) 
org.apache.spark.SparkException: Python worker exited unexpectedly (crashed) 
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:203) 
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207) 
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125) 
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70) 
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) 
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69) 
at org.apache.spark.rdd.RDD.iterator(RDD.scala:275) 
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70) 
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) 
at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) 
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) 
at org.apache.spark.scheduler.Task.run(Task.scala:89) 
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: java.io.EOFException 
at java.io.DataInputStream.readInt(DataInputStream.java:392) 
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:139) 
... 15 more 
"

Cause:
On reviewing  the Spark application History server, it was found that there were only two task that were run to process the job and it shared the entire RDD data within those 2 tasks.

The Spark workers shuffle the data across the node. Shuffling is the process of data transfer between stages.
In Spark, maximum allowed size of RDD block or Shuffle block is 2GB and this value is not configurable. If block size goes beyond 2GB, tasks will fail and driver will retry that task multiple times based on retry count. Eventually the jog/stage will fail. 

Hence, the issue the above error occurs when the Shuffle block exceeds more than 2 GB.

 
Solution:
To resolve the issue, increase the number of partitions to 100 [rdd.repartition(100)] to distribute the data processing across the node and keep the shuffle data below 2 GB.

About:
This article created by Hortonworks Support (Article: 000006438) on 2018-02-16 09:56
OS: Linux
Type: Executing_Jobs
Version: 2.5.3