"ERROR exec.Task (TezTask.java:execute(226)) - Failed to execute tez graph" when executing Hive jobs with Tez

SupportKB

Problem Description:
Hive job with Tez execution engine fails with the following error:

2018-08-14 15:55:27,581 ERROR exec.Task (TezTask.java:execute(226)) - Failed to execute tez graph. 
java.lang.InterruptedException 
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) 
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) 
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) 
at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.returnSession(TezSessionPoolManager.java:176)

Cause:
This issue occurs when a query is killed or cancelled. This interrupts with the background thread, which fails to put the session back to the pool. It is known issue related to  HIVE-15731 and HIVE-20415.
Solution:


Workaround Change the settings to:

hive.server2.tez.initialize.default.sessions=false

along with below:

hive.server2.idle.session.timeout=12h
hive.server2.idle.operation.timeout=8h
hive.server2.session.check.interval=4h
tez.session.am.dag.submit.timeout.secs=1200
Hold Containers to Reduce Latency=true
Number of Containers Held=

About:
This article created by Hortonworks Support (Article: 000007885) on 2019-02-01 08:09
OS: Linux
Type: Executing-Jobs
Version: 2-6-3