"ERROR: java.lang.OutOfMemoryError: Requested array size exceeds VM limit" when starting an HBase RegionServer

SupportKB

Problem Description:

When starting an HBase RegionServer with considerably large Ranger audit spool files, it crashes without any error. And when the RegionServer JVM option "-XX:OnOutOfMemoryError" is changed from "kill -9 %p" to "kill -3 %p", the following error is displayed in the stack trace: 

ERROR hbaseRegional.async.summary.batch_hbaseRegional.async.summary.batch.hdfs_destWriter queue.AuditFileSpool: 
Exception in destination writing thread.
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:2367)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)
at java.lang.StringBuffer.append(StringBuffer.java:322)
at java.io.BufferedReader.readLine(BufferedReader.java:363)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:812)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:758)
at java.lang.Thread.run(Thread.java:745)

This issue occurs because the Ranger audit spool directory contains large log spool files. While reading the files, the Ranger plugin-in within the RegionServer process requests for a large array size that exceeds the VM limit.

The reason for the audit spool file to increase so considerably in size could be a failure to access the HDFS in the past when Ranger audit destination is configured as HDFS.

1. Move older Spool log file to a temporary location.
2. Empty the json files inside spool directory before restarting the RegionServer.