"ERROR: java.lang.OutOfMemoryError: Requested array size exceeds VM limit" when starting an HBase RegionServer

SupportKB

When starting an HBase RegionServer with considerably large Ranger audit spool files, it crashes without any error. And when the RegionServer JVM option "-XX:OnOutOfMemoryError" is changed from "kill -9 %p" to "kill -3 %p", the following error is displayed in the stack trace: 

ERROR hbaseRegional.async.summary.batch_hbaseRegional.async.summary.batch.hdfs_destWriter queue.AuditFileSpool: 
Exception in destination writing thread.
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:2367)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)
at java.lang.StringBuffer.append(StringBuffer.java:322)
at java.io.BufferedReader.readLine(BufferedReader.java:363)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:812)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:758)
at java.lang.Thread.run(Thread.java:745)

This issue occurs because the Ranger audit spool directory contains large log spool files. While reading the files, the Ranger plugin-in within the RegionServer process requests for a large array size that exceeds the VM limit.

The reason for the audit spool file to increase so considerably in size could be a failure to access the HDFS in the past when Ranger audit destination is configured as HDFS.

1. Move older Spool log file to a temporary location.
2. Empty the json files inside spool directory before restarting the RegionServer.