Virtual Memory swapping can have a large impact on the performance of a Hadoop system. Because of the memory requirements of YARN containers and processes running on the nodes in a cluster, swapping process out of memory to disk can cause serious performance limitations. As such, the historical recommendations for setting the swappiness, or propensity to swap out a process, on a Hadoop system has been to disable swap altogether. With newer versions of the Linux kernel, Out Of Memory (OOM) situations can be more likely to indiscriminately kill important processes to reclaim valuable physical memory on the system with a swappiness of 0.
In order to prevent the system from swapping processes too frequently, but still allow for emergency swapping (instead of killing processes), the recommendation is now to set swappiness to 1 on Linux systems. This will still allow swapping, but with the least possible aggressiveness (for comparison, the default value for swappiness is 60).
To change the swappiness on a running machine, use the following command:
echo "1" > /proc/sys/vm/swappiness
To ensure the swappiness is set appropriately on reboot, use the following command:
echo "vm.swappiness=1" >> /etc/sysctl.conf
Kafka Topics and Brokers Metrics with Ambari 2.2.2
Optimizing HBase I/O for Large Scale Hadoop Implementations
Introducing Cloudbreak 2.9.0 GA
Google Cloud Platform fix for Cloudbreak - upgrade now to 2.4.1 or 1.16.6
Cloudbreak Credential Troubleshooting Tips
Get Started with HDP 2.6 in HDCloud for AWS 1.14.1
Design and Deployment Considerations for High Availability Kafka Service
How to remove all previous version HDP directories from /usr/hdp using Ambari.
This website uses cookies for analytics, personalisation and advertising. To learn more or change your cookie settings, please read our Cookie Policy. By continuing to browse, you agree to our use of cookies.
HCC Guidelines | HCC FAQs | HCC Privacy Policy | Privacy Policy | Terms of Service
© 2011-2019 Hortonworks Inc. All Rights Reserved.
Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.