Design and performance evaluation for Hadoop clusters on virtualized environment

Masakuni Ishii, Jungkyu Han, Hiroyuki Makino

Research output: Chapter in Book/Report/Conference proceedingConference contribution

20 Citations (Scopus)

Abstract

Hadoop an implementation of Google's MapReduce, is widely used in these days for big data analysis. Yahoo Inc. operated 25 PB with 25,000 nodes in 2010. The resource management for such large number of nodes is quite difficult from the aspects of configuration, deployment and efficient resource utilization. By deploying virtual machines (VMs), Hadoop management becomes much easier. Amazon already released the Hadoop on Xen-virtualized environment as Elastic MapReduce. However, Hadoop on VM clusters degrades its performance due to the overhead of the virtualization. Thus, it is important to minimize the overhead. We build a Hadoop performance model and examine how the performance is affected by changing VM configuration, allocation of VMs over physical machines, and multiplicity of jobs. We find that performance of the I/O-intensive jobs is more sensitive to the virtualization overhead than that of CPU-intensive jobs. The performance degradation caused by the VM configuration change is 55% at most and the one caused by allocation change is 18% at most for I/O-intensive jobs. For I/O intensive jobs, the best practice is to increase the number of VMs and not to increase the number of VCPUs in a VM, to allocate VMs widely over physical servers, and to decrease the number of simultaneous executed jobs. The main factor of virtualization overhead is disk I/O shared by multiple VMs in a physical server.

Original languageEnglish
Title of host publicationInternational Conference on Information Networking 2013, ICOIN 2013
Pages244-249
Number of pages6
DOIs
Publication statusPublished - 2013
Externally publishedYes
Event27th International Conference on Information Networking, ICOIN 2013 - Bangkok
Duration: 2013 Jan 272013 Jan 30

Other

Other27th International Conference on Information Networking, ICOIN 2013
CityBangkok
Period13/1/2713/1/30

Keywords

  • cluster
  • Hadoop performance evaluation
  • KVM
  • virtual machine
  • virtualized environment

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems

Fingerprint Dive into the research topics of 'Design and performance evaluation for Hadoop clusters on virtualized environment'. Together they form a unique fingerprint.

  • Cite this

    Ishii, M., Han, J., & Makino, H. (2013). Design and performance evaluation for Hadoop clusters on virtualized environment. In International Conference on Information Networking 2013, ICOIN 2013 (pp. 244-249). [6496384] https://doi.org/10.1109/ICOIN.2013.6496384