Combined load balancing and energy efficiency in Hadoop

doi:10.16511/j.cnki.qhdxxb.2016.26.016

Download: PDF(1321 KB)
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract Hadoop clusters are widely used in enterprises and research institutions but there are few tools in Hadoop to dynamically load balance and improve the energy efficiency. A dynamic load balancing method with negative feedback was developed for a dynamic management system for Hadoop systems and tested using classic Hadoop benchmark examples. This method reduces the total idle time of the Hadoop nodes by 25% and reduces energy consumption by 14% on average compared with other algorithms by improving the load balancing through reducing the load variations by 10%.

Keywords distributed computing Hadoop scheduling algorithm dynamic load-balancing energy-efficient scheduling

ZTFLH:

TP393

Issue Date: 15 November 2016

	Service

	E-mail this article
	E-mail Alert
	RSS
	Articles by authors

	TIAN Wenhong
	LI Guozhong
	CHEN Yu
	HUANG Chaojie
	YANG Wutong

Cite this article:

TIAN Wenhong,LI Guozhong,CHEN Yu, et al. Combined load balancing and energy efficiency in Hadoop[J]. Journal of Tsinghua University(Science and Technology), 2016, 56(11): 1226-1231.

URL:

http://jst.tsinghuajournals.com/EN/10.16511/j.cnki.qhdxxb.2016.26.016 OR http://jst.tsinghuajournals.com/EN/Y2016/V56/I11/1226

[1]	Leverich J, Kozyrakis C. On the energy (in) efficiency of Hadoop clusters[J]. ACM SIGOPS Operating Systems Review, 2010, 44(1):61-65.
[2]	田文洪, 赵勇. 云计算:资源调度管理[M]. 北京:国防工业出版社, 2011.TIAN Wenhong, ZHAO Yong, Cloud Computing:Resource Scheduling and Management[M]. Beijing:National Defense Press, 2011. (in Chinese)
[3]	王鹏. 云计算的关键技术与应用实例[M]. 北京:人民邮电出版社, 2010.WANG Peng. Cloud Computing:Key Technologies and Applications[M]. Beijing:People's Post and Telecommunication Press, 2010. (in Chinese)
[4]	Chen Y, Keys L, Katz R H. Towards energy efficient ""mapreduce""[J]. EECS University of California at Berkeley, 2009, UCB/EECS-2009-109.
[5]	陈涛, 陈启买. 分布式计算机系统负载平衡研究[J]. 计算机技术与发展, 2006, 16(5):33-35.CHEN Tao, CHEN Qimai. Research on load balancing in distributed computing system[J]. Computer Technology and Development, 2006, 16(5):33-35. (in Chinese)
[6]	Chen Y, Ganapathi A S, Fox A, et al. Statistical workloads for energy efficient mapreduce[J]. EECS University of California at Berkeley, 2010, UCB/EECS-2010-6.
[7]	Nedevschi S, Popa L, Iannaccone G, et al. Reducing network energy consumption via rate-adaptation and sleeping[J]. EECS Department, University of California, Berkeley, 2007, UCB/EECS-2007-128.
[8]	Polo J, Carrera D, Becerra Y, et al. Performance management of accelerated mapreduce workloads in heterogeneous clusters[C]//Proc 39th IEEE Conf on Parallel Processing. San Diego, UC:IEEE Press, 2010:653-662.
[9]	Xie J, Yin S, Ruan X, et al. Improving mapreduce performance through data placement in heterogeneous hadoop clusters[C]//IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), Atlanta, GA:IEEE Press, 2010:1-9.
[10]	Kim K H, Buyya R, Kim J. Power aware scheduling of bag-of-tasks applications with deadline constraints on DVS-enabled clusters[C]//IEEE International Symposium on CLUSTER Computing & the Grid. Rio:IEEE Computer Society, 2007:541-548.
[11]	Lee Y C, Zomaya A Y. Minimizing energy consumption for precedence-constrained applications using dynamic voltage scaling[C]//Proc 9th IEEE/ACM International Symposium on Cluster, Cloud and the Grid Computing. Washington, D.C., USA:IEEE Press, 2009:92-99.
[12]	Zhang X P. Electric power system analysis operation and control[J]. Electric Engineering, 2006, 2(3):1-42.
[13]	博韦·西斯特. 深入理解Linux内核[M]. 陈莉君, 张琼声,张宏伟, 译. 北京:中国电力出版社, 2005.Bovet D P. Understanding the Linux Kernel[M]. CHEN Lijun, ZHANG Qiongsheng, ZHANG Hongwei (Translation). Beijing:China Electric Power Press, 2005. (in Chinese)
[14]	Beloglazov A, Abawajy J, Buyya R. Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing[J]. Future Generation Computer Systems, 2012, 28(5):755-768.