一种兼顾负载均衡的Hadoop集群动态节能方法

田文洪, 李国忠, 陈瑜, 黄超杰, 杨吴同

清华大学学报(自然科学版) ›› 2016, Vol. 56 ›› Issue (11) : 1226-1231.

PDF(1321 KB)
PDF(1321 KB)
清华大学学报(自然科学版) ›› 2016, Vol. 56 ›› Issue (11) : 1226-1231. DOI: 10.16511/j.cnki.qhdxxb.2016.26.016
计算机科学与技术

一种兼顾负载均衡的Hadoop集群动态节能方法

  • 田文洪1,2, 李国忠1, 陈瑜1, 黄超杰1, 杨吴同1
作者信息 +

Combined load balancing and energy efficiency in Hadoop

  • TIAN Wenhong1,2, LI Guozhong1, CHEN Yu1, HUANG Chaojie1, YANG Wutong1
Author information +
文章历史 +

摘要

Hadoop集群广泛应用于企业和研究机构的大数据处理和并行计算中。该文针对Hadoop集群节点管理中缺少动态负载均衡和节能相互结合的调度技术的现状,提出一种动态负反馈调整算法,并设计和实现了一个用于Hadoop平台节点动态管理的系统。通过大量Hadoop经典测试用例测试,结果表明:该算法能够有效提高负载均衡并通过减少节点的空闲时间以有效地节能,与未使用本算法的结果相比,节点平均空闲休眠时间增加25%,节能14%。同时通过与其他算法相比,节点间均衡度有一定程度提升,平均负载方差减少10%。

Abstract

Hadoop clusters are widely used in enterprises and research institutions but there are few tools in Hadoop to dynamically load balance and improve the energy efficiency. A dynamic load balancing method with negative feedback was developed for a dynamic management system for Hadoop systems and tested using classic Hadoop benchmark examples. This method reduces the total idle time of the Hadoop nodes by 25% and reduces energy consumption by 14% on average compared with other algorithms by improving the load balancing through reducing the load variations by 10%.

关键词

分布式计算 / Hadoop / 调度算法 / 动态负载均衡 / 节能调度

Key words

distributed computing / Hadoop / scheduling algorithm / dynamic load-balancing / energy-efficient scheduling

引用本文

导出引用
田文洪, 李国忠, 陈瑜, 黄超杰, 杨吴同. 一种兼顾负载均衡的Hadoop集群动态节能方法[J]. 清华大学学报(自然科学版). 2016, 56(11): 1226-1231 https://doi.org/10.16511/j.cnki.qhdxxb.2016.26.016
TIAN Wenhong, LI Guozhong, CHEN Yu, HUANG Chaojie, YANG Wutong. Combined load balancing and energy efficiency in Hadoop[J]. Journal of Tsinghua University(Science and Technology). 2016, 56(11): 1226-1231 https://doi.org/10.16511/j.cnki.qhdxxb.2016.26.016
中图分类号: TP393   

参考文献

[1] Leverich J, Kozyrakis C. On the energy (in) efficiency of Hadoop clusters[J]. ACM SIGOPS Operating Systems Review, 2010, 44(1):61-65. [2] 田文洪, 赵勇. 云计算:资源调度管理[M]. 北京:国防工业出版社, 2011.TIAN Wenhong, ZHAO Yong, Cloud Computing:Resource Scheduling and Management[M]. Beijing:National Defense Press, 2011. (in Chinese) [3] 王鹏. 云计算的关键技术与应用实例[M]. 北京:人民邮电出版社, 2010.WANG Peng. Cloud Computing:Key Technologies and Applications[M]. Beijing:People's Post and Telecommunication Press, 2010. (in Chinese) [4] Chen Y, Keys L, Katz R H. Towards energy efficient ""mapreduce""[J]. EECS University of California at Berkeley, 2009, UCB/EECS-2009-109. [5] 陈涛, 陈启买. 分布式计算机系统负载平衡研究[J]. 计算机技术与发展, 2006, 16(5):33-35.CHEN Tao, CHEN Qimai. Research on load balancing in distributed computing system[J]. Computer Technology and Development, 2006, 16(5):33-35. (in Chinese) [6] Chen Y, Ganapathi A S, Fox A, et al. Statistical workloads for energy efficient mapreduce[J]. EECS University of California at Berkeley, 2010, UCB/EECS-2010-6. [7] Nedevschi S, Popa L, Iannaccone G, et al. Reducing network energy consumption via rate-adaptation and sleeping[J]. EECS Department, University of California, Berkeley, 2007, UCB/EECS-2007-128. [8] Polo J, Carrera D, Becerra Y, et al. Performance management of accelerated mapreduce workloads in heterogeneous clusters[C]//Proc 39th IEEE Conf on Parallel Processing. San Diego, UC:IEEE Press, 2010:653-662. [9] Xie J, Yin S, Ruan X, et al. Improving mapreduce performance through data placement in heterogeneous hadoop clusters[C]//IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), Atlanta, GA:IEEE Press, 2010:1-9. [10] Kim K H, Buyya R, Kim J. Power aware scheduling of bag-of-tasks applications with deadline constraints on DVS-enabled clusters[C]//IEEE International Symposium on CLUSTER Computing & the Grid. Rio:IEEE Computer Society, 2007:541-548. [11] Lee Y C, Zomaya A Y. Minimizing energy consumption for precedence-constrained applications using dynamic voltage scaling[C]//Proc 9th IEEE/ACM International Symposium on Cluster, Cloud and the Grid Computing. Washington, D.C., USA:IEEE Press, 2009:92-99. [12] Zhang X P. Electric power system analysis operation and control[J]. Electric Engineering, 2006, 2(3):1-42. [13] 博韦·西斯特. 深入理解Linux内核[M]. 陈莉君, 张琼声,张宏伟, 译. 北京:中国电力出版社, 2005.Bovet D P. Understanding the Linux Kernel[M]. CHEN Lijun, ZHANG Qiongsheng, ZHANG Hongwei (Translation). Beijing:China Electric Power Press, 2005. (in Chinese) [14] Beloglazov A, Abawajy J, Buyya R. Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing[J]. Future Generation Computer Systems, 2012, 28(5):755-768.

PDF(1321 KB)

Accesses

Citation

Detail

段落导航
相关文章

/