Please wait a minute...
 首页  期刊介绍 期刊订阅 联系我们 横山亮次奖 百年刊庆
最新录用  |  预出版  |  当期目录  |  过刊浏览  |  阅读排行  |  下载排行  |  引用排行  |  横山亮次奖  |  百年刊庆
清华大学学报(自然科学版)  2016, Vol. 56 Issue (11): 1226-1231    DOI: 10.16511/j.cnki.qhdxxb.2016.26.016
  计算机科学与技术 本期目录 | 过刊浏览 | 高级检索 |
田文洪1,2, 李国忠1, 陈瑜1, 黄超杰1, 杨吴同1
1. 电子科技大学 信息与软件工程学院, 成都 610054;
2. 中国科学院重庆绿色智能技术研究院, 重庆 400714
Combined load balancing and energy efficiency in Hadoop
TIAN Wenhong1,2, LI Guozhong1, CHEN Yu1, HUANG Chaojie1, YANG Wutong1
1. School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China;
2. Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
全文: PDF(1321 KB)  
输出: BibTeX | EndNote (RIS)      
摘要 Hadoop集群广泛应用于企业和研究机构的大数据处理和并行计算中。该文针对Hadoop集群节点管理中缺少动态负载均衡和节能相互结合的调度技术的现状,提出一种动态负反馈调整算法,并设计和实现了一个用于Hadoop平台节点动态管理的系统。通过大量Hadoop经典测试用例测试,结果表明:该算法能够有效提高负载均衡并通过减少节点的空闲时间以有效地节能,与未使用本算法的结果相比,节点平均空闲休眠时间增加25%,节能14%。同时通过与其他算法相比,节点间均衡度有一定程度提升,平均负载方差减少10%。
E-mail Alert
关键词 分布式计算Hadoop调度算法动态负载均衡节能调度    
Abstract:Hadoop clusters are widely used in enterprises and research institutions but there are few tools in Hadoop to dynamically load balance and improve the energy efficiency. A dynamic load balancing method with negative feedback was developed for a dynamic management system for Hadoop systems and tested using classic Hadoop benchmark examples. This method reduces the total idle time of the Hadoop nodes by 25% and reduces energy consumption by 14% on average compared with other algorithms by improving the load balancing through reducing the load variations by 10%.
Key wordsdistributed computing    Hadoop    scheduling algorithm    dynamic load-balancing    energy-efficient scheduling
收稿日期: 2016-06-29      出版日期: 2016-11-15
ZTFLH:  TP393  
田文洪, 李国忠, 陈瑜, 黄超杰, 杨吴同. 一种兼顾负载均衡的Hadoop集群动态节能方法[J]. 清华大学学报(自然科学版), 2016, 56(11): 1226-1231.
TIAN Wenhong, LI Guozhong, CHEN Yu, HUANG Chaojie, YANG Wutong. Combined load balancing and energy efficiency in Hadoop. Journal of Tsinghua University(Science and Technology), 2016, 56(11): 1226-1231.
链接本文:  或
  图1 Hadoop运行结构图
  图2 WordCount方差对比
  图3 TeraSort方差对比
  图4 WordCount节点闲置时间对比
  图5 TeraSort节点闲置时间对比
  图6 WordCount系统能耗对比
  图7 TeraSort系统能耗对比
[1] Leverich J, Kozyrakis C. On the energy (in) efficiency of Hadoop clusters[J]. ACM SIGOPS Operating Systems Review, 2010, 44(1):61-65.
[2] 田文洪, 赵勇. 云计算:资源调度管理[M]. 北京:国防工业出版社, 2011.TIAN Wenhong, ZHAO Yong, Cloud Computing:Resource Scheduling and Management[M]. Beijing:National Defense Press, 2011. (in Chinese)
[3] 王鹏. 云计算的关键技术与应用实例[M]. 北京:人民邮电出版社, 2010.WANG Peng. Cloud Computing:Key Technologies and Applications[M]. Beijing:People's Post and Telecommunication Press, 2010. (in Chinese)
[4] Chen Y, Keys L, Katz R H. Towards energy efficient ""mapreduce""[J]. EECS University of California at Berkeley, 2009, UCB/EECS-2009-109.
[5] 陈涛, 陈启买. 分布式计算机系统负载平衡研究[J]. 计算机技术与发展, 2006, 16(5):33-35.CHEN Tao, CHEN Qimai. Research on load balancing in distributed computing system[J]. Computer Technology and Development, 2006, 16(5):33-35. (in Chinese)
[6] Chen Y, Ganapathi A S, Fox A, et al. Statistical workloads for energy efficient mapreduce[J]. EECS University of California at Berkeley, 2010, UCB/EECS-2010-6.
[7] Nedevschi S, Popa L, Iannaccone G, et al. Reducing network energy consumption via rate-adaptation and sleeping[J]. EECS Department, University of California, Berkeley, 2007, UCB/EECS-2007-128.
[8] Polo J, Carrera D, Becerra Y, et al. Performance management of accelerated mapreduce workloads in heterogeneous clusters[C]//Proc 39th IEEE Conf on Parallel Processing. San Diego, UC:IEEE Press, 2010:653-662.
[9] Xie J, Yin S, Ruan X, et al. Improving mapreduce performance through data placement in heterogeneous hadoop clusters[C]//IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), Atlanta, GA:IEEE Press, 2010:1-9.
[10] Kim K H, Buyya R, Kim J. Power aware scheduling of bag-of-tasks applications with deadline constraints on DVS-enabled clusters[C]//IEEE International Symposium on CLUSTER Computing & the Grid. Rio:IEEE Computer Society, 2007:541-548.
[11] Lee Y C, Zomaya A Y. Minimizing energy consumption for precedence-constrained applications using dynamic voltage scaling[C]//Proc 9th IEEE/ACM International Symposium on Cluster, Cloud and the Grid Computing. Washington, D.C., USA:IEEE Press, 2009:92-99.
[12] Zhang X P. Electric power system analysis operation and control[J]. Electric Engineering, 2006, 2(3):1-42.
[13] 博韦·西斯特. 深入理解Linux内核[M]. 陈莉君, 张琼声,张宏伟, 译. 北京:中国电力出版社, 2005.Bovet D P. Understanding the Linux Kernel[M]. CHEN Lijun, ZHANG Qiongsheng, ZHANG Hongwei (Translation). Beijing:China Electric Power Press, 2005. (in Chinese)
[14] Beloglazov A, Abawajy J, Buyya R. Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing[J]. Future Generation Computer Systems, 2012, 28(5):755-768.
[1] 余嘉茵, 何玉林, 崔来中, 黄哲学. 针对大规模数据的分布一致缺失值插补算法[J]. 清华大学学报(自然科学版), 2023, 63(5): 740-753.
[2] 王志华, 庞海波, 李占波. 一种适用于Hadoop云平台的访问控制方案[J]. 清华大学学报(自然科学版), 2014, 54(1): 53-59.
Full text



版权所有 © 《清华大学学报(自然科学版)》编辑部
本系统由北京玛格泰克科技发展有限公司设计开发 技术支持