Short cv.
I interned at MSR Asia (SRG), where I investigated the intra-machine cost fairness of Xen-Based Cloud Systems.
Publications
- Maestro: Replica-Aware Map Scheduling for MapReduce
.
MapReduce has emerged as a leading programming
model for data-intensive computing. Many recent research
efforts have focused on improving the performance
of the distributed frameworks supporting this model. Many
optimizations are network-oriented and most of them mainly
address the data shuffling stage of MapReduce. Our studies
with Hadoop demonstrate that, apart from the shuffling phase,
another source of excessive network traffic is the high number
of map task executions which process remote data. That leads
to an excessive number of useless speculative executions of map
tasks and to an unbalanced execution of map tasks across
different machines. All these factors produce a noticeable
performance degradation. We propose a novel scheduling
algorithm for map tasks, named Maestro, to improve the
overall performance of the MapReduce computation. Maestro
schedules the map tasks in two waves: first, it fills the empty
slots of each data node based on the number of hosted map
tasks and on the replication scheme for their input data;
second, runtime scheduling takes into account the probability
of scheduling a map task on a given machine depending on
the replicas of the task’s input data. These two waves lead to
a higher locality in the execution of map tasks and to a more
balanced intermediate data distribution for the shuffling phase.
In our experiments on a 100-node cluster, Maestro achieves
around 95% local map executions, reduces speculative map
tasks by 80% and results in an improvement of up to 34% in
the execution time.
- Shadi Ibrahim, Hai Jin, Lu Lu , Bingsheng He, Gabriel Antoniu, Song Wu .
- The 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing CCGrid 2012, May 13-16, 2012, Ottawa, Canada (CCGRID2012).
- Efficient Disk I/O Scheduling with QoS guarantees for Xen-based Platforms.
In this paper, we address the problem of allocating
disk resources to guarantee specified latency and throughput
targets of VMs while keeping efficient disk I/O. Accordingly,
we present two-level scheduling framework, namely Flubber,
in Xen-based hosting platform that decouples latency and
throughput allocation. The high-level throughput control
regulates the pending requests from the VMs, in order to meet
the throughput requirements of different VMs and ensure
isolation. Meanwhile, the low-level latency control, by the
virtue of the batch and delay EDF mechanism, reorders all
pending requests from VMs based on the their deadlines, and
batches them to the disk device considering the locality of
accesses across VMs. We have implemented Flubber with
intensive evaluations on Xen-based host. The results show that
Flubber can simultaneously meet the different service
requirements of VMs while improving the efficiency of the
physical disk. In contrast to CFQ, besides that Flubber
achieves the desired QoS of each VM, Flubber speeds up the
sequential and random read by 17% and 25% due to the
efficient physical disk utilization.
- Xiao Ling, Hai Jin, Shadi Ibrahim, Wenzhi Cao, Song Wu .
- The 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing CCGrid 2012, May 13-16, 2012, Ottawa, Canada (CCGRID2012).
- Adaptive I/O Scheduling for MapReduce in Virtualized Environment
.Virtual machine (VM) interference has long been
a challenging problem for performance predictability and
system throughput for large-scale virtualized environments in
the cloud. Such interferences are contributed by intertwined
factors including the application’s type, the number of concurrent
VMs, and the VM scheduling algorithms used within
the host. Since MapReduce has become an important data
processing platform in the cloud, we investigate the impact
of disk schedulers in Hadoop. Interestingly, our experimental
results report a noticeable variation of the Hadoop performance
between different applications when applying different disk
pairs’ schedulers in both the hypervisor and the virtual
machines. Furthermore, a typical Hadoop application consists
of different interleaving stages, each requiring different I/O
workloads and patterns. As a result, the disk pairs’ schedulers
are not only sub-optimal for different MapReduce applications,
but also sub-optimal for different sub-phases of the whole
job. Accordingly, this paper presents a novel approach for
adaptively tuning the disk pairs’ schedulers in both the
hypervisor and the virtual machines during the execution of a
single MapReduce job. Our results show that MapReduce performance
can be significantly improved; specifically, adaptive
tuning of disk pairs’ schedulers achieves a 25% performance
improvement on a sort benchmark with Hadoop.
- Shadi Ibrahim, Hai Jin, Lu Lu , Bingsheng He, Song Wu .
- The 40th Annual International Conference on Parallel Processing ICPP 2011, Sep 13-16, Taiwan (ICPP2011).
- Towards Pay‐As‐You‐Consume Cloud Computing
.
Cloud computing enables users to perform their
computation tasks in the public virtualized cloud using a payas-
you-go style. Current pay-as-you-go pricing schemes
typically charge on the incurred virtual machine hours. Our
case studies demonstrate significant variations in the user
costs, indicating significant unfairness among different users
from the micro-economic perspective. Further studies reveal
the reason for such variations is interference among
concurrent virtual machines. The amount of interference cost
depends on various factors, including workload characteristics,
the number of concurrent VMs, and scheduling in the cloud. In
this paper, we adopt the concept of pricing fairness from micro
economics, and quantitatively analyze the impact of
interference on the pricing fairness. To solve the unfairness
caused by interference, we propose a pay-as-you-consume
pricing scheme, which charges users according to their
effective resource consumption excluding interference. The key
idea behind the pay-as-you-consume pricing scheme is a
machine learning based prediction model of the relative cost of
interference. Our preliminary results with Xen demonstrate
the accuracy of the prediction model, and the fairness of the
pay-as-you-consume pricing scheme.
- Shadi Ibrahim, Bingsheng He, Hai Jin.
- IEEE 8th International Conference on Services Computing SCC 2011, July 4-9, Washington, USA (SCC2011).
- The MapReduce Programming Model and Implementations
- Hai Jin, Shadi Ibrahim, Li Qi, Haijun Cao, Song Wu, Xuanhua Shi.
- Book Chapter in Cloud Computing: Principles and Paradigms, Wiley Press, 28 Mar 2011.
- LEEN: Locality/Fairness- aware key partitioning for MapReduce in the CloudThis paper investigates the problem of Partitioning
Skew1 in MapReduce-based system. Our studies with Hadoop,
a widely used MapReduce implementation, demonstrate that
the presence of partitioning skew causes a huge amount of data
transfer during the shuffle phase and leads to significant
unfairness on the reduce input among different data nodes. As
a result, the applications experience performance degradation
due to the long data transfer during the shuffle phase along
with the computation skew, particularly in reduce phase. We
develop a novel algorithm named LEEN for locality-aware and
fairness-aware key partitioning in MapReduce. LEEN
embraces an asynchronous map and reduce scheme. All
buffered intermediate keys are partitioned according to their
frequencies and the fairness of the expected data distribution
after the shuffle phase. We have integrated LEEN into
Hadoop-0.18.0. Our experiments demonstrate that LEEN can
efficiently achieve higher locality and reduce the amount of
shuffled data. More importantly, LEEN guarantees fair
distribution of the reduce inputs. As a result, LEEN achieves a
performance improvement of up to 40% on different
workloads.
- Shadi Ibrahim, Hai Jin, Lu Lu, Bingsheng He,Li Qi, Song Wui.
- 2nd IEEE International Conference on Cloud Computing Technology and Science, November 30-December 3, 2010, Indiana, USA (Cloudcom2010).
- Cloud Types and Services
.
The increasing popularity of Internet services such as the Amazon Web Services, Google App Engine and Microsoft Azure have drawn a lot of attention to the Cloud Computing paradigm. Although the term “Cloud Computing” is new, the technology is an extension of the remarkable achievements of grid, virtualization, Web 2.0 and Service Oriented Architecture (SOA) technologies, and the convergence of these technologies. Moreover, interest in Cloud Computing has been motivated by many factors such as the prevalence of multi-core processors and the low cost of system hardware, as well as the increasing cost of the energy needed to operate them. As a result, Cloud Computing, in just three years, has risen to the top of the IT revolutionary technologies, and has been announced as the top technology to watch in the year 2010.
- Hai Jin, Shadi Ibrahim, Tim Bell, Wei Gao, Dachuan Huang, Song Wu.
- Book Chapter in in the Handbook of Cloud Computing, Springer Press, 26 Sep 2010.
- Tools and technologies for building the Clouds
.
With cloud computing growing in popularity, tools and technologies are emerging to build, access, manage, and maintain the clouds. These tools need to manage the huge number of operations within a cloud transparently and without service interruptions. Cloud computing promises lower costs, faster implementation, and more flexibility using mixtures of technologies, and the associated tools are critical for achieving this.
In this chapter, we survey several state-of-the-art techniques for building clouds, starting with virtualization technology. We briefly introduce virtual machines (VMs) and their main features. Then, we introduce the main tools to manage VMs (hypervisors and virtual infrastructure managers) as well as the major technologies used to manage VMs in a public cloud. We then present MapReduce, a powerful model that makes it easier to write programs that take advantage of the power of cloud computing. We conclude by examining four web services tools and technologies that are built for cloud computing.
- Hai Jin, Shadi Ibrahim, Tim Bell, Li Qi, Haijun Cao, Song Wu, Xuanhua Shi.
- Book Chapter in Cloud Computing: Principles Systems and Applications, Springer Press, 2 Aug 2010.
- MR-Scope: A Real Time Tracing Tool for MapReduce
.
MapReduce programming model is emerging as an efficient tool for data-intensive applications. Hadoop, an open-source implementation of MapReduce, has been widely adopted and experienced by both academia and enterprise. Recently, lots of efforts have been done on improving the performance of MapReduce system and on analyzing the MapReduce process based on the log files generated during the Hadoop execution. Visualizing log files seems to be a very useful tool to understand the behavior of the Hadoop process. In this paper, we present MR-Scope, a real-time MapReduce tracing tool. MR-Scope provides a real-time insight of the MapReduce process, including the ongoing progress of every task hosted in Task Tracker. In addition, it displays the health of the Hadoop cluster data nodes, the distribution of the file system's blocks and their replicas and the content of the different block splits of the file system. We implement MR-Scope in native Hadoop 0.1. Experimental results demonstrat that MR-Scope's overhead is less than 4% when running wordcount benchmark.
- Dachuan Huang, Xuanhua Shi, Shadi Ibrahim, Lu Lu, Song Wu, Hai Jin.
- The First International Workshop on MapReduce and its Applications (MAPREDUCE'10) in conjunction with (ACM HPDC 2010), June 22nd, 2010, Chicago, IL, USA.
- Evaluating MapReduce on Virtual Machines : The Hadoop Case
.MapReduceis emerging as an important programming model for large scale parallel application. Meanwhile, Hadoop is an open source implementation of MapReduce enjoying wide popularity for developing data intensive applications in the cloud. As, in the cloud, the computing unit is virtual machine (VM) based; it is feasible to demonstrate the applicability of MapReduce on virtualized data center. Although the potential for poor performance and heavy load no doubt exists, virtual machines can instead be used to fully utilize the system resources, ease the management of such systems, improve the reliability, and save the power. In this paper, a series of experiments are conducted to measure and analyze the performance of Hadoop on VMs. Our experiments are used as a basis for outlining several issues that will need to be considered when implementing MapReduce to fit completely in the cloud.
- Shadi Ibrahim, Hai Jin, Lu Lu, Li Qi, Song Wu, Xuanhua Shi.
- 1st International conference on Cloud Computing (CloudCom2009), Dec 1‐4, 2009, Beijing, China.
- CLOUDLET: Towards MapReduce Implementation on Virtual Machine
.The existing MapReduce framework in virtualized environment suffers from poor performance, due to the heavy overhead of I/O virtualization, and management difficulty for storage and computation. To address the problems, we propose Cloudlet, a novel MapReduce framework on virtual machines. The aim of Cloudlet design is to overcome the overhead of VM while benefiting of the other features of VM (i.e. management and reliability issues).
- Shadi Ibrahim, Hai Jin, Cheng bin, HaiJun Cao, Song Wu, Li Qi.
- Poster Session, 18th International Symposium on High Performance Distributed Computing (HPDC-18), June 11-13, 2009, Munich, Germany.
- ImageFlow: Workflow based Image Processing with Legacy Program in GridAs image processing has been widely used in many scientific research areas, how to find a powerful and inexpensive approach to solve large-scale image processing problems poses great challenges. In this paper, based on workflow theory and Web service technology, we present a novel image processing framework called ImageFlow, which enables images to be processed by distributed legacy software coupled with interconnected target systems. The salient features of ImageFlow are: (1) Workflow based image processing; (2) General running service (GRS) for legacy programs; (3) Adaptive data transfer mechanism with SOAP attachment and GridFTP; and (4) Workflow based software deployment. The experimental results show that our approaches are feasible and efficient. SOAP attachment over HTTP is more preferable for transferring small size of data, while GridFTP performs better when transferring large-scale data. In addition, compared with the sequential deployment model, workflow based software deployment can gain higher speedup when constructing large-scale systems.
- Haijun Cao, Hai Jin, Song Wu, Shadi Ibrahim.
- 2009 International Conference on Future Information Technology and Management Engineering, (FITME 2009), Sanya, China 13-14 Dec 2009.
- Grid Maintenance: Challenges and Existing Models
.As the world of grid computing is continuously growing, scalability and complexity of grid systems are in an increasing rate. Moreover, grid systems are dynamic and subject to various changes. Management of such changes is a challenging issue facing grid communities. Namely, how to efficiently maintain grid system to adopt the environment modification, to meet the user demands and to manage the dynamic changes has become a big challenge. Manual- or script based maintenance approaches are tedious, time consuming and complex, in addition to being highly prone to errors. Therefore, state-of-the-art maintenance models are designed to support transparent runtime maintenance, fast and comprehensive respond to environmental changes and requirements, consequently, improve the availability and efficiency of the grid system. This paper focuses on investigating maintenance models and their common characteristics for grid. To this end, this paper surveys several representative grid maintenance model systems developed by various projects worldwide. The contributions of this paper are (i) discussing the motivations of maintenance for grid system, (ii) discussing the issues and the challenges in building maintenance mechanisms for grid system, (iii) reviewing maintenance models and compare their efficiencies for grid system, and (iv) discussing possible new directions.
- Shadi Ibrahim, Hai Jin, Li Qi, Chunqiang Zeng.
- 3rd IEEE International Conference on Information & Communication Technologies: from Theory to Applications (ICTTA'8), 7-11 April 2008, Damascus, Syria.
- The Implementation of Course Discussion System Using JXTA
- Shadi Ibrahim, Qingping Guo.
- 6th International Conference on Distributed Computing and Applications for business, engineering and sciences (DCABES2007), 14-17 August 2007, YiChang, China,vol I, pp. 1000-1003.
- A Proposal of Next Generation Grid-Operating System
- Raihan Ur Rasool, Qingping Guo, Guo Yucheng,Zhou Zhen, Shadi Ibrahim
- The 2005 International Conference on Grid Computing and Applications, GCA 2005, Las Vegas, Nevada, USA, June 20-23, 2005,pp. 90-98.
Projects
ChinaGrid , funded by Ministry of Education of China.
Basic Theory and Methodology of Virtualization Technology for Computing System, National 973 Basic Research Program, Ministry of Science and Technology of China.
Crane, MPI PaaS for the research institutions to provide parallel computing environment
Interested Fields
- Cloud Computing
- Virtualization Technology
- Distributed System
SHADI IBRAHIM (厦迪)
Services Computing Technology and System Lab.
Cluster and Grid Computing Lab.
School of Computer Science and Technology
Huazhong University of Science and Technology
Wuhan, Hubei, 430074, China
Email: shadi79h@hotmail.com , shadi@hust.edu.cn
te Name
- Grid Computing
- P2P Computing
Now, I am a post-doctoral researcher at INRIA Rennes, working on the MapReduce and Cloud Storage system with Gabriel Antoniu. From 2007 to 2011, I was a PhD. student working with Prof. Hai Jin in the the Services Computing Technology and System Laboratory (SCTS) at Huazhong University of Science & Technology (HUST).
My research is focused around Distributed Systems and Cloud Computing. I'm working on evaluating and improving MapReduce-based Cloud systems ( CLOUDLET and LEEN).
Professional Activities