MRPack: Multi-algorithm execution using compute-intensive approach in MapReduce
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
MRPack : Multi-algorithm execution using compute-intensive approach in MapReduce. / Idris, Muhammad; Hussain, Shujaat; Siddiqi, Muhammad Hameed; Hassan, Waseem; Bilal, Hafiz Syed Muhammad; Lee, Sungyoung.
In: PLoS ONE, Vol. 10, No. 8, e0136259, 25.08.2015.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - MRPack
T2 - Multi-algorithm execution using compute-intensive approach in MapReduce
AU - Idris, Muhammad
AU - Hussain, Shujaat
AU - Siddiqi, Muhammad Hameed
AU - Hassan, Waseem
AU - Bilal, Hafiz Syed Muhammad
AU - Lee, Sungyoung
N1 - Publisher Copyright: © 2015 Idris et al.This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
PY - 2015/8/25
Y1 - 2015/8/25
N2 - Large quantities of data have been generated from multiple sources at exponential rates in the last few years. These data are generated at high velocity as real time and streaming data in variety of formats. These characteristics give rise to challenges in its modeling, computation, and processing. Hadoop MapReduce (MR) is a well known data-intensive distributed processing framework using the distributed file system (DFS) for Big Data. Current implementations of MR only support execution of a single algorithm in the entire Hadoop cluster. In this paper, we propose MapReducePack (MRPack), a variation of MR that supports execution of a set of related algorithms in a single MR job. We exploit the computational capability of a cluster by increasing the compute-intensiveness of MapReduce while maintaining its data-intensive approach. It uses the available computing resources by dynamically managing the task assignment and intermediate data. Intermediate data from multiple algorithms are managed using multi-key and skew mitigation strategies. The performance study of the proposed system shows that it is time, I/O, and memory efficient compared to the default MapReduce. The proposed approach reduces the execution time by 200% with an approximate 50% decrease in I/O cost. Complexity and qualitative results analysis shows significant performance improvement.
AB - Large quantities of data have been generated from multiple sources at exponential rates in the last few years. These data are generated at high velocity as real time and streaming data in variety of formats. These characteristics give rise to challenges in its modeling, computation, and processing. Hadoop MapReduce (MR) is a well known data-intensive distributed processing framework using the distributed file system (DFS) for Big Data. Current implementations of MR only support execution of a single algorithm in the entire Hadoop cluster. In this paper, we propose MapReducePack (MRPack), a variation of MR that supports execution of a set of related algorithms in a single MR job. We exploit the computational capability of a cluster by increasing the compute-intensiveness of MapReduce while maintaining its data-intensive approach. It uses the available computing resources by dynamically managing the task assignment and intermediate data. Intermediate data from multiple algorithms are managed using multi-key and skew mitigation strategies. The performance study of the proposed system shows that it is time, I/O, and memory efficient compared to the default MapReduce. The proposed approach reduces the execution time by 200% with an approximate 50% decrease in I/O cost. Complexity and qualitative results analysis shows significant performance improvement.
UR - http://www.scopus.com/inward/record.url?scp=84942881313&partnerID=8YFLogxK
U2 - 10.1371/journal.pone.0136259
DO - 10.1371/journal.pone.0136259
M3 - Journal article
C2 - 26305223
AN - SCOPUS:84942881313
VL - 10
JO - PLoS ONE
JF - PLoS ONE
SN - 1932-6203
IS - 8
M1 - e0136259
ER -
ID: 388954442