Mr Share 11 Sep 2010
-
Upload
michalis-potamias -
Category
Technology
-
view
1.520 -
download
4
description
Transcript of Mr Share 11 Sep 2010
MRShare: Sharing Across Multiple Queries in MapReduce
Tomasz Nykiel (University of Toronto)
Michalis Potamias (Boston University)
Chaitanya Mishra (University of Toronto, currently Facebook)
George Kollios (Boston University)
Nick Koudas (University of Toronto)
1
Data management landscape
efficiency
flex
ibili
ty
σ π
2
• Time performance
• Arbitrary data• Large scale setups
MRShare – sharing framework for MR
MRShare – a sharing framework for Map Reduce
• MRShare framework:
– Inspired by sharing primitives from relational domain
– Introduces a cost model for Map Reduce jobs
– Searches for the optimal sharing strategies
– Does not change the Map Reduce computational model
3
Outline
• Introduction
• Map Reduce recap.
• MRShare – Sharing primitives in Map-Reduce
• MRShare – Cost based approach to sharing
• MRShare Evaluation
• Summary
4
Outline
• Map Reduce recap.
5
Map Reduce recap.
I
I
I
I
Map Reduce
6
Outline
• MRShare - Sharing primitives in Map-Reduce
7
Sharing primitives – sharing scans
• SELECT COUNT(*) FROM user GROUP BY hometown
• SELECT AVG(age) FROM user GROUP BY hometown
8
User_id Hometown Occupation Age
MRShare – sharing scans (map).
9
MRShare – sharing scans (reduce)
J1 J2 J3 J4 key value
Toronto 1
Toronto 1
Toronto 1
Toronto 17
Toronto 19
Toronto 2
Toronto 5
10
Outline
• MRShare - Sharing primitives in Map-Reduce
– Sharing scans
– Sharing intermediate data
11
Sharing primitives - Sharing intermediate data.
• SELECT COUNT(*) FROM user WHERE occupation=‘student’ GROUP BY hometown
• SELECT COUNT(*) FROM user WHERE age > 18 GROUP BY hometown
User_id Hometown Occupation Age
Occupation ?= ‘student’ Age ?> 18
12
MRShare – sharing intermediate data (map).
13
MRShare – sharing intermediate data (reduce).
J1 J2 J3 J4 key value
Toronto 1
Toronto 4
Toronto 1
Toronto 1
Toronto 2
Toronto 2
Toronto 5
14
Outline
• MRShare – Cost based approach to sharing
– Cost model for finding the optimal sharing strategy
– SplitJobs – cost based algorithm for sharing scans
– MultiSplitJobs – an improvement of SplitJobs
– γ-MultiSplitJobs – the algorithm for sharing intermediate data
15
Cost model for Map Reduce (single job)
• Reading – f(input size)
• Sorting – f(intermediate data size)
• Copying – f(intermediate data size)
• Writing – f(output size)
16
Cost of executing a group of jobs
17
Finding the optimal sharing strategy
18
• An optimization problem
“NoShare”
“GreedyShare”
Outline
• MRShare – Cost based approach to sharing
– Cost model for finding the optimal sharing strategy
– SplitJobs – cost based algorithm for sharing scans
– MultiSplitJobs – an improvement of SplitJobs
– γ-MultiSplitJobs – the algorithm for sharing intermediate data
19
Sharing scans - cost based optimization
• Savings come from reduced number of scans• The sorting cost might change• The costs of copying and writing the output do not
change
20
• We prove NP-hardness of the problem of finding the optimal sharing strategy
SplitJobs – a DP solution for sharing scans.
• We reduce the problem of grouping to the problem of splitting a sorted list of jobs – by approximating the cost of sorting.
22
• Using our cost model and the approximation, we employ a DP algorithm to find the optimal split points.
Outline
• MRShare – Cost based approach to sharing
– Cost model for finding the optimal sharing strategy
– SplitJobs – cost based algorithm for sharing scans
– MultiSplitJobs – an improvement of SplitJobs
– γ-MultiSplitJobs – the algorithm for sharing intermediate data
23
MultiSplitJobs – an improvement of SplitJobs
24
Outline
• MRShare – Cost based approach to sharing
– Cost model for finding the optimal sharing strategy
– SplitJobs – cost based algorithm for sharing scans
– MultiSplitJobs – an improvement of SplitJobs
– γ-MultiSplitJobs – the algorithm for sharing intermediate data
25
Sharing intermediate data - cost based optimization
• The sorting and copying costs change – depending on the size of the intermediate data
26
We need to estimate the size of the intermediate data of all combinations of jobs.
γ-MultiSplitJobs – the solution for sharing intermediate data
• Approximate the size of the intermediate data
27
• γ –MultiSplitJobs – applies MultiSplitJobs with modified cost function
• γ set heuristically
Outline
• MRShare Evaluation
28
Evaluation setup
• 40 EC2 small instance virtual machines
• Modified Hadoop engine
• 30 GB text dataset consisting of blogs
• Multiple grep-wordcount queries
– Counts words matching a regular expression
– Allows for variable intermediate data sizes
– Generic aggregation Map Reduce job
29
Evaluation goals
• Sharing is not always beneficial.
– ‘GreedyShare’ policy
• How much can we save on sharing scans?
– MRShare - MultiSplitJobs evaluation
• How much can we save on sharing intermediate data?
– MRShare - γ-MultiSplitJobs evaluation
30
Is sharing always beneficial?- ‘GreedyShare’ policy
Group of jobs
Group size
d=|intermediate data| / |input data|
H1 16 0.3 < d <0.7
H2 16 0.7 < d
H3 16 0.9 < d
31
How much we save on sharing scans –MRShare MultiSplitJobs
Group of jobs
Group size
d=|intermediate data| / |input data|
G1 16 0.7 < d
G2 16 0.2 < d < 0.7
G3 16 0.0 < d < 0.2
G4 16 0.0 < d < max
G5 64 0.0 < d < max
32
How much we save on sharing intermediate data -
MRShare - γ-MultiSplitJobs
33
Group of jobs
Group size
d=|intermediate data| / |input data|
G1 16 0.7 < d
G2 16 0.2 < d < 0.7
G3 16 0.0 < d < 0.2
Summary
• We introduced MRShare – a framework for automatic work sharing in Map Reduce.
• We identified sharing primitives and demonstrated the implementation thereof in a Map Reduce engine.
• We established a cost model and solved several work sharing optimization problems.
• We demonstrated vast savings when using MRShare.
34
Thank you!!!
Questions?
35
Ongoing work – sharing expensive computation
• Sharing across multiple Map Reduce jobs with expensive predicates.
36
Ongoing work – dynamic sharing
• Dynamic sharing.
37
time
pro
gres
s