Approximating Quantiles over Sliding Windows Srimathi Harinarayanan CMPS 565
description
Transcript of Approximating Quantiles over Sliding Windows Srimathi Harinarayanan CMPS 565
1
Approximating Quantiles over Sliding Windows
Srimathi Harinarayanan
CMPS 565
2
Streams Here, There, Everywhere!
10
11
1
0
1
0
0
1
1
Network Traffic Engineering.
Call Record Analysis.
Sensor Data Analysis.
Medical, Financial Monitoring.
Etc, etc, etc.
3
Problem Definition Data Stream Environment
One Pass
Data element is a value
Φ-quantile ( [0,1) )The element with rank Ceiling (ΦN) of an ordered sequence of N data elements.
4
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15
12
10
11
10
1 10
11
9 6 7 8 11
4 5 2 3
sort
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 10, 10, 11, 11, 11, 12
N = 16
0.5 quantile returns element ranked 8 ( 0.5*16)
which is 8
0.75 quantile returns element ranked 12 (0.75*16)
which is 10
5
3 Models Data Stream Model
Computing Φ-quantile for all the data items seen so far
Sliding Window Model Computing Φ-quantile against the N most
recent elements in a data stream seen so far n of N Model
For any n of N, computing Φ-quantile among the n most recent elements in a data stream seen so far
6
Sliding Window Model
….1 0 1 0 0 0 1 0 1 1 1 1 1 1 0 0 0 1 0 1 0 0 1 1…
Time Increases
Current Time
Window Size = N
• Most Recent N Elements
7
Sliding window model
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15
12 10 11 10 1 10 11 9 6 7 8 11 4 5 2 3
1 6 7 8 9 10 10 10 11 11 11 12
1 2 3 4 5 6 7 8 9 10 11 11
Window size = 12 , 0.5-quantile returns 10 at time t11
0.5-quantile returns 6 at time t15
8
n-of-N model
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15
12 10 11 10 1 10 11 9 6 7 8 11 4 5 2 3
1 6 7 8 9 10 11 11
2 3 4 5
N = 12, 0.5-quantile returns 8 at time t11 for n = 8,
0.5-quantile returns 3 at time t15 for n = 4
9
Applications - Sliding Window Model in Data Streams
Useful for Network Traffic Management, Sensor Data.
To find out Top Ranked Web pages from Most Recently accessed N pages
In the financial market, investors are often interested in finding out the most recent N bids.
10
Previous Work on Approximating Quantiles in One Scan of Data G. S. Manku, S. Rajagopalan, and B. G. Lindsay.
Approximate medians and other quantiles in one pass and with limited memory [1/1/єє log² log²єєN]N]
G. S. Manku, S. Rajagopalan, and B. G. Lindsay. Random sampling techniques for space efficient online computation of order statistics of large datasets.
M. Greenwald and S. Khanna. Space-efficient online computation of quantile summaries. [1/1/єє log log єєN] {GK N] {GK Algorithm}Algorithm}
GK Algorithm MOST EFFICIENT OWING TO LEAST SPACE USAGE + does not require advance knowledge of N
11
Definitions -Quantile: A -quantile ((0,1]) of an ordered
sequence of N data elements is the element with rank N .
Quantile Query: Given , find the data element with rank N among all elements in the stream. Variation: N recent elements (sliding window
model).
(-approximate): Find the element with rank r within the interval [r-N, r+N].
12
Computation of Quantile Summaries over Sliding Windows – 2 Methods
Continuously Maintaining Quantile Summaries of the Most Recent N Elements over a Data Stream, Xuemin Lin, Hongjun Lu, Jian Xu, Jeffrey Xu Yu, 2004 IEEE
Approximating frequency counts and quantiles using sliding window model, Arvind Arasu, Gurmeet Singh Manku,Stanford University, 2004
13
Computation of Quantile Summaries over Sliding Windows – LLXY04
GK Algorithm + Concept Of Aging (Computing quantiles over a Sliding Window of Most Recent N Elements)
Under sliding window model, a summary is maintained for the most recently seen N data elements.
Eliminate exact out-dated elements requires a space of O(N).
14
e-approximate
A quantile summary for a data sequence is e- approximate if, for any given rank r, it returns a value whose rank r’ is guaranteed to be within the interval [r -εN , r + εN ]
Example : A data stream with 100 elements,
0.5 – quantile with ε= 0.1 returns a value v.
The true rank of v is within [40,60]
15
Quantile Sketch Data structure
{ (vi , ri– ,ri
+) : 1 i m}≦ ≦ A value vi is one of the element seen so
far ri
– is the lower bound on the rank of vi
ri+ is the upper bound on the rank of vi
vi <= vi+1 , for 1 i m - 1≦ ≦ ri
– <= ri+1– , for 1 i m – 1≦ ≦
ri– < =ri <= ri
+ , where ri is the rank of vi
16
Example
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15
12 10 11 10 1 10 11 9 6 7 8 11 4 5 2 3
Quantile sketch consisting of 6 tuples
{(1,1,1), (2,2,9), (3,3,10), (5,4,10), (10,10,10), (12,16,16)}
17
e - approximate sketch
Theorem 1. r1
+ εN + 1≦ , 2. rm
– (1-ε)N,≧ 3. for 2 i m, ≦ ≦
Sketch S is e - approximate, That is for each Φ(0,1] , there is a (vi , ri
– ,ri+) in S such that
N NrrNN ii
Nrr ii 21
18
Query
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15
12 10 11 10 1 10 11 9 6 7 8 11 4 5 2 3
Quantile sketch consisting of 6 tuples ε= 0.25
{(1,1,1), (2,2,9), (3,3,10), (5,4,10), (10,10,10), (12,16,16)}
0.5 – quantile return the vi of rank 8 , εN = 4
4848 ii rNrrNr Find the first tuple to satisfy the rule, and return vi
(4,4,10) => return 4
19
One-Pass summary for sliding windows
Continuously divide a stream into the buckets based on the arrival ordering of data elements
The capacity of each bucket is For each bucket, we maintain an -
approximate continuously by GK-algorithm Once a bucket is full its - approximate
sketch is compressed into an - approximate sketch
The oldest bucket is expired if currently the total number of elements is N+1
2
N
4
4
2
20
Current bucket
the most recent N elements
elements 2
N
elements 2
N
elements 2
N
elements 2
N ….
expired bucket
Compressed - approximate sketch in each bucket2
GK
Summary Technique
21
-approximate sketch4
-approximate sketch
2
ExampleN = 8 , ε= 1 , = 4
1 2 3 4
2
N
5 6 7 8 9
Current bucket
Expire
Current bucket Current bucket
Full , compress
-approximate sketch4
-approximate sketch
2
22
Compress Compress an - approximate sketch
into e-approximate sketch Memory space is most Why not use - approximate sketch in
each bucket directly? Compress technique takes about half
of the number of tuples given by - approximate sketch
2
21
2
2
23
Merge There are h data stream Di ,and each Di
has Ni data elements. Suppose each Si is an e-approximate sketch of Di.
Smerge is a sketch of
|Smerge| =
Suppose each Si is an e-approximate sketch. Then, Smerge is also an e-approximate sketch
ihi D1
ihi S1
24
Another Problem
5, 6, 7, 8,1, 2, 3, 4,
Expired
9
Current
ε=1 and N = 8
2
Approximate sketch 3,3,7,1,1,5 1 ,1 ,9
The first tuple in Smerge is , but the rank of 5 is 4. Smerge is not an - approximate sketch
5.3 ,1 ,5
2
25
Lift To solve the pervious problem, we
use a “lift” operation to lift the value of by for each tuple i
If S is an - approximate sketch, then Slift is an e-approximate sketch
That is why the bucket size is and we maintain - approximate sketch of each bucket summary
ir
2
N
2
N
2
2
26
Query
Step1. merge the local sketch
…N2
N
2
N
2
N
2
N
2
Smerge
Step2. lift Smerge lift
Slift
Current bucket
Step3. for a given rank r = ,find the first tuple
in Slift such that , return vi
N iii rrv ,,
NrrrNr ii
27
Space – Sliding Window LLXY ‘04 O(1/1/єє²² +(log (+(log (єє²N)/²N)/єє))))
Reason: Reason: Sketch in each bucket produced by Sketch in each bucket produced by
the GK algorithm takes O (the GK algorithm takes O (log log ((єє²N)/²N)/єє)) space which will be space which will be compressed to O(1/compressed to O(1/єє) once the ) once the bucket is fullbucket is full
O(1/O(1/єє) buckets) buckets
28
Performance Studies Sliding window model
Compare with the ARS-algorithm Avg Errors Space Consumption Distributions
n-of-N model Compare with the heuristic algorithm nN’
Avg Errors Space Consumption Query performance
29
Conclusion
This work presented is among the first attempts to develop space efficient, one pass, deterministic quantile summary algorithms with performance guarantees under the sliding window model of data streams
30
Approximating quantiles using sliding window model - Manku’s
Approximating Quantiles: GK Algorithm + Concept of Aging Improves over [ LLXY `04 ]
[LLXY `04] space: O(1/1/єє²² +(log (+(log (єє²N)/ ²N)/ єє)))) Manku’s Space: Manku’s Space: O(1/1/єє(log (1/(log (1/єє log N))) log N))) The space complexity is achieved by minimising The space complexity is achieved by minimising
the space used for maintaining the state the space used for maintaining the state at any point in time,e-approximate quantiles, for any (0; 1]) over the current contents of the sliding window can be computed using the maintained state.
The goal is to minimize the space required for maintaining the state.
31
Overview
N
32
Overview
N
33
Overview
N
34
Overview
N
35
Overview
N
36
Overview
N
37
Overview
N
38
Overview
N
39
Details
N
єN4
1є
log ( )є1є0
є2
= O(єN)
40
Space Requirement O(1/1/єє(log (log (1/(1/єє log N))) log N)))
Space required for level-ℓ blocks:
1єℓ
xN
Nℓ
Size of a quantile sketchNumber of “active” blocks
N
єN / log(1є )
= =
1є
1є
log( ) x
Space required for GK Algorithm = 1/1/єє log log єєNN
1/1/єє log log єєN =N =O(1/1/єє(log (1/(log (1/єє log N))) log N)))
1є
1є
log( )
41
Conclusion
The work presented is better than the first method with respect to space.
This paper also provides a randomized quantile finding algorithm with further improvement in space.
42
Any Question?