Exact and Stable Covariance Estimation from Quadratic Sampling

Yuxin Chen? Yuejie Chi† Andrea J. Goldsmith?? Electrical Engineering, Stanford University † Electrical and Computer Engineering, Ohio State University

• Covariance Estimationin – second-order statistics Σ ∈ Rn×nin – cornerstone of many information processing tasks

• Quadratic Samplingin – obtain measurements of the form

y ≈ a>Σa (1)

Objectives

binary data stream by Kazmin

• Covariance Sketchingin – data stream: real-time data {xt}∞t=1 arrivingsequentially at a high rate...

• Challengesin – limited memoryin – computational efficiencyin – a single pass over the data

Application: Data Stream Processing

• Proposed qudratic sketching method!  Quadratic)Sketching)Scheme!

!x1"

!x2"

!x3"

!x4"

!x5"

!x6"

!x7"

!xN!

a1T"

!xi!

amT"

random))quadratic)sampling:)"a1T xixi

Ta1

xiT" !a1"

y1 =1N

a1T xixi

Ta1i=1

N

∑

aggregate)all)sketches"

N"data"instances"!"m"sketches"

random)sampling)"

n6dim)

1. sketching:

– at each time t, randomly choose a sketching vector ai– observe a quadratic sketch (aTi xt)

2

!  Quadratic)Sketching)Scheme!

!x1"

!x2"

!x3"

!x4"

!x5"

!x6"

!x7"

!xN!

a1T"

!xi!

amT"

random))quadratic)sampling:)"a1T xixi

Ta1

xiT" !a1"

y1 =1N

a1T xixi

Ta1i=1

N

∑

aggregate)all)sketches"

N"data"instances"!"m"sketches"

n6dim)

2. aggregation:

– all sketches are aggregated into m measurements

yi ≈ E(a>i xtx

>t ai

)= a>i Σai (1 ≤ i ≤ m)

3. goal (estimation)

– estimate the covariance matrix Σ from y := {yi}mi=1

• Benefits

– one pass

– minimal storage (as will be shown)

Sketching*

Aggrega.on*

Es.ma.on*

• High-frequency wireless and signal processing

– Stationary processes / signals (possibly sparse)

t1 t2 t3– In high-frequency regime, energy measurements are more reliable

– Goal: recovery the power spectral density from energy measurements

∗ Frequency spikes can assume any continuous value (off-the-grid).

Application: Spectral Estimation

• Suppose that rank(Σ) = r � n

The Task

C = A* B*+

Low-rank Matrix

Unknown rank, eigenvectors

Sparse “Errors” Matrix

Unknown support, values

GivenComposite

matrix

• Convex Relaxation (TraceMin)

minimize trace (M )

s.t. ‖A (M )− y‖1 ≤ ε1,

M � 0.

– This coincides with PhaseLift (Candes et. al.) forrank-1 cases.

Formulation

• Given m (� n2) quadratic measurements y = {yi}mi=1

yi = a>i Σai + ηi, i = 1, · · · ,m, (2)

we wish to recover Σ ∈ Rn×n.

– ai : sketching/sampling vectors

– η = {ηi}mi=1: inaccuracy / noise terms

– More concise operator form: y = A(Σ) + η

• Sampling model

– sub-Gaussian i.i.d. sketching / sampling vectors

Algorithm: general low-rank structure

• Convex Relaxation (ToeplitzMin)

minimize M1,1

s.t. ‖A (M )− y‖1 ≤ ‖η‖1 ,M � 0,

M is Toeplitz.

– Can acommodate off-grid frequencies.

Algorithm: spectrally sparse processes

Piet Mondrian

Theoretical Guarantee

Theorem. (Non-stationary) If ‖η‖1 ≤ ε1, then the solution to TraceMin obeys∥∥∥Σ̂−Σ∥∥∥

F≤ C1

‖Σ−Σr‖∗√r

+ C2ε

m, (3)

where Σr is the best rank-r approximation of Σ, provided that m > Θ(nr).

(Stationary) If ‖η‖2 ≤ ε2, then the solution to ToeplitzMin obeys∥∥∥Σ̂−Σ∥∥∥

F≤ C3ε2√

m, (4)

provided that rank(Σ) ≤ r and m > Θ(r · poly log(n)).

m / (n*n)

r/n

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

information theoretic limit

m: number of measurements

r: r

ank

5 10 15 20 25 30 35 40 45 500

5

10

15

20

25

30

35

40

45

50

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

theoretic sampling limit

(Left) Non-stationary case; (Right) Stationary case.

• Exact and Universal Recovery: from minimal noiseless measurements;

• Stable Recovery: inaccuracy proportional to noise level;

• Robust Recovery: robust to imperfect structural assumptions.

Discussion

• Work for other structural models

– Sparse covariance matrix

– Sparse phase retrieval (rank-1)

– Simultaneous sparse and low-rank covariance model

• Paper: Exact and Stable Covariance Estimation from Quadratic Samplingvia Convex Programming (http://arxiv.org/abs/1310.0807)