Princeton MIT Michiganweb.mit.edu/~ecprice/www/slides/fjlt.pdf · Faster Sparse matrices Less...

Fast RIP matrices with fewer rows

Jelani Nelson Eric Price Mary Wootters

Princeton MIT Michigan

2013-04-05

σlarge σsmall

Eric Price (MIT) Fast RIP matrices with fewer rows 2013-04-05 1 / 52

Outline

1 IntroductionCompressive sensingJohnson Lindenstrauss TransformsOur result

2 Concentration of measure: a toolboxOverviewSymmetrizationGaussian ProcessesLipschitz Concentration

3 ProofOverviewCovering Number

4 Conclusion


Outline




4 Conclusion


Compressive Sensing

Given: A few linear measurements of an (approximately) k -sparsevector x ∈ Rn.Goal: Recover x (approximately).

x

=

y

Φm

n


Compressive Sensing Algorithms: Two Classesx

=

y

Φm

n



=

y

Φm

n

Recovery algorithmtied to matrix structure

(e.g. Count-Sketch)

Structure-aware

FasterSparse matricesLess robust

Often:



=

y

Φm

n


(e.g. Count-Sketch)

Structure-aware


Often:

Recovery algorithmsjust multiply by Φ,ΦT

(e.g. L1 minimization)

Structure-oblivious



=

y

Φm

n


(e.g. Count-Sketch)

Structure-aware


Often:



Structure-oblivious

SlowerDense matricesMore robust



=

y

Φm

n


(e.g. Count-Sketch)

Structure-aware


Often:



Structure-oblivious


Yesterday:Fourier→→→ sparse



=

y

Φm

n


(e.g. Count-Sketch)

Structure-aware


Often:



Structure-oblivious


Yesterday:Fourier→→→ sparse

Today


Algorithms for compressive sensing

Goal: recover approximately k -sparse x from y = Φx .

A lot of people use convex optimization:

min ‖x‖1s.t. Φx = y

Also Iterative Hard Thresholding, CoSaMP, OMP, StOMP, ROMP....For all of these:

I the time it takes to multiply by Φ or ΦT is the bottleneck.I the Restricted Isometry Property is a sufficient condition.



Goal: recover approximately k -sparse x from y = Φx .A lot of people use convex optimization:








Also Iterative Hard Thresholding, CoSaMP, OMP, StOMP, ROMP....

For all of these:







I the time it takes to multiply by Φ or ΦT is the bottleneck.

I the Restricted Isometry Property is a sufficient condition.


Restricted Isometry Property (RIP)

Φ

k

All of these submatricesare well conditioned.

(1− ε)‖x‖22 ≤ ‖Φx‖22 ≤ (1 + ε)‖x‖22

for all k -sparse x ∈ Rn.


Goals

What properties should an RIP matrix have?

Good compression: m small

I Random Gaussian matrix: Θ(k log (n/k)) rows.

Fast multiplication:

I Reconstruction dominated by log n multiplications by Φ,ΦT .I Random Gaussian matrix: Θ(nk log n) time.

Goal: an RIP matrix with O(n log n) multiplication and small m.

* Talk will assume n0.1 < k < n0.9, so log k ' log n ' log(n/k).


Goals


Good compression: m smallI Random Gaussian matrix: Θ(k log (n/k)) rows.






Goals


Good compression: m smallI Random Gaussian matrix: Θ(k log n) rows.






Goals



Fast multiplication:I Reconstruction dominated by log n multiplications by Φ,ΦT .

I Random Gaussian matrix: Θ(nk log n) time.




Goals



Fast multiplication:I Reconstruction dominated by log n multiplications by Φ,ΦT .I Random Gaussian matrix: Θ(nk log n) time.




An open question

A = Fm

n

Let A contain random rows from a Fourier matrix.

You can multiply by A in O(n log n) time.How many rows do you need to ensure that A has the RIP?

m = O(k log4 n) [CT06,RV08,CGV13].

Ideal:m = O(k log n).

(Related: how about partial circulant matrices?)m = O(k log4 n) [RRT12,KMR12].


An open question

A = Fm

n

Let A contain random rows from a Fourier matrix.You can multiply by A in O(n log n) time.

How many rows do you need to ensure that A has the RIP?





An open question

A = Fm

n

Let A contain random rows from a Fourier matrix.You can multiply by A in O(n log n) time.How many rows do you need to ensure that A has the RIP?





Outline




4 Conclusion


Another motivation:Johnson Lindenstrauss (JL) Transforms

High dimensional dataS ⊂ Rn




Linear map Φ

Low dimensional sketchΦ(S) ∈ Rm




Linear map Φ


Φ preserves the geometry of S




Linear map Φ



(1− ε)‖x‖2 ≤ ‖Φx‖2 ≤ (1 + ε)‖x‖2




Linear map Φ



(1− ε)‖x‖2 ≤ ‖Φx‖2 ≤ (1 + ε)‖x‖2〈Φx ,Φy〉 = 〈x , y〉 ± ε‖x‖2‖y‖2


Johnson-Lindenstrauss Lemma

Theorem (variant of Johnson-Lindenstrauss ’84)Let x ∈ Rn. A random Gaussian matrix Φ will have

(1− ε)‖x‖2 ≤ ‖Φx‖2 ≤ (1 + ε)‖x‖2

with probability 1− δ, so long as

m &1ε2

log(1/δ)

Set δ = 1/2k : embed 2k points into O(k) dimensions.


What do we want in a JL matrix?

Target dimension should be small (close to 1ε2

k for 2k points).Fast multiplication.

I Approximate numerical algebra problems (e.g., linear regression,low-rank approximation)

I k -means clustering




k for 2k points).

Fast multiplication.I Approximate numerical algebra problems (e.g., linear regression,

low-rank approximation)I k -means clustering




k for 2k points).Fast multiplication.

I Approximate numerical algebra problems (e.g., linear regression,low-rank approximation)

I k -means clustering


How do we get a JL matrix?

Gaussians

I Dimension O( 1ε2 k).

I 1ε2 nk multiplication time.

Best way known for fast JL: by [Krahmer-Ward ’11], RIP⇒ JL.

I Existing results: dimension O( 1ε2 k log4 n).

I n log n multiplication time.

And by [BDDW ’08], JL⇒ RIP; so equivalent.

1

1Round trip loses log n factor in dimension



GaussiansI Dimension O( 1

ε2 k).

I 1ε2 nk multiplication time.





1





ε2 k).I 1

ε2 nk multiplication time.





1





ε2 k).I 1

ε2 nk multiplication time.Best way known for fast JL: by [Krahmer-Ward ’11], RIP⇒ JL.




1





ε2 k).I 1

ε2 nk multiplication time.Best way known for fast JL: by [Krahmer-Ward ’11], RIP⇒ JL.



And by [BDDW ’08], JL⇒ RIP; so equivalent.1

1Round trip loses log n factor in dimensionEric Price (MIT) Fast RIP matrices with fewer rows 2013-04-05 14 / 52

Outline




4 Conclusion


Our result: a fast RIP matrix with fewer rows

H A = Φ

Random sign flipsSubsampled Fourier

B

m

New construction of fast RIP matrices: sparse times Fourier.k log3 n rows and n log n multiplication time.

TheoremIf m ' k log3 n, B ' logc n, and A is a random partial Fourier matrix,then Φ has the RIP with probability at least 2/3.


GeneralizationOur approach is actually works for more general A:

H A = Φ


B

m

Am

A2

A1

...

If A is a “decent” RIP matrix:

A has RIP (whp), but too many (mB) rows.RIP-ness degrades “gracefully” as number of rows decreases:

I For all Ai the RIP constant, although 1, is still controlled.

Then Φ is a good RIP matrix:Φ has the RIP (whp) with m = O(k log3 n) rows.Time to multiply by Φ = time to multiply by A + mB.



H A = Φ


B

m

Am

A2

A1

...

If A is a “decent” RIP matrix:A has RIP (whp), but too many (mB) rows.

RIP-ness degrades “gracefully” as number of rows decreases:





H A = Φ


B

m

Am

A2

A1

...

If A is a “decent” RIP matrix:A has RIP (whp), but too many (mB) rows.RIP-ness degrades “gracefully” as number of rows decreases:




Results in the areaConstruction Measurements m Multiplication Time

Sparse JL matrices [KN12]1ε2

k log n εmn

Partial Fourier [RV08,CGV13]1ε2

k log4 n n log n

Partial Circulant [KMR12]1ε2

k log4 n n log n

Our result: Hash of partial Fourier1ε2

k log3 n n log n

Our result: Hash of partial circulant1ε2

k log3 n n log n

Iterated Fourier [AC06,AL09,AR13]1ε2

k log n† n log n

† Requires k ≤ n1/2−δ. This is the “easy” case:

n k log4 n

n log n

Dimension:

Time:

k log n

k2 log5 n

Gaussian[RV08]


Concentration of Measure

Let Σk is unit-norm k -sparse vectors.We want to show for our distribution Φ on matrices that

E supx∈Σk

∣∣∣‖Φx‖22 − ‖x‖22∣∣∣ < ε,

(Expectation of *) = *

Expected deviation of ΦT Φ from mean In, in a funny norm.

Probabilists have lots of tools to analyze this.


Outline




4 Conclusion


Tools

Screwdriver

Drill

Bit sets

Bit


Tools

Common interface: m drivers, n bits =⇒ mn combinations.

Hex shanks

Common interfacefor drill bits

Gaussians

Common interfacefor probability


A Probabilist’s Toolbox

Convert to Gaussians

Symmetrization

Subgaussians

Berry-Esseen

Gaussian concentration

Hoeffding bound

Dudley’s entropy integral

Lipschitz concentration

Gaussian

Will prove: symmetrization and Dudley’s entropy integral.


Outline




4 Conclusion


SymmetrizationLemma (Symmetrization)Suppose X1, . . . ,Xt are i.i.d. with mean µ. For any norm ‖·‖,

E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]

≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]

where si ∈ ±1 independently.

How well does X concentrate about its mean?

Example (RIP)For some norm ‖·‖, RIP constant of subsampled Fourier

‖AT A− I‖ = ‖∑

ATi Ai − I‖.



E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]

≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]




‖AT A− I‖

= ‖∑

ATi Ai − I‖.



E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]

≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]




‖AT A− I‖ = ‖∑

ATi Ai − I‖.



E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]

≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]


Proof.

Draw X ′1, . . . ,X′t independently from the same distribution.

E[‖1t

∑Xi − E[

1t

∑X ′i ]‖] ≤ E[‖1

t

∑(Xi − X ′i )‖]

= E[‖1t

∑si(Xi − X ′i )‖]

and apply the triangle inequality.



E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]

≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]


Proof.Draw X ′1, . . . ,X

′t independently from the same distribution.

E[‖1t

∑Xi − E[

1t

∑X ′i ]‖] ≤ E[‖1

t

∑(Xi − X ′i )‖]

= E[‖1t

∑si(Xi − X ′i )‖]




E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]

≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]




E[‖1t

∑Xi − E[

1t

∑X ′i ]‖]

≤ E[‖1t

∑(Xi − X ′i )‖]

= E[‖1t

∑si(Xi − X ′i )‖]




E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]

≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]




E[‖1t

∑Xi − E[

1t

∑X ′i ]‖] ≤ E[‖1

t

∑(Xi − X ′i )‖]

= E[‖1t

∑si(Xi − X ′i )‖]




E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]

≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]




E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]

where si ∈ ±1 independently and gi ∼ N(0,1) independently.



E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]


Proof.

We have E[|gi |] ≈ .8 > 2/3.

2E[‖∑

siXi‖]

≤ 3E[‖∑

siEEE[[[|||gi|||]]]Xi‖]

≤ 3E[‖∑

si|||gi|||Xi‖]

= 3E[‖∑

giXi‖].

Because gi = si E[|gi |] and E[|gi |] > 2/3.



E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]


Proof.We have E[|gi |] ≈ .8 > 2/3.

2E[‖∑

siXi‖]

≤ 3E[‖∑

siEEE[[[|||gi|||]]]Xi‖]

≤ 3E[‖∑

si|||gi|||Xi‖]

= 3E[‖∑

giXi‖].




E

[∥∥∥∥∥1t

∑i

Xi − µ

∥∥∥∥∥]≤ 2E

[∥∥∥∥∥1t

∑i

siXi

∥∥∥∥∥]≤ 3E

[∥∥∥∥∥1t

∑i

giXi

∥∥∥∥∥]


Proof.We have E[|gi |] ≈ .8 > 2/3.

2E[‖∑

siXi‖] ≤ 3E[‖∑

siEEE[[[|||gi|||]]]Xi‖]

≤ 3E[‖∑

si|||gi|||Xi‖]

= 3E[‖∑

giXi‖].



Outline




4 Conclusion


Gaussian ProcessesGaussian process Gx : a Gaussian at each point x ∈ T .

Standard problem: E supx∈T Gx .

Example (Maximum singular value of random Gaussian matrix)Let A be a random m × n Gaussian matrix. For any u ∈ Rm andv ∈ Rn, define

Gu,v := uT Av = 〈uvT ,A〉.

Then Gu,v ∼ N(0, ‖uvT‖2F ).

E‖A‖2 = E supu,v∈Sm−1×Sn−1

uT Av = E supu,v∈Sm−1×Sn−1

Gu,v

Depends on the geometry of T .Distance: ‖x − y‖ is standard deviation of Gx −Gy .In example: ‖(u, v)− (u′, v ′)‖ = ‖uvT − u′v ′T‖F .


Gaussian ProcessesGaussian process Gx : a Gaussian at each point x ∈ T .Standard problem: E supx∈T Gx .






Gu,v









Gu,v

Depends on the geometry of T .

Distance: ‖x − y‖ is standard deviation of Gx −Gy .In example: ‖(u, v)− (u′, v ′)‖ = ‖uvT − u′v ′T‖F .








Gu,v

Depends on the geometry of T .Distance: ‖x − y‖ is standard deviation of Gx −Gy .

In example: ‖(u, v)− (u′, v ′)‖ = ‖uvT − u′v ′T‖F .








Gu,v



Gaussian ProcessesGoal: E supx∈T Gx , where Gx −Gy ∼ N(0, ‖x − y‖2).

Ignoring geometry:

I Pr[Gx > σmax t ] ≤ e−t2/2

I Union bound: with high probability, Gx . σmax√

log n.I E supx∈T Gx . σmax

√log n

Two levels: σmax√

log 4 + σsmall√

log n.


Gaussian ProcessesGoal: E supx∈T Gx , where Gx −Gy ∼ N(0, ‖x − y‖2).

Ignoring geometry:




√log n


log 4 + σsmall√

log n.

σmax

Position = xColor = Gx

G0 = 0


Gaussian ProcessesGoal: E supx∈T Gx , where Gx −Gy ∼ N(0, ‖x − y‖2).Ignoring geometry:




√log n


log 4 + σsmall√

log n.

σmax


G0 = 0





log n.

I E supx∈T Gx . σmax√

log n


log 4 + σsmall√

log n.

σmax


G0 = 0






√log n


log 4 + σsmall√

log n.

σmax


G0 = 0






√log n


log 4 + σsmall√

log n.

σmax

σsmall


G0 = 0


Gaussian Processes: chainingBound E supx∈T Gx , where Gx −Gy has variance ‖x − y‖2.Two levels: σmax

√log 4 + σsmall

√log n.

Why stop at two?

E supx∈T

Gx . σ1√

log N(σ2) +

σ2√

log N(σ3) + σ3√

log N(σ4) +

· · ·

σmax

σsmall



√log 4 + σsmall

√log n.

Why stop at two?

E supx∈T

Gx . σ1√

log N(σ2) +

σ2√

log N(σ3) + σ3√

log N(σ4) +

· · ·

T



√log 4 + σsmall

√log n.

Why stop at two?

E supx∈T

Gx . σ1√

log N(σ2) +

σ2√

log N(σ3) + σ3√

log N(σ4) +

· · ·

T

σ1



√log 4 + σsmall

√log n.

Why stop at two?

E supx∈T

Gx . σ1√

log N(σ2) +

σ2√

log N(σ3) + σ3√

log N(σ4) +

· · ·

T

σ1σ2



√log 4 + σsmall

√log n.

Why stop at two?

E supx∈T

Gx . σ1√

log N(σ2) +

σ2√

log N(σ3) + σ3√

log N(σ4) +

· · ·

T

σ1# balls necessary:

N(σ2)

(covering numberdepends on T , ‖·‖)

σ2


Gaussian Processes: chainingBound E supx∈T Gx , where Gx −Gy has variance ‖x − y‖2.Two levels: σ1

√log N(σ2) + σ2

√log n.

Why stop at two?

E supx∈T

Gx . σ1√

log N(σ2) +

σ2√

log N(σ3) + σ3√

log N(σ4) +

· · ·

T

σ1# balls necessary:

N(σ2)

(covering numberdepends on T , ‖·‖)

σ2



√log N(σ2) + σ2

√log n.

Why stop at two?

E supx∈T

Gx . σ1√

log N(σ2) + σ2√

log N(σ3) +

σ3√

log N(σ4) +

· · ·

T

σ3



√log N(σ2) + σ2

√log n.

Why stop at two?

E supx∈T

Gx . σ1√

log N(σ2) + σ2√

log N(σ3) + σ3√

log N(σ4) +

· · ·

T

σ4



√log N(σ2) + σ2

√log n.

Why stop at two?

E supx∈T

Gx . σ1√

log N(σ2) + σ2√

log N(σ3) + σ3√

log N(σ4) + · · ·

T

σr

= σ1/2r



√log N(σ2) + σ2

√log n.

Why stop at two?

E supx∈T

Gx .∞∑

r=0

σ1

2r

√log N

( σ1

2r+1

)

T

σr = σ1/2r



√log N(σ2) + σ2

√log n.

Why stop at two?

E supx∈T

Gx .∞∑

r=0

σ1

2r

√log N

( σ1

2r+1

)

T

σr = σ1/2r

σ

√log N(σ)



√log N(σ2) + σ2

√log n.

Why stop at two?

E supx∈T

Gx .∫ ∞

0

√log N(σ)dσ

T

σr = σ1/2r

σ

√log N(σ)


Gaussian ProcessesDudley’s Entropy Integral, Talagrand’s generic chaining

Theorem (Dudley’s Entropy Integral)Define the norm ‖·‖ of a Gaussian process G by

‖x − y‖ = standard deviation of (Gx −Gy ).

Then

γ2(T , ‖·‖) :=

E supx∈T

Gx .∫ ∞

0

√log N(T , ‖·‖,u)du

Bound a random variable using geometry.


Gaussian ProcessesDudley’s Entropy Integral, Talagrand’s generic chaining

Theorem (Dudley’s Entropy Integral)Define the norm ‖·‖ of a Gaussian process G by

‖x − y‖ = standard deviation of (Gx −Gy ).

Thenγ2(T , ‖·‖) := E sup

x∈TGx .

∫ ∞0

√log N(T , ‖·‖,u)du

Bound a random variable using geometry.


Outline




4 Conclusion


Lipschitz Concentration of Gaussians

TheoremIf f : Rn → R is C-Lipschitz and g ∼ N(0, In), then for any t > 0,

Pr[f (g) > E[f (g)] + Ct ] ≤ e−Ω(t2).

f concentrates as well as individual Gaussians.Can replace f with −f to get lower tail bound.

ExampleIf g ∼ N(0, In), then with probability 1− δ,

‖g‖2 ≤√

n + O(√

log(1/δ)).

For n = O(1/ε2 log(1/δ)), this is 1± ε approximation.

=⇒ the Johnson-Lindenstrauss lemma.


A Probabilist’s Toolbox (recap)

Convert to Gaussians

Symmetrization

Subgaussians

Berry-Esseen

Gaussian concentration

Hoeffding bound

Dudley’s entropy integral

Lipschitz concentration

Gaussian


Outline




4 Conclusion


Goal

H A = Φ

Random sign flips

B

m

logc nmB rows of Fourier matrix

For Σk denoting unit-norm k -sparse vectors, we want

E supx∈Σk

∣∣∣‖Φx‖22 − ‖x‖22∣∣∣ < ε,

(Expectation of *) = *Eric Price (MIT) Fast RIP matrices with fewer rows 2013-04-05 35 / 52

Proof outline: Rudelson-VershyninRudelson-Vershynin: subsampled Fourier, O(k log4 n) rows.

Nelson-P-Wootters: sparse times Fourier, O(k log3 n) rows.

Expectedsup deviation

E sup‖AT A − I‖

Symmetrization

Expectedsup norm

of Gaussian

KrahmerMendelsonRauhut ’13

Dudley

Coveringnumber

N(Σk , ‖·‖,u)

Maurey:randomize

Expecteddeviation

Symmetrization

Expectednorm

of Gaussian

Union bound

Answer!

log2 n loss log n loss

γ2 : supremum of Gaussian process

Σk : k -sparse unit vectors

‖·‖ : a norm that depends on A

(specified in a few slides)






Symmetrization

Expectedsup norm

of Gaussian


γ2(Σk , ‖·‖)

Dudley

Coveringnumber

N(Σk , ‖·‖,u)

Maurey:randomize

Expecteddeviation

Symmetrization

Expectednorm

of Gaussian

Union bound

Answer!











Symmetrization

Expectedsup norm

of Gaussian


γ2(Σk , ‖·‖)

Dudley

Coveringnumber

N(Σk , ‖·‖,u)

Maurey:randomize

Expecteddeviation

E ‖z− E[z]‖

Symmetrization

Expectednorm

of Gaussian

Union bound

Answer!











Symmetrization

Expectedsup norm

of Gaussian


γ2(Σk , ‖·‖)

Dudley

Coveringnumber

N(Σk , ‖·‖,u)

Maurey:randomize

Expecteddeviation

E ‖z− E[z]‖

Symmetrization

Expectednorm

of GaussianE ‖g‖

Union bound

Answer!







Proof outlineRudelson-Vershynin: subsampled Fourier, O(k log4 n) rows.Nelson-P-Wootters: sparse times Fourier, O(k log3 n) rows.


E sup‖ΦT Φ− I‖

Symmetrization

Expectedsup norm

of Gaussian


γ2(Σk , ‖·‖)

Dudley

Coveringnumber

N(Σk , ‖·‖,u)

Maurey:randomize

Expecteddeviation

E ‖z− E[z]‖

Symmetrization

Expectednorm


Union bound

Answer!







Proof outlineRudelson-Vershynin: subsampled Fourier, O(k log4 n) rows.Nelson-P-Wootters: sparse times Fourier, O(k log3 n) rows.



Symmetrization

Expectedsup norm

of Gaussian


γ2(Σk ,‖‖‖‖‖‖‖‖‖·‖‖‖‖‖‖‖‖‖)

Dudley

Coveringnumber

N(Σk ,‖‖‖‖‖‖‖‖‖·‖‖‖‖‖‖‖‖‖,u)

Maurey:randomize

Expecteddeviation

E‖‖‖‖‖‖‖‖‖z− E[z]‖‖‖‖‖‖‖‖‖

Symmetrization

Expectednorm

of GaussianE‖‖‖‖‖‖‖‖‖g‖‖‖‖‖‖‖‖‖

Union bound

Answer!







Proof part I: triangle inequality

H A = Φ

E supx∈Σk

∣∣∣‖Φx‖22 − ‖x‖22∣∣∣

≤ E supx∈Σk

∣∣∣‖Φx‖22 − ‖Ax‖22∣∣∣+ E sup

x∈Σk

∣∣∣‖Ax‖22 − ‖x‖22∣∣∣

= E supx∈Σk

∣∣∣‖XAs‖22 − Es ‖XAs‖22∣∣∣+ (RIP constant of A),

where XA is some matrix depending x and A, and s is the vector ofrandom sign flips used in H.



E supx∈Σk

∣∣‖XAs‖22 − Es ‖XAs‖22∣∣ + (RIP constant of A)



E supx∈Σk


By assumption, this is small.(Recall A has extra rows)



E supx∈Σk


By assumption, this is small.(Recall A has extra rows)

This is a Rademacher Chaos Process.We have to do some work to show that it is small.


Proof part II: probability and geometry

By [KMR12] and some manipulation, can bound the Rademacherchaos using

γ2(Σk , ‖ · ‖A)

Some norm induced by A

The supremum of a Gaussian process over Σk with norm ‖·‖A

Dudley’s entropy integral: can estimate this by bounding the coveringnumber N(Σk , ‖·‖A,u).


Definition of the Norm

N(Σk , ‖·‖A,u)

for the norm ‖x‖A:

Ax= subset of x A1x

`2

B

A2x

`2

B

A3x

`2

B

A4x

`2

B

`∞

m

‖x‖A = maxi∈[m]‖Aix‖2.

Rudelson-Vershynin: estimates N(Σk , ‖·‖A,u) when B = 1.


Definition of the Norm

N(Σk , ‖·‖A,u)

for the norm ‖x‖A:

Ax= subset of x A1x

`2

B

A2x

`2

B

A3x

`2

B

A4x

`2

B

`∞

m

‖x‖A = maxi∈[m]‖Aix‖2.

Rudelson-Vershynin: estimates N(Σk , ‖·‖A,u) when B = 1.Eric Price (MIT) Fast RIP matrices with fewer rows 2013-04-05 40 / 52

Progress



Symmetrization

Expectedsup norm

of Gaussian


γ2(Σk , ‖·‖)

Dudley

Coveringnumber

N(Σk , ‖·‖,u)

Maurey:randomize

Expecteddeviation

E ‖z− E[z]‖

Symmetrization

Expectednorm


Union bound

Answer!



Outline




4 Conclusion


Covering Number Bound

N(Σk , ‖·‖A,u)

≤ N(B1, ‖·‖A,u/√

k)

u

Σk = k -sparse x | ‖x‖2 ≤ 1

⊂√

kB1 = x | ‖x‖1 ≤√

k


Covering Number Bound

N(Σk , ‖·‖A,u) ≤ N(B1, ‖·‖A,u/√

k)

u

Σk = k -sparse x | ‖x‖2 ≤ 1⊂√

kB1 = x | ‖x‖1 ≤√

k


Covering number bound

N(B1, ‖·‖A,u)

Simpler to imagine: what about `2?How many `2 balls of radius u required to cover B1?

N(B1, ‖·‖2,u) .

(1/u)O(n)

by an easy volume argumentnO(1/u2) trickier; next few slides

Latter bound is better when u 1/√

n.Maurey’s empirical method: generalizes to arbitrary norms



N(B1, ‖·‖A,u)

Simpler to imagine: what about `2?

How many `2 balls of radius u required to cover B1?

N(B1, ‖·‖2,u)

.

(1/u)O(n)






N(B1, ‖·‖A,u)


N(B1, ‖·‖2,u)

.

(1/u)O(n)






N(B1, ‖·‖A,u)


N(B1, ‖·‖2,u) .

(1/u)O(n) by an easy volume argument

nO(1/u2) trickier; next few slides





N(B1, ‖·‖A,u)


N(B1, ‖·‖2,u) .

(1/u)O(n) by an easy volume argumentnO(1/u2) trickier; next few slides





N(B1, ‖·‖A,u)


N(B1, ‖·‖2,u) .



n.

Maurey’s empirical method: generalizes to arbitrary norms



N(B1, ‖·‖A,u)


N(B1, ‖·‖2,u) .






N(B1, ‖·‖A,u)


N(B1, ‖·‖A,u) .

(√

B/u)O(n) by an easy volume argumentnO(B/u2) trickier; next few slides




Covering Number BoundMaurey’s empirical method

How many balls of radius u required to cover B

+

1 ?

Consider any x ∈ B+1 .

Let z1, . . . , zt be i.i.d. randomized roundings of x to simplex.The sample mean z = 1

t∑

zi converges to x as t →∞.Let t be large enough that, regardless of x ,

E[‖z− x‖] ≤ u.

All x lie within u of at least one possible z.

I Then N(B1, ‖·‖,u) ≤ number of z

≤ (n + 1)t .I Only (n + 1)t possible tuples (z1, . . . , zt ) =⇒ z.



How many balls of radius u required to cover B+1 ?



t∑


E[‖z− x‖] ≤ u.






x




t∑


E[‖z− x‖] ≤ u.






x

z1



Let z1, . . . , zt be i.i.d. randomized roundings of x to simplex.

The sample mean z = 1t∑


E[‖z− x‖] ≤ u.






x

z2






E[‖z− x‖] ≤ u.






x

z3






E[‖z− x‖] ≤ u.






xz4






E[‖z− x‖] ≤ u.






x

z5






E[‖z− x‖] ≤ u.









t∑

zi converges to x as t →∞.

Let t be large enough that, regardless of x ,

E[‖z− x‖] ≤ u.






Radius u




t∑


E[‖z− x‖] ≤ u.






Radius u1/t




t∑


E[‖z− x‖] ≤ u.






Radius u1/t




t∑


E[‖z− x‖] ≤ u.

All x lie within u of at least one possible z.I Then N(B1, ‖·‖,u) ≤ number of z

≤ (n + 1)t .

I Only (n + 1)t possible tuples (z1, . . . , zt ) =⇒ z.



Radius u1/t




t∑


E[‖z− x‖] ≤ u.

All x lie within u of at least one possible z.I Then N(B1, ‖·‖,u) ≤ number of z ≤ (n + 1)t .I Only (n + 1)t possible tuples (z1, . . . , zt ) =⇒ z.



Radius u1/t




t∑


E[‖z− x‖] ≤ uE[‖z− x‖] ≤ uE[‖z− x‖] ≤ u.




Radius u1/t




t∑


E[‖z− x‖] ≤ uE[‖z− x‖] ≤ uE[‖z− x‖] ≤ u.


Will show: E[‖z− x‖A] ≤√

B/t

=⇒ N(T , ‖·‖A,u) ≤ nB/u2



Radius u1/t




t∑


E[‖z− x‖] ≤ uE[‖z− x‖] ≤ uE[‖z− x‖] ≤ u.


Will show: E[‖z− x‖A] ≤√

B/t =⇒ N(T , ‖·‖A,u) ≤ nB/u2


Covering Number BoundGoal: E[‖z− x‖A] .

√B/t .

Symmetrize!

E[‖1t

∑zi − x‖A] . E[‖1

t

∑gizi‖A]

=:1√tE[‖g‖A]

where g ∈ Rn has

gj ∼ N(0,number of zi at ej

t)

independently in each coordinate.Hence E[‖g‖22] = (fraction of zi that are nonzero) ≤ 1.

Goal: E[‖g‖A] ≤√

B.

(Note: E[‖g‖2] ≤ 1 =⇒ N(B1, `2,u) ≤ n1/u2.)



√B/t .

Symmetrize!

E[‖1t

∑zi − x‖A]

. E[‖1t

∑gizi‖A]

=:1√tE[‖g‖A]

where g ∈ Rn has


t)

independently in each coordinate.

Hence E[‖g‖22] = (fraction of zi that are nonzero) ≤ 1.


B.

(Note: E[‖g‖2] ≤ 1 =⇒ N(B1, `2,u) ≤ n1/u2.)



√B/t .

Symmetrize!

E[‖1t

∑zi − x‖A] . E[‖1

t

∑gizi‖A]

=:1√tE[‖g‖A]

where g ∈ Rn has


t)

independently in each coordinate.

Hence E[‖g‖22] = (fraction of zi that are nonzero) ≤ 1.


B.

(Note: E[‖g‖2] ≤ 1 =⇒ N(B1, `2,u) ≤ n1/u2.)



√B/t .

Symmetrize!

E[‖1t

∑zi − x‖A] . E[‖1

t

∑gizi‖A]

=:1√tE[‖g‖A]

where g ∈ Rn has


t)

independently in each coordinate.Hence E[‖g‖22] = (fraction of zi that are nonzero) ≤ 1.


B.

(Note: E[‖g‖2] ≤ 1 =⇒ N(B1, `2,u) ≤ n1/u2.)


Progress



Symmetrization

Expectedsup norm

of Gaussian


γ2(Σk , ‖·‖)

Dudley

Coveringnumber

N(Σk , ‖·‖,u)

Maurey:randomize

Expecteddeviation

E ‖z− E[z]‖

Symmetrization

Expectednorm


Union bound

Answer!



Bounding the norm (intuition)Just want to bound E[‖g‖A].

Ag= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞

g ∈ Rn has Gaussian coordinates, k -sparse, total variance 1.

Each is N(0,1)

Each coordinate gj = Fjg ∼ N(0,1).Rudelson-Vershynin: B = 1

I Naive: ‖g‖∞ .√

log n.I Gives the log n loss in their bound.

If the gi were independent:

‖Aig‖2 ≤√

B + O(√

log n) w.h.p.

Would get E[‖g‖A] .√

B

so union bound just loses a constant.

They’re not independent... but the Ai satisfy “very weak” RIP.

I Bound ‖Aig‖2 using ‖g‖2, which has independent entries.

Lipschitz concentration(just like

√n +

√log(1/δ) in tutorial)



Ag= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞


Each is N(0,1)

Each coordinate gj = Fjg ∼ N(0,1).

Rudelson-Vershynin: B = 1




‖Aig‖2 ≤√

B + O(√

log n) w.h.p.


B





√n +




Ag= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞


Each is N(0,1)





‖Aig‖2 ≤√

B + O(√

log n) w.h.p.


B





√n +




Ag= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞


Each is N(0,1)



log n.

I Gives the log n loss in their bound.If the gi were independent:

‖Aig‖2 ≤√

B + O(√

log n) w.h.p.


B





√n +




Ag= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞


Each is N(0,1)





‖Aig‖2 ≤√

B + O(√

log n) w.h.p.


B





√n +




Ag= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞


Each is N(0,1)





‖Aig‖2 ≤√

B + O(√

log n) w.h.p.


B

so union bound just loses a constant.They’re not independent... but the Ai satisfy “very weak” RIP.



√n +




Ag= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞


Each is N(0,1)





‖Aig‖2 ≤√

B + O(√

log n) w.h.p.


B so union bound just loses a constant.




√n +




Ag= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞


Each is N(0,1)





‖Aig‖2 ≤√

B + O(√

log n) w.h.p.


B so union bound just loses a constant.They’re not independent... but the Ai satisfy “very weak” RIP.



√n +




Ag= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞


Each is N(0,1)





‖Aig‖2 ≤√

B + O(√

log n) w.h.p.


B so union bound just loses a constant.They’re not independent... but the Ai satisfy “very weak” RIP.

I Bound ‖Aig‖2 using ‖g‖2, which has independent entries.Lipschitz concentration

(just like√

n +√

log(1/δ) in tutorial)


Bounding the norm (by example)Ag

= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞

Just want to bound E[‖g‖A].

Recall: x ∈ Σk/√

k −→ z1, . . . , zt −→ g.The full proof is complicated.Will assume

gi ∼ N(0, xi)

(the “large t” limit).Two examples:

1 Very concentrated: x1 = 1/√

k , rest is zero.2 Very spread out: x1 = · · · = xk = 1/k .

Example (1):‖x‖1 1.Fourier transform g is Gaussian with variance 1/

√k .

‖g‖∞ . k−1/4 ·√

log n 1.Hence ‖Aig‖2

√B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞



k −→ z1, . . . , zt −→ g.

The full proof is complicated.Will assume

gi ∼ N(0, xi)





√k .

‖g‖∞ . k−1/4 ·√


√B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞



k −→ z1, . . . , zt −→ g.The full proof is complicated.

Will assumegi ∼ N(0, xi)





√k .

‖g‖∞ . k−1/4 ·√


√B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞




gi ∼ N(0, xi)

(the “large t” limit).

Two examples:




√k .

‖g‖∞ . k−1/4 ·√


√B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞




gi ∼ N(0, xi)





√k .

‖g‖∞ . k−1/4 ·√


√B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞

x

i

xi




gi ∼ N(0, xi)



k , rest is zero.

2 Very spread out: x1 = · · · = xk = 1/k .


√k .

‖g‖∞ . k−1/4 ·√


√B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞

x

i

xi




gi ∼ N(0, xi)





√k .

‖g‖∞ . k−1/4 ·√


√B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞

x

i

xi




gi ∼ N(0, xi)




Example (1):

‖x‖1 1.Fourier transform g is Gaussian with variance 1/

√k .

‖g‖∞ . k−1/4 ·√


√B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞

x

i

xi




gi ∼ N(0, xi)




Example (1):‖x‖1 1.

Fourier transform g is Gaussian with variance 1/√

k .‖g‖∞ . k−1/4 ·

√log n 1.

Hence ‖Aig‖2 √

B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞

x

i

xi




gi ∼ N(0, xi)





√k .

‖g‖∞ . k−1/4 ·√


√B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞

x

i

xi




gi ∼ N(0, xi)





√k .

‖g‖∞ . k−1/4 ·√

log n 1.

Hence ‖Aig‖2 √

B for all i .



= subset of g

A1g

`2

A2g

`2

A3g

`2

A4g

`2

`∞

x

i

xi




gi ∼ N(0, xi)





√k .

‖g‖∞ . k−1/4 ·√


√B for all i .


Bounding the norm (example 2)Just want to bound E[‖g‖A] when gi ∼ N(0,1/k) for i ∈ [k ].

‖Aig‖2 is C-Lipschitz with factor

C = ‖Ai‖RIP · 1/√

k

Then with high probability,

‖Aig‖2 .√

B + C√

log n

Naive bound:

‖Ai‖RIP ≤ ‖Ai‖F =√

Bk =⇒ C ≤√

B

“Very weak” RIP bound, with B = logc n:

‖Ai‖RIP . ‖Ai‖F/√

log n

SoC .

√B/ log n =⇒ ‖Aig‖2 .

√B w.h.p

So E‖g‖A = max‖Aig‖2 .√

B.


Bounding the norm (example 2)Just want to bound E[‖g‖A] when gi ∼ N(0,1/k) for i ∈ [k ].


C = ‖Ai‖RIP · 1/√

k

Then with high probability,

‖Aig‖2 .√

B + C√

log n

Naive bound:

‖Ai‖RIP ≤ ‖Ai‖F =√

Bk =⇒ C ≤√

B

“Very weak” RIP bound, with B = logc n:

‖Ai‖RIP . ‖Ai‖F/√

log n

SoC .

√B/ log n =⇒ ‖Aig‖2 .

√B w.h.p


B.

Aig

`2


Unrolling everything



Symmetrization

Expectedsup norm

of Gaussian


γ2(Σk , ‖·‖)

Dudley

Coveringnumber

N(Σk , ‖·‖,u)

Maurey:randomize

Expecteddeviation

E ‖z− E[z]‖

Symmetrization

Expectednorm


Union bound

Answer!


√B

√B/t

nkB/u2

√kB log3 n

√k log3 n

m

Union bound just loses a constant factorSample mean z expects to lie within u of x for t ≥ B/u2Covering number of B1 is (n + 1)B/u2Entropy integral is

√kB log3 nRIP constant ε .

√k log3 n

m


Summary and Open Questions

We get fast RIP matrices with O(k log3 n) rows.

Is the extra log2 n necessary?

I Loss seems to be from Dudley’s entropy integral:

sup∑≤∑

sup

I Generic chaining: tight but harder to use. [Fernique, Talagrand]

For JL, is the extra log3 n necessary?

I [Krahmer-Ward] only needs “model-based” RIP; could save log n.

Any other constructions?Lots of cool techniques in the field; can we use them elsewhere?

Thanks!




Is the extra log2 n necessary?I Loss seems to be from Dudley’s entropy integral:

sup∑≤∑

sup


For JL, is the extra log3 n necessary?

I [Krahmer-Ward] only needs “model-based” RIP; could save log n.


Thanks!





sup∑≤∑

sup


For JL, is the extra log3 n necessary?I [Krahmer-Ward] only needs “model-based” RIP; could save log n.


Thanks!





sup∑≤∑

sup



Any other constructions?

Lots of cool techniques in the field; can we use them elsewhere?

Thanks!





sup∑≤∑

sup




Thanks!


Thoughts on lossRecall that

N(B1, `2,u) ≤ n1/u2

So the entropy integral gives

γ2(B1, `2) ≤ log3/2 n.

But the associated Gaussian process is just:

γ2(B1, `2) = E supx∈B1

〈g, x〉

for g ∼ N(0, In). (Check: E[(〈g, x〉 − 〈g, y〉)2] = ‖x − y‖22)We can compute this directly:

E supx∈B1

〈g, x〉 = ‖g‖∞ =√

log n.

Generic chaining: there exists a partition A1,A2, . . . such that

γ2 ' supx

∑√log|Ai+1|d(x ,Ai)

Dudley: choose Ai so sup d(x ,Ai) ≤ σ1/2i .



N(B1, `2,u) ≤ n1/u2


γ2(B1, `2) ≤ log3/2 n.


γ2(B1, `2) = E supx∈B1

〈g, x〉

for g ∼ N(0, In). (Check: E[(〈g, x〉 − 〈g, y〉)2] = ‖x − y‖22)

We can compute this directly:

E supx∈B1

〈g, x〉 = ‖g‖∞ =√

log n.


γ2 ' supx





N(B1, `2,u) ≤ n1/u2


γ2(B1, `2) ≤ log3/2 n.


γ2(B1, `2) = E supx∈B1

〈g, x〉


E supx∈B1

〈g, x〉 = ‖g‖∞ =√

log n.


γ2 ' supx





N(B1, `2,u) ≤ n1/u2


γ2(B1, `2) ≤ log3/2 n.


γ2(B1, `2) = E supx∈B1

〈g, x〉


E supx∈B1

〈g, x〉 = ‖g‖∞ =√

log n.


γ2 ' supx


Dudley: choose Ai so sup d(x ,Ai) ≤ σ1/2i .Eric Price (MIT) Fast RIP matrices with fewer rows 2013-04-05 54 / 52


Answer is nt , where t is such that

E := E[‖1t

∑zi − x‖] ≤ u.

E[1t∑

zi ]

Symmetrize:

E .1tE[‖∑

gizi‖]

for gi ∼ N(0,1) i.i.d.Then g :=

∑gizi is an independent Gaussian in each coordinate.

In `2,

1tE[‖g‖2] ≤ 1

tE[‖g‖22]1/2 =

√number nonzero zi

t≤ 1√

t.

giving an nO(1/u2) bound.




E := E[‖1t

∑zi − x‖] ≤ u.

E[1t∑

zi ]

Symmetrize:

E .1tE[‖∑

gizi‖]

for gi ∼ N(0,1) i.i.d.

Then g :=∑

gizi is an independent Gaussian in each coordinate.In `2,

1tE[‖g‖2] ≤ 1

tE[‖g‖22]1/2 =


t≤ 1√

t.





E := E[‖1t

∑zi − x‖] ≤ u.

E[1t∑

zi ]

Symmetrize:

E .1tE[‖∑

gizi‖]

for gi ∼ N(0,1) i.i.d.Then g :=

∑gizi is an independent Gaussian in each coordinate.

In `2,

1tE[‖g‖2] ≤ 1

tE[‖g‖22]1/2 =


t≤ 1√

t.



Bounding the norm in our case (part 1)

x ∈ Σk/√

k ⊂ B1 rounded to z1, . . . , zt symmetrized to g.

G(x) = Ez,g‖g‖A

First: split x into “large” and “small” coordinates.

G(x) ≤ G(xlarge) + G(xsmall)

xlarge: Locations where xi > (log n)/k

I Bound:‖xlarge‖1

≤ 1/ log n.

I Given ‖x‖22 ≤ 1/k , maximal ‖xlarge‖1 if spread out.

I k/(log2 n) of value (log n)/kI Absorbs the loss from union bound.

So can focus on ‖x‖∞ < (log n)/k .



x ∈ Σk/√


G(x) = Ez,g‖g‖A



xlarge: Locations where xi > (log n)/kI Bound:

‖xlarge‖1

≤ 1/ log n.






x ∈ Σk/√


G(x) = Ez,g‖g‖A




‖xlarge‖1

≤ 1/ log n.


I k/(log2 n) of value (log n)/k

I Absorbs the loss from union bound.




x ∈ Σk/√


G(x) = Ez,g‖g‖A




‖xlarge‖1 ≤ 1/ log n.


I k/(log2 n) of value (log n)/k

I Absorbs the loss from union bound.




x ∈ Σk/√


G(x) = Ez,g‖g‖A




‖xlarge‖1 ≤ 1/ log n.





Bounding the norm in our case (part 2)k -sparse x rounded to z1, . . . , zt symmetrized to g.‖x‖∞ < (log n)/k

gi ∼ N(0, σ2i ) for σ2

i = #zj at vertex ei/t2

≈ xi/t .


C = ‖Ai‖RIP · ‖σ‖∞Naive bound:

C . ‖Ai‖F ·√‖x‖∞/t

≤√

Bk ·√

log n/(kt) =√

B log n/t

“Very weak” RIP bound:

for some B = logc n,

‖Ai‖RIP . log4 n(√

B +√

k)

≤ ‖Ai‖F/ log n.

GivesC .

√B/(t log n)

So with high probability, ‖Aig‖2 .√

B/t + C√

log n .√

B/t .So E‖g‖A = max‖Aig‖2 .

√B/t .


Bounding the norm in our case (part 2)k -sparse x rounded to z1, . . . , zt symmetrized to g.‖x‖∞ < (log n)/kgi ∼ N(0, σ2

i ) for σ2i = #zj at vertex ei/t2

≈ xi/t .‖Aig‖2 is C-Lipschitz with factor


C . ‖Ai‖F ·√‖x‖∞/t

≤√

Bk ·√

log n/(kt) =√

B log n/t




B +√

k)


GivesC .

√B/(t log n)


B/t + C√

log n .√


√B/t .



i ) for σ2i = #zj at vertex ei/t2 ≈ xi/t .



C . ‖Ai‖F ·√‖x‖∞/t

≤√

Bk ·√

log n/(kt) =√

B log n/t




B +√

k)


GivesC .

√B/(t log n)


B/t + C√

log n .√


√B/t .





C = ‖Ai‖RIP · ‖σ‖∞

Naive bound:

C . ‖Ai‖F ·√‖x‖∞/t

≤√

Bk ·√

log n/(kt) =√

B log n/t




B +√

k)


GivesC .

√B/(t log n)


B/t + C√

log n .√


√B/t .

`2






C . ‖Ai‖F ·√‖x‖∞/t

≤√

Bk ·√

log n/(kt) =√

B log n/t




B +√

k)


GivesC .

√B/(t log n)


B/t + C√

log n .√


√B/t .

`2






C . ‖Ai‖F ·√‖x‖∞/t ≤

√Bk ·

√log n/(kt) =

√B log n/t




B +√

k)


GivesC .

√B/(t log n)


B/t + C√

log n .√


√B/t .

`2






C . ‖Ai‖F ·√‖x‖∞/t ≤

√Bk ·

√log n/(kt) =

√B log n/t

“Very weak” RIP bound: for some B = logc n,


B +√

k) ≤ ‖Ai‖F/ log n.

GivesC .

√B/(t log n)


B/t + C√

log n .√


√B/t .

`2






C . ‖Ai‖F ·√‖x‖∞/t ≤

√Bk ·

√log n/(kt) =

√B log n/t



B +√


GivesC .

√B/(t log n)


B/t + C√

log n .√

B/t .


B/t .

`2






C . ‖Ai‖F ·√‖x‖∞/t ≤

√Bk ·

√log n/(kt) =

√B log n/t



B +√


GivesC .

√B/(t log n)


B/t + C√

log n .√


√B/t .

`2


Princeton MIT Michiganweb.mit.edu/~ecprice/www/slides/fjlt.pdf · Faster Sparse matrices Less...

Documents

Transcript of Princeton MIT Michiganweb.mit.edu/~ecprice/www/slides/fjlt.pdf · Faster Sparse matrices Less...

Minimization and Reduction of Plane Curves · Minimization and Reduction of Plane Curves MichaelStoll UniversitätBayreuth (jointworkwithStephanElsenhans) Rational Points on Irrational

Lecture Notes. Cost Minimization Before looked at maximizing Profits (π) = TR – TC or π =pf(L,K) – wL – rK But now also look at cost minimization That.

Harmonic Minimization using PSO of meld μgird-MLCI

A Framework for Efficient and Composable Oblivious Transferweb.eecs.umich.edu/~cpeikert/pubs/slides-ot-crypto08.pdfA Framework for Efﬁcient and Composable Oblivious Transfer Chris

Minimization of Symbolic Automata Presented By: Loris D’Antoni Joint work with: Margus Veanes 01/24/14, POPL14.

Lecture 7: Minimization or maximization of functions ...neufeld/numerical/lecturenotes7.pdfLecture 7: Minimization or maximization ... • As in the case of root finding, ... (a.k.a.

DICTIONARY LEARNING SPARSE REPRESENTATIONShebergement.u-psud.fr/mkowalski/doc/M2_ATSI_dic_learning.pdfSPARSE REPRESENTATION — ATSI DICTIONARY LEARNING: ALTERNATING MINIMIZATION Idea:

arXiv:math/0201308v1 [math.NA] 30 Jan 2002 · · 2008-02-01The minimization is a solution of the governing equations. Careful ... let us ﬁrst suppose that the unknown ψis constant

Digital Signal Processors (DSPs). DSP Advanced signal processor circuits MAC (Multiply and Accumulate) unit (s) - provides fast multiplication of two.

Linear Time Approximation Schemes for the Gale-Berlekamp Game and Related Minimization Problems

Maximal Sparsity Representation via l Minimization...Maximal Sparsity Representation via l1 Minimization David L. Donoho and Michael Elady August 15, 2002 Abstract Finding a sparse

Convergence of the k-Means Minimization Problem using ...homepages.warwick.ac.uk/~masfk/Gamma_Convergence... · The k-means method addresses this problem by clustering data points

20/16-bit AGC Architecture · Web viewP01 Double precision multiply, MPAC * memory, preserving max truncation. R02 triple precision. ... 11 01244 RED OperAddr Go to big-end address.

BIOANALYSIS CATALOG · 2016. 5. 31. · Comparison of Recovery of BOTH Singly and Multiply Phosphorylated Peptides NEW Titansphere™ Phos-TiO MP Kit The above experiment was done

Synthesis of Combinational Logic - dropzone.tamu.edudropzone.tamu.edu/~wshi/475/MIT_Logic_Synthesis.pdf · 6.004 - Fall 2002 9/19/02 L05 – Logic Synthesis 2 Boolean Minimization:

RADAR imaging from multiply scattered waves · 2005-10-28 · Introduction Traditional SAR Geometric Structure of Λ Scattering from Environment RADAR imaging from multiply scattered

PanORAMa: Oblivious RAM with Logarithmic Overhead · PanORAMa: Oblivious RAM with Logarithmic Overhead Sarvar Patel 1, Giuseppe Persianoy1,2, Mariana Raykovaz1,3, and Kevin Yeox ...

Support Vector Machines - EEMB DERSLER · Support Vector Machines Content Introduction The VC Dimension & Structure Risk Minimization Linear SVM The Separable case Linear SVM The

Approximation schemes Scheduling problems. Polynomial Time Approximation Scheme (PTAS) Let Π be a minimization problem. An approximation scheme for problem.

Fourier transforms - ACRUska2014/materials/... · function is simply the sum of the individual fourier transforms. (2) if k is any constant, F[kf(t)] = kF(ω) (2) if we multiply a