Data-Driven Approaches for Spectral Analysisof Ergodic Dynamical Systems

Dimitris GiannakisCenter for Atmosphere Ocean Science

Courant Institute of Mathematical SciencesNew York University

Center for Informatics and Computational ScienceUniversity of Notre Dame

November, 2018

Collaborators: Tyrus Berry, Shuddho Das, John Harlim, Joanna Slawinska

f (x) = x1, x = (x1, x2, x3)

f (Φt(x)), t = 0.2

f (Φt(x)), t = 0.4

f (Φt(x)), t = 0.6

f (Φt(x)), t = 0.8

f (Φt(x)), t = 1

t = 0

t = 0.2

t = 0.4

t = 0.6

t = 0.8

t = 1

t = 2

t = 3

t = 4

t = 5

t = 15

t = 20

Objectives

Given time-ordered measurements F (x0), . . . ,F (xN−1) of an observable of adynamical system:

1 Identify coherent observables of the system

2 Perform prediction of F , or other observables

Outline

1 Approximation of evolution operators in a data-driven basis(Berry et al. 2015)

2 Computation of Koopman eigenfunctions(G. et al. 2015; G. 2017; Das & G. 2017)

3 RKHS compactification for systems with continuous spectrum(G. et al 2018)

Basic assumptions

• C r flow Φt : M → M, t ∈ R, on a manifold M

• Φt preserves an ergodic probability measure µ ∈ B(M) with compact supportA ⊆ M

• There exists a compact, forward-invariant manifold X ⊆ M containing A

• Observations made through a continuous map F : M → Y taking values in amanifold Y

• A dataset y0, . . . , yN−1, yn = F (xn), is available, with xn = Φn ∆t(x0) and∆t > 0, n ∈ Z

• The sampling interval ∆t is such that µ is ergodic for the discrete-time mapΦn ∆t : M → M, n ∈ Z

Operator-theoretic formalism

• Focus on the action of the dynamical system on spaces of observables

• Natural choices in the present context are C 0(A) and L2(µ)

• For either choice, Φt induces a group of Koopman operators U t acting onobservables via composition with the flow map; e.g.,

U t : L2(µ)→ L2(µ), U t f = f Φt

• These operators act on C 0(A) and L2(µ) by isometries; in the L2(µ) case theyform a unitary group

• Koopman operators and their duals (transfer operators) have receivedsignificant attention in data-driven techniques (Dellnitz & Junge 1999; Dellnitz et al.

2000; Mezic & Banaszuk 2004; Mezic 2005; Rowley et al. 2009; . . . )

Spectral properties of unitary Koopman groups

• Under the basic assumptions, the unitary group U tt∈R is fully characterizedby its generator; a skew-adjoint, densely defined operator

V : D(V )→ L2(µ), D(V ) ⊂ L2(µ), Vf = limt→0

U t f − f

t

• Associated with the generator is a unique projection-valued measureE : B(R)→ L(L2(µ)) such that

V =

∫R

iω dE(ω), U t = etV =

∫R

e iωt dE(ω)

Relationship with Fourier spectra

• Given an observable f ∈ L2(µ), the PVM E induces a non-negative, finitemeasure Ef : B(R)→ R, s.t. Ef (Ω) = 〈f ,E(Ω)f 〉L2(µ)

• Autocorrelation function: Cf : R→ C,

Cf (t) = 〈f ,U t f 〉L2(µ) =

∫R

e iωt dEf (ω)

• Power spectral density: If dEf LebR,

Cf (t) =

∫R

e iωtρf (ω) dω, ρf =dEf

dLebR

Statistical prediction

Given an observable f ∈ L2(µ) and a probability measure ν withρ = dν

dµ∈ L2(µ), we seek to compute

EνU t f = 〈ρ,U t f 〉L2(µ)

• Since U t is bounded, given an orthonormal basis φj of L2(µ) with projectionoperators Πl : L2(µ)→ spanφ0, . . . , φl−1,

〈Πlρ,UtΠl f 〉L2(µ) −−−→

l→∞〈ρ,U t f 〉L2(µ)

• Moreover,

〈Πlρ,UtΠl f 〉L2(µ) =

l∑i,j=0

ρiU(t)ij fj ,

ρj = 〈φj , ρ〉L2(µ), fj = 〈φj , f 〉L2(µ), U(t)ij = 〈φi ,U

tφj〉L2(µ)

• Data-driven approach (Berry et al. 2015). Approximate the matrix elements〈φi ,U

tφj〉L2(µ) in a basis constructed from data using kernel algorithms (Belkin

& Niyogi 2003; Coifman & Lafon 2005; Von Luxburg et al. 2008; Berry & Harlim 2016)

Statistical prediction

Given an observable f ∈ L2(µ) and a probability measure ν withρ = dν

dµ∈ L2(µ), we seek to compute

EνU t f = 〈ρ,U t f 〉L2(µ)

• Since U t is bounded, given an orthonormal basis φj of L2(µ) with projectionoperators Πl : L2(µ)→ spanφ0, . . . , φl−1,

〈Πlρ,UtΠl f 〉L2(µ) −−−→

l→∞〈ρ,U t f 〉L2(µ)

• Moreover,

〈Πlρ,UtΠl f 〉L2(µ) =

l∑i,j=0

ρiU(t)ij fj ,

ρj = 〈φj , ρ〉L2(µ), fj = 〈φj , f 〉L2(µ), U(t)ij = 〈φi ,U

tφj〉L2(µ)

• Data-driven approach (Berry et al. 2015). Approximate the matrix elements〈φi ,U

tφj〉L2(µ) in a basis constructed from data using kernel algorithms (Belkin

& Niyogi 2003; Coifman & Lafon 2005; Von Luxburg et al. 2008; Berry & Harlim 2016)

Kernels and their associated integral operators

• In the present context, a kernel is a continuous function k : M ×M → R

• Associated with k is an integral operator K : L2(µ)→ C 0(X ),

Kf =

∫M

k(·, x)f (x) dµ(x)

• Moreover, there are operators G : L2(µ)→ L2(µ) and G ′ : C 0(X )→ C 0(X )with

G = ι K , G ′ = K ι, ι : f ∈ C 0(X )→ [f ]µ ∈ L2(µ)

• G and G ′ are compact, and there exists an orthonormal basis φ0, φ1, . . . ofL2(µ) with

Gφj = λjφj , λ0 ≥ λ1 ≥ · · ·

• (λj , φj) is an eigenpair of G with λj 6= 0 iff (λj , φ′j) with φ′j = λ−1

j Kφj is aneigenpair of G ′

Kernels and their associated integral operators

• In the present context, a kernel is a continuous function k : M ×M → R

• Associated with k is an integral operator K : L2(µ)→ C 0(X ),

Kf =

∫M

k(·, x)f (x) dµ(x)

• Moreover, there are operators G : L2(µ)→ L2(µ) and G ′ : C 0(X )→ C 0(X )with

G = ι K , G ′ = K ι, ι : f ∈ C 0(X )→ [f ]µ ∈ L2(µ)

• G and G ′ are compact, and there exists an orthonormal basis φ0, φ1, . . . ofL2(µ) with

Gφj = λjφj , λ0 ≥ λ1 ≥ · · ·

• (λj , φj) is an eigenpair of G with λj 6= 0 iff (λj , φ′j) with φ′j = λ−1

j Kφj is aneigenpair of G ′

Pullback kernels from data space

• We focus on “data-driven” kernels of the form

k(x , x ′) = κ(F (x),F (x ′)),

where κ : Y × Y → R is a kernel on data space

• Examples for Y = Rd include

k(x , x ′) = 〈F (x),F (x ′)〉Y (covariance kernel)

k(x , x ′) = exp

(−‖F (x)− F (x ′)‖2

Y

ε

)(radial Gaussian kernel)

• Gaussian kernels are generally preferable to covariance kernels since in theformer case G can have infinite rank even in finite data space dimensions

• In practice, we work with a variable-bandwidth analog of the radial Gaussiankernel (Berry & Harlim 2016), and also perform a Markov normalization (Coifman &

Lafon 2006; Coifman & Hirn 2013)

Data-driven basis

• Assume that µ is a physical measure, i.e., there exists a set Bµ ⊆ X ofpositive Lebesgue measure, such that for all x0 ∈ Bµ the sampling measuresµN =

∑N−1n=0 δxn/N on the orbit xn = Φn ∆t(x0) weak-converge to µ,∫

M

f dµN −−−−→N→∞

∫M

f dµ, ∀f ∈ C 0(X )

• Integral operators KN : L2(µN)→ C 0(X ), GN : L2(µN)→ L2(µN), andG ′N : C 0(X )→ C 0(X ) defined as in the case of µ

• There exists an orthonormal basis φ0,N , . . . , φN−1,N of L2(µN) such that

GNφj,N = λj,Nφj,N , λ0 ≥ λ1 ≥ · · · ≥ λN−1,N

Theorem. For any starting point x0 ∈ Bµ, and every nonzero eigenvalue λj ofG (including multiplicities), the sequence of eigenvalues λj,N of GN convergesas N →∞ to λj . Moreover, for any eigenfunction φj of G at nonzeroeigenvalue λj , there exists a sequence of eigenfunctions φj,N of GN ateigenvalue λj,N such that φ′j,N converges to φ′j in C 0(X ) norm.

Proof. Following the approach of Von Luxburg et al. (2008), establish that theoperators G ′N converge to G ′ compactly (here, due to µ being physical)

Data-driven forecasts

Given f ∈ C 0(X ), a probability measure ν µ with ρ = dνdµ∈ C 0(X ), and lead

time t = q ∆t, q ∈ N:

1 Approximate 〈φi ,Utφj〉L2(µ) by U

(q)N,ij = 〈φi,N ,U

(q)N φj,N〉L2(µN ), where

U(q)N : L2(µN)→ L2(µN) is the q-step shift operator:

U(q)N,ij =

1

N

N−1∑n=0

φi,N(xn)φj,N(xn+q)

2 Choose a spectral truncation parameter l ≤ N − 1, and evaluate the forecast

〈Πl,Nρ,U(q)N Πl,N f 〉L2(µN ) =

l−1∑i,j=0

ρi,NU(q)N,ij fj,N ,

ρi,N = 〈φi,N , ρ〉L2(µN ), fj,N = 〈φj,N , f 〉L2(µN ), U(q)N,ij = 〈φi,N ,U

(q)N φj,N〉L2(µN )

Basic consistency result. If G has no zero eigenvalues,

liml→∞

limN→∞

〈Πl,Nρ,U(q)N Πl,N f 〉L2(µN ) = 〈ρ,U t f 〉L2(µ) = EνU t f

Lorenz 63 prediction results (Berry et al. 2015)

NONPARAMETRIC FORECASTING OF LOW-DIMENSIONAL . . . PHYSICAL REVIEW E 91, 032915 (2015)

0 10 20 30 40 50 60 70 80 900

2

4

6

8

10

12

Forecast Steps (∆ t = 0.1)

RM

SE

Ensemble ForecastEnsemble Error EstimateDiffusion ForecastDiffusion Error EstimateLocal Linear ForecastLocal Linear Error EstimateIterated Local Linear ForecastIterated Local Linear Error EstimateInvariant Measure

0 1 2 3 4 5 6 7 8 910−1

100

101

102

Forecast Steps (∆ t = 0.5)

RM

SE

Ensemble ForecastEnsemble Error EstimateDiffusion ForecastDiffusion Error EstimateLocal Linear ForecastLocal Linear Error EstimateIterated Local Linear ForecastIterated Local Linear Error EstimateInvariant Measure

−10010 −200

20

10

20

30

40

−10010 −200

20

10

20

30

40

−10010 −200

20

10

20

30

40

−10010 −200

20

10

20

30

40

FIG. 2. (Color online) Comparing forecast methods for Lorenz-63. Top: !t = 0.1. Middle: !t = 0.5. Bottom: Eigenfunctionsϕ40,ϕ500,ϕ1500, and ϕ4000 of the coarse approximation of the fractalattractor by a manifold.

We chose this very small perturbation to demonstrate thediffusion forecast for an initial condition which is almostperfect; as the amount of noise increases, the advantage ofthe diffusion forecast over the linear methods is even moresignificant.

The diffusion forecast is performed with 4500 eigenfunc-tions ϕj , constructed with the diffusion maps algorithm with avariable bandwidth [5] (we show examples in Fig. 2). The locallinear forecast uses ordinary least squares to fit an affine model

to the n-step shift map on the 15 nearest neighbors to the initialstate. The iterated local linear forecast completes this processfor one step and then recomputes the 15 nearest neighbors tothe 1-step forecast and then repeats the process. The varianceestimate of the local linear models is given by conjugating thecovariance matrix of p0 with the linear part of the appropriateaffine forecast model. We compute the root-mean-squarederror (RMSE) between each mean forecast and the true state,averaged over the verification period of 5000 steps. We alsoshow the standard deviation of the forecast density, so that aforecasting method has good uncertainty quantification (UQ)if the standard deviation agrees with the RMSE.

Of course, the ensemble forecast with the true model givesthe best forecast; however the diffusion forecast is a consid-erable improvement over the local linear forecast. For short!t = 0.1, the iterated local linear forecast is comparable to thediffusion forecast except in the long term where the iteratedlocal linear forecast exhibits significant bias. Moreover, theiterated local linear forecast significantly overestimates theerror variance in the intermediate to long term forecast. Thisoverestimation is due to the positive Lyapunov exponent,which is implicitly estimated by the product of the iteratedlocal linearizations. In contrast, the direct local linearizationis unbiased in the long term, but converges very quickly tothe invariant measure and underestimates the variance. Thisunderestimation is because no single linearization can capturethe information creation introduced by the positive Lyapunovexponent. For long!t = 0.5, the bias in the local linear modelsleads them to diverge far beyond the invariant measure for evenintermediate term forecasts. The ensemble forecast providesthe most consistent UQ since it has access to the true model;however the diffusion forecast produces reasonable estimateswithout knowing the true model as shown in Fig. 2. Thelocal linear forecast error estimates vary widely and do notrobustly provide a useful UQ whereas the diffusion forecastis robust across multiple sampling times as suggested by thetheory. We include a video showing good long-term agreementbetween the diffusion forecast density and an ensemble in theSupplemental Material [10].

The diffusion forecast is able to give a reasonable estimateof the evolution of the density by building a consistentfinite-dimensional Markovian approximation of the dynamics.This Markovian system incorporates global knowledge of theattractor structure via the smoothing with the adapted basisϕj . This Markovian approximation of the Lorenz-63 modelimplicitly uses a small Brownian forcing to replicate theentropy generation of the positive Lyapunov exponent.

El Nino data set. We now apply our method to a real-worlddata set, where the validity of our theory is unverifiable, namelythe Nino-3.4 index, which records the monthly anomaliesof sea surface temperature (SST) in the central equatorialPacific region (the raw data set is available from NOAA [12]).In applying this method, we implicitly assume that there isan underlying dynamics on a low-dimensional manifold thatgenerates these SST anomalies. Since the time series is one-dimensional, we apply the time-delay embedding technique tothe data set, following [13–15], which will recover this low-dimensional manifold if it exists. We use a 5-lag embedding,empirically chosen to maximize the forecast correlation skillbetween the true time series and the mean estimate. We

032915-3

Decomposition of the Koopman spectrum

There exists a U t-invariant splitting L2(µ) = Ha ⊕ Hc and unique PVMsEa : B(R)→ L(Ha) and Ec : B(R)→ L(Hc) such that:

1 E = Ea ⊕ Ec

2 Ea is purely atomic, and Ec is continuous

• There exists an orthonormal basis zj of Ea consisting of eigenfunctions of V :

Vzj = iωjzj , ωj ∈ R

• The corresponding eigenfrequencies ωj are the atoms of Ea:

Ea(ωj) = 〈zj , ·〉L2(µ)zj

• Every observable f ∈ Ha has a quasiperiodic evolution,

U t f =∑j

〈zj , f 〉L2(µ)eiωj tzj

• In contrast, every f ∈ Hc has the weak-mixing property,

1

t

∫ t

0

|〈g ,Us f 〉L2(µ)| ds −−−→t→∞

0, ∀g ∈ L2(µ)

Coherent observables through Koopman eigenfunctions

• Koopman eigenfunctions form a distinguished class of observables that evolveunder the dynamics by multiplication by a periodic phase factor:

U tzj = e iωj tzj

• By ergodicity, all Koopman eigenvalues are simple, and the eigenfunctions canbe normalized s.t. |zj | = 1, µ-a.e.

• If zj is continuous, it provides a topological semiconjugacy with a circlerotation R t

ωj: S1 → S1 at frequency ωj :

A A

S1 S1

Φt

zj zj

Rtωj

• Group property: If zj , zk are eigenfunctions at eigenfrequencies ωj , ωk ,

V (zjzk) = i(ωj + ωk)zjzk

• If all Koopman eigenfunctions are continuous, they and their correspondingeigenfrequencies can be generated given knowledge of finitely manyeigenfunctions corresponding to rationally independent eigenfrequencies

Challenges with data-driven approximation

Issues that must be confronted by numerical algorithms include:

• The generator V is unbounded

• A non-trivial continuous spectrum may be present

• Even in the absence of continuous spectrum, the set of eigenfrequencies willgenerally be a dense subset of the real line

We address these issues by constructing a family of kernel integral operatorshaving common eigenspaces with U t

(G. 2017; Das & G. 2017)

Basic observation. A kernel integral operator G : L2(µ)→ L2(µ) associatedwith a shift-invariant kernel,

k(Φt(x),Φt(x ′)) = k(x , x ′), ∀t ∈ R, µ× µ-a.e. (x , x ′) ∈ M ×M,

is a compact operator commuting with U t :

[G ,U t ] = GU t − U tG = 0

• The eigenspaces of G at nonzero corresponding eigenvalues arefinite-dimensional U t -invariant subspaces of Ha, and restricted to thesesubspaces, V is unitarily diagonalizable

Kernels with delay-coordinate maps

• Start from the pseudometric function dQ : M ×M → R, Q ∈ N,

d2Q(x , x ′) =

1

Q

Q−1∑q=0

‖F (Φq∆t(x))− F (Φq∆t(x ′))‖2Y

• Define the kernel kQ : M ×M → R with kQ(x , x ′) = e−d2Q (x,x′)/ε

Theorem. The following hold:

1 d∞ = limQ→∞ dQ is well defined as a function in L2(µ× µ)

2 The projection of d∞ onto the continuous spectrum subspace Hc ⊗ Hc isµ× µ-a.e. constant

3 As a result, ker G∞ ⊇ Hc , where G∞ : L2(µ)→ L2(µ) is the integral operator

associated with k∞ = e−d2∞/ε

4 As Q →∞, the integral operators GQ : L2(µ)→ L2(µ) associated with kQ

converge in operator norm (and thus in spectrum) to G∞

Galerkin approximation

• Ha,F = ran G∞ is an invariant subspace of Ha; it admits an orthonormal basisφj consisting of eigenfunctions of G∞ at positive corresponding eigenvalue λj

• Define the Sobolev spaces

Hpa,F =

∞∑j=0

cjηpj φj :

∞∑j=0

|cj |2ηpj <∞

, ηj =

1/λj − 1

ε

and the Laplace-like operator ∆ : D(∆)→ Ha,F , with

D(∆) = H2a,F ⊂ D(V ), ∆φj = ηjφj

• Formulate a Petrov-Galerkin method for the regularized generator

L = V − θ∆, θ > 0,

using H2a,F and Ha,F as trial and test spaces, respectively

• Because V and ∆ commute, the effect of ∆ is to add a real part to theeigenvalues of V equal to the Dirichlet energy of the correspondingeigenfunctions:

Lzj = γjzj =⇒ γj = −〈zj ,∆zj〉L2(µ) + iωj

• Data-driven analog of the scheme constructed as in the prediction case, usingtemporal finite differences to approximate V ; convergence holds in a suitablelimit ∆t → 0, l ,Q,N →∞

Variable-speed rotation on T2

x1

−1.5

−1

−0.5

0

0.5

1

1.5

t

x3

0 5 10 15 20 25−0.5

0

0.5

v =2∑µ=1

vµ∂

∂θµ

v 1 = 1 + β cos θ1

v 2 = ω(1− β sin θ2)

ω =√

30, β =√

1/2

Numerical Koopman eigenfunctions for variable-speed rotation on T2

Koopman eigenfunctions from noisy data

Koopman eigenfunctions for the variable-speed flow on T2 recovered from datafrom data corrupted with i.i.d. Gaussian noise in R3 with SNR ' 1

Recovering convective organization in the tropical atmosphere

20oW 40

oW 60

oW 80

oW 100

oW 120

oW 140

oW 160

oW 180

oW 160

oE 140

oE 120

oE 100

oE 80

oE 60

oE 40

oE 20

oE 0

o

5oN

0o

10oS

15oS

10oN

15oN

5oS

20oW 40

oW 60

oW 80

oW 100

oW 120

oW 140

oW 160

oW 180

oW 160

oE 140

oE 120

oE 100

oE 80

oE 60

oE 40

oE 20

oE 0

o

5oN

0o

10oS

15oS

10oN

15oN

5oS

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

20oW 40

oW 60

oW 80

oW 100

oW 120

oW 140

oW 160

oW 180

oW 160

oE 140

oE 120

oE 100

oE 80

oE 60

oE 40

oE 20

oE 0

o

5oN

0o

10oS

15oS

10oN

15oN

5oS

20oW 40

oW 60

oW 80

oW 100

oW 120

oW 140

oW 160

oW 180

oW 160

oE 140

oE 120

oE 100

oE 80

oE 60

oE 40

oE 20

oE 0

o

5oN

0o

10oS

15oS

10oN

15oN

5oS

20oW 40

oW 60

oW 80

oW 100

oW 120

oW 140

oW 160

oW 180

oW 160

oE 140

oE 120

oE 100

oE 80

oE 60

oE 40

oE 20

oE 0

o

5oN

0o

10oS

15oS

10oN

15oN

5oS

day -1

day -2

day 0

day 1

day 2

Convectively coupled equatorial waves recovered from satellite observationsof brightness temperature via approximate Koopman eigenfunctions (Slawinska &

G. 2016)

Coherent observables in the presence of continuous spectrum

• Koopman eigenfunctions do not provide a notion of coherent observables in thecontinuous spectrum subspace Hc

• In principle, such observables could be identified through eigenfunctions of aregularized generator L = V − θ∆, but this approach encounters problems dueto domain issues and/or non-normality of L

• As an alternative approach (G. et al. 2018), we regularize V by mapping it into acompact, skew-adjoint operator acting on a suitable reproducing kernelHilbert space (RKHS) of functions on M

Basic properties of RKHSs

Let k : X × X → R be a kernel with the properties:

1 k(x , x ′) = k(x ′, x), ∀x , x ′ ∈ X (symmetry)

2∑N−1

m,n=0 c∗mk(xm, xn)cn ≥ 0, ∀xm ∈ X , ∀cm ∈ C (non-negativity)

Then, there exists a unique Hilbert space of functions H on X such that:

1 The kernel sections k(x , ·) lie in H for all x ∈ X

2 The reproducing property, f (x) = 〈k(x , ·), f 〉H, holds for every x ∈ X

• The reproducing property implies that the pointwise evaluation functionalsJx : H → C, Jx f = f (x), are continuous at every x ∈ X

• If k is C r , then H is a dense subspace of C r (X ), and convergence in H normimplies convergence in C r (X ) norm

RKHSs and kernel integral operators

• Associated with k is an integral operator K : L2(µ)→ H,

Kf =

∫X

k(·, x)f (x) dµ(x)

• The adjoint, K∗ : H → L2(µ), is the inclusion operator, K∗f = [f ]µ

• G = K∗K : L2(µ)→ L2(µ) is a positive-semidefinite, self-adjoint,Hilbert-Schmidt integral operator, and there exists an orthonormal basisφj∞j=0 of L2(µ) s.t.,

Gφj = λjφj , λ0 ≥ λ1 · · · ≥ 0

• For j such that λj > 0, define ψj = Kφj/λ1/2j ; then, ψj is an orthonormal

basis of ran K ⊆ H• If all λj are positive, H|A is universal (Micchelli et al. 2006), i.e., lies dense in C 0(A)

• For a universal RKHS, N : D(N )→ H, with

D(N ) =

∞∑j=0

cjφj ∈ L2(µ) :∞∑j=0

|cj |2/λj <∞

, Nφj = ψj/λ

1/2j

is a densely-defined extension operator s.t. D(N ) = ran K∗, ranN = ran K ,

K∗N f = f , NK∗g = g , ∀f ∈ D(N ), ∀g ∈ ranN

RKHS compactification schemes for the generator

Given a symmetric positive-definite kernel in C 1(X × X ),

ran G ⊆ ran K∗ ⊂ D(V )

Pre-smoothing:

• A = VG is a Hilbert-Schmidt integral operator on L2(µ) with kernelk ′ ∈ C 0(X × X ), k ′(·, x) = Vk(·, x), i.e.,

Af =

∫X

k ′(·, x)f (x) dµ(x)

Post-smoothing:

• GV : D(V )→ L2(µ) is a densely-defined bounded operator, whose closure,GV = (GV )∗∗ =: B is equal to −A∗, i.e.,

GV ⊂ −A∗ = B, Bf = −∫X

k ′(x , ·)f (x) dµ(x)

RKHS compactification schemes for the generator

Given a symmetric positive-definite kernel in C 1(X × X ),

ran G ⊆ ran K∗ ⊂ D(V )

Pre-smoothing:

• A = VG is a Hilbert-Schmidt integral operator on L2(µ) with kernelk ′ ∈ C 0(X × X ), k ′(·, x) = Vk(·, x), i.e.,

Af =

∫X

k ′(·, x)f (x) dµ(x)

Post-smoothing:

• GV : D(V )→ L2(µ) is a densely-defined bounded operator, whose closure,GV = (GV )∗∗ =: B is equal to −A∗, i.e.,

GV ⊂ −A∗ = B, Bf = −∫X

k ′(x , ·)f (x) dµ(x)

RKHS compactification schemes for the generator

Given a symmetric positive-definite kernel in C 1(X × X ),

ran G ⊆ ran K∗ ⊂ D(V )

Pre-smoothing:

• A = VG is a Hilbert-Schmidt integral operator on L2(µ) with kernelk ′ ∈ C 0(X × X ), k ′(·, x) = Vk(·, x), i.e.,

Af =

∫X

k ′(·, x)f (x) dµ(x)

Post-smoothing:

• GV : D(V )→ L2(µ) is a densely-defined bounded operator, whose closure,GV = (GV )∗∗ =: B is equal to −A∗, i.e.,

GV ⊂ −A∗ = B, Bf = −∫X

k ′(x , ·)f (x) dµ(x)

RKHS compactification schemes for the generator

Skew-adjoint compactification on H:

• W = KVK∗ is a skew-adjoint, Hilbert-Schmidt operator on H given by

Wf = −∫X

k ′(x , ·)f (x) dµ(x)

• The densely defined operator K∗WN : D(N )→ L2(µ) is a restriction of B

Skew-adjoint compactification on L2(µ):

• G 1/2VG 1/2 is a densely-defined, bounded, antisymmetric operator, whoseclosure is a skew-adjoint, Hilbert-Schmidt operator V : L2(µ)→ L2(µ)

• V is unitarily equivalent to W , V = U∗WU , where U : L2(µ)→ H is a unitaryoperator s.t. K = UG 1/2 (polar decomposition)

RKHS compactification schemes for the generator

Skew-adjoint compactification on H:

• W = KVK∗ is a skew-adjoint, Hilbert-Schmidt operator on H given by

Wf = −∫X

k ′(x , ·)f (x) dµ(x)

• The densely defined operator K∗WN : D(N )→ L2(µ) is a restriction of B

Skew-adjoint compactification on L2(µ):

• G 1/2VG 1/2 is a densely-defined, bounded, antisymmetric operator, whoseclosure is a skew-adjoint, Hilbert-Schmidt operator V : L2(µ)→ L2(µ)

• V is unitarily equivalent to W , V = U∗WU , where U : L2(µ)→ H is a unitaryoperator s.t. K = UG 1/2 (polar decomposition)

Spectra of the compactified generators

• A, B, V , and W have the same, purely imaginary, spectra, includingmultiplicities of eigenvalues

• There exist orthonormal bases z0, z1, . . . and ζ0, ζ1, . . . of L2(µ) and H,respectively, s.t.,

V zj = iωjzj , W ζj = iωjζj , ωj ∈ R, ζj = Uzj

If k is Markovian, i.e.,∫X

k(x , ·) dµ = 1, ∀x ∈ X , then:

• 0 is a simple eigenvalue of A, B, V , and W corresponding to constanteigenfunctions

• z ′j = G−1/2zj and zj = G 1/2zj are eigenfunctions of A and B, respectively, andz ′0, z ′1, . . . and z0, z1, . . . form biorthogonal, unconditional Schauder basesof L2(µ)

• The following expansions converge in operator norm:

A =∞∑j=0

iωj〈zj , ·〉L2(µ)z′j , B =

∞∑j=0

iωj〈z ′j , ·〉L2(µ)zj , V =∞∑j=0

iωj〈zj , ·〉L2(µ)zj ,

W =∞∑j=0

iωj〈ζj , ·〉Hζj ,

Spectra of the compactified generators

• A, B, V , and W have the same, purely imaginary, spectra, includingmultiplicities of eigenvalues

• There exist orthonormal bases z0, z1, . . . and ζ0, ζ1, . . . of L2(µ) and H,respectively, s.t.,

V zj = iωjzj , W ζj = iωjζj , ωj ∈ R, ζj = Uzj

If k is Markovian, i.e.,∫X

k(x , ·) dµ = 1, ∀x ∈ X , then:

• 0 is a simple eigenvalue of A, B, V , and W corresponding to constanteigenfunctions

• z ′j = G−1/2zj and zj = G 1/2zj are eigenfunctions of A and B, respectively, andz ′0, z ′1, . . . and z0, z1, . . . form biorthogonal, unconditional Schauder basesof L2(µ)

• The following expansions converge in operator norm:

A =∞∑j=0

iωj〈zj , ·〉L2(µ)z′j , B =

∞∑j=0

iωj〈z ′j , ·〉L2(µ)zj , V =∞∑j=0

iωj〈zj , ·〉L2(µ)zj ,

W =∞∑j=0

iωj〈ζj , ·〉Hζj ,

Functional calculus

• Associated with V and W are unique, purely-atomic PVMsE : B(R)→ L(L2(µ)) and E : B(R)→ L(H), s.t.

E(Ω) =∑

j :ωj∈Ω

〈zj , ·〉L2(µ)zj , E(Ω) =∑

j :ωj∈Ω

〈ζj , ·〉Hζj , E(Ω) = U∗E(Ω)U

V =

∫R

iω dE(ω), W =

∫R

iω dE(ω)

• Given a Borel-measurable function Z : iR→ C, we compute

Z(V ) =

∫R

Z(iω) dE(ω) =∞∑j=0

Z(iω)〈zj , ·〉L2(µ)zj ,

Z(W ) =

∫R

Z(iω) dE(ω) =∞∑j=0

Z(iω)〈ζj , ·〉Hζj

• Being non-normal, A and B generally do not have associated PVMs, but it isstill possible to define a, more-restrictive, holomorphic functional calculus

Spectral convergence

Let kττ>0 be a continuous, one-parameter family of C 1, symmetric, strictlypositive kernels on X , s.t., as τ → 0+,

1 Gτ = K∗τ Kτ → I , pointwise on L2(µ)

2 Vτ → V , pointwise on D(V 2) ⊂ D(V )

D(V 2) is a core for V , and as a result condition 2 implies that Vτ converges toV in strong resolvent sense

The latter, in conjunction with compactness and skew-adjointness of theapproximating operators Vτ implies in turn the following:

• As τ → 0+, the operators Aτ and Bτ converge strongly to V on D(V )

• For every bounded continuous function Z : iR→ C, and as τ → 0+, Z(Vτ )and K∗τ Z(Wτ )Nτ converge strongly to Z(V )

• For every bounded Borel-measurable set Ω ⊂ R such that E(∂Ω) = 0, and asτ → 0+, Eτ (Ω) and K∗τ Eτ (Ω)Nτ converge strongly to E(Ω)

• For every element of the spectrum iω of V , there exists a sequence ofeigenvalues iωτ of Vτ (and Aτ , Bτ , Wτ ) depending continuously on τ , andconverging to iω as τ → 0+

Markov semigroups

To construct a one-parameter family of kernels meeting the previously statedconditions:

1 Start from a C 2, symmetric, strictly positive-definite, Markov kernelp : X × X → R (constructed, e.g., by normalization of an unnormalized kernel(Coifman & Hirn 2013))

2 Compute the eigenvalues and eigenfunctions of the associated operatorG = P∗P : L2(µ)→ L2(µ),

Gφj = λjφj , 1 = λ0 > λ1 ≥ λ2 ≥ · · ·

3 For every τ > 0 and x , x ′ ∈ X , define

pτ (x , x ′) =∞∑j=0

e−τηjφj(x)φj(x ′), ηj =1

λj− 1

Then, the following hold:

• The series for pτ (x , x ′) converges to a C 2 kernel on X , uniformly on thesupport of µ

• Defining G0 = I and Gτ = P∗τPτ , the family Gττ≥0 is a strongly continuous,self-adjoint, Markov semigroup

• As τ → 0+, the corresponding compactified generator Vτ converges to V ,pointwise on D(V 2), as required

Prediction of observables

• Compute Z(Wτ ) for the family of functions et : iω 7→ e iωt , t ∈ R

• etWτ can be viewed as an approximation of the Koopman operator on theRKHS Hτ

• Given a prediction observable f ∈ L2(µ) and error bound ε > 0, there existsfε ∈ ∩τ>0Hτ such that ‖U t f − U tK∗τ fε‖L2(µ) < ε for all τ > 0 and t ∈ R, and

limτ→0+

‖U t fε − etWτ fε‖L2(µ) = 0

• In practice, fε is computed by projection of f onto a subspace of D(Nτ )

• The resulting predictor etWτ fε can be evaluated at arbitrary x ∈ X , includingpreviously unseen states

Torus rotation—eigenfunctions of Wτ

Torus rotation—prediction of observables using etWτ

F : T2 → R3, (θ1, θ2) 7→

(1 + R cos θ2) cos θ1

(1 + R cos θ2) sin θ1

sin θ2

L63 system—eigenfunctions of Wτ

L63 system—prediction of observables using etWτ

Rossler system—eigenfunctions of Wτ

Rossler system—prediction of observables using etWτ

Indo-Pacific sea surface temperature (Slawinska & G. 2017; G. & Slawinska 2018)

Conclusions

• Kernel methods provide a useful framework for data-driven prediction andspectral decomposition in ergodic dynamical systems

• Advantages of the approach include the ability to construct bases for functionspaces of appropriate regularity with minimal prior knowledge about the system

• Regularization approach based on compactification of the generator in a spaceof functions of high regularity (RKHS), as opposed to adding diffusion, leads toa notion of coherent observables in systems with continuous spectra and anapproximation framework for the functional calculus of the generator

Ongoing and future work:

-40 -20 0 20 40

x+y

5

10

15

20

25

30

35

40

45

z

• Data assimilation and control

• Spectral approximation of vector fields anddifferential forms (exterior calculus)(Berry & G. 2018)

• Applications to climate and fluid dynamics(Slawinska & G. 2018)

References

• Berry, T., D. Giannakis, J. Harlim (2015). Nonparametric forecasting oflow-dimensional dynamical systems. Phys. Rev. E, 91, 032915

• Giannakis, D., J. Slawinska, Z. Zhao (2015). Spatiotemporal feature extractionwith data-driven Koopman operators. J. Mach. Learn. Res. Proceedings, 44,103–115

• Giannakis, D. (2017). Data-driven spectral decomposition and forecasting ofergodic dynamical systems. Appl. Comput. Harmon. Anal.doi:10.1016/j.acha.2017.09.001

• Das, S., D. Giannakis (2017). Delay-coordinate maps and the spectra ofKoopman operators. arXiv:1706.08544

• Giannakis, D., S. Das, and J. Slawinska (2018). Reproducing kernel Hilbertspace compactification of unitary evolution groups. arXiv:1808.01515

• Slawinska, J. and D. Giannakis (2018). Characterizing and predicting climatevariability through the spectral theory of dynamical systems. In preparation