Oceanography 569Oceanographic Data Analysis Laboratory

Kathie KellyApplied Physics Laboratory

515 Ben Hall IR Bldgclass web site:

faculty.washington.edu/kellyapl/classes/ocean569_2014/

Exercise 3a: Error Estimates

Known errors for Q, T and H

Need error estimates for

• Q/(ρ cpH)• dT/dt

Exercise 3a: Error Estimates

Terms quite similar:• Q/(ρ cpH)• dT/dt

Compare LHS variance<[dT/dt - Q/(ρ cpH)]2>

with error variance estimate

1) Compute estimated relative error variance for heating term

2) Convert to error variance by multiplying by variance

3) Compute error variance for dT/dt (assuming uncorrelated errors)

4) Total error variance is sum of these terms5) Compare with variance of LHS6) Error variance > LHS variance => other terms negligible comared with errors

Exercise 3a: Error Estimates

Exercise 3b: Significance Test

Is the recent global temperature change significantly different from the previous values?

Probability of year-to-yeartemperature differences

Exercise 3b: Significance Test

Differences ΔT can be normalized to N(0,1) using the Z-transform

Probability p of Z score (or lower) found from a table OR

probability of ΔT score can be found from Matlab:

p = normcdf(Z,0,1) [Matlab function]

Is the most recent temperature change significantly different from previous values at 5%? (Is the probability of recent value smaller than 5%?)

The error variance in the derivative dT/dt is given by

How is the error variance affected by the choice of time step Δt?

Consider two cases:

1) errors are point-to-point random2) errors have time scales long relative to T

Derivatives and Errors: strategy

1) For errors with little serial correlation the error variance in the derivative decreases with increasing interval Δt

2) For errors that are highly correlated (more like a bias) there is little advantage to increasing the interval size because the error terms nearly cancel

Derivatives and Errors

Analysis of Variance (ANOVA)

Evaluate a dynamical or statistical model using observations[mean and periodic signals removed]

Example: linear function

The error is the difference between the model and the observations. The data variance NOT explained by model (error variance) is

The fraction of data variance σ2 that IS explained by the model is usually called the skill

Signal-to-noise ratio:

Chi-SquaredA related measure of the goodness of fit of a least-squares estimator is chi-squared (Χ2)

For errors that are all about the same size this is just the error variance divided by the data variance

which is the fraction of variance in the error so in this caseskill is given by

Analysis of Variance: correlation vs. skill

What is the relationship between correlation and skill for

By definition:

In terms of variance:

Eliminate y:

which is the skill S. This is a special case where amplitude of y is nearly the same as that of x and

Analysis of Variance for a Model

What is the relationship between skill and correlation for

In terms of variance:

So the skill S is given by

Exercise 4: Lagged correlations

SSH: longitude-time

Δt

Δx

Lag correlate the SSH at one location with SSH at every other location to get this image:

Measuring Similarity

The correlation gives a measure of the similarity of two time series but not their magnitude Example of a model that is highly correlated with databut smaller in magnitude

The fraction of variance explained by the model (skill) does not distinguish between correlations and magnitudes.

Does any metric describe both?

To compare both correlations and relative magnitudes (Taylor, 2001)

Given a model of an observed value

The mean squared error is given by

which reduces to (1)

because

Compare with law of cosines for triangle

(2)

Taylor Diagram

σm

σo

ε

θ

Truth ---

Taylor Diagram & Normalized VersionEqns (1) and (2) match if we define

Triangle shows error contributions of both correlation and magnitude geometrically

Normalized version:

Divide through by variance of observations σo

or

where r is relative magnitude

Analysis of Variance for a Model

What is the relationship between skill and correlation for

In terms of variance:

So the skill S is given by

where . For large a, S can be negative. (For small a the correlation will be small and the assumption is violated.)

Skill: Empirical vs. ModelA good estimator has a skill near 1 (small squared error).

A linear regression is an empirical fit to data that minimizes the squared error (least-squares fit). Its skill is positive by design.

Skill or fraction of variance explained can also be used to evaluate a model

dm(x,t) = f[u(x,t])

However, models do not optimize the fit to the observations (unless data assimilation is used) so the skill can be negative.

For example, a model that predicts the seasonal cycle of ocean temperature, with a good match of phase, but an underestimate of amplitude, could give a negative skill.

Therefore, Taylor diagrams are frequently used to evaluate models.

Lowpass Filter

Smooth data to remove errors. What is the assumption?

filter removes half the power at specified time (or space) scale

Input to function “butter” in signal processing toolbox:

Wn = 2*t/half_power

2*t is twice the grid spacing (Nyquist frequency)

Other Filters

Highpass filter: removes low frequencies (or large spatial scales)

Bandpass filter: removes low and high frequencies (small and large scales)

Filtering and Correlations

Lowpass filter: what happens to the integral length scale?

How does this affect N*?

How does this affect the significance level for correlations?

Linear Algebra Review (2)

Linear Algebra Review (3)

B=A\C

Linear Algebra Review (1)

Linear Regression

Linear Regression

Linear Regression (cont’d)

x

minimum error

observationz

y

estimate

estimate error

best estimate

Linear Regression (cont’d)

y=X

=X\y

Linear Regression: limiting the number of variables

Note: fitting data to a curve is a simple form of linear regression in which variables X are 1, x, x2, x3, ...

Coefficients are optimized to give best fit to the data. For each variable X added to the regression squared error decreases, because coefficients also fit random components (“noise”).

However, on another set of data the same coefficients will not fit random components and the fit may be worse. The amount by which the estimator overfits data is sometimes called “artificial skill.”

Minimize “artificial skill” by limiting regression to only significant variables, which are determined using an F test on the variance reduction (skill) of the estimator.

We check the estimator by comparing the errors from regression and errors from another set of data.

Significance of Linear Regression

k is number of additional parameters

Code for Linear Regression (1)

Code for Linear Regression (2)

Exercise 6: Linear Regressioncreate linear estimator for heat flux

Linear regressionTest each variable (hindcast)

Regression on single variables for latent heat flux

[use only half the data]

cosine air-sea temp.

wind speed humidity diff.

Multiple RegressionTest combinations of variables (hindcast)

Find variable(s) least correlated with best single variable

correlated = redundant

F test for evaluating additional variables

humidity humidity + wind

humidity + wind + cosine

Multiple RegressionExamine residuals

plot residual

check histogram (nearly normal?)

Is there a pattern in residual?

residual

humidity + wind + cosine

Linear regressionTest regressions on new data (forecast)

Compare hindcast and forecast errors

Do estimators perform as predicted?

Check for patterns in residual

humidity + wind