Generalized Additive Models:an introduction with R

COPYRIGHT CRC DO NOT DISTRIBUTE

Simon N. Wood

Contents

Preface xi

1 Linear Models 1

1.1 A simple linear model 2

Simple least squares estimation 3

1.1.1 Sampling properties of 3

1.1.2 So how old is the universe? 5

1.1.3 Adding a distributional assumption 7

Testing hypotheses about 7

Confidence intervals 9

1.2 Linear models in general 10

1.3 The theory of linear models 12

1.3.1 Least squares estimation of 12

1.3.2 The distribution of 13

1.3.3 (i i)/i tnp 141.3.4 F-ratio results 15

1.3.5 The influence matrix 16

1.3.6 The residuals, , and fitted values, 16

1.3.7 Results in terms of X 17

1.3.8 The Gauss Markov Theorem: whats special about least

squares? 17

1.4 The geometry of linear modelling 18

1.4.1 Least squares 19

1.4.2 Fitting by orthogonal decompositions 20

iii

iv CONTENTS

1.4.3 Comparison of nested models 21

1.5 Practical linear models 22

1.5.1 Model fitting and model checking 23

1.5.2 Model summary 28

1.5.3 Model selection 30

1.5.4 Another model selection example 31

A follow up 34

1.5.5 Confidence intervals 35

1.5.6 Prediction 36

1.6 Practical modelling with factors 36

1.6.1 Identifiability 37

1.6.2 Multiple factors 39

1.6.3 Interactions of factors 40

1.6.4 Using factor variables in R 41

1.7 General linear model specification in R 44

1.8 Further linear modelling theory 45

1.8.1 Constraints I: general linear constraints 45

1.8.2 Constraints II: contrasts and factor variables 46

1.8.3 Likelihood 48

1.8.4 Non-independent data with variable variance 49

1.8.5 AIC and Mallows statistic, 50

1.8.6 Non-linear least squares 52

1.8.7 Further reading 54

1.9 Exercises 55

2 Generalized Linear Models 59

2.1 The theory of GLMs 60

2.1.1 The exponential family of distributions 62

2.1.2 Fitting Generalized Linear Models 63

2.1.3 The IRLS objective is a quadratic approximation to the

log-likelihood 66

CONTENTS v

2.1.4 AIC for GLMs 67

2.1.5 Large sample distribution of 68

2.1.6 Comparing models by hypothesis testing 68

Deviance 69

Model comparison with unknown 70

2.1.7 and Pearsons statistic 70

2.1.8 Canonical link functions 71

2.1.9 Residuals 72

Pearson Residuals 72

Deviance Residuals 73

2.1.10 Quasi-likelihood 73

2.2 Geometry of GLMs 75

2.2.1 The geometry of IRLS 76

2.2.2 Geometry and IRLS convergence 79

2.3 GLMs with R 80

2.3.1 Binomial models and heart disease 80

2.3.2 A Poisson regression epidemic model 87

2.3.3 Log-linear models for categorical data 92

2.3.4 Sole eggs in the Bristol channel 96

2.4 Likelihood 101

2.4.1 Invariance 102

2.4.2 Properties of the expected log-likelihood 102

2.4.3 Consistency 105

2.4.4 Large sample distribution of 107

2.4.5 The generalized likelihood ratio test (GLRT) 107

2.4.6 Derivation of 2 2r under H0 108

2.4.7 AIC in general 110

2.4.8 Quasi-likelihood results 112

2.5 Exercises 114

vi CONTENTS

3 Introducing GAMs 119

3.1 Introduction 119

3.2 Univariate smooth functions 120

3.2.1 Representing a smooth function: regression splines 120

A very simple example: a polynomial basis 120

Another example: a cubic spline basis 122

Using the cubic spline basis 124

3.2.2 Controlling the degree of smoothing with penalized regres-

sion splines 126

3.2.3 Choosing the smoothing parameter, : cross validation 128

3.3 Additive Models 131

3.3.1 Penalized regression spline representation of an additive

model 132

3.3.2 Fitting additive models by penalized least squares 132

3.4 Generalized Additive Models 135

3.5 Summary 137

3.6 Exercises 138

4 Some GAM theory 141

4.1 Smoothing bases 142

4.1.1 Why splines? 142

Natural cubic splines are smoothest interpolators 142

Cubic smoothing splines 144

4.1.2 Cubic regression splines 145

4.1.3 A cyclic cubic regression spline 147

4.1.4 P-splines 148

4.1.5 Thin plate regression splines 150

Thin plate splines 150

Thin plate regression splines 153

Properties of thin plate regression splines 154

Knot based approximation 156

4.1.6 Shrinkage smoothers 156

CONTENTS vii

4.1.7 Choosing the basis dimension 157

4.1.8 Tensor product smooths 158

Tensor product bases 158

Tensor product penalties 161

4.2 Setting up GAMs as penalized GLMs 163

4.2.1 Variable coefficient models 164

4.3 Justifying P-IRLS 165

4.4 Degrees of freedom and residual variance estimation 166

4.4.1 Residual variance or scale parameter estimation 167

4.5 Smoothing Parameter Estimation Criteria 168

4.5.1 Known scale parameter: UBRE 168

4.5.2 Unknown scale parameter: Cross Validation 169

Problems with Ordinary Cross Validation 170

4.5.3 Generalized Cross Validation 171

4.5.4 GCV/UBRE/AIC in the Generalized case 173

Approaches to GAM GCV/UBRE minimization 175

4.6 Numerical GCV/UBRE: performance iteration 177

4.6.1 Minimizing the GCV or UBRE score 177

Stable and efficient evaluation of the scores and derivatives 178

The weighted constrained case 181

4.7 Numerical GCV/UBRE optimization by outer iteration 182

4.7.1 Differentiating the GCV/UBRE function 182

4.8 Distributional results 185

4.8.1 Bayesian model, and posterior distribution of the parameters,

for an additive model 185

4.8.2 Structure of the prior 187

4.8.3 Posterior distribution for a GAM 187

4.8.4 Bayesian confidence intervals for non-linear functions of

parameters 190

4.8.5 P-values 190

4.9 Confidence interval performance 192

viii CONTENTS

4.9.1 Single smooths 192

4.9.2 GAMs and their components 195

4.9.3 Unconditional Bayesian confidence intervals 198

4.10 Further GAM theory 200

4.10.1 Comparing GAMs by hypothesis testing 200

4.10.2 ANOVA decompositions and Nesting 202

4.10.3 The geometry of penalized regression 204

4.10.4 The natural parameterization of a penalized smoother 205

4.11 Other approaches to GAMs 208

4.11.1 Backfitting GAMs 209

4.11.2 Generalized smoothing splines 211

4.12 Exercises 213

5 GAMs in practice: mgcv 217

5.1 Cherry trees again 217

5.1.1 Finer control of gam 219

5.1.2 Smooths of several variables 221

5.1.3 Parametric model terms 224

5.2 Brain Imaging Example 226

5.2.1 Preliminary Modelling 228

5.2.2 Would an additive structure be better? 232

5.2.3 Isotropic or tensor product smooths? 233

5.2.4 Detecting symmetry (with by variables) 235

5.2.5 Comparing two surfaces 237

5.2.6 Prediction with predict.gam 239

Prediction with lpmatrix 241

5.2.7 Variances of non-linear functions of the fitted model 242

5.3 Air Pollution in Chicago Example 243

5.4 Mackerel egg survey example 249

5.4.1 Model development 250

5.4.2 Model predictions 255

CONTENTS ix

5.5 Portuguese larks 257

5.6 Other packages 261

5.6.1 Package gam 261

5.6.2 Package gss 263

5.7 Exercises 265

6 Mixed models: GAMMs 273

6.1 Mixed models for balanced data 273

6.1.1 A motivating example 273

The wrong approach: a fixed effects linear model 274

The right approach: a mixed effects model 276

6.1.2 General principles 277

6.1.3 A single random factor 278

6.1.4 A model with two factors 281

6.1.5 Discussion 286

6.2 Linear mixed models in general 287

6.2.1 Estimation of linear mixed models 288

6.2.2 Directly maximizing a mixed model likelihood in R 289

6.2.3 Inference with linear mixed models 290

Fixed effects 290

Inference about the random effects 291

6.2.4 Predicting the random effects 292

6.2.5 REML 293

The explicit form of the REML criterion 295

6.2.6 A link with penalized regression 296

6.3 Linear mixed models in R 297

6.3.1 Tree Growth: an example using lme 298

6.3.2 Several levels of nesting 303

6.4 Generalized linear mixed models 303

6.5 GLMMs with R 305

6.6 Generalized Additive Mixed Models 309

x CONTENTS

6.6.1 Smooths as mixed model components 309

6.6.2 Inference with GAMMs 311

6.7 GAMMs with R 312

6.7.1 A GAMM for sole eggs 312

6.7.2 The Temperature in Cairo 314

6.8 Exercises 318

A Some Matrix Algebra 325

A.1 Basic computational efficiency 325

A.2 Covariance matrices 326

A.3 Differentiating a matrix inverse 326

A.4 Kronecker product 327

A.5 Orthogonal matrices and Householder matrices 327

A.6 QR decomposition 328

A.7 Choleski decomposition 328

A.8 Eigen-decomposition 329

A.9 Singular value decomposition 330

A.10 Pivoting 331

A.11 Lanczos iteration 331

B Solutions to exercises 335

B.1 Chapter 1 335

B.2 Chapter 2 340

B.3 Chapter 3 345

B.4 Chapter 4 347

B.5 Chapter 5 354

B.6 Chapter 6 363

Bibliography 373

Index 378

Preface

This book is designed for readers wanting a compact, but thorough, introduction to

linear models, generalized linear models , generalized additive models, and the mixed

model extension of these, with particular emphasis on generalized additive models.

The aim is to provide a full, but concise, theoretical treatment, explaining how the

models and methods work, in order to underpin quite extensive material on practical

application of the models using R.

Linear models are statistical models in which a univariate response is modelled as the

sum of a linear predictor and a zero mean random error term. The linear predictor

depends on some predictor variables, measured with the response variable, and some

unknown parameters, which must be estimated. A key feature of linear models is

that the linear predictor depends linearly on these parameters. Statistical inference

with such models is usually based on the assumption that the response variable has

a normal distribution. Linear models are used widely in most branches of science,

both in the analysis of designed experiments, and for other modeling tasks, such as

polynomial regression. The linearity of the models endows them with some rather

elegant theory, which is explored in some depth in Chapter 1, alongside practical

examples of their use.

Generalized linear models (GLMs) somewhat relax the strict linearity assumption of

linear models, by allowing the expected value of the response to depend on a smooth

monotonic function of the linear predictor. Similarly the assumption that the response

is normally distributed is relaxed by allowing it to follow any distribution from the

exponential family (for example, normal, Poisson, binomial, gamma etc.). Inference

for GLMs is based on likelihood theory, as is explained, quite fully, in chapter 2,

where the practical use of these models is also covered.

A Generalized Additive Model (GAM) is a GLM in which part of the linear pre-

dictor is specified in terms of a sum of smooth functions of predictor variables. The

exact parametric form of these functions is unknown, as is the degree of smoothness

appropriate for each of them. To use GAMs in practice requires some extensions to

GLM methods:

1. The smooth functions must be represented somehow.

2. The degree of smoothness of the functions must be made controllable, so that

models with varying degrees of smoothness can be explored.

xi

xii PREFACE

3. Some means for estimating the most appropriate degree of smoothness from data

is required, if the models are to be useful for more than purely exploratory work.

This book provides an introduction to the framework for Generalized Additive Mod-

elling in which (i) is addressed using basis expansions of the smooth functions, (ii) is

addressed by estimating models by penalized likelihood maximization, in which wig-

gly models are penalized more heavily than smooth models in a controllable manner,

and (iii) is performed using methods based on cross validation or sometimes AIC or

Mallows statistic. Chapter 3 introduces this framework, chapter 4 provides details

of the theory and methods for using it, and chapter 5 illustrated the practical use of

GAMs using the R package mgcv.

The final chapter of the book looks at mixed model extensions of linear, general-

ized linear, and generalized additive models. In mixed models, some of the unknown

coefficients (or functions) in the model linear predictor are now treated as random

variables (or functions). These random effects are treated as having a covariance

structure that itself depends on some unknown fixed parameters. This approach en-

ables the use of more complex models for the random component of data, thereby

improving our ability to model correlated data. Again theory and practical applica-

tion are presented side by side.

I assume that most people are interested in statistical models in order to use them,

rather than to gaze upon the mathematical beauty of their structure, and for this rea-

son I have tried to keep this book practically focused. However, I think that practical

work tends to progress much more smoothly if it is based on solid understanding of

how the models and methods used actually work. For this reason, the book includes

fairly full explanations of the theory underlying the methods, including the underly-

ing geometry, where this is helpful. Given that the applied modelling involves using

computer programs, the book includes a good deal of material on statistical mod-

elling in R. This approach is now fairly standard when writing about practical sta-tistical analysis, but in addition Chapter 3 attempts to introduce GAMs by having

the reader build their own GAM using R: I hope that this provides a useful way ofquickly gaining a rather solid familiarity with the fundamentals of the GAM frame-

work presented in this book. Once the basic framework is mastered from chapter 3,

the theory in chapter 4 is really filling in details, albeit practically important ones.

The book includes a moderately high proportion of practical examples which re-

flect the reality that statistical modelling problems are usually quite involved, and

rarely require only straightforward brain-free application of some standard model.

This means that some of the examples are fairly lengthy, but do provide illustration

of the process of producing practical models of some scientific utility, and of check-

ing those models. They also provide much greater scope for the reader to decide that

what Ive done is utter rubbish.

Working through this book from Linear Models, through GLMs to GAMs and even-

tually GAMMs, it is striking that as model flexibility increases, so that the models

become better able to describe the reality that we believe generated a set of data, so

the methods for inference become less well founded. The linear model class is quite

PREFACE xiii

restricted, but within it, hypothesis testing and interval estimation are exact, while

estimation is unbiased. For the larger class of GLMs this exactness is generally lost

in favour of the large sample approximations of general likelihood theory, while esti-

mators themselves are consistent, but not necessarily unbiased. Generalizing further

to GAMs, penalization lowers the convergence rates of estimators, hypothesis testing

is only approximate, and satisfactory interval estimation seems to require the adop-

tion of a Bayesian approach. With time, improved theory will hopefully reduce these

differences. In the meantime, this book is offered in the belief that it is usually better

to be able to say something approximate about the right model, rather than something

very precise about the wrong model.

Life is too short to spend too much of it reading statistics text books. This book is of

course an exception to this rule and should be read from cover to cover. However, if

you dont feel inclined to follow this suggestion, here are some alternatives.

For those who are already very familiar with linear models and GLMs, but wantto use GAMs with a reasonable degree of understanding: work through Chapter 3

and read chapter 5, trying some exercises from both, use chapter 4 for reference.

Perhaps skim the other chapters.

For those who want to focus only on practical modelling in R, rather than theory.Work through the following: 1.5, 1.6.4, 1.7, 2.3, Chapter 5, 6.3, 6.5 and 6.7.

For those familiar with the basic idea of setting up a GAM using basis expansionsand penalties, but wanting to know more about the underlying theory and practical

application: work through Chapters 4 and 5, and probably 6.

For those not interested in GAMs, but wanting to know about linear models,GLMs and mixed models. Work through Chapters 1 and 2, and Chapter 6 up

to section 6.6.

The book is written to be accessible to numerate researchers and students from the

last two years of an undergraduate programme upwards. It is designed to be used ei-

ther for self study, or as the text for the regression modelling strand of mathematics

and/or statistics degree programme. Some familiarity with statistical concepts is as-

sumed, particularly with notions such as random variable, expectation, variance and

distribution. Some knowledge of matrix algebra is also needed, although Appendix

A is intended to provide all that is needed beyond the fundamentals.

Finally, Id like to thank the people who have in various ways helped me out in the

writing of this book, or in the work that lead to writing it. Among these, are Lucy

Augustin, Nicole Augustin, Miguel Bernal, Steve Blythe, David Borchers, Mark

Bravington, Steve Buckland, Richard Cormack, Jose Pedro Granadeiro ,Chong Gu,

Bill Gurney, John Halley, Joe Horwood, Sharon Hedley, Peter Jupp, Alain Le Tetre,

Stephan Lang, Mike Lonergan, Henric Nilsson, Roger D. Peng, Charles Paxton,

Bjorn Stollenwerk, Yorgos Stratoudaki, the R core team in particular Kurt Hornikand Brian Ripley, the Little Italians and the RiederAlpinists. I am also very grateful

to the people who have sent me bug reports and suggestions which have greatly im-

proved the the mgcv package over the last few years: the list is rather too long to

reproduce here, but thankyou.

CHAPTER 1

Linear Models

How old is the universe? The standard big-bang model of the origin of the universe

says that it expands uniformly, and locally, according to Hubbles law:

y = x

where y is the relative velocity of any two galaxies separated by distance x, and isHubbles constant(in standard astrophysical notation y v, x d and H0).1 gives the approximate age of the universe, but is unknown and must somehowbe estimated from observations of y and x, made for a variety of galaxies at differentdistances from us.

Figure 1.1 plots velocity against distance for 24 galaxies, according to measurements

made using the Hubble Space Telescope. Velocities are assessed by measuring the

Doppler effect red shift in the spectrum of light observed from the Galaxies con-

cerned, although some correction for local velocity components is required. Dis-

5 10 15 20

50

01

00

01

50

0

Distance (Mpc)

Velo

city (

km

s1)

Figure 1.1 A Hubble diagram showing the relationship between distance, x, and velocity, y,for 24 Galaxies containing Cepheid stars. The data are from the Hubble Space Telescope key

project to measure the Hubble constant as reported in Freedman et al. (2001).

1

2 LINEAR MODELS

tance measurement is much less direct, and is based on the 1912 discovery, by Hen-

rietta Leavit, of a relationship between the period of a certain class of variable stars,

known as the Cepheids, and their luminosity. The intensity of Cepheids varies regu-

larly with a period of between 1.5 and something over 50 days, and the mean intensity

increases predictably with period. This means that, if you can find a Cepheid, you can

tell how far away it is, by comparing its apparent brightness to its period predicted

intensity.

It is clear, from the figure, that the observed data do not follow Hubbles law exactly,

but given the measurement process, it would be surprising if they did. Given the

apparent variability, what can be inferred from these data? In particular: (i) what

value of is most consistent with the data? (ii) what range of values is consistentwith the data and (iii) are some particular theoretically derived values of consistentwith the data? Statistics is about trying to answer these three sorts of question.

One way to proceed is to formulate a linear statistical model of the way that the data

were generated, and to use this as the basis for inference. Specifically, suppose that,

rather than being governed directly by Hubbles law, the observed velocity is given

by Hubbles constant multiplied by the observed distance plus a random variability

term. That is

yi = xi + i i = 1 . . . 24 (1.1)

where the i terms are independent random variables such that E(i) = 0 andE(2i ) =

2. The random component of the model is intended to capture the fact

that if we gathered a replicate set of data, for a new set of galaxies, Hubbles law

would not change, but the apparent random variation from it would be different, as a

result of different measurement errors. Notice that it is not implied that these errors

are completely unpredictable: their mean and variance are assumed to be fixed, it is

only their particular values, for any particular galaxy, that are not known.

1.1 A simple linear model

This section develops statistical methods for a simple linear model of the form (1.1).

This allows the key concepts of linear modelling to be introduced without the dis-

traction of any mathematical difficulty.

Formally, consider n observations, xi, yi, where yi is an observation on random vari-able, Yi, with expectation, i E(Yi). Suppose that an appropriate model for therelationship between x and y is:

Yi = i + i where i = xi. (1.2)

Here is an unknown parameter and the i are mutually independent zero meanrandom variables, each with the same variance 2. So the model says that Y is givenby x multiplied by a constant plus a random term. Y is an example of a responsevariable, while x is an example of a predictor variable. Figure 1.2 illustrates thismodel for a case where n = 8.

A SIMPLE LINEAR MODEL 3

x

i

Yi

xi

Figure 1.2 Schematic illustration of a simple linear model with one explanatory variable.

Simple least squares estimation

How can , in model (1.2) be estimated from the xi, yi data? A sensible approachis to choose a value of that makes the model fit closely to the data. To do this weneed to define a measure of how well, or how badly, a model with a particular fitsthe data. One possible measure is the residual sum of squares of the model:

S =n

i=1

(yi i)2 =n

i=1

(yi xi)2

If we have chosen a good value of , close to the true value, then the model pre-dicted i should be relatively close to the yi, so that S should be small, whereas poorchoices will lead to i far from their corresponding yi, and high values of S. Hence can be estimated by minimizing S w.r.t. and this is known as the method of leastsquares.

To minimize S, differentiate w.r.t. :S

= n

i=1

2xi(yi xi)

and set the result to zero to find , the least squares estimate of :

n

i=1

2xi(yi xi) = 0n

i=1

xiyi n

i=1

x2i = 0 =n

i=1

xiyi/

n

i=1

x2i .

1.1.1 Sampling properties of

To evaluate the reliability of the least squares estimate, , it is useful to consider thesampling properties of . That is, we should consider some properties of the distri-bution of values, which would be obtained from repeated independent replication

2S/2 = 2P

x2i which is clearly positive, so a minimum of S has been found.

4 LINEAR MODELS

of the xi, yi data used for estimation. To do this, it is helpful to introduce the conceptof an estimator, which is obtained by replacing the observations, yi, in the estimateof by the random variables, Yi, to obtain

=

n

i=1

xiYi/

n

i=1

x2i .

Clearly the estimator, , is a random variable and we can therefore discuss its distri-bution. For now, consider only the first two moments of that distribution.

The expected value of is obtained as follows:

E() = E

(

n

i=1

xiYi/

n

i=1

x2i

)

=

n

i=1

xiE(Yi)/

n

i=1

x2i =

n

i=1

x2i/

n

i=1

x2i = .

So is an unbiased estimator its expected value is equal to the true value of theparameter that it is supposed to estimate.

Unbiasedness is a reassuring property, but knowing that an estimator gets it right on

average, does not tell us much about how good any one particular estimate is likely to

be: for this we also need to know how much estimates would vary from one replicate

data set to the next we need to know the estimator variance.

From general probability theory we know that if Y1, Y2, . . . , Yn are independent ran-dom variables and a1, a2, . . . an are real constants then

var

(

i

aiYi

)

=

i

a2i var(Yi).

But we can write

=

i

aiYi where ai = xi/

i

x2i ,

and from the original model specification we have that var(Yi) = 2 for all i. Hence,

var() =

i

x2i /

(

i

x2i

)2

2 =

(

i

x2i

)1

2. (1.3)

In most circumstances 2 itself is an unknown parameter and must also be estimated.Since 2 is the variance of the i, it makes sense to estimate it using the variance ofthe estimated i, the model residuals, i = yi xi. An unbiased estimator of 2 is:

2 =1

n 1

i

(yi xi)2

(proof of unbiasedness is given later for the general case). Plugging this into (1.3)

obviously gives an unbiased estimate of the variance of .

A SIMPLE LINEAR MODEL 5

1.1.2 So how old is the universe?

The least squares calculations derived above are available as part of the statisticalpackage and environment R. The function lm fits linear models to data, includingthe simple example currently under consideration. The Cepheid distance velocitydata shown in figure 1.1 are stored in a data frame hubble. The following R codefits the model and produces the (edited) output shown.

> data(hubble)

> hub.mod summary(hub.mod)

Call:

lm(formula = y x - 1, data = hubble)

Coefficients:

Estimate Std. Error

x 76.581 3.965

The call to lm passed two arguments to the function. The first is a model formula,

yx-1, specifying the model to be fitted: the name of the response variable is to

the left of while the predictor variable is specified on the right; the -1 term

indicates that the model has no intercept term, i.e. that the model is a straight line

through the origin. The second (optional) argument gives the name of the data frame

in which the variables are to be found. lm takes this information and uses it to fit the

model by least squares: the results are returned in a fitted model object, which in

this case has been assigned to an object called hub.mod for later examination.

6 LINEAR MODELS

500 1000 1500

6

00

2

00

02

00

40

06

00

fitted values

resid

ua

ls

a

500 1000 1500

3

00

1

00

01

00

20

0

fitted values

resid

ua

ls

b

Figure 1.3 Residuals against fitted values for (a) the model (1.1) fitted to all the data in figure

1.1 and (b) the same model fitted to data with two substantial outliers omitted.

as the fitted values increase. A trend in the mean violates the independence assump-tion, and is usually indicative of something missing in the model structure, while atrend in the variability violates the constant variance assumption. The main problem-atic feature of figure 1.3(a) is the presence of two points with very large magnituderesiduals, suggesting a problem with the constant variance assumption. It is proba-bly prudent to repeat the model fit, with and without these points, to check that theyare not having undue influence on our conclusions. The following code omits theoffending points and produces a new residual plot shown in figure 1.3(b).

> hub.mod1 summary(hub.mod1)

Call:

lm(formula = y x - 1, data = hubble[-c(3, 15), ])

Coefficients:

Estimate Std. Error

x 77.67 2.97

> plot(fitted(hub.mod1),residuals(hub.mod1),

+ xlab="fitted values",ylab="residuals")

The omission of the two large outliers has improved the residuals and changed somewhat, but not drastically.

The Hubble constant estimates have units of (km)s1 (Mpc)1. A Mega-parsec is3.09 1019km, so we need to divide by this amount, in order to obtain Hubblesconstant with units of s1. The approximate age of the universe, in seconds, is thengiven by the reciprocal of . Here are the two possible estimates expressed in years:

A SIMPLE LINEAR MODEL 7

> hubble.const age age/(602*24*365)

12794692825 12614854757

Both fits give an age of around 13 billion years. So we now have an idea of the best

estimate of the age of the universe, but what range of ages would be consistent with

the data?

1.1.3 Adding a distributional assumption

So far everything done with the simple model has been based only on the model

equations and the two assumptions of independence and equal variance, for the re-

sponse variable. If we wish to go further, and find confidence intervals for , or testhypotheses related to the model, then a further distributional assumption will be nec-

essary.

Specifically, assume that i N(0, 2) for all i, which is equivalent to assumingYi N(xi, 2). Now we have already seen that is just a weighted sum of Yi,but the Yi are now assumed to be normal random variables, and a weighted sum ofnormal random variables is itself a normal random variable. Hence the estimator, ,must be a normal random variable. Since we have already established the mean and

variance of , we have that

N(

,(

xi

)12)

. (1.4)

Testing hypotheses about

One thing we might want to do is to try and evaluate the consistency of some hy-

pothesized value of with the data. For example some Creation Scientists estimatethe age of the universe to be 6000 years, based on a reading of the Bible. This would

imply that = 163 106. The consistency with data of such a hypothesized valuefor , can be based on the probability that we would have observed the actuallyobtained, if the true value of was the hypothetical one.

Specifically, we can test the null hypothesis, H0 : = 0, versus the alternativehypothesis, H1 : 6= 0, for some specified value 0, by examining the probabilityof getting the observed , or one further from 0, assuming H0 to be true. If

2 were

known then we could work directly from (1.4), as follows.

The probability required is known as the p-value of the test. It is the probability of

getting a value of at least as favourable to H1 as the one actually observed, if H0 is

This isnt really valid, of course, since the Creation Scientists are not postulating a big bang theory.

8 LINEAR MODELS

actually true. In this case it helps to distinguish notationally between the estimate,obs , and estimator . The p-value is then

p = Pr[

| 0| |obs 0|

H0

]

=

[

| 0|

|obs 0|

H0

]

= Pr[|Z| > |z|]

where Z N(0, 1), z = (obs 0)/ and = (

x2i )12. Hence, having

formed z, the p-value is easily worked out, using the cumulative distribution functionfor the standard normal, built into any statistics package. Small p-values suggest that

the data are inconsistent with H0, while large values indicate consistency. 0.05 isoften used as the boundary between small and large in this context.

In reality 2 is usually unknown. Broadly the same testing procedure can still beadopted, by replacing with , but we need to somehow allow for the extra uncer-tainty that this introduces (unless the sample size is very large). It turns out that if

H0 : = 0 is true then

T 0

tn1

where n is the sample size, = (

x2i )1, and tn1 is the t-distribution with

n 1 degrees of freedom. This result is proven in section 1.3. It is clear that largemagnitude values of T favour H1, so using T as the test statistic, in place of wecan calculate a p-value by evaluating

p = Pr[|T | > |t|]

where T tn1 and t = (obs 0)/ . This is easily evaluated using the c.d.f.of the t distributions, built into any decent statistics package. Here is some code toevaluate the p-value for H0 : the Hubble constant is 163000000.

> cs.hubble t.stat pt(t.stat,df=21)*2 # 2 because of |T| in p-value defn.

3.906388e-150

i.e. as judged by the test statistic, t, the data would be hugely improbable if =1.63 108. It would seem that the hypothesized value can be rejected rather firmly(in this case, using the data with the outliers increases the p-value by a factor of 1000

or so).

Hypothesis testing is particularly useful when there are good reasons to want to stick

with some null hypothesis, until there is good reason to reject it. This is often the

This definition holds for any hypothesis test, if the specific is replaced by the general a test statis-tic.

A SIMPLE LINEAR MODEL 9

case when comparing models of differing complexity: it is often a good idea to retain

the simpler model until there is quite firm evidence that it is inadequate. Note one

interesting property of hypothesis testing. If we choose to reject a null hypothesis

whenever the p-value is less than some fixed level, (often termed the significancelevel of a test), then we will inevitably reject a proportion, , of correct null hypothe-ses. We could try and reduce the probability of such mistakes by making verysmall, but in that case we pay the price of reducing the probability of rejecting H0when it is false!

Confidence intervals

Having seen how to test whether a particular hypothesized value of is consistentwith the data, the question naturally arises of what range of values of would beconsistent with the data. To do this, we need to select a definition of consistent: a

common choice is to say that any parameter value is consistent with the data if it

results in a p-value of 0.05, when used as the null value in a hypothesis test.

Sticking with the Hubble constant example, and working at a significance level of

0.05, we would have rejected any hypothesized value for the constant, that resulted

in a t value outside the range (2.08, 2.08), since these values would result in p-values of less than 0.05. The R function qt can be used to find such ranges: e.g.qt(c(0.025,0.975),df=21) returns the range of the middle 95% of t21 ran-dom variables. So we would accept any 0 fulfilling:

2.08 0

2.08

which re-arranges to give the interval

2.08 0 + 2.08 .Such an interval is known as a 95% Confidence interval for .

The defining property of a 95% confidence interval is this: if we were to gather an

infinite sequence of independent replicate data sets, and calculate 95% confidence

intervals for from each, then 95% of these intervals would include the true , and5% would not. It is easy to see how this comes about. By construction, a hypothesis

test with a significance level of 5%, rejects the correct null hypothesis for 5% of

replicate data sets, and accepts it for the other 95% of replicates. Hence 5% of 95%

confidence intervals must exclude the true parameter, while 95% include it.

For the Hubble example, a 95% CI for the constant (in the usual astro-physicistsunits) is given by:

> sigb h.ci h.ci

[1] 71.49588 83.84995

10 LINEAR MODELS

This can be converted to a confidence interval for the age of the universe, in years, asfollows:

> h.ci sort(1/h.ci)

[1] 11677548698 13695361072

i.e. the 95% CI is (11.7,13.7) billion years. Actually this Hubble age is the age

of the universe if it has been expanding freely, essentially unfettered by gravitation.

If the universe is really matter dominated then the galaxies should be slowed by

gravity over time so that the universe would actually be younger than this, but it is

time to get on with the subject of this book.

1.2 Linear models in general

The simple linear model, introduced above, can be generalized by allowing the re-

sponse variable to depend on multiple predictor variables (plus an additive constant).

These extra predictor variables can themselves be transformations of the original pre-

dictors. Here are some examples, for each of which a response variable datum, yi,is treated as an observation on a random variable, Yi, where E(Yi) i, the i arezero mean random variables, and the j are model parameters, the values of whichare unknown and will need to be estimated using data.

1. i = 0 + xi1, Yi = i + i, is a straight line relationship between y andpredictor variable, x.

2. i = 0 +xi1 +x2i2 +x

3i3, Yi = i+i, is a cubic model of the relationship

between y and x.

3. i = 0 + xi1 + zi2 + log(xizi)3, Yi = i + i, is a model in which ydepends on predictor variables x and z and on the log of their product.

Each of these is a linear model because the i terms and the model parameters, j ,enter the model in a linear way. Notice that the predictor variables can enter the model

non-linearly. Exactly as for the simple model, the parameters of these models can be

estimated by finding the j values which make the models best fit the observed data,in the sense of minimizing

i(yii)2. The theory for doing this will be developedin section 1.3, and that development is based entirely on re-writing the linear model

using using matrices and vectors.

To see how this re-writing is done, consider the straight line model given above.

Writing out the i equations for all n pairs, (xi, yi), results in a large system of

LINEAR MODELS IN GENERAL 11

linear equations:

1 = 0 + x11

2 = 0 + x21

3 = 0 + x31

. .

. .

n = 0 + xn1

which can be re-written in matrix-vector form as

123..

n

=

1 x11 x21 x3. .. .1 xn

[

01

]

.

So the model has the general form = X, i.e. the expected value vector is givenby a model matrix (also known as a design matrix), X, multiplied by a parameter

vector, . All linear models can be written in this general form.

As a second illustration, the cubic example, given above, can be written in matrix

vector form as

123..

n

=

1 x1 x21 x

31

1 x2 x22 x

32

1 x3 x23 x

33

. . . .

. . . .1 xn x

2n x

3n

0123

.

Models in which data are divided into different groups, each of which are assumed

to have a different mean, are less obviously of the form = X, but in fact theycan be written in this way, by use of dummy indicator variables. Again, this is most

easily seen by example. Consider the model

i = j if observation i is in group j,

and suppose that there are three groups, each with 2 data. Then the model can be

re-written

123456

=

1 0 01 0 00 1 00 1 00 0 10 0 1

012

.

Variables indicating the group to which a response observation belongs, are known as

factor variables. Somewhat confusingly, the groups themselves are known as levels

12 LINEAR MODELS

of a factor. So the above model involves one factor, group, with three levels. Mod-

els of this type, involving factors, are commonly used for the analysis of designed

experiments. In this case the model matrix depends on the design of the experiment

(i.e on which units belong to which groups), and for this reason the terms design

matrix and model matrix are often used interchangeably. Whatever it is called ,X

is absolutely central to understanding the theory of linear models, generalized linear

models and generalized additive models.

1.3 The theory of linear models

This section shows how the parameters, , of the linear model

= X, y N(, In2)can be estimated by least squares. It is assumed that X is a matrix, with n rowsand p columns. It will be shown that the resulting estimator, , is unbiased, has thelowest variance of any possible linear estimator of , and that, given the normality

of the data, N(, (XTX)12). Results are also derived for setting confidencelimits on parameters and for testing hypotheses about parameters in particular the

hypothesis that several elements of are simultaneously zero.

In this section it is important not to confuse the length of a vector with its dimension.

For example (1, 1, 1)T has dimension 3 and length

3. Also note that no distinctionhas been made notationally between random variables and particular observations of

those random variables: it is usually clear from the context which is meant.

1.3.1 Least squares estimation of

Point estimates of the linear model parameters, , can be obtained by the method of

least squares, that is by minimizing

S =n

i=1

(yi i)2,

with respect to . To use least squares with a linear model, written in general matrix-

vector form, first recall the link between the Euclidean length of a vector and the sum

of squares of its elements. If v is any vector of dimension, n, then

v2 vTv n

i=1

v2i .

Hence

S = y 2 = y X2

Since this expression is simply the squared (Euclidian) length of the vector yX,its value will be unchanged if y X is rotated. This observation is the basis for a

THE THEORY OF LINEAR MODELS 13

practical method for finding , and for developing the distributional results required

to use linear models.

Specifically, like any real matrix, X can always be decomposed

X = Q

[

R

0

]

= QfR (1.5)

where R is a p p upper triangular matrix, and Q is an n n orthogonal matrix,the first p columns of which form Qf (see A.6). Recall that orthogonal matricesrotate vectors, but do not change their length. Orthogonality also means that QQT =QTQ = In. Applying Q

T to y X implies that

y X2 = QTy QTX2 =

QTy [

R

0

]

2

.

Writing QTy =

[

f

r

]

, where f is vector of dimension p, and hence r is a vector of

dimension n p, yields

y X2 =

[

f

r

]

[

R

0

]

2

= f R2 + r2.

The length of r does not depend on , while f R2 can be reduced to zero bychoosing so that R equals f . Hence

= R1f (1.6)

is the least squares estimator of . Notice that r2 = yX2, the residual sumof squares for the model fit.

1.3.2 The distribution of

The distribution of the estimator, , follows from that of QTy. Multivariate normal-

ity of QTy follows from that of y, and since the covariance matrix of y is In2, the

covariance matrix of QTy is

VQTy = QTInQ

2 = In2.

Furthermore

E

[

f

r

]

= E(QTy) = QTX =

[

R

0

]

E(f) = R and E(r) = 0.i.e. we have that

f N(R, Ip2) and r N(0, Inp2)

i.e. Ri,j = 0 if i > j. If the last equality isnt obvious recall that x2 = Pi x2i , so if x =

v

w

, x2 = Pi v2i +P

i w2i = v2 + w2.

14 LINEAR MODELS

with both vectors independent of each other.

Turning to the properties of itself, unbiasedness follows immediately:

E() = R1E(f) = R1R = .

Since the covariance matrix of f is Ip2, it also follows that the covariance matrix of

is

V = R1IpR

T2 = R1RT2. (1.7)

Furthermore, since is just a linear transformation of the normal random variables

f , it must follow a multivariate normal distribution,

N(,V).

The forgoing distributional result is not usually directly useful for making inferences

about , since 2 is generally unknown and must be estimated, thereby introducingan extra component of variability that should be accounted for.

1.3.3 (i i)/i tnp

Since the elements of r are i.i.d. N(0, 2) random variables,

1

2r2 = 1

2

np

i=1

r2i 2np.

The mean of a 2np r.v. is n p, so this result is sufficient (but not necessary: seeexercise 7) to imply that

2 = r2/(n p) (1.8)is an unbiased estimator of 2. The independence of the elements of r and f alsoimplies that and 2 are independent.

Now consider a single parameter estimator, i, with standard deviation, i , given

by the square root of element i, i of V . An unbiased estimator of V is V =

V 2/2 = R1RT2, so an estimator, i , is given by the square root of element

i, i of V , and it is clear that i = i /. Hence

i ii

=i ii /

=(i i)/i

12 r2/(n p)

N(0, 1)2np/(n p)

tnp

(where the independence of i and 2 has been used). This result enables confidence

intervals for i to be found, and is the basis for hypothesis tests about individual is(for example H0 : i = 0).

Dont forget that r2 = y X2.

THE THEORY OF LINEAR MODELS 15

1.3.4 F-ratio results

It is also of interest to obtain distributional results for testing, for example, the si-

multaneous equality to zero of several model parameters. Such tests are particularly

useful for making inferences about factor variables and their interactions, since each

factor (or interaction) is typically represented by several elements of .

First consider partitioning the model matrix into two parts so that X = [X0 : X1],where X0 and X1 have p q and q columns, respectively. Let 0 and 1 be thecorresponding sub-vectors of .

QTX0 =

[

R00

]

where R0 is the first p q rows and columns of R (Q and R are from (1.5)). Sorotating y X00 using QT implies that

y X002 = f0 R002 + f12 + r2

where f has been partitioned so that f =

[

f0f1

]

(f1 being of dimension q). Hence

f12 is the increase in residual sum of squares that results from dropping X1 fromthe model (i.e. setting 1 = 0).

Now, under

H0 : 1 = 0,

E(f1) = 0 (using the facts that E(f) = R and R is upper triangular), and wealready know that the elements of f1 are i.i.d. normal r.v.s with variance

2. Hence,

if H0 is true,1

2f12 2q .

So, forming an F-ratio statistic, F , assuming H0, and recalling the independence off and r we have

F =f12/q

2=

12 f12/q

12 r2/(n p)

2q/q

2np/(n p) Fq,np

and this result can be used to test the significance of model terms.

In general if is partitioned into sub-vectors 0,1 . . . ,m (each usually relatingto a different effect), of dimensions q0, q1, . . . , qm, then f can also be so partitioned,fT = [fT0 , f

T

1 , . . . , fT

m], and tests of

H0 : j = 0 vs. H1 : j 6= 0are conducted using the result that under H0

F =fj2/qj

2 Fqj ,np

with F larger than this suggests, if the alternative is true. This is the result used todraw up sequential ANOVA tables for a fitted model, of the sort produced by a single

16 LINEAR MODELS

argument call to anova in R. Note, however, that the hypothesis test about j isonly valid in general if k = 0 for all k such that j < k m: this follows fromthe way that the test was derived, and is the reason that the ANOVA tables resulting

from such procedures are referred to as sequential tables. The practical upshot is

that, if models are reduced in a different order, the p-values obtained will be different.

The exception to this is if the js are mutually independent, in which case the all

tests are simultaneously valid, and the ANOVA table for a model is not dependent on

the order of model terms: such independent js usually arise only in the context ofbalanced data, from designed experiments.

Notice that sequential ANOVA tables are very easy to calculate: once a model has

been fitted by the QR method, all the relevant sums of squares are easily calculated

directly from the elements of f , with the elements of r providing the residual sum of

squares.

1.3.5 The influence matrix

One matrix which will feature extensively in the discussion of GAMs is the influence

matrix (or hat matrix) of a linear model. This is the matrix which yields the fitted

value vector, , when post-multiplied by the data vector, y. Recalling the definition

of Qf , as being the first p columns of Q, f = Qfy and so

= R1QTf y.

Furthermore = X and X = QfR so

= QfRR1QTf y = QfQ

T

f y

i.e. the matrix A QfQTf is the influence (hat) matrix such that = Ay.The influence matrix has a couple of interesting properties. Firstly, the trace of the

influence matrix is the number of (identifiable) parameters in the model, since

tr (A) = tr(

QfQT

f

)

= tr(

QTf Qf)

= tr (Ip) = p.

Secondly, AA = A, a property known as idempotency . Again the proof is simple:

AA = QfQT

f QfQT

f = QfIpQT

f = QfQT

f = A.

1.3.6 The residuals, , and fitted values,

The influence matrix is helpful in deriving properties of the fitted values, , and

residuals, . is unbiased, since E() = E(X) = XE() = X = . Thecovariance matrix of the fitted values is obtained from the fact that is a linear

transformation of the random vector y, which has covariance matrix In2, so that

V = AInAT2 = A2,

by the idempotence (and symmetry) of A. The distribution of is degenerate multi-

variate normal.

THE THEORY OF LINEAR MODELS 17

Similar arguments apply to the residuals.

= (IA)y,so

E() = E(y) E() = = 0.As in the fitted value case, we have

V = (In A)In(In A)T2 = (In 2A + AA)2 = (In A) 2

Again, the distribution of the residuals will be degenerate normal. The results for the

residuals are useful for model checking, since they allow the residuals to be stan-

dardized, so that they should have constant variance, if the model is correct.

1.3.7 Results in terms of X

The presentation so far has been in terms of the method actually used to fit linear

models in practice (the QR decomposition approach), which also greatly facilitatesthe derivation of the distributional results required for practical modelling. However,

for historical reasons, these results are more usually presented in terms of the model

matrix, X, rather than the components of its QR decomposition. For completeness

some of the results are restated here, in terms of X.

Firstly consider the covariance matrix of . This turns out to be (XTX)12, whichis easily seen to be equivalent to (1.7) as follows:

V

= (XTX)12 =(

RTQTf QfR)1

2 =(

RTR)1

2 = R1RT2.

The expression for the least squares estimates is = (XTX)1XTy, which is equiv-alent to (1.6):

= (XTX)1XTy = R1RTRTQTf y = R1QTf y = R

1f .

Given this last result it is easy to see that the influence matrix can be written:

A = X(XTX)1XT.

These results are of largely historical and theoretical interest: they should not be used

for computational purposes, and derivation of the distributional results is much more

difficult if one starts from these formulae.

1.3.8 The Gauss Markov Theorem: whats special about least squares?

How good are least squares estimators? In particular, might it be possible to find

better estimators, in the sense of having lower variance while still being unbiased?

A few programs still fit models by solution of XTX = XTy, but this is less computationally stablethan the rotation method described here, although it is a bit faster.

18 LINEAR MODELS

The Gauss Markov theorem shows that least squares estimators have the lowest vari-

ance of all unbiased estimators that are linear (meaning that the data only enter the

estimation process in a linear way).

Theorem: Suppose that E(Y) = X and Vy = 2I, and let = cTY be anyunbiased linear estimator of = tT, where t is an arbitrary vector. Then:

Var() Var()where = tT, and = (XTX)1XTY is the least squares estimator of . Noticethat, since t is arbitrary, this theorem implies that each element of is a minimum

variance unbiased estimator.

Proof: Since is a linear transformation of Y, Var() = cTc2. To compare vari-

ances of and it is also useful to express Var() in terms of c. To do this, notethat unbiasedness of implies that

E(cTY) = tT cTE(Y) = tT cTX = tT cTX = tT.So the variance of can be written as

Var() = Var(tT) = Var(cTX).

This is the variance of a linear transformation of , and the covariance matrix of

is (XTX)12, so

Var() = Var(cTX) = cTX(XTX)1XTc2 = cTAc2

(where A is the influence or hat matrix). Now the variances of the two estimators can

be directly compared, and it can be seen that

Var() Var()iff

cT(IA)c 0.This condition will always be met, because it is equivalent to:

[(IA)c]T(IA)c 0by the idempotency of (IA), but this last condition is saying that a sum of squarescan not be less than 0, which is clearly true. 2

Notice that this theorem uses independence and equal variance assumptions, but does

not assume normality. Of course there is a sense in which the theorem is intuitively

rather unsurprising, since it says that the minimum variance estimators are those

obtained by seeking to minimize the residual variance.

1.4 The geometry of linear modelling

A full understanding of what is happening when models are fitted by least squares

is facilitated by taking a geometric view of the fitting process. Some of the results

derived in the last few sections become rather obvious when viewed in this way.

THE GEOMETRY OF LINEAR MODELLING 19

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

x

y

1

2

3

1

2

3

Figure 1.4 The geometry of least squares. The left panel shows a straight line model fitted to

3 data by least squares. The right panel gives a geometric interpretation of the fitting process.

The 3 dimensional space shown is spanned by 3 orthogonal axes: one for each response vari-

able. The observed response vector, y, is shown as a point () within this space. The columnsof the model matrix define two directions within the space: the thick and dashed lines from

the origin. The model states that E(y) could be any linear combination of these vectors: i.e.anywhere in the model subspace indicated by the grey plane. Least squares fitting finds the

closest point in the model sub space to the response data (): the fitted values. The shortthick line joins the response data to the fitted values: it is the residual vector.

1.4.1 Least squares

Again consider the linear model,

= X, y N(, In2),where X is an n p model matrix. But now consider an n dimensional Euclideanspace,

20 LINEAR MODELS

1

2

3

1

2

3

Figure 1.5 The geometry of fitting via orthogonal decompositions. The left panel illustrates

the geometry of the simple straight line model of 3 data introduced in figure 1.4. The right

hand panel shows how this original problem appears after rotation by QT, the transpose of

the orthogonal factor in a QR decomposition of X. Notice that in the rotated problem the

model subspace only has non-zero components relative to p axes (2 axes for this example),while the residual vector has only zero components relative to those same axes.

the vectors [1, 1, 1]T and [.2, 1, .6]T. As the right hand panel of figure 1.4 illustrates,fitting the model by least squares amounts to finding the particular linear combination

of the columns of these vectors, that is as close to y as possible (in terms of Euclidean

distance).

1.4.2 Fitting by orthogonal decompositions

Recall that the actual calculation of least squares estimates involves first forming the

QR decomposition of the model matrix, so that

X = Q

[

R

0

]

,

where Q is an n n orthogonal matrix and R is a p p upper triangular matrix.Orthogonal matrices rotate vectors (without changing their length) and the first step

in least squares estimation is to rotate both the response vector, y, and the columns

of the model matrix, X, in exactly the same way, by pre-multiplication with QT.

Figure 1.5 illustrates this rotation for the example shown in figure 1.4. The left panel

shows the response data and model space, for the original problem, while the right

In fact the QR decomposition is not uniquely defined, in that the sign of rows of Q, and correspondingcolumns of R, can be switched, without changing X these sign changes are equivalent to reflectionsof vectors, and the sign leading to maximum numerical stability is usually selected in practice. Thesereflections dont introduce any extra conceptual difficulty, but can make plots less easy to understand,so I have surpressed them in this example.

THE GEOMETRY OF LINEAR MODELLING 21

1

2

3

1

2

3

Figure 1.6 The geometry of nested models.

hand panel shows the data and space after rotation by QT. Notice that, since the

problem has simply been rotated, the relative position of the data and basis vectors

(columns of X) has not changed. What has changed is that the problem now has

a particularly convenient orientation relative to the axes. The first two components

of the fitted value vector can now be read directly from axes 1 and 2, while the

third component is simply zero. By contrast, the residual vector has zero components

relative to axes 1 and 2, and its non-zero component can be read directly from axis

3. In terms of section 1.3.1, these vectors are [fT,0T]T and [0T, rT]T, respectively.

The corresponding to the fitted values is now easily obtained. Of course we usually

require fitted values and residuals to be expressed in terms of the un-rotated problem,

but this is simply a matter of reversing the rotation using Q. i.e.

= Q

[

f

0

]

, and = Q

[

0

r

]

.

1.4.3 Comparison of nested models

A linear model with model matrix X0 is nested within a linear model with model

matrix X1 if they are models for the same response data, and the columns of X0span a subspace of the space spanned by the columns of X1. Usually this simply

means that X1 is X0 with some extra columns added.

The vector of the difference between the fitted values of two nested linear models

is entirely within the subspace of the larger model, and is therefore orthogonal to

the residual vector for the larger model. This fact is geometrically obvious, as figure

1.6 illustrates, but it is a key reason why F ratio statistics have a relatively simple

distribution (under the simpler model).

Figure 1.6 is again based on the same simple straight line model that forms the ba-

sis for figures 1.4 and 1.5, but this time also illustrates the least squares fit of the

22 LINEAR MODELS

simplified model

yi = 0 + i,

which is nested within the original straight line model. Again, both the original and

rotated versions of the model and data are shown. This time the fine continuous line

shows the projection of the response data onto the space of the simpler model, while

the fine dashed line shows the vector of the difference in fitted values between the

two models. Notice how this vector is orthogonal both to the reduced model subspace

and the full model residual vector.

The right panel of figure 1.6 illustrates that the rotation, using the transpose of the

orthogonal factor Q, of the full model matrix, has also lined up the problem very

conveniently for estimation of the reduced model. The fitted value vector for the

reduced model now has only one non-zero component, which is the component of

the rotated response data () relative to axis 1. The residual vector has gained thecomponent that the fitted value vector has lost, so it has zero component relative to

axis one, while its other components are the positions of the rotated response data

relative to axes 2 and 3.

So, much of the work required for estimating the simplified model has already been

done, when estimating the full model. Note, however, that if our interest had been in

comparing the full model to the model

yi = 1xi + i,

then it would have been necessary to reorder the columns of the full model matrix,

in order to avoid extra work in this way.

1.5 Practical linear models

This section covers practical linear modelling, via an extended example: the analysis

of data reported by Baker and Bellis (1993), which they used to support a theory

of sperm competition in humans. The basic idea is that it is evolutionarily advan-

tageous for males to (sub-conciously) increase their sperm count in proportion to

the opportunities that their mate may have had for infidelity. Such behaviour has

been demonstrated in a wide variety of other animals, and using a sample of student

and staff volunteers from Manchester University, Baker and Bellis set out to see if

there is evidence for similar behaviour in humans. Two sets of data will be examined:

sperm.comp1 contains data on sperm count, time since last copulation and propor-

tion of that time spent together, for single copulations, from 15 heterosexual couples;

sperm.comp2 contains data on median sperm count, over multiple copulations,

for 24 heterosexual couples, along with the weight, height and age of the male and

female of each couple, and the volume of one teste of the male. From these data,

Baker and Bellis concluded that sperm count increases with the proportion of time,

since last copulation, that a couple have spent apart, and that sperm count increases

with female weight.

In general, practical linear modelling is concerned with finding an appropriate model

PRACTICAL LINEAR MODELS 23

lm Estimates a linear model by least squares. Returns a fitted model ob-

ject of class lm containing parameter estimates plus other auxiliary

results for use by other functions.

plot Produces model checking plots from a fitted model object.

summary Produces summary information about a fitted model, including pa-

rameter estimates, associated standard errors and p-values, r2 etc.anova Used for model comparison based on F ratio testing.

AIC Extract Akaikes information criterion for a model fit.

residuals Extract an array of model residuals form a fitted model.

fitted Extract an array of fitted values from a fitted model object.

predict Obtain predicted values from a fitted model, either for new values

of the predictor variables, or for the original values. Standard errors

of the predictions can also be returned.

Table 1.1 Some standard linear modelling functions. Strictly all of these functions except lm

itself end .lm, but when calling them with an object of class lm this may be omitted.

to explain the relationship of a response (random) variable to some predictor vari-

ables. Typically, the first step is to decide on a linear model that can reasonably be

supposed capable of describing the relationship, in terms of the predictors included

and the functional form of their relationship to the response. In the interests of ensur-

ing that the model is not too restrictive this full model is often more complicated

than is necessary, in that the most appropriate value for a number of its parameters

may, in fact, be zero. Part of the modelling process is usually concerned with model

selection: that is deciding which parameter values ought to be zero. At each stage

of model selection it is necessary to estimate model parameters by least squares fit-

ting, and it is equally important to check the model assumptions (particularly equal

variance and independence) by examining diagnostic plots. Once a model has been

selected and estimated, its parameter estimates can be interpreted, in part with the aid

of confidence intervals for the parameters, and possibly with other follow up analy-

ses. In R these practical modelling tasks are facilitated by a large number of functionsfor linear modelling, some of which are listed in table 1.1.

1.5.1 Model fitting and model checking

The first thing to do with the sperm competition data is to have a look at it.

pairs(sperm.comp1[,-1])

produces the plot shown in figure 1.7. The columns of the data frame are plotted

against each other pairwise (with each pairing transposed between lower left and

upper right of the plot); the first column has been excluded from the plot as it sim-

ply contains subject identification lables. The clearest pattern seems to be of some

decrease in sperm count as the proportion of time spent together increases.

24 LINEAR MODELS

time.ipc

0.0 0.2 0.4 0.6 0.8 1.0

40

80

120

160

0.0

0.2

0.4

0.6

0.8

1.0

prop.partner

40 60 80 100 120 140 160 100 200 300 400 500

100

300

500

count

Figure 1.7 Pairs plot of the sperm competition data from Baker and Bellis 1993. Count

is sperm count (millions) from one copulation, time.ipc is time (hours) since the previous

copulation and prop.partner is the proportion of the time since the previous copulation that

the couple have spent together.

Following Baker and Bellis, a reasonable initial model might be,

yi = 0 + ti1 + pi2 + i, (1.9)

where yi is sperm count (count), ti is the time since last inter pair copulation(time.ipc) and pi is the proportion of time, since last copulation, that the pairhave spent together (prop.partner). As usual, the j are unknown parametersand the i are i.i.d. N(0,

2) random variables. Really this model defines the classof models thought to be appropriate: it is not immediately clear whether either of 1and 2 are non-zero.

The following fits the model (1.9) and stores the results in an object called sc.mod1.

sc.mod1

PRACTICAL LINEAR MODELS 25

100 200 300 400 500

200

0100

200

Fitted values

Re

sid

ua

ls

Residuals vs Fitted

9

114

1 0 1

2.0

1.0

0.0

1.0

Theoretical Quantiles

Sta

nd

ard

ize

d r

esid

ua

ls

Normal QQ plot

9

7

1

100 200 300 400 500

0.0

0.4

0.8

1.2

Fitted values

Sta

nd

ard

ize

d r

esid

ua

ls

ScaleLocation plot9

71

2 4 6 8 10 12 14

0.0

00.1

00.2

00.3

0

Obs. number

Co

oks

dis

tan

ce

Cooks distance plot

27

9

Figure 1.8 Model checking plots for the linear model fitted to the sperm.comp1 data.

> model.matrix(sc.mod1)

(Intercept) time.ipc prop.partner

1 1 60 0.20

2 1 149 0.98

3 1 70 0.50

4 1 168 0.50

5 1 48 0.20

6 1 32 1.00

7 1 48 0.02

8 1 56 0.37

9 1 31 0.30

10 1 38 0.45

11 1 48 0.75

12 1 54 0.75

13 1 32 0.60

14 1 48 0.80

15 1 44 0.75

Having fitted the model, it is important to check the plausibility of the assumptions,graphically.

par(mfrow=c(2,2)) # split the graphics device into 4 panels

plot(sc.mod1) # (uses plot.lm as cs.mod1 is class lm)

The resulting plots, shown in figure 1.8, require some explanation. Note that, in two

of the plots, the residuals have been scaled, by dividing them by their estimated

standard deviation (see section 1.3.6). If the model assumptions are met, then this

standardization should result in residuals that look like N(0, 1) random deviates.

26 LINEAR MODELS

The upper left plot shows the model residuals, i, against the model fitted values,i, where = X and = y . The residuals should be evenly scatteredabove and below zero (the distribution of fitted values is not of interest). A trend

in the mean of the residuals would violate the assumption of independent response

variables, and usually results from an erroneous model structure: e.g. assuming a

linear relationship with a predictor, when a quadratic is required, or omitting an

important predictor variable. A trend in the variability of the residuals suggests

that the variance of the response is related to its mean, violating the constant

variance assumption: transformation of the response or use of a GLM may help,

in such cases. The plot shown does not indicate any problem.

The lower left plot is a scale-location plot. The square root of the absolute value ofeach standardized residual is plotted against the equivalent fitted value. It can be

easier to judge the constant variance assumption from such a plot, and the square

root transformation reduces the skew in the distribution, which would otherwise

be likely to occur. Again, the plot shown gives no reason to doubt the constant

variance assumption.

The upper right panel is a normal Q-Q (quantile-quantile) plot. The standardizedresiduals are sorted and then plotted against the quantiles of a standard normal dis-

tribution. If the residuals are normally distributed then the resulting plot should

look like a straight line relationship, perturbed by some random scatter. The cur-

rent plot fits this description, so the normality assumption seems plausible.

The lower left panel shows Cooks distance for each observation. Cooks distanceis a measure of how much influence each observation has on the fitted model. If

[k]i is the i

th fitted value, when the kth datum is omitted from the fit, then Cooksdistance is

dk =1

(p + 1)2

n

i=1

([k]i i)2, (1.10)

where p is the number of parameters and n the number of data. A very large valueof dk indicates a point that has a substantial influence on the model results. Ifthe Cooks distance values indicate that model estimates may be very sensitive

to just one or two data, then it usually prudent to repeat any analysis without the

offending points, in order to check the robustness of the modelling conclusions.

In this case none of the points look wildly out of line.

By default the most extreme three points in each plot are labelled with their row in-dex in the original data frame, so that the corresponding data can be readily checked.The 9th datum is flagged in all 4 plots in figure 1.8. It should be checked:

> sperm.comp1[9,]

subject time.ipc prop.partner count

9 P 31 0.3 76

This subject has quite a low count, but not the lowest in the frame. Examination of the

plot of count against prop.partner indicates that the point adds substantially

to the uncertainty surrounding the relationship, but its hard to see a good reason to

PRACTICAL LINEAR MODELS 27

remove it, particularly since, if anything, it is obscuring the relationship, rather than

exaggerating it.

Since the assumptions of model (1.9) appear reasonable, we can proceed to examinethe fitted model object. Typing the name of an object in R causes the default printmethod for the object to be invoked (print.lm in this case).

> sc.mod1

Call:

lm(formula=counttime.ipc+prop.partner,data=sperm.comp1)

Coefficients:

(Intercept) time.ipc prop.partner

357.418 1.942 -339.560

The intercept parameter (0) is estimated to be 357.4. Notionally, this would be thecount expected if time.ipc and prop.partner were zero, but the value is bi-

ologically implausible if interpreted in this way. Given that the smallest observed

time.ipc was 31 hours we cannot really expect to predict the count at zero. The

remaining two parameter estimates are 1 and 2, and are labelled by the name ofthe variable to which they relate. In both cases they give the expected increase in

count for a unit increase in their respective predictor variable. Note the important

point that the absolute values of the parameter estimates are only interpretable rela-

tive to the variable which they multiply. For example, we are not entitled to conclude

that the effect of prop.partner is much greater than that of time.ipc, on the

basis of the relative magnitudes of the respective parameters: they are measured in

completely different units.

One point to consider, is whether prop.partner is the most appropriate predictorvariable. Perhaps the total time spent together (in hours) would be a better predictor.

sc.mod2

28 LINEAR MODELS

1.5.2 Model summary

The summary function provides a good deal more information about the fittedmodel.

> summary(sc.mod1)

Call:

lm(formula=counttime.ipc+prop.partner,data=sperm.comp1)

Residuals:

Min 1Q Median 3Q Max

-239.740 -96.772 2.171 96.837 163.997

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) 357.4184 88.0822 4.058 0.00159 **time.ipc 1.9416 0.9067 2.141 0.05346 .

prop.partner -339.5602 126.2535 -2.690 0.01969 *---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 136.6 on 12 degrees of freedom

Multiple R-Squared: 0.4573, Adjusted R-squared: 0.3669

F-statistic: 5.056 on 2 and 12 DF, p-value: 0.02554

The explanation of the parts of this output are as follows:

Call simply reminds you of the call that generated the object being summarized.

Residuals gives a five figure summary of the residuals: this should indicate any

gross departure from normality, for example a very skewed set of residuals might

lead to very different magnitudes for Q1 and Q2, or to the median residual being a

long way from 0 (the mean residual is always zero if the model includes an intercept:

see exercise 6).

Coefficients gives a table relating to the estimated parameters of the model.

The first two columns are the least squares estimates (js) and the estimated stan-dard errors associated with those estimates (j ), respectively. The standard error

calculations follow sections 1.3.2 and 1.3.3. The third column gives the parameter

estimates divided by their estimated standard errors:

Tj jj

.

Note that calling the summary function with a fitted linear model object, x, actually results in thefollowing: summary looks at the class of the x, finds that it is lm and passes it to summary.lm;summary.lm calculates a number of interesting quantities from x which it returns in a list, y,of class lm.summary; unless y is assigned to an object, R prints it, using the print methodprint.lm.summary.

PRACTICAL LINEAR MODELS 29

Tj is a standardized measure of how far each parameter estimate is from zero. It wasshown, in section 1.3.3, that under H0 : j = 0,

Tj tnp, (1.11)where n is the number of data and p the number of model parameters estimated.i.e. if the null hypothesis is true, then the observed Tj should be consistent withhaving been drawn from a tnp distribution. The final Pr(>|t|) column providesthe measure of that consistency, namely the probability that the magnitude of a tnprandom variable would be at least as large as the observed Tj . This quantity is knownas the p-value of the test of H0 : j = 0. A large p-value indicates that the dataare consistent with the hypothesis, in that the observed Tj is quite a probable valuefor a tnp deviate, so that there is no reason to doubt the hypothesis underpinning(1.11). Conversely a small p-value suggests that the hypothesis is wrong, since the

observed Tj is a rather improbable observation from the tnp distribution implied byj = 0. Various arbitrary significance levels are often used as the boundary p-valuesfor deciding whether to accept or reject hypotheses. Some common ones are listed at

the foot of the table, and the p-values are flagged according to which, if any, they fall

below.

Residual standard error gives where 2 =

(2i )/(n p) (see section1.3.3). n p is the residual degrees of freedom.

Multiple R-squared is an estimate of the proportion of the variance in the data

explained by the regression:

r2 = 1

2i /n

(yi y)2/nwhere y is the mean of the yi. The fraction in this expression is basically an estimateof the proportion variance not explained by the regression.

Adjusted R-squared. The problem with r2 is that it always increases when anew predictor variable is added to the model, no-matter how useless that variable

is for prediction. Part of the reason for this is that the variance estimates used to

calculate r2 are biased in a way that tends to inflate r2. If unbiased estimators areused we get the adjusted r2

r2adj = 1

2i /(n p)

(yi y)2/(n 1).

A high value of r2adj indicates that the model is doing well at explaining the variabilityin the response variable.

F-statistic. The final line, giving an F-statistic and p-value, is testing the null

hypothesis that the data were generated from a model with only an intercept term,

against the alternative that the fitted model generated the data. This line is really

about asking if the whole model is of any use. The theory of such tests is covered in

section 1.3.4.

30 LINEAR MODELS

So the summary of sc.mod1 suggests that there is evidence that the model is better

than one including just a constant (p-value = 0.02554). There is quite clear evidence

that prop.partner is important in predicting sperm count (p-value = 0.019), but

less evidence that time.ipc matters (p-value = 0.053). Indeed, using the conven-

tional significance level of 0.05, we might be tempted to conclude that time.ipc

does not affect count at all. Finally note that the model leaves most of the variability

in count unexplained, since r2adj is only 37%.

1.5.3 Model selection

From the model summary it appears that time.ipc may not be necessary: the as-

sociated p-value of 0.053 does not provide strong evidence that the true value of 1 isnon-zero. By the true value is meant the value of the parameter in the model imag-

ined to have actually generated the data; or equivalently, the value of the parameter

applying to the whole population of couples from which, at least conceptually, our

particular sample has been randomly drawn. The question then arises of whether a

simpler model, without any dependence on time.ipc, might be appropriate. This

is a question of model selection. Usually it is a good idea to avoid over-complicated

models, dependent on irrelevant predictor variables, for reasons of interpretability

and efficiency. Inferences about causality will be made more difficult if a model con-

tains spurious predictors, but estimates using such a model will also be less precise,

as more parameters than necessary have been estimated from the finite amount of

uncertain data available.

Several approaches to model selection are based on hypothesis tests about modelterms, and can be thought of as attempting to find the simplest model consistent witha set of data, where consistency is judged relative to some threshold p-value. For thesperm competition model the p-value for time.ipc is greater than 0.05, so thispredictor might be a candidate for dropping.

> sc.mod3 summary(sc.mod3)

(edited)

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) 451.50 86.23 5.236 0.000161 ***prop.partner -292.23 140.40 -2.081 0.057727 .

---

Residual standard error: 154.3 on 13 degrees of freedom

Multiple R-Squared: 0.25, Adjusted R-squared: 0.1923

F-statistic: 4.332 on 1 and 13 DF, p-value: 0.05773

These results provide a good example of why it is dangerous to apply automatic

model selection procedures unthinkingly. In his case dropping time.ipc has made

the estimate of the parameter multiplying prop.partner less precise: indeed this

term also has a p-value greater than 0.05 according to this new fit. Furthermore, the

PRACTICAL LINEAR MODELS 31

new model has a much reduced r2, while the models overall p-value does not givestrong evidence that it is better than a model containing only an intercept. The only

sensible choice here is to revert to sc.mod1. The statistical evidence indicates that

it is better than the intercept only model, and dropping its possibly non-significant

term has lead to a much worse model.

Hypothesis testing is not the only approach to model selection. One alternative is totry and find the model that gets as close as possible to the true model, rather than tofind the simplest model consistent with data. In this case we can attempt to find themodel which does the best job of predicting the E(yi). Selecting models in order tominimize Akaikes Information Criterion (AIC) is one way of trying to do this (seesection 1.8.5). In R, the AIC function can be used to calculate the AIC statistic fordifferent models.

> sc.mod4 AIC(sc.mod1,sc.mod3,sc.mod4)

df AIC

sc.mod1 4 194.7346

sc.mod3 3 197.5889

sc.mod4 2 199.9031

This alternative model selection approach also suggests that the model with both

time.ipc and prop.partner is best.

So, on the basis of sperm.comp1, there seems to be reasonable evidence that sperm

count increases with time.ipc but decreases with prop.partner: exactly as

Baker and Bellis concluded.

1.5.4 Another model selection example

The second dataset from Baker and Bellis (1993) is sperm.comp2. This gives me-

dian sperm count for 24 couples, along with ages (years), heights (cm) and weights

(kg) for the male and female of each couple and volume (cm3) of one teste for themale of the couple (m.vol). There are quite a number of missing values for the pre-

dictors, particularly for m.vol, but, for the 15 couples for which there is an m.vol

measurement, the other predictors are also available. The number of copulations over

which the median count has been taken varies widely from couple to couple. Ideally

one should probably allow within couple and between couple components to the ran-

dom variability component of the data, to allow for this, but this will not be done

here. Following Baker and Bellis it seems reasonable to start from a model including

linear effects of all predictors. i.e.

counti = 0 + 1 f.agei + 2 f.weighti + 3 f.heighti + 4 m.agei+ 5 m.weighti + 6 m.heighti + 7 m.vol + i

The following estimates and summarizes the model, and plots diagnostics.

> sc2.mod1

32 LINEAR MODELS

150 200 250 300 350 400

200

0100

Fitted values

Re

sid

ua

ls

Residuals vs Fitted

9

1912

1 0 1

2

1

01

2

Theoretical Quantiles

Sta

nd

ard

ize

d r

esid

ua

ls

Normal QQ plot

19

2

12

150 200 250 300 350 400

0.0

0.5

1.0

1.5

Fitted values

Sta

nd

ard

ize

d r

esid

ua

lsScaleLocation plot

19

2

12

2 4 6 8 10 12 14

0.0

1.0

2.0

3.0

Obs. number

Co

oks

dis

tan

ce

Cooks distance plot

19

2

7

Figure 1.9 Model checking plots for the sperm competition model.

+ m.weight+m.vol,sperm.comp2)

> plot(sc2.mod1)

> summary(sc2.mod1)

[edited]

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) -1098.518 1997.984 -0.550 0.600

f.age 10.798 22.755 0.475 0.650

f.height -4.639 10.910 -0.425 0.683

f.weight 19.716 35.709 0.552 0.598

m.age -1.722 10.219 -0.168 0.871

m.height 6.009 10.378 0.579 0.581

m.weight -4.619 12.655 -0.365 0.726

m.vol 5.035 17.652 0.285 0.784

Residual standard error: 205.1 on 7 degrees of freedom

Multiple R-Squared: 0.2192, Adjusted R-squared: -0.5616

F-statistic: 0.2807 on 7 and 7 DF, p-value: 0.9422

The resulting figure 1.9, looks reasonable, but datum 19 appears to produce the most

extreme point on all 4 plots. Checking row 19 of the data frame, shows that the male

of this couple is rather heavy (particularly for his height), and has a large m.vol

measurement, but a count right near the bottom of the distribution (actually down at

the level that might be expected to cause fertility problems, if this is typical). Clearly,

whatever we conclude from these data will need to be double checked without this

observation. Notice, from the summary, how poorly this model does at explaining

PRACTICAL LINEAR MODELS 33

the count variability: the adjusted r2 is actually negative, an indication that we havea large number of irrelevant predictors in the model.

There are only 15 data from which to estimate the 8 parameters of the full model: it

would be better to come up with something more parsimonious. In this case using

AIC suggests a rather complicated model with only m.weight dropped. It seems

quite sensible to switch to hypothesis testing based model selection, and ask whether

there is really good evidence that all these terms are necessary? One approach is to

perform backwards model selection, by repeatedly removing the single term with

highest p-value, above some threshold (e.g. 0.05), and then refi