� Linear Regression

� Polynomial Regression

� Multiple Linear Regression

Numerical Methods for Numerical Methods for CivilCivil EngineersEngineers

Mongkol JIRAVACHARADET

S U R A N A R E E INSTITUTE OF ENGINEERING

UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

Lecture Lecture 77 Curve FittingCurve Fitting

LINEAR REGRESSION

x

y

Candidate lines for curve fit

y = α x +β

No exact solution but many approximated solutions

x

y Observation: [ xi yi ]

Error Between Model and Observation

Model: y = α x + β

Error: ei = yi - α xi + β

Criteria for a “Best” Fit

Find the BEST line which minimize the sum of error for all data

Least-Square Fit of a Straight Line

Minimize sum of the square of the errors

( )∑∑==

−−==n

i

ii

n

i

ir xyeS1

2

1

2 αβ

Differentiate with respect to each coefficient:

])[(2

)(2

iiir

iir

xxyS

xyS

αβα

αββ

−−∑−=∂∂

−−∑−=∂∂

Setting derivatives = 0 :

20

0

iiii

ii

xxxy

xy

αβ

αβ

∑−∑−∑=

∑−∑−∑=

From Σβ = n β , express equations as set of 2 unknowns ( β , α )

iiii

ii

xyxx

yxn

∑=∑+∑

∑=∑+

2αβ

αβ

Solve equations simultaneously:

( )22

1

1

i i i i

i i

x y x yn

x xn

α∑ − ∑ ∑

=∑ − ∑

y xβ α= −

xyxy and of mean the are and where

1xy i i i iS x y x y

n= Σ − Σ ΣDefine:

( )22 1xx i iS x x

n= Σ − Σ

( )22 1yy i iS y y

n= Σ − Σ

y xα β= +

Approximated y for any x is

xy

xx

S

Sα =

Example: Fit a straight line to x and y values

xi yi

1

2

3

4

5

6

7

0.5

2.5

2.0

4.0

3.5

6.0

5.5 4286.37

24

47

28

7

==

==

=

y

x

n

2

(119.5) (28)(24) / 70.8393

(140) (28) / 7

3.4286 0.8393(4) 0.0714

α

β

−= =

−

= − =0714.08393.0 += xy

Least-square fit:

xi2 xi yi

1

4

9

16

25

36

49

0.5

5.0

6.0

16.0

17.5

36.0

38.5

yi2

0.25

6.25

4

16

12.25

36

30.25

Σ 28 24 140 119.5 105

How good is our fit?

Sum of the square of the errors:

( )22

1 1

n n

r i i i

i i

S e y xβ α= =

= = − −∑ ∑

Standard errors of the estimation:/

2

ry x

Ss

n=

−

Sum of the square around the mean: ( )21

n

t i

i

S y y=

= −∑

Standard deviation:2

ty

Ss

n=

−

( )2 /xx yy xy xxS S S S= −

yyS=

Coefficient of determination 2 t r

t

S Sr

S

−=

For perfect fit Sr = 0 and r = r2 = 1

y

sy sy/x

Linear regression

sy > sy/x

2

xy

xx yy

S

S S=

r2 ������������� ����������� y ���������������������� x

Example: error analysis of the linear fit

xi yi

1

2

3

4

5

6

7

0.5

2.5

2.0

4.0

3.5

6.0

5.5

0.8393

0.0714

αβ=

=

3.4286y =

/

22.7143

7 2

2.131

2.9911

7 2

0.773

y

y x

s

s

=−

=

=−

=

( yi - β - α xi)2

8.5765

0.8622

2.0408

0.3265

0.0051

6.6122

4.2908

0.1687

0.5626

0.3473

0.3265

0.5896

0.7972

0.1993

Σ 28 24 22.7143 2.9911

( )2iy y−

St SrSince sy/x < sy , linear regression has merit.

22.7143 2.99110.868 0.932

22.7143r

−= = =

Linear model explains 86.8% of original uncertainty.

xi yi

1

2

3

4

5

6

7

0.5

2.5

2.0

4.0

3.5

6.0

5.5

xi2 xi yi

1

4

9

16

25

36

49

0.5

5.0

6.0

16.0

17.5

36.0

38.5

yi2

0.25

6.25

4

16

12.25

36

30.25

Σ 28 24 140 119.5 105

OR Example: error analysis of the linear fit

2140 28 / 7 28xxS = − =

2105 24 / 7 22.7yyS = − =

119.5 28 24 / 7 23.5xyS = − × =

2(28 22.7 23.5 ) / 28

2.977

rS = × −

=

/

22.72.131

7 2

2.9770.772

7 2

y

y x

s

s

= =−

= =−

Since sy/x < sy , linear regression has merit.

22 23.5

0.86928 22.7

r = =×

Linear model explains 86.9% of original uncertainty.

Confidence Interval (CI)

A confidence interval is an interval in which a measurement or trial

falls corresponding to a given probability.

x

y

xi

ˆi

y

ˆiy ± ∆

x

y

For CI 95%, you can be 95% confident that the two curved

confidence bands enclose the true best-fit linear regression line,

leaving a 5% chance that the true line is outside those boundaries.

2

/ 2 /

( )1ˆ i

i y x

xx

x xy t s

n Sα

−± +

A 100 (1 - α) % confidence interval for yi is given by

Confidence interval 95% → α = 0.05

xi yi

1

2

3

4

5

6

7

0.5

2.5

2.0

4.0

3.5

6.0

5.5

Example: to estimate y when x is 3.4 using 95% confidence interval:

ˆ 0.8363(3.4) 0.0714 2.9148y xα β= + = + =

95% Confidence → α = 0.05 → tα/2 = t0.025(df = n-2 = 5) = 2.571

Interval:21 (3.4 4)

2.9148 (2.571) (0.772)7 28

−± +

2.9148 0.7832±

T-Distribution

t

95%

99%

t 0.025 t 0.025

t 0.005 t 0.005

Probability density function of

the t distribution:

where B is the beta function and

ν is a positive integer

shape parameter.

The formula for the beta function is

2 ( 1) / 2(1 / )( )

(0.5,0.5 )

xf x

B

νν

ν ν

− ++=

1

1 1

0

( , ) (1 )B t t dtα βα β − −= −∫

The following is the plot of the t probability density function for 4 different

values of the shape parameter.

In fact, the t distribution with ν equal to 1 is a Cauchy distribution.

The t distribution approaches a normal distribution as ν becomes large.

The approximation is quite good for values of ν > 30.

ν = df

Degree of freedom

Critical Values of t

80% 90% 95% 98% 99% 99.8%

df 0.10 0.05 0.025 0.01 0.005 0.001

1 3.078 6.314 12.706 31.821 63.657 318.313

2 1.886 2.920 4.303 6.965 9.925 22.327

3 1.638 2.353 3.182 4.541 5.841 10.215

4 1.533 2.132 2.776 3.747 4.604 7.173

5 1.476 2.015 2.571 3.365 4.032 5.893

6 1.440 1.943 2.447 3.143 3.707 5.208

7 1.415 1.895 2.365 2.998 3.499 4.782

8 1.397 1.860 2.306 2.896 3.355 4.499

9 1.383 1.833 2.262 2.821 3.250 4.296

10 1.372 1.812 2.228 2.764 3.169 4.143

11 1.363 1.796 2.201 2.718 3.106 4.024

12 1.356 1.782 2.179 2.681 3.055 3.929

13 1.350 1.771 2.160 2.650 3.012 3.852

14 1.345 1.761 2.145 2.624 2.977 3.787

15 1.341 1.753 2.131 2.602 2.947 3.733

16 1.337 1.746 2.120 2.583 2.921 3.686

17 1.333 1.740 2.110 2.567 2.898 3.646

18 1.330 1.734 2.101 2.552 2.878 3.610

19 1.328 1.729 2.093 2.539 2.861 3.579

20 1.325 1.725 2.086 2.528 2.845 3.552

Confidence Interval

80% 90% 95% 98% 99% 99.8%

0.10 0.05 0.025 0.01 0.005 0.001

1.323 1.721 2.080 2.518 2.831 3.527

1.321 1.717 2.074 2.508 2.819 3.505

1.319 1.714 2.069 2.500 2.807 3.485

1.318 1.711 2.064 2.492 2.797 3.467

1.316 1.708 2.060 2.485 2.787 3.450

1.315 1.706 2.056 2.479 2.779 3.435

1.314 1.703 2.052 2.473 2.771 3.421

1.313 1.701 2.048 2.467 2.763 3.408

1.311 1.699 2.045 2.462 2.756 3.396

1.310 1.697 2.042 2.457 2.750 3.385

1.309 1.696 2.040 2.453 2.744 3.375

1.309 1.694 2.037 2.449 2.738 3.365

1.308 1.692 2.035 2.445 2.733 3.356

1.307 1.691 2.032 2.441 2.728 3.348

1.306 1.690 2.030 2.438 2.724 3.340

1.306 1.688 2.028 2.434 2.719 3.333

1.305 1.687 2.026 2.431 2.715 3.326

1.304 1.686 2.024 2.429 2.712 3.319

1.304 1.685 2.023 2.426 2.708 3.313

1.303 1.684 2.021 2.423 2.704 3.307

Confidence Interval

df

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

80% 90% 95% 98% 99% 99.8%

0.10 0.05 0.025 0.01 0.005 0.001

1.303 1.683 2.020 2.421 2.701 3.301

1.302 1.682 2.018 2.418 2.698 3.296

1.302 1.681 2.017 2.416 2.695 3.291

1.301 1.680 2.015 2.414 2.692 3.286

1.301 1.679 2.014 2.412 2.690 3.281

1.300 1.679 2.013 2.410 2.687 3.277

1.300 1.678 2.012 2.408 2.685 3.273

1.299 1.677 2.011 2.407 2.682 3.269

1.299 1.677 2.010 2.405 2.680 3.265

1.299 1.676 2.009 2.403 2.678 3.261

1.298 1.675 2.008 2.402 2.676 3.258

1.298 1.675 2.007 2.400 2.674 3.255

1.298 1.674 2.006 2.399 2.672 3.251

1.297 1.674 2.005 2.397 2.670 3.248

1.297 1.673 2.004 2.396 2.668 3.245

1.297 1.673 2.003 2.395 2.667 3.242

1.297 1.672 2.002 2.394 2.665 3.239

1.296 1.672 2.002 2.392 2.663 3.237

1.296 1.671 2.001 2.391 2.662 3.234

1.296 1.671 2.000 2.390 2.660 3.232

Confidence Interval

df

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

80% 90% 95% 98% 99% 99.8%

0.10 0.05 0.025 0.01 0.005 0.001

1.296 1.670 2.000 2.389 2.659 3.229

1.295 1.670 1.999 2.388 2.657 3.227

1.295 1.669 1.998 2.387 2.656 3.225

1.295 1.669 1.998 2.386 2.655 3.223

1.295 1.669 1.997 2.385 2.654 3.220

1.295 1.668 1.997 2.384 2.652 3.218

1.294 1.668 1.996 2.383 2.651 3.216

1.294 1.668 1.995 2.382 2.650 3.214

1.294 1.667 1.995 2.382 2.649 3.213

1.294 1.667 1.994 2.381 2.648 3.211

1.294 1.667 1.994 2.380 2.647 3.209

1.293 1.666 1.993 2.379 2.646 3.207

1.293 1.666 1.993 2.379 2.645 3.206

1.293 1.666 1.993 2.378 2.644 3.204

1.293 1.665 1.992 2.377 2.643 3.202

1.293 1.665 1.992 2.376 2.642 3.201

1.293 1.665 1.991 2.376 2.641 3.199

1.292 1.665 1.991 2.375 2.640 3.198

1.292 1.664 1.990 2.374 2.640 3.197

1.292 1.664 1.990 2.374 2.639 3.195

Confidence Interval

df

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

80% 90% 95% 98% 99% 99.8%

0.10 0.05 0.025 0.01 0.005 0.001

1.292 1.664 1.990 2.373 2.638 3.194

1.292 1.664 1.989 2.373 2.637 3.193

1.292 1.663 1.989 2.372 2.636 3.191

1.292 1.663 1.989 2.372 2.636 3.190

1.292 1.663 1.988 2.371 2.635 3.189

1.291 1.663 1.988 2.370 2.634 3.188

1.291 1.663 1.988 2.370 2.634 3.187

1.291 1.662 1.987 2.369 2.633 3.185

1.291 1.662 1.987 2.369 2.632 3.184

1.291 1.662 1.987 2.368 2.632 3.183

1.291 1.662 1.986 2.368 2.631 3.182

1.291 1.662 1.986 2.368 2.630 3.181

1.291 1.661 1.986 2.367 2.630 3.180

1.291 1.661 1.986 2.367 2.629 3.179

1.291 1.661 1.985 2.366 2.629 3.178

1.290 1.661 1.985 2.366 2.628 3.177

1.290 1.661 1.985 2.365 2.627 3.176

1.290 1.661 1.984 2.365 2.627 3.175

1.290 1.660 1.984 2.365 2.626 3.175

1.290 1.660 1.984 2.364 2.626 3.174

1.282 1.645 1.960 2.326 2.576 3.090

Confidence Interval

df

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

∞∞∞∞

Polynomial Regression

Second-order polynomial:

y = a0 + a1x + a2 x2

Sum of the squares of the residuals:

22

210 )( iiir xaxaayS −−−∑=

Take derivative with respect to each coefficients:

)(2

)(2

)(2

2

210

2

2

2

210

1

2

210

0

iiiir

iiiir

iiir

xaxaayxa

S

xaxaayxa

S

xaxaaya

S

−−−∑−=∂∂

−−−∑−=∂∂

−−−∑−=∂∂

Normal equations:

( ) ( )( ) ( ) ( )( ) ( ) ( ) iiiii

iiiii

iii

yxaxaxax

yxaxaxax

yaxaxan

2

2

4

1

3

0

2

2

3

1

2

0

2

2

10

∑=∑+∑+∑

∑=∑+∑+∑

∑=∑+∑+

For mth-order polynomial: y = a0 + a1x + a2 x2+ . . . + amx

m

We have to solve m+1 simultaneous linear equations.

MATLAB polyfit Function

For second-order polynomial, we can define

211 1

1222 2

2

32

-1

1

1, ,

1

and show that ( ' ) ' or =

mm m

yx xc

yx xc

cyx x

−

= = =

= 1

A Y C

C A A A Y C A Y

�� � �

>> C = polyfit(x, y, n)

>> [C, S] = polyfit(x, y, n)

x = independent variable

y = dependent variable

n = degree of polynomial

C = coeff. of polynomial in

descending power

S = data structure for polyval

function

Fit norm Fit QR

Example: Fit a second-order polynomial to the data.

xi yi

0

1

2

3

4

5

2.1

7.7

13.6

27.2

40.9

61.1

( yi - a0 - a1xi - a2xi2)2

544.44

314.47

140.03

3.12

239.22

1272.11

0.14332

1.00286

1.08158

0.80491

0.61951

0.09439

Σ 15 152.6 2513.39 3.74657

( )2iy y−

4

2 2

3

2 15 979

6 152.6 585.6

2.5 55 585.6

25.433 225

i i

i i i

i i i

i

m x x

n y x y

x x x y

y x

= ∑ = ∑ =

= ∑ = ∑ =

= ∑ = ∑ =

= ∑ =

From the given data:

Simultaneous linear equations

0

1

2

6 15 25 152.6

15 55 225 585.6

55 225 979 2488.8

a

a

a

=

Solving these equation gives a0 = 2.47857, a1 = 2.35929, and

a2 = 1.86071.

Least-squares quadratic equation:

y = 2.47857 + 2.35929x + 1.86071x2

Coefficient of determination:

2513.39 3.746570.99851 0.99925

2513.39r

−= = =

Solving by MATLAB polyfit Function

>> x = [0 1 2 3 4 5];

>> y = [2.1 7.7 13.6 27.2 40.9 61.1];

>> c = polyfit(x, y, 2)

>> [c, s] = polyfit(x, y, 2)

>> st = sum((y - mean(y)).^2)

>> sr = sum((y - polyval(c, x)).^2)

>> r = sqrt((st - sr) / st)

MATLAB polyval Function

Evaluate polynomial at the points defined by the input vector

Example: y = 1.86071x2+ 2.35929x + 2.47857

Y = c(1)*xn + c(2)*x(n-1) + ... + c(n)*x + c(n+1)

>> y = polyval(c, x)

where x = Input vector

y = Value of polynomial evaluated at x

c = vector of coefficient in descending order

>> c = [1.86071 2.35929 2.47857]

0 1 2 3 4 50

10

20

30

40

50

60

70

x

y

Polynomial Interpolation

>> y2 = polyval(c,x)

>> plot(x, y, ’o’, x, y2)

Error Bounds

By passing an optional second output parameter from polyfit as an input

to polyval.

Interval of ±2∆ = 95% confidence interval

>> [y2,delta] = polyval(c,x,s)

>> [c,s] = polyfit(x,y,2)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')

Linear Regression Example:

>> [y2,delta] = polyval(c,x,s)

>> [c,s] = polyfit(x,y,1)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')

xi yi

1

2

3

4

5

6

7

0.5

2.5

2.0

4.0

3.5

6.0

5.5

Multiple Linear Regression

y = c0 + c1x1+ c2x2+ . . .+ cpxp

Example case: two independent variables y = c0 + c1x1+ c2x2

2

0 1 1 2 2( )r i i iS y c c x c x= ∑ − − −Sum of squares of the residual:

0 1 1 2 2

0

1 0 1 1 2 2

1

2 0 1 1 2 2

2

2 ( )

2 ( )

2 ( )

ri i i

ri i i i

ri i i i

Sy c c x c x

c

Sx y c c x c x

c

Sx y c c x c x

c

∂= − ∑ − − −

∂

∂= − ∑ − − −

∂

∂= − ∑ − − −

∂

Differentiate with respect to unknowns:

Setting partial derivatives = 0 and expressing result in matrix form:

1 2 0

2

1 1 1 2 1 1

2

2 1 2 2 2 2

i i i

i i i i i i

i i i i i i

n x x c y

x x x x c x y

x x x x c x y

∑ ∑ ∑ ∑ ∑ ∑ = ∑ ∑ ∑ ∑ ∑

Example:

x1 x2 y

0

2

2.5

1

4

7

0

1

2

3

6

2

5

10

9

0

3

27

0

1

2

0

1

2

6 16.5 14 54

16.5 76.25 48 243.5

14 48 54 100

5

4

3

c

c

c

c

c

c

=

=

=

= −

Multivariate Fit in MATLAB

c0 + c1x11+ c2x12+ . . .+ cpx1p = y1

.

.

.

c0 + c1x21+ c2x22+ . . .+ cpx2p = y2

c0 + c1xm1 + c2xm2+ . . .+ cpxmp= ym

11 12 1 0 1

21 22 2 1 2

1 2

1

1, , and y

1

p

p

m m mp p m

x x x c y

x x x c y

x x x c y

= = =

A c

�

�

� � � � � � �

�

Overdetermined system of equations: A c = y

Fit norm >> c = (A’*A)\(A’*y)

Fit QR >> c = A\y

Example:

x1 x2 y

0

2

2.5

1

4

7

0

1

2

3

6

2

5

10

9

0

3

27

>> x1=[0 2 2.5 1 4 7]';

>> x2=[0 1 2 3 6 2]';

>> y=[5 10 9 0 3 27]';

>> A=[x1 x2 ones(size(x1))];

>> c=A\y