© Crary Group, 2004 1/31
Optimal Design of Experimentspresentation for Stanford University Statistics 252 class
Data Mining and Electronic Business
July 7, 2004
Selden B. Crary, Ph.D.
Crary Group, Palo Alto
WebDOE™WebDOE™
© Crary Group, 2004 2/31
Design of Experiments Makes a Difference
Weighing two apples
Simple method:
W1= 0.63 kg ± 2σ
W2= 0.57 kg ± 2σ
σ = std. dev. of measurement
Is there a better method?
WebDOE™WebDOE™
© Crary Group, 2004 3/31
R1 = W1 + W2R2 = W1 – W2
Solve for W1 and W2:
W1 = ½ (R1 + R2)
W2 = ½ (R1 – R2)
[ ]
2.dev.Std
2)R(Var)R(Var
21)W(Var)W(Var
2
21221
σ
σ
=
=+==
Errors in weights reduced by 29%
WebDOE™WebDOE™
© Crary Group, 2004 4/31
Many industrial products and processes benefit from designed experiments
Spaghetti Sauce
.
.
.
Semiconductors
WebDOE™WebDOE™
© Crary Group, 2004 5/31
Statistical Design of Experiments: Response-Surface Methodology (RSM)
Application Areas:Improvement of products and processes
Three Examples:Quadratic response: one factor
Sensor calibration: two factors
Direct mail marketing campaign: multiple factors
WebDOE™WebDOE™
© Crary Group, 2004 6/31
Examples: Features:Agriculture (static designs) Easy to find
Pharma (sequential designs) Sample space ~ uniformly
Quality control in manufacturing Not optimal
Full-factorial design Fractional-factorial design
··········
·· ·······
···· ···
Classical Design of Experiments
WebDOE™WebDOE™
© Crary Group, 2004 7/31
Problem: The size of factorial and fractional-factorial designs grows exponentially withthe number of variables.
0
100
200
300
0 2 4 6 8 10 12
Num
ber o
f tria
ls
Number of factors
OptimalDesigns
Factorial and Fractional-Factorial Designs
WebDOE™WebDOE™
© Crary Group, 2004 8/31
Methods used to design experiments depended upon the level of computation available:
Classical designs (Fisher)
Orthogonal arrays (Taguchi)
D-optimal designs
I-optimal, G-optimal, andapplication-specificoptimal designs
WebDOE™WebDOE™
© Crary Group, 2004 9/31
Optimal Statistical Design of Experiments
Application Areas
• Further improvement of products and processes(50% smaller confidence bounds on prediction are typical)
• Increased accuracy of standard-cell libraries
• Optimization of hospital staffing
• Optimization of marketing campaigns
WebDOE™WebDOE™
© Crary Group, 2004 10/31
Theory
WebDOE™WebDOE™
© Crary Group, 2004 11/31
ModelPolynomial response model:
e.g., Y = β0 + β1 x1 + β2 x2 + β3 x12 + β4 x22 + β5 x1 x2or Y = fT β (linear in the βι’s)
Measurements:
e.g., Y = β0 + β1 x1 + β2 x2 + β3 x12 + β4 x22 + β5 x1 x2 + ε
{Yi} i=1, …, N
or Y = X β + ε,
ε ~ i.i.d. with mean zero and variance σ2
WebDOE™WebDOE™
© Crary Group, 2004 12/31
Example of X, the design matrix
=
2)N(1)N(22)N(
21)N(2)N(1)N(
2)2(1)2(22)2(
21)2(2)2(1)2(
2)1(1)1(22)1(
21)1(2)1(1)1(
xxxxxx1..................xxxxxx1xxxxxx1
X
Y = β0 + β1 x1 + β2 x2 + β3 x12 + β4 x22 + β5 x1 x2 + ε
or, Y = X β + ε
WebDOE™WebDOE™
© Crary Group, 2004 13/31
( )( )
( )designsintpoNofsettheiswhere
,||XX||min
:optimalityD
XX)ˆ(Var
YXXXˆ
N
1T
1T2
T1T
N
−
−
=
=
−
−
−
ω
σβ
β
ω
D-optimality:minimize area of confidence interval
.
βiiβ̂
jβ̂
Optimal Designs for Parameter Estimation95% confidence interval for parameter estimationParameters β βj
WebDOE™WebDOE™
© Crary Group, 2004 14/31
Example 1:
D-optimal design forfitting a parabolawith N=5 points
Y = β0 + β1x1 + β2x12
Var of prediction =
DIV*
0.667
0.2
0.4
0.6
0.8
1.0
1.2
-1 -0.5 0 0.5 1
Varia
nce
of P
redi
ctio
nx
D
( ) ( )
−=
2YŶEŶVar
*IV=Integrated Variance: defined on next slide
WebDOE™WebDOE™
© Crary Group, 2004 15/31
Optimal Designs for Prediction
( ) ( )
( ) ( )
( )VarianceIntegrated
dx...
dx)x(Y)x(ŶE...
IV
dxfXXf...mindx)x(Y)x(ŶE...min
overerrorpredictionaverageimizemin:optimalityI
fXXfmaxmin)x(Y)x(ŶEmaxmin
regionaovererrorpredictioncaseworstectedexpimizemin:optimalityG
x
x
2
x
1TT
x
2
1TT2
NN
NN
=
−
=
=
−
−
=
−
−−
∫ ∫
∫ ∫
∫ ∫∫ ∫
∈
∈
∈
−
∈
−
χ
χ
χω
χω
χωχω
χ
χ
WebDOE™WebDOE™
© Crary Group, 2004 16/31
D-, G-, and I-optimal designs for fitting a parabola with N=5 points
Y = β0 + β1x1 + β2x12
GI
DIV
0.6670.5510.444
0.2
0.4
0.6
0.8
1.0
1.2
-1 -0.5 0 0.5 1
Varia
nce
of P
redi
ctio
nx
D
I
G
WebDOE™WebDOE™
© Crary Group, 2004 17/31
Interesting Observations Re I-optimal Designs
Asymmetric optimal designs can result from symmetric problem statements.
Phase transitions exist between designs of various symmetries, as weights assigned to different regions are changed continuously.
WebDOE™WebDOE™
© Crary Group, 2004 18/31
Extensions of I-optimal Designs (1)
Region need not be square, cube, or hyper-cube,i.e., inequality constraints can be handled.
Penalty for variance can be weighted.
Equality constraints can be handled,e.g., mixtures.
Variance of responses can be taken into account,i.e., heteroscedasticity can be treated.
WebDOE™WebDOE™
© Crary Group, 2004 19/31
Extensions of I-optimal Designs (2)
Points can be forced to be in the design.
Optimal augmented designs are possible.
Prior information on the distribution of β’s,e.g., from previous measurements,
can be taken into account(Bayesian approach).
I-optimality can be extended to deterministicerror models (computer experiments).
WebDOE™WebDOE™
© Crary Group, 2004 20/31
Example 2: Optimal Sensor Calibration
MEMS pressure sensor in an automotive setting
Sensor capacitance C=C(P,T)
Find an efficient design for determining a compensation formula:
C = C(P,T) = β0+ β1T + β2P + β3P2 + β4P3
Design must have at least Nmin=5 points
Changing P is less expensive than changing T.
WebDOE™WebDOE™
© Crary Group, 2004 21/31
I-optimal Designs for
C(P,T) = β0+ β1T + β2P + β3P2 + β4P3
N=5 N=40
P
T
WebDOE™WebDOE™
© Crary Group, 2004 22/31
Variance Contours of I-optimal Designs for
C(P,T) = β0+ β1T + β2P + β3P2 + β4P3
0.6
0.8
0.6
0.06
0.07
0.08
N=5 N=40
WebDOE™WebDOE™
© Crary Group, 2004 23/31
0.1
1.0
10 100
IV
N
slope ~N-1
Integrated Variance vs. Number of PointsC(P,T) = β0+ β1T + β2P + β3P2 + β4P3
Compensated Pressure = P(C,T) + ∆
WebDOE™WebDOE™
© Crary Group, 2004 24/31
If some regions are more important for prediction than others, use weighted I-optimal designs
Example: C(P,T) = β0+ β1T + β2P + β3P2 + β4P3, N=7
( ) ( ) dCdTefXXfmin 222N
~2/TC1T
C T
T σ
ω
+−−∫ ∫
P
T
N=7, σ=0.1∞
P
T
N=7, σ=
WebDOE™WebDOE™
© Crary Group, 2004 25/31
Example 3: Similar to On-Line Marketing:
Direct Mail Marketing of Credit-Cards* (1)
Challenges:
Response rates down: 2% 0.6%
Balance surfers: People move accounts to 0% APR
Adverse selection: bankruptcy
Solicitation volumes up: 6,000,000,000/yr
Costs per account activation up
*Source: Gerald Fahner, Fair Isaac seminar at Stanford, Jan. 31, 2003
WebDOE™WebDOE™
© Crary Group, 2004 26/31
Direct Mail Marketing of Credit-Cards (2)
Segmentation factors: Risk score, income, credit usage
Treatment factors: APR’s, fees, promotional incentives,
credit line, communications
(words, fonts, colors, layout,
envelopes, etc.)
Strategy: Should an offer be made, and, if so,
what treatment should be applied?:
{segmentation factors X} {treatment factors T}
Statistical question: How to make splits on decision tree?
WebDOE™WebDOE™
© Crary Group, 2004 27/31
Direct Mail Marketing of Credit-Cards (3)
Segmentation factors: {X}
Treatment factors: {T}
Profit = f(T,X)
Goal: Find: T* = argmax f(T, given X){T}
A possible optimal-design strategy: Find the design that minimizes the average expected squared error of prediction (i.e., IV) of the profit function f(T,X).
N.B.: The form of the profit function would need to be determined in order to use the approach given here.
WebDOE™WebDOE™
© Crary Group, 2004 28/31
Aside:
Assume cost of $0.50 per piece mailed
Fact: 10% of mailings are designed experiments
$300,000,000/yr is expended on designed experiments?
If 50% is saved using optimal design of experiments, then
savings of $150,000,000 are possible.
Direct Mail Marketing of Credit-Cards (4)
WebDOE™WebDOE™
© Crary Group, 2004 29/31
SummaryOptimal designs use 50%* fewer points
and/orare 50%* more accurate for predictionDesirable features:
I- and G-optimal designs focus on accurate predictionRegion of greatest interest can be weightedComplex cost functionals can be envisionedAdaptive, sequential approach possible(Computer experiments are accommodated)
* Typical
WebDOE™WebDOE™
© Crary Group, 2004 30/31
SourcesBooks: www.WebDOE.com/Resources
Classical designs: WebDOE (registration req’d.)
Optimal designs: ditto
Training: WebDOE home page
Questions: [email protected]
WebDOE™WebDOE™
© Crary Group, 2004 31/31WebDOE™WebDOE™