Traditional methods - Accueil · I The \traditional" methods were introduced as heuristics. For...

90
LHS Coordinate Search Nelder-Mead References Traditional methods MTH8418 S. Le Digabel, Polytechnique Montr´ eal Winter 2020 (v3) MTH8418: Traditional methods 1/55

Transcript of Traditional methods - Accueil · I The \traditional" methods were introduced as heuristics. For...

LHS Coordinate Search Nelder-Mead References

Traditional methods

MTH8418

S. Le Digabel, Polytechnique Montreal

Winter 2020(v3)

MTH8418: Traditional methods 1/55

LHS Coordinate Search Nelder-Mead References

IntroductionI We consider the unconstrained version of the problem with

Ω = Rn:

minx∈Rn

f(x)

I Possible to treat constrained problems with meta-strategiessuch as the extreme barrier or the augmented Lagrangianapproaches (discussed in Lecture #9)

I The “traditional” methods were introduced as heuristics. Forsome of them, convergence was studied only in the 90’. Thisopened the era of modern DFO methods

I This lecture describes three of these methods

MTH8418: Traditional methods 2/55

LHS Coordinate Search Nelder-Mead References

Plan

Latin Hypercube Sampling

The Coordinate Search method

The Nelder-Mead method

References

MTH8418: Traditional methods 3/55

LHS Coordinate Search Nelder-Mead References

Latin Hypercube Sampling

The Coordinate Search method

The Nelder-Mead method

References

MTH8418: Traditional methods 4/55

LHS Coordinate Search Nelder-Mead References

Latin Hypercube SamplingI Traditional statistical method from [McKay et al., 1979]

I Often used in the design of computer experiments

I Latin square: With n = 2, a grid containing one sample ineach row and each column

I Latin hypercube: Generalization in dimension n

I Bounds for each variable are necessary

I Stochastic method:I Use a good Random Number Generator (RNG)I Use the process identification number (PID) as the random

seedI Change the seed for different resultsI Remember the seed for reproducibility

MTH8418: Traditional methods 5/55

LHS Coordinate Search Nelder-Mead References

Latin Hypercube Sampling

I Optimization:

1. Construct sampling: Generate p sample points2. Evaluate f at the sample points3. Remember the best solution

I Bad algorithm for optimization

I But good strategy to generate a set of starting solutions forother methods

MTH8418: Traditional methods 6/55

LHS Coordinate Search Nelder-Mead References

Latin Hypercube Sampling

One strategy to generate p points in dimension n:

I The range of each variable is divided into p intervals

I The intersections of these intervals define rectangular regions

I Two steps of random draws:

1. Permutations for each variable, to choose one particular region

2. Random location inside the region

MTH8418: Traditional methods 7/55

LHS Coordinate Search Nelder-Mead References

Latin Hypercube Sampling: Pseudocode

vector<Point> LHS ( n , p , lb , ub )

vector<Point> pts;

for ( k = 0 ; k < p ; ++k )

Point x(n);

for ( i = 0 ; i < n ; ++i )

x[i] = lb[i] + ( ub[i]-lb[i] ) *

( permut(p) + rand(0.0,1.0) ) / p;

pts.insert(x);

return pts;

MTH8418: Traditional methods 8/55

LHS Coordinate Search Nelder-Mead References

Latin Hypercube Sampling: Example

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100LHS with p=10

x

y

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100LHS with p=100

x

y

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100LHS with p=1000

x

y

n = 2 and p ∈ 10, 100, 1000

MTH8418: Traditional methods 9/55

LHS Coordinate Search Nelder-Mead References

Latin Hypercube Sampling

The Coordinate Search method

The Nelder-Mead method

References

MTH8418: Traditional methods 10/55

LHS Coordinate Search Nelder-Mead References

The Coordinate Search methodI Coordinate Search (CS), or Compass Search

I The CS is the ancestor of the modern direct search methods:I Hooke & Jeeves method: The original Pattern Search method

(also defined “direct search”) [Hooke and Jeeves, 1961]

I Generalized Pattern Search (GPS) [Torczon, 1997]

I Mesh Adaptive Direct Search(MADS) [Audet and Dennis, Jr., 2006]

I Generating Set Search (GSS) [Kolda et al., 2003]

I [Fermi and Metropolis, 1952] (first known reference)

I Tested on one of the first computers, the MANIAC(Mathematical Analyzer, Numerical Integrator, and Computer or Mathematical Analyzer, Numerator,Integrator, and Computer)

MTH8418: Traditional methods 11/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Algorithm

I Initialization:x0 : starting point in Rn

∆0 > 0 : initial step size

I Poll step: for k = 0, 1, . . .if f(t) < f(xk) for t ∈ Pk := xk ±∆kei : i = 1, 2, . . . , n:

xk+1 ← t∆k+1 ← ∆k

else (failure): xk is a local minimum relatively to Pk:xk+1 ← xk∆k+1 ← ∆k/2

MTH8418: Traditional methods 12/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Algorithm

I Initialization:x0 : starting point in Rn

∆0 > 0 : initial step size

I Poll step: for k = 0, 1, . . .if f(t) < f(xk) for t ∈ Pk := xk ±∆kei : i = 1, 2, . . . , n:

xk+1 ← t∆k+1 ← ∆k

else (failure): xk is a local minimum relatively to Pk:xk+1 ← xk∆k+1 ← ∆k/2

MTH8418: Traditional methods 12/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Algorithm

I Initialization:x0 : starting point in Rn

∆0 > 0 : initial step size

I Poll step: for k = 0, 1, . . .if f(t) < f(xk) for t ∈ Pk := xk ±∆kei : i = 1, 2, . . . , n:

xk+1 ← t∆k+1 ← ∆k

else (failure): xk is a local minimum relatively to Pk:xk+1 ← xk∆k+1 ← ∆k/2

MTH8418: Traditional methods 12/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Details

I Evaluation of x0 is counted

I Extension with bounds: If the algorithm generates a trialpoint outside of the boundary, then do not evaluate andconsider as a failure

I If the algorithm generates a trial point x found in the cache,then do not evaluate and consider as a failure

MTH8418: Traditional methods 13/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

•f=4176

MTH8418: Traditional methods 14/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

•f=4176

MTH8418: Traditional methods 14/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

•f=4176

MTH8418: Traditional methods 14/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

•f=4176

MTH8418: Traditional methods 14/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

•f=4176

MTH8418: Traditional methods 14/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x1=(1,2)>,∆1=1

•f=166

•f=4401

•f=262

•f=81

•f=106

MTH8418: Traditional methods 15/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x1=(1,2)>,∆1=1

•f=166

•f=4401

•f=262

•f=81

•f=106

MTH8418: Traditional methods 15/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x2=(0,2)>,∆2=1

•f=81

•f=166

•f=152

•f=2646

•f=36

MTH8418: Traditional methods 16/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x2=(0,2)>,∆2=1

•f=81

•f=166

•f=152

•f=2646

•f=36

MTH8418: Traditional methods 16/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x3=(0,1)>,∆3=1

•f=36

•f=106

•f=81

•f=2466

•f=17

MTH8418: Traditional methods 17/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x3=(0,1)>,∆3=1

•f=36

•f=106

•f=81

•f=2466

•f=17

MTH8418: Traditional methods 17/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x4=(0,0)>,∆4=1

•f=17

•f=82

•f=36

•f=2402

•f=24

•f=17

x4 is a mesh local optimizer

MTH8418: Traditional methods 18/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x4=(0,0)>,∆4=1

•f=17

•f=82

•f=36

•f=2402

•f=24

•f=17

x4 is a mesh local optimizer

MTH8418: Traditional methods 18/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x4=(0,0)>,∆4=1

•f=17

•f=82

•f=36

•f=2402

•f=24

•f=17

x4 is a mesh local optimizer

MTH8418: Traditional methods 18/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x5=(0,0)>,∆5= 12

After 20 eval.:

f(x) = 17

Can we do better?

•f=17

MTH8418: Traditional methods 19/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Basic algorithm

x5=(0,0)>,∆5= 12

After 20 eval.:

f(x) = 17

Can we do better?

•f=17

MTH8418: Traditional methods 19/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

MTH8418: Traditional methods 20/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

MTH8418: Traditional methods 20/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

MTH8418: Traditional methods 20/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

MTH8418: Traditional methods 20/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x1=(1,2)>,∆1=1

•f=166

•f=4401

•f=262

•f=81

MTH8418: Traditional methods 21/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x1=(1,2)>,∆1=1

•f=166

•f=4401

•f=262

•f=81

MTH8418: Traditional methods 21/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x2=(0,2)>,∆2=1

•f=81

•f=166

•f=152

•f=2646

•f=36

MTH8418: Traditional methods 22/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x2=(0,2)>,∆2=1

•f=81

•f=166

•f=152

•f=2646

•f=36

MTH8418: Traditional methods 22/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x3=(0,1)>,∆3=1

•f=36

•f=106

•f=81

•f=2466

•f=17

MTH8418: Traditional methods 23/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x3=(0,1)>,∆3=1

•f=36

•f=106

•f=81

•f=2466

•f=17

MTH8418: Traditional methods 23/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x4=(0,0)>,∆4=1

•f=17

•f=82

•f=36

•f=2402

•f=24

MTH8418: Traditional methods 24/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x4=(0,0)>,∆4=1

•f=17

•f=82

•f=36

•f=2402

•f=24

MTH8418: Traditional methods 24/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x5(0,0)>,∆5= 12

•f=17

•f=1.0625

MTH8418: Traditional methods 25/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x5(0,0)>,∆5= 12

•f=17

•f=1.0625

MTH8418: Traditional methods 25/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x6=(0.5,0)>,∆6= 12

After 20 eval.:

f(x) = 1.0625

Can we do better?

•f=1.0625

MTH8418: Traditional methods 26/55

LHS Coordinate Search Nelder-Mead References

Opportunistic Coordinate Search

x6=(0.5,0)>,∆6= 12

After 20 eval.:

f(x) = 1.0625

Can we do better?

•f=1.0625

MTH8418: Traditional methods 26/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

MTH8418: Traditional methods 27/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

MTH8418: Traditional methods 27/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

MTH8418: Traditional methods 27/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

MTH8418: Traditional methods 27/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x0=(2,2)>,∆0=1

•f=4401

•f=29286

•f=4772

•f=166

MTH8418: Traditional methods 27/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x1=(1,2)>,∆1=1

•f=166

•f=81

MTH8418: Traditional methods 28/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x1=(1,2)>,∆1=1

•f=166

•f=81

MTH8418: Traditional methods 28/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x2=(0,2)>,∆2=1

•f=81

•f=2646

•f=166

•f=152

•f=36

MTH8418: Traditional methods 29/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x2=(0,2)>,∆2=1

•f=81

•f=2646

•f=166

•f=152

•f=36

MTH8418: Traditional methods 29/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x2=(0,2)>,∆2=1

•f=81

•f=2646

•f=166

•f=152

•f=36

MTH8418: Traditional methods 29/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x2=(0,2)>,∆2=1

•f=81

•f=2646

•f=166

•f=152

•f=36

MTH8418: Traditional methods 29/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x2=(0,2)>,∆2=1

•f=81

•f=2646

•f=166

•f=152

•f=36

MTH8418: Traditional methods 29/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x3=(0,1)>,∆3=1

•f=36

•f=17

MTH8418: Traditional methods 30/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x3=(0,1)>,∆3=1

•f=36

•f=17

MTH8418: Traditional methods 30/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x4=(0,0)>,∆4=1

•f=17

•f=24

•f=2402

•f=82

•f=36

MTH8418: Traditional methods 31/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x4=(0,0)>,∆4=1

•f=17

•f=24

•f=2402

•f=82

•f=36

MTH8418: Traditional methods 31/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x5=(0,0)>,∆5= 12

•f=17

•f=17.25

•f=411

•f=1.0625

MTH8418: Traditional methods 32/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x5=(0,0)>,∆5= 12

•f=17

•f=17.25

•f=411

•f=1.0625

MTH8418: Traditional methods 32/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x6=(0.5,0)>,∆6= 12

•f=1.0625

•f=82

•f=0.375

MTH8418: Traditional methods 33/55

LHS Coordinate Search Nelder-Mead References

Dynamic Coordinate Search

x6=(0.5,0)>,∆6= 12

•f=1.0625

•f=82

•f=0.375

MTH8418: Traditional methods 33/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: 3 strategies

Basic (complete) Opportunistic Dynamic

x> f(x) x> f(x) x> f(x)

After 20 evaluations of f

(0,0) 17 (.5,0) 1.0625 (.5,-.5) .375

After 50 evaluations of f

(.375, -.375) 1.8E-2 (.375, -.312) 5.7E-3 (.375, -.344) 3.1E-3

MTH8418: Traditional methods 34/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Convergence analysis

I Special case of GPS for which convergence was provedby [Torczon, 1997]

I If the sequence of iterates xk is bounded, thenI lim

k∆k = 0

I There exists x ∈ Rn, the limit of a subsequence of mesh localoptimizers, whose size is infinitely small

I If f is continuously differentiable at x, then ∇f(x) = 0

MTH8418: Traditional methods 35/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Failure

f(x) = ‖x‖∞ from x0 = (1, 1)>:

-

6

?

x1

x2Level set:

x ∈ R2 : f(x) = 1 s x0 = (1, 1)>

-

6

?

Algorithm “stuck” at x0

MTH8418: Traditional methods 36/55

LHS Coordinate Search Nelder-Mead References

Coordinate Search: Failure

f(x) = ‖x‖∞ from x0 = (1, 1)>:

-

6

?

x1

x2Level set:

x ∈ R2 : f(x) = 1 s x0 = (1, 1)>

-

6

?

Algorithm “stuck” at x0

MTH8418: Traditional methods 36/55

LHS Coordinate Search Nelder-Mead References

Latin Hypercube Sampling

The Coordinate Search method

The Nelder-Mead method

References

MTH8418: Traditional methods 37/55

LHS Coordinate Search Nelder-Mead References

The Nelder-Mead method

I The Nelder-Mead (NM) algorithm, or the other Simplexmethod [Nelder and Mead, 1965]

I No link with the Simplex method from Dantzig for linearprogramming

I 1965: Good timing with the emergence of blackboxoptimization problems

I Most used and cited method for unconstrained optimization

MTH8418: Traditional methods 38/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: IdeaI Simplex in dimension n: set of n+ 1 points

I At each iteration, such a simplex is defined:

Y = y0, y1, . . . , yn

with f(y0) ≤ f(y1) ≤ . . . ≤ f(yn)

I The centroid yc of the first n points is defined as

yc =

n−1∑i=0

yi/n

I Depending on the success of the iteration, the shape and thesize of the simplex are updated

MTH8418: Traditional methods 39/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Transformations (1/2)

I Five different transformations:

I Reflection

I Expansion

I Outside contraction

I Inside contraction

I Shrink

I Except for the shrink, the worst vertex of the simplex isreplaced by one of the reflection, expansion, or contractionpoints

MTH8418: Traditional methods 40/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Transformations (2/2)

I Reflection point: yr = yc + δr(yc − yn)

I Expansion point: ye = yc + δe(yc − yn)

I Outside contraction point: yoc = yc + δoc(yc − yn)

I Inside contraction point: yic = yc + δic(yc − yn)

I Shrink points: ysi = y0 + γs(yi − y0) for i = 1, 2, . . . , n

I 0 < γs < 1, −1 < δic < 0 < δoc < δr < δe

I Standard choices: δic = −0.5, γs = δoc = 0.5, δr = 1, δe = 2

MTH8418: Traditional methods 41/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Reflection, expansion, contractions

Taken from [Conn et al., 2009]

MTH8418: Traditional methods 42/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Shrink

Taken from [Conn et al., 2009]

MTH8418: Traditional methods 43/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Algorithm (1/4)

[0] Initializationsevaluate f at Y0 = y00 , y10 , . . . , yn0 0 < γs < 1, −1 < δic < 0 < δoc < δr < δe

k ← 0

[1] Iteration k[1.1] Order

order the points in Yk such thatf0 = f(y0k) ≤ f1 = f(y1k) ≤ . . . ≤ fn = f(ynk )

[1.2] Reflectconstruct yc and yr and evaluate fr = f(yr)if (f0 ≤ fr < fn−1)

Yk+1 ← y0k, y1k, . . . , yn−1k , yr

k ← k + 1goto [1]

. . .

MTH8418: Traditional methods 44/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Algorithm (2/4)

[1] Iteration k. . .

[1.3] Expand if (fr < f0)Construct ye and evaluate fe = f(ye)if (fe ≤ fr)

Yk+1 ← y0k, y1k, . . . , yn−1k , ye

otherwiseYk+1 ← y0k, y1k, . . . , y

n−1k , yr

k ← k + 1goto [1]

. . .

MTH8418: Traditional methods 45/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Algorithm (3/4)

[1] Iteration k. . .

[1.4] Contract (fr ≥ fn−1)Outside contraction if (fr < fn)

construct yoc and evaluate foc = f(yoc)if (foc ≤ fr)

Yk+1 ← y0k, y1k, . . . , yn−1k , yoc

k ← k + 1, goto [1]otherwise goto [1.5]

Inside contraction if (fr ≥ fn)construct yic and evaluate f ic = f(yic)if (f ic < fn)

Yk+1 ← y0k, y1k, . . . , yn−1k , yic

k ← k + 1, goto [1]. . .

MTH8418: Traditional methods 46/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Algorithm (4/4)

[1] Iteration k. . .

[1.5] Shrink (fr ≥ fn−1)construct the n points ysi , i = 1, 2, . . . , nevaluate f at these n pointsYk+1 ← y0k, ys1 , ys2 , . . . , ysnk ← k + 1goto [1]

MTH8418: Traditional methods 47/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Other details

I Stopping criteria:

I Minimal diameter size

I Number of evaluations

I Number of shrinks

I etc.

I Number of evaluations at each iteration:

I 1 if the iteration is a reflection

I 2 if the iteration is an expansion or a contraction

I n+ 2 if the iteration is a shrink

MTH8418: Traditional methods 48/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Execution example

See the animation NM animation.pdf taken from Wikipedia, onthe Rosenbrock function f(x, y) = (1− x)2 + 100(y − x2)2

MTH8418: Traditional methods 49/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Convergence (1/2)

I It is a heuristic. No convergence theory. But at the time, themethod proved to be very efficient compared to what wasavailable before

I From [Swann, 1972]: “Although the methods described abovehave been developed heuristically and no proofs ofconvergence have been derived for them, in practice they havegenerally proved to be robust and reliable”

I With time, deficiencies appeared, including the famousMcKinnon family of strictly convex functions, for which NMexecutes an infinite sequence of inside contractions and failsto converge [McKinnon, 1998]

MTH8418: Traditional methods 50/55

LHS Coordinate Search Nelder-Mead References

Nelder-Mead: Convergence (2/2)

I Modified versions of the NM method are in use nowadays.They correct its known deficiencies with:I Use of a sufficient decrease instead of a simple decreaseI Restart when the simplex becomes ill-conditionedI Lexicographic rules to break the ties when ordering

I However NM remains an “old” method and modernalgorithms should be preferred

I But NM is still very popular amongst engineers

I NM is still analyzed to understand why it works so well insome instances

MTH8418: Traditional methods 51/55

LHS Coordinate Search Nelder-Mead References

Latin Hypercube Sampling

The Coordinate Search method

The Nelder-Mead method

References

MTH8418: Traditional methods 52/55

LHS Coordinate Search Nelder-Mead References

References I

Audet, C. and Dennis, Jr., J. (2006).

Mesh adaptive direct search algorithms for constrained optimization.SIAM Journal on Optimization, 17(1):188–217.(MADS).

Conn, A., Scheinberg, K., and Vicente, L. (2009).

Introduction to Derivative-Free Optimization.MOS-SIAM Series on Optimization. SIAM, Philadelphia.(NM description, Section 8.1).

Fermi, E. and Metropolis, N. (1952).

Numerical solution of a minimum problem.Los Alamos Unclassified Report LA–1492, Los Alamos National Laboratory, Los Alamos, USA.(Coordinate Search).

Hooke, R. and Jeeves, T. (1961).

“Direct Search” Solution of Numerical and Statistical Problems.Journal of the Association for Computing Machinery, 8(2):212–229.(H&J).

Kolda, T., Lewis, R., and Torczon, V. (2003).

Optimization by direct search: New perspectives on some classical and modern methods.SIAM Review, 45(3):385–482.(GSS).

MTH8418: Traditional methods 53/55

LHS Coordinate Search Nelder-Mead References

References II

McKay, M., Beckman, R., and Conover, W. (1979).

A comparison of three methods for selecting values of input variables in the analysis of output from acomputer code.Technometrics, 21(2):239–245.(Latin Hypercube Sampling).

McKinnon, K. (1998).

Convergence of the Nelder-Mead simplex method to a nonstationary point.SIAM Journal on Optimization, 9(1):148–158.(NM failure).

Nelder, J. and Mead, R. (1965).

A simplex method for function minimization.The Computer Journal, 7(4):308–313.(NM).

Swann, W. (1972).

Direct Search Methods.In Murray, W., editor, Numerical Methods for Unconstrained Optimization, pages 13–28. Academic Press,London and New York.(Remark on NM).

MTH8418: Traditional methods 54/55

LHS Coordinate Search Nelder-Mead References

References III

Torczon, V. (1997).

On the convergence of pattern search algorithms.SIAM Journal on Optimization, 7(1):1–25.(GPS).

Wright, M. (2012).

Nelder, Mead, and the Other Simplex Method.In Grotschel, M., editor, Documenta Mathematica Extra Volume: Optimization Stories, pages 271–276.Journal der Deutschen Mathematiker-Vereinigung Gegrundet 1996, Berlin.(NM history).

MTH8418: Traditional methods 55/55