MULTI VARIABLE OPTIMIZATION

Min f(x1, x2, x3,----- xn)

UNIDIRECTIONAL SEARCH

- CONSIDER A DIRECTION S

sxx rr αα +=)(

- REDUCE TO

)(αfMin- SOLVE AS

A SINGLE VARIABLE PROBLEM

srMin Point

Uni directional search (example)

Min f(x1, x2) = (x1-10)2 + (x2-10)2

S = (2, 5) (search direction)

X = (2, 1) (Initial guess)

DIRECT SERACH METHODS

- SEARCH THROUGH MANY DIRECTIONS- FOR N VARIABLES 2N DIRECTIONS-Obtained by altering each of the n values and taking all combinations

EVOLUTIONARY OPTIMIZATION METHOD

- COMPARE ALL 2N+1 POINTS & CHOOSE THE BEST.- CONTINUE TILL THERE IS AN IMPROVE MENT - ELSE DECREASE INCREAMENT

STEP 1: x0 = INITIAL POINT

= STEP REDUCTION PARAMETER FOR EACH VARIABLE

= TERMINATION PARA METER

i∆

∈

2&

2;2/:4)12(:3

2/2

:2

0

0

GOTOxxELSE

GOTOxxIFSTEPMinxSTEP

xxPOINTSCREATEELSE

STOPIFSTEP

ii

N

iii

N

=

∆=∆−=

+=

∆±=

<∈∆

Hooke Jeeves pattern search

• Pattern Search ---– Create a set of search directions iteratively– Should be linearly independent

• A combination of exploratory and pattern moves– Exploratory – find the best point in the vicinity of the

current point– Pattern – Jump in the direction of change, if better then

continue, else reduce size of exploratory move and continue

Exploratory move

• Current solution is xc; set i = 1; x = xc

• S1: f = f(x), f+ = f(xi + ∆i), f- = f(xi - ∆i)• S2: fmin = min (f, f+, f-); set x corresponding

to fmin

• S3: If i = N, go to 4; else i = i + 1, go to 1• S4: If x ≠ xc, success, else failure

Pattern Move

• S1: Choose x(0), ∆I, for I = 1, 2, …N, ε, and set k = 0

• S2: Perform exploratory move with xk as base point; – If success, xk+1 = x, go to 4 else goto 3

• S3: If |∆| < ε, terminate– Else set ∆i = ∆i / α --- ∀i, go to 2

Pattern Move (contd)

• S4: k = k+1; xpk+1 = xk + (xk – xk-1)

• S5: Perform another exploratory move with xp

k+1 as the base point; Result = xk+1

• S6: If f(xk+1) < f(xk), goto S4– Else goto S3

Example :• Consider the Himmelblau function:

• SolutionStep 1 Selection of initial conditions1. Initial Point : 2. Increment vector : 3. Reduction factor : 4. Termination parameter :5. Iteration counter : k = 0

2221

22

2121 )7()11(),( −++−+= xxxxxxf

Tx )0,0()0( =T)5.0,5.0(=∆

2=α310 −=ε

• Step 2Perform an Iteration of the exploratory move with base point as

Thus we set

The exploratory move will be performed with the following steps

1)0,0()0( === iandxx T

)0(xx =

Steps for the Exploratory moveStep 1 : Explore the vicinity of the variable x1

Calculate the function values at three points

Step 2 : Take the Minimum of above function and corresponding point

Step 3 : As : all variables are not explored Increment counter i=2 and explore second variable.First iteration completed

TTxx )5.0,5.0()( )0(1

)0( =∆+

Tx )0,0()0( =

TTxx )5.0,5.0()( )0(1

)0( −=∆−

81.157))5.0,5.0(( ==+ Tff

170))0,0(( == Tff

81.171))5.0,5.0(( =−=− Tff

1≠i

Step1: At this point the base point is explore the variable x2 and calculate the function values.

Step 2 : fmin = 144.12 and point , x=(0.5,0.5)

Step 3 : As i=2 move to the step 4 of the exploratory moveStep 4 : ( of the Exploratory move )Since the move is success and we set

Tx )0,5.0(=

12.144))5.0,5.0(( ==+ Tff

81.157))0,5.0(( == Tff

62.165))5.0,5.0(( =−=− Tff

cxx ≠Tx )5.0,5.0(=

• As the move is success, set moveto step 4

STEP 4 : We set k=1 and perform Pattern move

Step 5 : Perform another exploratory move as before and with as the base point.

The new point is Set the new point Step 6 : is smaller than Proceed to next step to perform another pattern move

Txx )5.0,5.0()1( ==

TTTTp xxxx )1,1()0,0()5.0,5.0(2))(( )0()1()1()2( =−=−+=

)2(

pxTx )5.1,5.1(=

Txx )5.1,5.1()2( ==12.63)( )2( =xf 12.144)( )1( =xf

STEP 4 : Set k=2 and create a new point

Note: as is better than , a jump along the direction is made, this will take search closer to

true minimum

STEP 5 : Perform another exploratory move to find any better point around the new point.

Performing the move on both variables we have New point

This point is the true minimum point

Tp xxx )5.2,5.2()2( )1()2()3( =−=

)2(x )1(x)( )1()2( xx −

Tx )0.2,0.3()3( =

In the example the minimum of the Hookes-Jeevesalgorithm happen in two iterations : this may not be the case always

Even though the minimum point is reached there is no way of finding whether the optimum is reachedor notThe algorithm proceeds until the norm if the increment vector is small.

STEP 6 : function value at new point

Thus move on to step 412.63)(0)( 23 =<= xfxf

STEP 4 : The iteration counter k = 3 and the new point

STEP 5 : With the new point as base the search is success and x=(4.0,2.0) and thus we set

STEP 6 : The function value is 50, which is larger that the earlier i.e. 0 . Thus we move to step 3

Step 3 : Since we reduce the increment vector

and proceed to Step 2 to perform the iterations

Tp xxx )5.2,5.4()2( )2()3()4( =−=

Tx )0.2,0.4()4( =

ε⊄=∆ 5.0TT )25.0,25.0(2/)5.0,5.0( =−=∆

Step 2 : Perform an exploratory move with the following as current pointThe exploratory move on both the variables is failure and we obtain thus we proceed to Step 3

Step 3 : Since is not small reduce the increment vector and move to Step 2.

The new increment vector is The algorithm now continues with step 2 and step 3

until is smaller than the termination factor. The final solution is with the function

value 0

Tx )0.2,0.3()3( =

Tx )0.2,0.3()3( =

∆

T)125.0,125.0(=∆

Tx )0.2,0.3(* =

∆

POWELL’S CONJUGATE DIRECTION METHOD

For a quadratic function IN 2 VARIABLES- TAKE 2 POINTS x1 & x2 AND- A DIRECTION ‘d’

IF y1 IS A SOLUTION OF MIN &)( 1 dxf λ+y2 IS A SOLUTION OF MIN )( 2 dxf λ+

THEN (y2- y1) IS CONJUGATE TO d

OPTIMUM LIES ALONG (y2- y1)

x2

x1

y1

y2

OR

FROM x1 GET y1 ALONG (1,0)y1 GET x2 ALONG (0,1)x2 GET y2 ALONG (1,0)

TAKE (y2 - y1 )

Alternate to the above method - One point and both coordinate

directions- Can be used to create a pair of conjugate

directions

)( 1x))1,0()0,1(( TT and

))(( 12 yyandd −

• Point obtained by unidirectional search along from the point .

• Point obtained by unidirectional search along from the point .

• Point obtained by unidirectional search along from the point .

The figure shown also follows the Parallel Subspace Property.

This requires Three Unidirectional searches.

)( 1yT)0,1( )( 1x

)( 2xT)1,0( )( 1y

)( 2y)( 2xT)0,1(

Example :

• Consider the Himmelblau function:

• SolutionStep1 : Begin with a pointInitial direction and Step 2: Find minimum along the first search direction.

Any point along that direction can be written as

2221

22

2121 )7()11(),( −++−+= xxxxxxf

Tx )4,0()0( =Ts )0,1()1( = Ts )1,0()2( =

)1()0()( sxx p α+=

Thus the point xp can be written as xp=(α,4)T

Now the two variable function can be expressed in terms of one variable

We are looking for the point which the function value is minimum.

== Following the procedure of numerical differentiation. Using the bounding phase method, the minimum is bracketed in the interval (1,4), and using the golden search method the minimum α*=2.083 with three decimals places of accuracy. Thus x1 = (2.083,4.00)T

222 )9()7()( ++−= αααF

Similarly find the minimum point along the second search direction. A general point on the line is

Using similar approach as said earlier we have α*=-1.592 x2 = (2.083,2.408)T

From the above point perform a unidirectional search along the first search direction and obtain the minimum point

x3 = (2.881,2.408)T

Tsxx ))4(,083.2()()( 21 +=+= ααα

Step 3 : According to the parallel subspace property, we find the new conjugate direction

Step 4 : the magnitude of search vector d is not small. Thus the new conjugate search direction are

This completes one iteration of Powell’s conjugate direction method.

Ts )0,1()2( =

T

TTs

)894.0,448.0(

)592.1,798.0(/)592.1,798.0()1(

−=

−−=

Step 2 : A single variable minimization along the search direction s(1) from the point x(3)=(2.881,2.408)T results in the new point x = (3.063,2.045)

One more unidirectional search along the s2 from the point x4 results in the point x5. Another minimization along s1 results in x6

Step 3 : the new conjugate direction is

The unit vector along this direction is ( 0.816,-0.578)

Txxd )039.0,055.0()( )4()6( −=−=

Step 4 : The new pair of conjugate search direction are

The search direction d ( before normalizing ) may be considered to be small and therefore the algorithm may be terminated

Ts )039.0,055.0()2( −=Ts )894.0,448.0()1( −=

EXTENDED PARALLEL SUBSPACE PROPERTY.

Let us assume that the point is found after unidirectional searches along each of m ( < N ) conjugate directions from a chosen point and similarly, the point is found after unidirectional searches along each of m conjugate directions from another point . The vector is the conjugate to all m search directions.

1x

2x

2y

)( 12 yy −

1y

• ALGORITHM Step 1 : Choose starting point

And a set of N linearly independent directionStep 2 : Minimize along N unidirectional search directions

using the previous minimum point to begin the next search.

Step 3 : Form a new conjugate direction “d ” using the extended parallel subspace property.

Step 4 : If || d || is small or search directions are linearly dependent,

TERMINATE,Else replace starting point for N directions, set s(1)=d /||d||And go to Step 2

FOR N VARIABLES

STEP 1: TAKE x0 & N LINEARLY INDEPENDENT DIRECTIONS s1, s2, s3, s4,------- sN si = ei

STEP 2: - MINIMIZE ALONG N UNI-DIRECTIONAL SEARCHDIRECTIONS,USING PREVIOUS BEST EVERY TIME

- PERFORM ANOTHER SEARCH ALONG s1

STEP 3: FROM THE CONJUGATE DIRECTION “d”

STEP 4: IF IS SMALL TERMINATE

ELSE

d

2/

&1

1

GOTOdds

ss jjj

=

∀= −

Powell’s method with N variables

• Start from x1

• Get y1 by doing search along s1

• Find y2 by doing search along s2, s3, s4 .. sn, s1

• (y2 – y1) is conjugate to s1

• Replace sn by (y2-y1) and start same procedure starting from s2

GRADIENT BASED METHODS• The methods exploit the derivative information

of the function and are faster.• Cannot be applied to problems where the

objective function is discrete or discontinuous.• Efficient when the derivative information is easily

available• Some algorithms require first order derivatives

while some require first and second orderderivatives of the objective function.

• The derivatives can be obtained by numerical computations

• Methods in Gradient Search.– Decent Direction– Cauchy’s (steepest decent) method– Newton’s Method– Marquardt’s Method– Conjugate gradient method– Variable-metric method

• By definition , the first derivative at any point represents the direction of the maximum increase of the function value.

• If we are interested in finding a point with the minimum function value, ideally we should be searching along the opposite to the first derivative direction, that is , we should search along

• Any search made in this direction will have smaller function value.

)( )(txf∇)( )(tx

)( )(txf∇−

• DECENT DIRECTIONA search direction is a decent direction at a point if the condition is satisfied in the vicinity of the point

It can also be proved by comparing function values at two points along any decent direction.

The magnitude of the vector for a decent direction specifies how decent the search direction is.The above statement can be explained with the help of an example on the next side

)( )(td)( )(tx 0).( )()( ≤∇ tt dxf

)( )(tx

)()( ).( tt dxf∇)( )(td

• If is used, the quantity

is maximally negative

Thus the search direction is called the steepest decent direction

Note :

)(. )()(2

tt xfd −∇= )()( ).( tt dxf∇

)( )(txf∇−

⎥⎦

⎤⎢⎣

⎡∂∂

−−−−−∂∂

∂∂

∂∂

=∇nxxxx 321

Example• Consider the Himmelblau function:

We would like to determine whether the direction at the point is a decent

direction or not. Refer to the figure below

2221

22

2121 )7()11(),( −++−+= xxxxxxf

Ttd )0,1()( )( = Ttx )1,1()( )( =

• It is clear from the figure that if we move locally along from the point will reduce the function value

• Investigate this aspect by calculating the derivative at the point.

• Derivative as calculated numerically is

• Taking the dot product we obtain and

• The above is a negative quantity, thus the search direction is a Decent direction

)( )(td )( )(tx

)( )(txf∇

Ttxf )38,46()( )( −−=∇

4601)38,46()()( )( −=⎟

⎠⎞

⎜⎝⎛−−=∇ TTt tdxf

)( )(tx)( )(txf∇

• The amount of non-negativity suggests the extentof decent in the direction

• If the search direction is used, the magnitude of the above dot product is

• The direction (46,38) or (0.771,0.637) is more descent than

• The above direction is the steepest direction at the point x(t)

TTtxftd )38,46()()( )( −−=−∇=

560,33846)38,46( −=⎟

⎠⎞

⎜⎝⎛−− T

• In nonlinear function the steepest decent direction at any point may not exactly passthrough the true minimum.

• The steepest decent direction is a direction which is a local best direction

• It is not guaranteed that moving along the steepest decent direction will always take the search closer to the true minimum

CAUCHY’S ( STEEPEST DECENT) METHOD

• Search direction used is the negative of the gradient at any particular point x(t) :

• As the direction gives the maximum decent in function values, it is also known as Steepest Decent Methods

• The algorithm guarantees improvement in the function value at every iteration

)( )(kk xfs −∇=

• Algorithm in brief ;• At every iteration ;

– To find the minimum point along direction– Compute derivative at current point– Perform unidirectional search in the negative

to this derivative direction• The minimum point becomes the current point

and the search continues from this point ;

• Algorithm continues until a point having a small enough gradient vector is found.

Algorithm in StepsSTEP 1: CHOOSE M (MAX. NO. OF ITERATIONS)

STEP 2: CALCULATE

STEP 3: IF TERMINATE

IF TERMINATE

STEP 4: UNIDIRECTIONAL SEARCH USING 2

1

021

)(

)(

0,,,

∈

≥

≤∈∇

∇

=∈∈

Mk

xf

xf

kx

k

k

1

))(()(

1

1

1

+=

≤∈−

∇−=

+

+

kk

x

xx

xfxfxf

k

kk

kkk αMIN.

STEP 5: IF

ELSE GOTO STEP 2

- METHOD WORKS WELL WHEN xk IS FAR FROM x* (OPTIMUM)- IF POINT IS CLOSE TO x* THEN CHANGE IN GRADIENT VECTOR IS

VERY SMALL.- OTHER METHODS USE VARIATION

-SECOND DERIVATIVES(NEWTON’S)-COMBIMATION-CONJUGATE GRADIENT METHOD

Example• Consider the Himmelblau function:

Step 1: Choose large value of M for proper convergenceM ≈100 is normally chosen.

Initial conditions :M = 100x(0) = (0,0)T

ε1= ε2=10-3

k=0

2221

22

2121 )7()11(),( −++−+= xxxxxxf

• Step 2 : The derivative at the initial point x0 is calculated and is found to be (- 14, - 22)T

• Step 3 : As the magnitude of the derivative is not small and k = 0 < M = 100. do not terminate proceed to step 4

• Step 4 : perform a line search form x(0) in the direction such that the function value is minimum.Along that direction, any point can be expressed by fixing a value for the parameter α 0 in the equation.

Using Golden section search in the interval (0,1)α 0*= 0.127 minimum point along the direction is x1 = (1.788,2.810)T

• Step 5 : As x1 and x0 are quite different, we do not terminate, but go back to step 2, this completes one iteration.

)( )0(xf∇−

( )Txfxx 00)0(00 22,14)( ααα =∇−=

• Step 2: The derivative vector at this point is• ( - 30.707, 18.803 )T• Step 3 : The magnitude of the derivative vector is

not smaller than ε1, thus, we continue with step 4.• Step 4 : Another unidirectional search along• ( 30.707,-18.803) from the point

x1=(1.788,2.810) using the golden section search finds the new point x2=(23.008,1.99) with a function value equal to 0.018

• Continue the process until the termination criteria is reached

Penalty function approach

• Transformation method- convert to a sequence of unconstrained

problems.• Give penalty to (violated) constraints.• Add to objective function.• Solve.• Use result as starting point for next iteration.• Alter penalties ands repeat.

• MinimiseSubjected toPenalty =

2221

22

21 )7()11( −++−+ xxxx

026)5( 22

21 ≥−+− xx

222

21 026)5(1.0 >≥−+−< xx

1x

2x

Minimum point

Infeasible regionFeasible region

Process 1. Choose2. Form modified objective function

3. Start with . Find so as to minimize P.(use )

4. If Terminate.

5. Else k=k+1; go to step 2.

.,,, 21 ΩRεε

))(),(,()(),( kkkkkk xhxgRxfRxP Ω+=

kx 1+kx

1ε

211 ),(),( ε<− −+ kkkk RxPRxP

,1 cRRk =+

• At any stage minimize P(x,R) = f(x)+Ω(R,g(x),h(x))R = set of penalty parametersΩ= penalty function

Types of penalty function• Parabolic penalty

- for equality constraints- only for infeasible points

• Interior penalty functions- penalize feasible points

• Exterior penalty functions- penalize infeasible points

• Mixed penalty functions- combination of both

2Rh(x)=Ω

• Infinite barrier penalty

- inequality constraints.- R is very large.- Exterior.

• Log penaltyΩ=-R ln[g(x)] -inequality constraints- for feasible points- interior.-initially large R.- larger penalty close to border.

∑=Ω )(xgR j

• Inverse penalty

- interior.- larger penalty close to border.- initially large R

• Bracket order penalty

- <A>=A if A<0.- exterior.- initially small R.

⎥⎦

⎤⎢⎣

⎡=Ω)(

1xg

R

2)( >< xgR

Direct search

• Variable elimination- for equality constraints- Express one variable as a function of others

and- Eliminate one variable - Remove all equality constraints

Complex search

- generate a set of points at random- If a point is infeasible reflect beyond

centroid of remaining points- Take worst point and push towards centroid

of feasible points.

Complex Search Algo

• S1: Assume a bound in x (xL, xU), a reflection parameter α, and ε & δ

• S2: Generate a set of P (= 2N) initial points – For each point

• Sample N times to determine xi(P), in the given

bound• if x(P) is infeasible, calculate xxx (centroid) of the

current set of points and set xP = xP+1/2(xxx-xP) until xP is feasible.

• If xP is feasible, continue till u get P points

Complex Search Algo (contd)

• S3: Reflection step– select xR such that f(xR) = max (f(xP)) = Fmax

– calculate x,x,x, centroid of remaining points– xm = xxx + α (xxx - xR)– If xm is feasible and f(xm) > Fmax, retrtact half

the distance to xxx, and continue till f(xm) < Fmax

– If xm is feasible and f(xm) < Fmax, Go to S5– If xm is infeasible, Goto S4

Complex Search Algo (contd)

• S4: Check for feasibility of the solution– For all i, reset violated variable bounds

• if xim < xi

L, xim = xi

L

• if xim > xi

U, xim = xi

U

– If the resulting xim is infeasible, retract half

the distance to the centroid, repeat till xm is feasible

Complex Search Algo (contd)

• S5: Replace xR by xm, check for termination– fmean = mean of f(xP), – xmean = mean (xP)

δ

ε

≤−

≤−

∑

∑

pmean

p

pmean

p

xx

fxf

2

2))((

Complex Search Algo (contd)

Characteristics of complex search

• For complex feasible region• If the optimum is well inside the search space,

the algo is efficient• Not so good if the search space is narrow, or

the optimum is close to the constraint boundary