Miroslav KrsticUniversity of California, San Diego

Extremum Seeking Feedback Tools forReal-Time Optimization

Large Scale Robust Optimization, Sandia, August 2005

1

Example of Single-Parameter Maximum Seeking

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

2

Example of Single-Parameter Maximum Seeking

3

Theory

•History

•Single parameter ES, how it works, and stability analysis byaveraging

•ES with plant dynamics and compensators for performanceimprovement

• Internal model principle for tracking parameter changes

•Multi-parameter ES

•Slope seeking

•ES in discrete time

•Limit cycle minimization via ES

4

Applications

•Anti-skid braking

•Compressor instabilities in jet engines

•Combustion instabilities

•Flow separation control in diffusers

•Thermoacoustic coolers

•Automotive engine mapping

•Beam matching in particle accelerators

•Formation flight

•Bioreactors

•PID tuning

•Autonomous vehicles without position sensing

5

History• Leblanc (1922) - electric railways

• Early Russian literature (1940’s) - many papers

• Drapper and Li (1951) - application to IC engine spark timing tuning

• Tsien (1954) - a chapter in his book on Engineering Cybernetics

• Feldbaum (1959) - book Computers in Automatic Control Systems

• Blackman (1962 chap. in book by Westcott) - nice intuitive presentation of ES

• Wilde (1964) - a book

• Chinaev (1969) - a handbook on self-tuning systems

• Papers by[Morosanov], [Ostrovskii], [Pervozvanskii], [Kazakevich], [Frey, Deem,and Altpeter], [Jacobs and Shering], [Korovin and Utkin] - late 50s - early 70’s

• Meerkov (1967, 1968) - papers with averaging analysis

• Sternby (1980) - survey

• Astrom and Wittenmark (1995 book) - rates ES as one of the most promisingareas for adaptive control

6

Recent Developments

• Krstic and Wang (2000, Automatica) - stability proof for single-parametergeneral dynamic nonlinear plants

• Choi, Ariyur, Wang, Krstic - discrete-time, limit cycle minimization, IMC forparameter tracking, etc.

• Rotea; Walsh; Ariyur - multi-parameter ES

• Ariyur - slope seeking

• Tan, Nesic, Mareels (2005) - semi-global stability of ES

• Other approaches: Guay, Dochain, Titica, and coworkers; Zak, Ozguner, andcoworkers; Banavar, Chichka, Speyer; Popovic, Teel; etc.

• Applications outside of my group:o Electromechanical valve actuator (Peterson and Stephanopoulou)o Artificial heart (Antaki and Paden)o Exercise machine (Zhang and Dawson)o Chemical reactors and petroleum processing (several research groups)o Shape optimization for magnetic head in hard disk drives (UCSD)

7

ES Book

8

Basic Extremum Seeking - Static Map

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

y = output to be minimizedf * = minimum of the mapf " = second derivative (positive - f (θ) has a min.)θ* = unknown parameter

θ̂ = estimate of θ*

k = adaptation gain (positive) of the integrator 1s

a = amplitude of the probing signalω = frequency of the probing signal

h = cut-off frequency of the "washout filter" ss + h

+/× = modulation/demodulation

9

How Does It Work?

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

Estimation error: θ = θ* − θ̂

y = f * + a

2 f "4

+f "2θ 2 − af " θ sinωt + a

2 f "4cos2ωt

y ≈ f * + a

2 f "4

+f "2θ 2 − af " θ sinωt + a

2 f "4cos2ωt

Loc. Analysis - neglect quadratic terms:

10

How Does It Work?

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

y ≈ f * + a

2 f "4

− af " θ sinωt + a2 f "4cos2ωt

ss + h

[y] ≈ f * + a2 f "4

− af " θ sinωt + a2 f "4cos2ωt

11

How Does It Work?

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

ξ = sinωt s

s + h[y] ≈ −af " θ sin2ωt + a

2 f "4cos2ωt sinωt

Demodulation:

ξ ≈ −

a2 f "4θ +

a2 f "4θ cos2ωt + a

2 f "8

sinωt − sin 3ωt( )

12

How Does It Work?

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

θ = − ̂θthen

Since θ = θ* − θ̂

13

How Does It Work?

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

θ ≈ks

−a2 f "4θ +

a2 f "4θ cos2ωt + a

2 f "8

sinωt − sin 3ωt( )⎡

⎣⎢

⎤

⎦⎥

high frequency terms - attenuated by integrator

14

How Does It Work?

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

θ ≈ −

ka2 f "4θ

Stable because k,a, f " > 0

15

Stability Proof by Averaging

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

θ = θ* − θ̂

e = f * − hs + h

y[ ]τ =ωt

ddτθ =

kω

f "2θ − asinτ( )2 − e⎛

⎝⎜⎞⎠⎟sinτ

ddτ

e = hω

−e − f "2θ − asinτ( )2⎛

⎝⎜⎞⎠⎟

Full nonlinear time-varying model:

16

Stability Proof by Averaging

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

θ = θ* − θ̂

e = f * − hs + h

y[ ]τ =ωt

ddτθav = −

kaf "2ωθav

ddτ

eav =hω

−eav −f "2θ 2av +

a2

2⎛⎝⎜

⎞⎠⎟

⎛

⎝⎜⎞

⎠⎟

Average system:

θav = 0

eav = −a2 f "4

Average equilibrium:

17

Stability Proof by Averaging

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

θ = θ* − θ̂

e = f * − hs + h

y[ ]τ =ωt

Jav =−kaf "2ω

0

0 −hω

⎡

⎣

⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥

Jacobian of the average system:

18

Stability Proof by Averaging

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

θ = θ* − θ̂

e = f * − hs + h

y[ ]τ =ωt

θ2π /ω (t) + e2π /ω (t) −a2 f "4

≤O 1ω

⎛⎝⎜

⎞⎠⎟,→→∀t ≥ 0

Theorem. For sufficiently large ω there exists a unique exponentially stableperiodic solution of period 2π/ω and it satisfies

Speed of convergence proportional to 1/ω, a2, k, f "

19

Stability Proof by Averaging

f (θ) = f * + f ''2

θ −θ*( )2

asinωt sinωt

−ks

ss + h+ ×

y

ξ

θ

θ̂

f * θ* Plant

θ = θ* − θ̂

e = f * − hs + h

y[ ]τ =ωt

y − f * → f "O 1ω 2 + a

2⎛⎝⎜

⎞⎠⎟

Output performance:

Extremum-Seeking Control of Combustion Instabilities

Andrzej Banaszuk

United Technologies Research Center, E. Hartford, CT, U.S.A.

Acknowledgements:

Youping Zhang, Numerical Technologies

Kartik Ariyur, Honeywell

Miroslav Krstic, UCSD

Jeff Cohen, Pratt & Whitney

Clas Jacobson, UTRC

Workshop on Real-Time Optimization by Extremum-Seeking Control, ACC 05

Partially sponsored by AFOSR

Thermo-acoustic instability

Fluctuating acoustic pressure and velocity driven by unsteady heat release

Acoustics

Combustion

Coupling of acoustics with heat release results in pressure oscillations

Fuel

CombustorFuel/air mixing nozzle

Air

Fluctuating heat release driven by fluctuating velocity

•Power generation: pressure oscillations prevent low emission operation

•Military aeroengines: afterburner screech and rumble limit performance, passive fixes increase weight, maintenance costs

• Rockets: passive fixes increase weight, limit payload

Thermo-acoustic instabilities affect cost and performance of gas turbines and rocket engines

Fuel/Air Ratio

01020304050N

Ox 60

708090

100

0.400.420.440.460.480.500.520.540.560.580.60

NOx

CombustionInstability

25MW Pratt & Whitney FT8 gas turbine engine

Active fuel control can suppress oscillations

Fuel

CombustorFuel/air mixing nozzle

Air Acoustics

Combustion

Control

Fuel valve

Controller

Pressure measurement

0 200 400 600 800 10000

2

4

6

8

10

12

14

Am

plit

ude

(psi

)

Uncontrolled

Controlled

6X reduction in amplitudeDecrease in NOX & CO emissions

Experiment in UTRC 4MW Single Nozzle Rig

• Numerous demonstrations at universities and industrial rigs

• Rolls-Royce demonstrated control in afterburner

• Siemens implemented control in 260MW gas turbine engine

• Phase-shifting control effective, but no models to guide selection of parameters

Summary of results: two extremum seeking schemes were successfully demonstrated on UTRC 4MW single nozzle rig in August 1998.

Excellent performance at high power Improved performance at low power

0 5 10 15 20 25 30−150

−100

−50

0

50

file r54p18 ,Date & Time of Capture: Thu Aug 13 11:57:35 1998

contr

ol ph

ase

0 5 10 15 20 25 3001234567

Performance measure 1 = 8.0355, Performance measure 2 = 20.6

magn

itude

of bu

lk mo

de

0 5 10 15 20 25 30

−50

0

50

100

file r56p37 ,Date & Time of Capture: Wed Aug 19 12:00:34 1998

contr

ol ph

ase

0 5 10 15 20 25 300

2

4

6

8

10

Performance measure 1 = 27.9418, Performance measure 2 = 79.3275

magn

itude

of bu

lk mo

de

−200 −150 −100 −50 0 50 100 150 2001

2

3

4

5

6

7

control phase

pres

sure

and i

ts ma

gnitu

de

file r54p18 ,Date & Time of Capture: Thu Aug 13 11:57:35 1998

−200 −150 −100 −50 0 50 100 150 2004

5

6

7

8

9

10

11

control phase

pres

sure

and i

ts ma

gnitu

de

file r56p37 ,Date & Time of Capture: Wed Aug 19 12:00:34 1998

Control phase in degrees

Optimal level

Uncontrolled level

Optimal level

Phase-magnitude map

Uncontrolled level

Optimal phase

Time in seconds Time in seconds

Control phase in degrees Control phase in degrees

Pressure magnitude in psi

Pressure magnitude in psi

1

Formation Flight Optimization

(Acknowledgement: Paolo Binetti)

2

Benefits of Formation Flight

• Reduction in power demand

• Alleviation of airspace congestion

Obstacles to Attainment

• Sensitivity to positioning

• Difficulty in precise measurements

• Absence of precise modeling

6

Physics of Aircraft Wakes and Wake Modeling

• Wake structure—modeled as two NASA counter-rotating vortices

• Wake transport—a modeling uncertainty

• Wake decay—neglected

( ) ( ) bbbW

rrr

rrV red

redc 4! ,

"V ,

!2 22

2

# ==Γ+

Γ=∞

5

Selection of the Problem

Why the C-5?

• Large savings in fuel consumption

• Representative of large transports

• They will be in service for the next 40 years

• Availability of wake data

13

Extremum Seeking for Formation Flightz!

Aircraft with formation autopilot

in the wake

sk 1−

1hss+

sk 2−

2hss+

zref

yref+

Estimator

Objective:-

( )ta 11 "sin ( )11"sin ϕ−t

( )22"sin ϕ−t( )ta 22 "sin

w!

18

Simulation for brief CAT Encounter …

Extremum Seeking Control of Thermoacoustic Coolers

Mario RoteaJune 7, 2005

Purdue Universityrotea@purdue.edu

REF. Extremum Seeking Control of Tunable Thermo-acoustic Coolers, Y. Li, M. Rotea, G. Chiu, L. Mongeau, I. Paek, IEEE Transactions on Control Systems Technology, Vol. 13, No. 4, July 2005

Thermoacoustic CoolingElectric energy Acoustic energy Heat pumping

Standing sound wave creates the refrigeration cycle

Resonance tubeStackHot-end heat

exchangers

Electro-dynamic driver

Cold-end heat exchangers

Pressurized He-Ar mixture

32

Heat Pumping

1-2: adiabatic compression and displacement2-3: isobaric heat transfer (gas to solid)3-4: adiabatic expansion and displacement4-1: isobaric heat transfer (solid to gas)

Solid surface (stack plate)

Gas particle in a standing wave

1

4

QLQH

Thermoacoustic Cooling• Benefits

– Environmentally benign (inert gases)– Simple, easy to maintain configuration

• Limitations– Very hard to model ! hard to tune key parameters (driver, stack

location, duct geometry) for best performance• Existing prototypes

– Acoustic Stirling Heat Engine (Los Alamos National Lab)– Triton 10 kW refrigerator (Penn State)– Space Thermo-acoustic Refrigerator (NASA)– Purdue 300 W standing wave unit– Miniature thermo-acoustic cooler (Rockwell Science Center)

Tunable Thermoacoustic Cooler

Moving piston (varying resonator’s stiffness)

Heat exchangers

Helmholtz resonator

Neck (mass) Volume (stiffness)

Electro-dynamicDriver

Performance and efficiency sensitive to variations in – Piston position (acoustic impedance)– Driver frequency

Tuning Variables

Extremum Seeking Controller

PDIntegratorLPF + +sin( )x x xa t! "#sin( )xt!

PDIntegratorLPF + +

sin( )f f fa t! "#sin( )f t!

HPF

11fT s #Tunable

Cooler

CoolingPower

Calculation 11xT s #

POS Command

FREQ Command

cQ!

ESC Parameters– High pass filter (HPF)– Low pass filters (LPF)– Dither frequencies, amplitudes

and phases (!x,ax,"x; !x,ax,"x)– PD gains (4 gains)

(Tuning mechanism)

Experiment – Varying Operating Condition

0 50 100 150 200 250 300 350 4000

50

100

Coo

ling

Pow

er (W

att)

0 50 100 150 200 250 300 350 4004

6

8

10

12

Pis

ton

Pos

ition

(in)

0 50 100 150 200 250 300 350 400140

145

150

155

Driv

ing

Freq

uenc

y (H

z)

0 50 100 150 200 250 300 350 40020

40

60

80

100

Time (sec)

Flow

Rat

e (m

l/s)

Cold SideHot Side

ESC ON

Flow Rate Change

ESC tracks optimum after cold-side flow rate is increased

Extremum-Seeking Control of Flow Separation in a Planar Diffuser

Andrzej Banaszuk

United Technologies Research Center, E. Hartford, CT, U.S.A.

Acknowledgements:

Satish Narayanan, UTRC: experiment

Youping Zhang, Numerical Technologies: algorithm

Workshop on Real-Time Optimization by Extremum-Seeking Control, ACC 05

Partially sponsored by AFOSR

Objective of pressure recovery control

Control effortPerformance

221

)(inlet

U

pptC inletexit

p !"

#HU

hutC

inlet

neck

2

2

)( #$

H

h

inletp exitp

neckuinlet

U

$C

Cp bang

buck=

Speaker command

Experimental Setup

% & 14o% & 18o

Pressure recovery as function of diffuser angle (no control)

%

no stall

•Optimum uncontrolled performance•Insignificant improvement with control

fully-developed2D stall jet flow

unsteady stall

•Poor uncontrolled performance•Significant improvement with control

)(tCpUTRC diffuser rig•fully turbulent BL•40,000 < ReH < 140,000•Re%e > 300; M < 0.1•Actuation: C$ ~ 0.001

Need: control algorithm to optimize performance

-150 -100 -50 0 50 100 15010

15

20

25

30

35

40

45

50

0.05

0.05

0.1

0.1

0.1

0.15

0.15

0.2

Frequency f

Phase !

+

-150 -100 -50 0 50 100 15010

15

20

25

30

35

40

45

50

-0.05

0

0

0

0.05

0.05

0.05

0.1

0.1

0.15

Frequency f

Phase !

+

Optimal operating point: f=31Hz, ! =60, Cp =0.21 Optimal operating point: f=36Hz, ! =60, Cp =0.16

U = 30m/sU = 20m/s

Objective:•Optimize performance without exhaustive search

Challenges:•Noisy measurement•Flow transients•Keeping up with operating condition change

Two frequency control law: U(t)=A1*(sin(2*"*f*t) + sin(2*"*2f*t-!))

Adaptive control used to optimize performance

Speaker command

Pressure recoverycalculation

Pressure recoverycalculation

Multi-frequency forcing

Multi-frequency forcing

Extremum-seeking algorithm

Extremum-seeking algorithm

Adjustable parameters )2sin()(1

ii

N

ii tfAtU !" #$%

$NifA iii ..1,,, $!

)(tCp

Filter noise, wait for transient to settle, adapt parameters to improve performance

Excite multiple vortices, explore their interactions

Measure performance

consttC $)(&

Automatic Control Parameter Tuning to Optimum Values

On-line optimization of pressure recovery using extremum-seeking algorithm demonstrated.

0 10 20 30 40 50 600

0.2

0.4

0 10 20 30 40 50 600

20

40

60

Frequency

0 10 20 30 40 50 60-200

0

200

Control phase

Time in seconds

•Mean pressure recovery, control frequency, and phase in four independent adaptive control experiments. •The control frequency and phase initialized away from the optimal values.

Transient behaviorOptimal pressure recovery reached & held

-150 -100 -50 0 50 100 15010

15

20

25

30

35

40

45

50

0.05

0.05

0.1

0.1

0.1

0.15

0.15

0.2

Phase

Fre

quen

cy

Optimal pressure recovery:Cp =0.21

optimal control parametersf=31Hz! =60

Dynamics determines transient time!

Pressure recovery

Autonomous Vehicle Controlvia Extremum Seeking

Antranik A. Siranosian and Miroslav Krstic

2

Model

ω ,v

Unicycle Dynamics

x = vcos(θ)y = vsin(θ)θ =ω

Unicycle Inputs

peak) itsat output (from leUnobservab

and ) inputs (from blencontrollalinearly U is System

f(x,y)

v,w

Unicycle

3

Control InputsUnicycle

4

Block DiagramUnicycle

5

Simulation Results – TrajectoryUnicycle

6

Simulation Results – ES ValueUnicycle

BEAM MATCHING IN PARTICLE ACCELERATORS

Eugenio Schuster

Complex Control Systems LaboratoryMechanical Engineering and Mechanics

Workshop: Real Time Optimization by Extremum Seeking ControlAmerican Control Conference 2005

June 7, 2005

LEHIGHU N I V E R S I T Y

PARTICLE ACCELERATORS

The Spallation Neutron Source (SNS), being built in Oak Ridge, Tennessee, by the U.S. Department of Energy with a cost of $1.4 billion, is the most intense accelerator-based neutron source in the world.

SPALLATION NEUTRON SOURCE

BEAM MATCHING CHANNEL

! "z#

z

1$#3$#

2$#

4$#

qL dL

L

! "

! " 02

02

3

2

3

2

%&'

&'((

%&'

&&((

YYXKYzY

XYXKXzX

Y

X

)#

)#Drift Lens Drift Lens Drift DriftLens Lens Drift

z

1# 2# 3# 4#

0 L

inix finx

BEAM MATCHING OPTIMIZATION

332211 JkJkJkJ ''%

****

+

,

----

.

/

(

(%

****

+

,

----

.

/

(

(%

****

+

,

----

.

/

(

(%

tar

tar

tar

tar

tar

fin

fin

fin

fin

fin

ini

ini

ini

ini

ini

YYXX

x

YYXX

x

YYXX

x ,,Drift Lens Drift Lens Drift DriftLens Lens Drift

z

1# 2# 3# 4#

0 L

inix finx

! " ! "! " ! "! " ! " ! "0 12 &'&%

(&('(&(%

&'&%

L

o desiYdesiX

tarfindYtarfindX

tarfinYtarfinX

dzzYzYKzXzXKzwJ

YYKXXKJ

YYKXXKJ

223

222

221

)()()()(

BEAM MATCHING OPTIMIZATION

! "#J

*# *J

hqq'&1

1&&q3

! "ka 4cos ! "54 &kb cos

67! "k8

Plant

Low-PassFilter

High-PassFilter

! "k# ! "! "kJ #

! "k#̂ ! "kJ f

! " ! " ! " ! "! " ! " ! "

! " ! " ! "! " ! " ! ")1(cos1ˆ1

ˆ1ˆcos

11

'''%'

&%'

&%

&&'&&%

kakk

kkk

kbkJkkJkJkhJkJ

f

ff

4##

38##

548

BEAM MATCHING OPTIMIZATION – 4D

50 100 150 200 250 300 350 400 450 500−55

−50

−45

−40

−35

−30

−25

−20

−15

−10

J [d

B]

Iteration

Cost Function

0 5000 10000 15000−70

−60

−50

−40

−30

−20

−10

0

10

Case Number

J [d

B]

50 100 150 200 250 300 350 400 450 500−50

−40

−30

−20

−10

0

10

20

30

40

50

Iteration

Filte

red θ

Parameter Evolution

θ1θ2θ3θ4

0 0.2 0.4 0.6 0.80

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5 x 10−3 Beam profile for estimated θ

z

X Y XtarYtar

., k, kk, KK,, K, K, KK iYiXdYdXYX

0110 1110002000

321 !!!!!!!!!

""""

#

$

%%%%

&

'(

!

011034.0003289.0006730.0

001070.0

finx

""""

#

$

%%%%

&

'(

!

011726.0003290.0007864.0

001094.0

tarx

TERMINAL CONSTRAINTS ONLY

) *) *) *) *) * dB - if J ....a

dB -J dB if -....adB -J dB if -....a

dB -J dB if -....aJdB if -...a

,,,

,,,

,,,

,,,

,,,

60025002500100105560050050050050

505510102501040505050750250

40517512252

4321

4321

4321

4321

4321

+!

+,!

+,!

+,!

,!

PID Tuning using ExtremumSeeking

Extremum Seeking Tuning Scheme

Stepfunction

!-

"kExtremum

Seeking Algorithm

# $krC "

# $kyC "

Gy(t)!r(t) u(t)

!"#$%#&"&'()%*+

J("k)

,%'-.+$+()%*+

)201)(101(

51)(4 ss

ssG

!!"

#

c) Step Response of output

b) Evolution of PID Parameters

d) Step Response of controller

a) Evolution of Cost Function