Mixed Integer Nonlinear Programming

IMA New Directions Short Course on Mathematical OptimizationJeff Linderoth and Jim Luedtke

Department of Industrial and Systems EngineeringUniversity of Wisconsin-Madison

August 12, 2016

Introduction

Mixed-Integer Nonlinear Optimization

Mixed Integer Nonlinear Program: (MINLP)

zminlp = minx

η

subject to gi (x) ≤ 0, i = 1, . . . ,m

f (x) ≤ ηx ∈ X , xI ∈ Z|I |

f : Rn → R, g : Rn → Rm smooth, sometimes convex functions.X ∈ Rn bounded, polyhedral set, e.g. X = {x : l ≤ AT x ≤ u}I ⊆ {1, . . . , n} subset of integer variablesWe may assume that f is a linear function.

NP-Super-Hard

Combines challenges of handling nonlinearitieswith combinatorial explosion of integer variables

Introduction

The MINLP Family

The great watershed inoptimization isn’t betweenlinearity and nonlinearity,but convexity andnonconvexity.

- R. Tyrrell Rockafellar

If f and g are convex functions,then we have a convex MINLP

If f and g are not convex, thenwe have a nonconvex MINLP

Optimization 101

Know the convexity properties of your instance!

Algorithms for two different classes of problems are different

We can solve significantly larger convex MINLP instances thatnonconvex MINLP instances

Algorithms designed for convex-MINLP are often used as heuristicsfor nonconvex MINLPs

Introduction

Specializations

Functional Form Problem Type

gi (x) = ‖Aix + bi‖2 − pTi x + qi MISOCP

gi (x) = polynomial MIPP

gi (x) = xTQix + pTi + qi MIQP

gi (x) = aTi x − bi MILP

MISOCP: Mixed Integer Second Order Cone ProgramAre convex-MINLP

MIPP: Mixed Integer Polynomial ProgramMIQP: Mixed Integer Quadratic Program

May be convex or nonconvexConvex MIQP is a special case of MISOCPIf f is convex quadratic and c is an affine mapping, then there arespecialized algorithms for convex-MIQP

MILP: Mixed Integer Linear Program

Introduction

Aside – MISOCP

The feasible regions of surprisingly many MINLP problems can beexpressed as MISOCP:

Hyperbolic constraints: Product of (non-negative) variables ≥ anorm2.

wTw ≤ xy , x ≥ 0, y ≥ 0 ⇔∥∥∥∥[ 2w

x − y

]∥∥∥∥ ≤ x + y

Convex Quadratic: ci (x) = xTQix + pTi x + qi ≤ t

Qi is positive semidefinite, so it has Cholesky factorization Qi = LTL

y = Lx ⇒ g(x) = yT y + pTi x + qi ≤ t

{(x , t) ∈ Rn+1 | xTQx + pTi x + qi ≤ t} =

Proj(x ,t){(x , y , t, p) ∈ R2n+2 | y = Lx , p + qT x + r ≤ t, p ≥ ‖y‖2}

Introduction

MINLP Tree

MINLP

NonconvexMINLP

ConvexMINLP

NonconvexNLP

ConvexNLP

PolynomialOpt.

MI-Polynomial

Opt.MISOCP SOCP

NonconvexQP

NonconvexMIQP

ConvexMIQP

Convex QP

MILP LP

Introduction

How Big Is It?

The Leyffer-Linderoth-Luedtke (LLL) Measure of Complexity

You have a problem of class X that has Y decision variables? What is thelargest value of Y for which one of Sven, Jim, and Jeff would be willingto bet $50 that a “state-of-the-art” solver could solve the problem?

Convex NonconvexProblem Class (X ) # Var (Y ) Problem Class (X ) # Var (Y )

MINLP 500 MINLP 100NLP 5× 104 NLP 100

MISOCP 1000 MIPP 150SOCP 105 PP 150MIQP 1000 MIQP 300

QP 5× 105 QP 300MILP 2× 104

LP 5× 107

Introduction Software Tools and Online Resources

Solvers for MINLP

Convex Nonconvexα-ECP α-BBBONMIN ANTIGONEDICOPT BARONFilMINT COCONUTGUROBI† COUENNEKNITRO CPLEX‡MILANO LGOMINLPBB LindoGlobalMINOPT SCIPMINOTAURSBBXpress††MIQP ‡MIQP/convex MIQP

Open-source and commercial solvers

Introduction Software Tools and Online Resources

Software for MINLP

Convex MINLP

ALPHA-ECP, Bonmin, DICOPT, FilMINT, KNITRO, MINLP-BB,SBB

(Convex) MIQP

CPLEX, GUROBI, MOSEK, XPRESS

(Nonconvex) MIQP

GLOMIQO

General Global Optimization

BARON, Couenne, LINDOGlobal, SCIP

Try ’em on NEOS

http://www.neos-server.org/neos/solvers/index.html

Introduction Basic Concepts

Simple Example

(x1 − 2)2 + (x2 + 1)2 ≤ 36

3x1 − x2 ≤ 6

−2.5(x1 − 1)2 − x2 ≤ −4

0 ≤ x1, x2 ≤ 5

x2 ∈ Z

x1

x2

Doubly NP-Hard!

Feasible region is not convex for two reasons—Integrality of x2 andnonconvexity of constraint

Introduction Basic Concepts

Q: What to Do?!

Relax!

Remove some constraints.Solution of relaxed problem provide a lower bound on optimal solutionvalue

Introduction Basic Concepts

Natural Relaxation

“Natural” relaxation is toremove constraints xi ∈ Z ∀i ∈ I

But this still leaves a nonconvexNLP relaxation

x1

x2

Introduction Basic Concepts

Relax More

We need a mechanism forrelaxing the nonconvexconstraints.

Secant underestimator forconcave functions

“Easy” to get polyhedralouterapproximation ofnonconvex constraints

Ideally, we would like to get theconvex hull of the feasible points

x1

x2

Introduction Basic Concepts

Linear Objective Is Important Here!

min(x1 − 1/2)2 + (x2 − 1/2)2

s.t. x1 ∈ {0, 1}, x2 ∈ {0, 1}

η ≥ (x1 − 1/2)2 + (x2 − 1/2)2

x1

x2

(x1, x2)

η

Without linear objective, optimal solution may be interior to theconvex hull ⇒ convexifying may do you no good!

Introduction Basic Concepts

Algorithm Ingredients

1 Relaxation

Used to compute a lower bound onthe optimum valueObtained by enlarging feasible set; e.g.ignore constraintsShould be “tractable” and “tight”

2 Constraint Enforcement

Exclude solutions from relaxations not feasible to MINLPRefine or tighten of relaxation—Branching and Cutting

3 Upper Bounds

Obtained from any feasible point; e.g. solve NLP for fixed xI .(NLP(xI )).Other heuristic techniques have been developed

Modeling and Applications

Models are Important!

Writing “good” models is extremelyimportant in MILP

Probably more important for MINLP

More Complexity

Tradeoff between physical accuracy and mathematical tractability

Convex reformulations? Linearize expressions?

MINLP Modeling Preference

We want strong formulations—Relaxations that can be relaxed tosomething that closely approximates the convex hull

We prefer linear over convex over nonconvex formulations

Modeling and Applications

Some MINLP Applications

1 Supply Chain—Convex MINLPDiscrete: Fixed charges for opening facilitiesNonlinear: Nonlinear transportation costs

2 Gas/Water Network Design—Nonconvex MINLPDiscrete: Pipe connections/sizesNonlinear: Pressure Loss

3 Pooling/Petrochemical—Nonconvex MINLPDiscrete: Which process to use?Nonlinear: Product Blending (among others)

Luedtke Theorem

97.5% of all practical MINLPs have integer variables for one of twopurposes...

1 Binary variables to make a “multiple choice” selection2 Binary indicator variables that turn on/off continuous variables

and/or constraints.

Modeling and Applications Supply Chain

Uncapacitated Facility Location

Problem introduced by Gunluk, Lee,and Weismantel (’07) and classes ofstrong cutting planes derived

M: Facility

N: Customer

xij : percentage of customer j ∈ N demand met by facility i ∈ M

zi = 1⇔ facility i ∈ M is built

Fixed cost for opening facility i ∈ M

Quadratic cost for meeting demand j ∈ N from facility i ∈ M

Modeling and Applications Supply Chain

Quadratic Uncapacitated Facility Location Problem

A very simple MIQP

z∗def= min

∑i∈M

cizi +∑i∈M

∑j∈N

qijx2ij

subject to

xij ≤ zi ∀i ∈ M,∀j ∈ N∑i∈M

xij = 1 ∀j ∈ N

xij ≥ 0 ∀i ∈ M, ∀j ∈ N

zi ∈ {0, 1} ∀i ∈ M

Modeling and Applications Supply Chain

Base Case of Induction to “Luedtke Theorem”

Note that binary variables are used as indicators: zi = 0⇒ xij = 0

If z = 1, then we need to model the epigraph of x2ij

Intager Programmerz ’r’ Smrt

Mixed Integer Linear Programmerscarefully study simple problemstructures to come up with“good” formulations for problems

Good formulations closelyapproximate convex hull offeasible solutions

Important to understand thestructure of a special MINLP withindicator variables

Modeling and Applications Supply Chain

A Very Simple Structure

Rdef={

(x , y , z) ∈ R2 × B | y ≥ x2, 0 ≤ x ≤ uz}

z = 0⇒ x = 0, y ≥ 0

z = 1⇒ x ≤ u, y ≥ x2

x

y

z = 1

z

y ≥ x2

Deep Insights

conv(R) ≡ line connecting (0, 0, 0) to y = x2 in the z = 1 plane

Modeling and Applications Supply Chain

Characterization of Convex Hull

Deep Theorem #1

R ={

(x , y , z) ∈ R2 × B | y ≥ x2, 0 ≤ x ≤ uz}

conv(R) ={

(x , y , z) ∈ R3 | yz ≥ x2, 0 ≤ x ≤ uz , 0 ≤ z ≤ 1, y ≥ 0}

x2 ≤ yz , y , z ≥ 0 ≡

Second Order Cone Programming

There are effective, robust algorithms for optimizing over conv(R)

Modeling and Applications Supply Chain

In General—Giving You Some Perspective

For a convex function f : Rn → R, the perspective functionP : Rn+1 → R of f is

P(x , z)def=

{0 if z = 0zf (x/z) if z > 0

The epigraph of P(x , z) is a cone pointed at the origin whose lowershape is f (x)

Exploiting Your Perspective

If zi is an indicator that the (nonlinear, convex) inequalityf (x) ≤ 0 must hold, (otherwise x = 0), replace the inequality withits perspective version:

zi f (x/zi ) ≤ 0

The resulting (convex) inequality is a much tighter relaxation ofthe feasible region.

Modeling and Applications Supply Chain

Perspectivizing Quadratic Uncapacitated Facility Location

z∗def= min

∑i∈M

cizi +∑i∈M

∑j∈N

qijx2ijyij

xij ≤ zi ∀i ∈ M, ∀j ∈ N∑i∈M

xij = 1 ∀j ∈ N

xij ≥ 0 ∀i ∈ M,∀j ∈ N

zi ∈ {0, 1} ∀i ∈ M

x2ij − ziyij ≤ 0 ∀i ∈ M,∀j ∈ N

Steps

1 Make ObjectiveLinear

2 ApplyPerspective

3 Make GiantProfit

Modeling and Applications Supply Chain

Strength of Relaxations

zR : Value of NLP relaxation

zGLW : Value of NLP relaxation after GLW cuts

zP : Value of perspective relaxation

z∗: Optimal solution value

|M| |N| zR zGLW zP z∗

10 30 140.6 326.4 346.5 348.715 50 141.3 312.2 380.0 384.120 65 122.5 248.7 288.9 289.325 80 121.3 260.1 314.8 315.830 100 128.0 327.0 391.7 393.2

Cheers!

Modeling and Applications Supply Chain

Impact of SOCP

m = 30, n = 100

Convex MINLP, Original: 16697 CPU seconds, 45901 nodes

Convex MINLP, w/GLW Ineq.: 21206 CPU seconds, 29277 nodes

MISOCP, Perspective, 23 CPU seconds, 44 B&B nodes

Larger Instances

m n T N

30 200 141.9 6340 100 76.4 5440 200 101.3 4550 100 61.6 4950 200 140.4 47

“Poifect”

Modeling and Applications Water Network Design

Design of Water Distribution Networks

Goal: Design minimum costnetwork from discrete pipediameters to meet waterdemand

Sets

N nodes in network

S source nodes

A: arcs in the network

K: Pipe diameters

Modeling and Applications Water Network Design

Design of Water Distribution Networks

Variables

qij : flow pipe (i , j) ∈ Ahi : hydraulic head at node i ∈ Ndij : diameter of pipe (i , j) ∈ A, where dij ∈ {P1, . . . ,Pr}aij : area of cross section (i , j) ∈ Ayijk : Binary variable = 1 if pipe of size k ∈ K chosen for (i , j) ∈ A

Constraints

Conservation of flow∑(i ,j)∈A

qij −∑

(j ,i)∈A

qji = Di , ∀i ∈ N \ S.

Flow bounds: (linear in a, but since aij = 0.25πd2ij , nonlinear in d)

−Vmaxaij ≤ qij ≤ Vmaxaij , ∀(i , j) ∈ A.

Modeling and Applications Water Network Design

Design of Water Distribution Networks

Discrete pipe sizes dij ∈ {P1, . . . ,Pr}Use “multiple choice” integrality to remove nonlinearity aij = πd2

ij/4

This is in general a good modeling practice

1 Binary variables yijk ∈ {0, 1} for k ∈ K2 Model discrete choice as∑

k∈Kyijk = 1, and

∑k∈K

Pkyijk = dij . ∀(i , j) ∈ A

3 Model aij = πd2ij/4 (nonlinear) as

aij =∑k∈K

(π4

Pk

)yijk ∀(i , j) ∈ A

Modeling and Applications Water Network Design

Design of Water Distribution Networks

Nonsmooth, nonlinear pressure loss as a function of flow qij along arc(i , j) ∈ A and diameter dij :

hi − hj = κijd−c1ij sgn(qij)|qij |c2 ∀(i , j) ∈ A

Another trick: disaggregate flow into flow variables for each pipe size:

qij =∑k∈K

qijk

(−Vmaxaij)yijk ≤ qijk ≤ (Vmaxaij)yijk

Modeling and Applications Water Network Design

Why On Earth?

Doing this allows us to write the(nonlinear) headloss as afunction of a single variable qijk

Often in applications, thenonlinearity is low-dimensional(low ∈ {1, 2})Therefore a piecewise linearapproximation (or relaxation)may be sufficiently accurate

Luedtke’s Theorem!

Note also thatyijk = 0⇒ qijk = 0

q

f (q) = sgn(q)|q|c2

Modeling and Applications Water Network Design

Modeling Piecewise-Linear Functions

Take Your Pick

SOS2 Model [Beale and Tomlin, 1970, Beale and Forrest, 1976]

Incremental Model [Markowitz and Manne, 1957]

Multiple Choice Model [Jeroslow and Lowe, 1984]

Convex Combination Model [Dantzig, 1960, Padberg, 2000]

Disaggregated Convex Combination Model [Meyer, 1976]

Logarithmic Model [Vielma and Nemhauser, 2011]

Modeling and Applications Water Network Design

Multiple Choice Model

Want to model (∀i)

f (q) = miq + ci , q ∈ [Bi−1,Bi ]

Introduce New Variables

wi : = q if q ∈ [Bi−1,Bi ]

bi : = 1 if q ∈ [Bi−1,Bi ]

t ≈ f (q)

q

f (q) ≈ sgn(q)|q|c2

B0 B1 B2

B3 B4

q =n∑

i=1

wi ,

t =n∑

i=1

(miwi + cibi )

Bi−1bi ≤ wi ≤ Bibi ∀i ∈ 1, . . . , n

1 =n∑

i=1

bi

B0y ≤ q ≤ Bny

Modeling and Applications Water Network Design

A Simple Trick

Instead of modeling the piecewise-linear indicator in the standard way:

B0y ≤ q ≤ Bny , 1 =n∑

i=1

bi

Model it as

y =n∑

i=1

bi

Resulting formulation is provably stronger (locally-ideal)

All piecewise-linear functions turned on/off by an indicator have asimilar modeling trick [Sridhar et al., 2013]

It is the exact extension of the “perspective reformulation” to thisnonconvex case

Modeling and Applications Pooling

The Pooling Problem

The Pooling Problem

A canonical problem in the petrochemical industry

Decide how to blend (pool) feeds together to make productsmeeting “quality requirements” at minimum cost

Feeds I Pools J Products K

Generally each product has multiple attributes

Modeling and Applications Pooling

Pooling Problem

Basic variables

xij : Flow of input i to pool j

xik : Flow of input i to product k

xjk : Flow from pool j to product k

Key Parameters

λi : Concentration of attribute in feed i

`k , υk : Bounds on attribute concentration in product k

Ci ,Cj ,Ck : Capacities on feeds, pools, and outputs

Uij ,Uik ,Ukj : Flow capacities (some may be zero)

Simple (Linear) Constraints

Linear Constraints: Flow balance, bounds on flows,feed/pool/output capacity

Modeling and Applications Pooling

P-formulation

(For example) Input capacity:∑`∈L

xi` +∑j∈J

xij ≤ Ci ∀i ∈ I

Flow Balance: ∑i∈I

xi` =∑j∈J

x`j ∀` ∈ L

Pool Quality: ∑i∈I

λixi` = p`∑i∈I

xi` ∀` ∈ L

Product Quality:∑`∈L

p`x`j +∑i∈I

λixij = pj

(∑`∈L

x`j +∑i∈I

xij

)∀j ∈ J

`j ≤ pj ≤ υj ∀j ∈ J

Modeling and Applications Pooling

Modeling the Quality Constraints

Q-formulation [Ben-Tal et al., 1994]

New variables qij : proportion of flow to pool j coming from feed i :∑i∈I

qij = 1

qij satisfy the relations:

qij =xij∑t∈I xtj

⇔ xij = qij

∑t∈I

xtj

Reformulate using flow balance at pool j :∑

t∈I xtj =∑

k∈K xjk

xij = qij

∑k∈K

xjk

Concentration of attribute at pool i is then:∑

i∈I λiqij

Modeling and Applications Pooling

Modeling the Quality Constraints (2)

Concentration of attribute at pool i :∑

i∈I λiqij

Quality constraint on product k:

`k ≤∑

i∈I λixik +∑

j∈J∑

i∈I λiqijxjk∑i∈I xik +

∑j∈J xjk

≤ υk

Multiply and gather terms (e.g., for upper bound):∑i∈I

(λi − υk)xik +∑j∈J

∑i∈I

(λi − υk)qijxjk ≤ 0

Introduce variables wijk to represent bilinear terms qijxjk :

⇒ xij = qij

∑k∈K

xjk =∑k∈K

wijk

Modeling and Applications Pooling

Modeling the Quality Constraints (3)

Bringing it all together∑i∈I

qij = 1, j ∈ J

xij =∑k∈K

wijk , i ∈ I , j ∈ J∑i∈I

(λi − υk)xik +∑j∈J

∑i∈I

(λi − υk)wijk ≤ 0, k ∈ K

∑i∈I

(`k − λi )xik +∑j∈J

∑i∈I

(`k − λi )wijk ≥ 0 k ∈ K

wijk = qijxjk , i ∈ I , j ∈ J, k ∈ K

Modeling and Applications Pooling

PQ-Formulation [Tawarmalani and Sahinidis, 2002]

1 For each j ∈ J, multiply the constraint∑

i∈I qij = 1 with xjk :∑i∈I

qijxjk = xjk

2 For each j ∈ J, multiply the constraint:∑

k∈K xjk ≤ Cj with qij :∑k∈K

qijxjk ≤ Cjqij

3 Replace qijxjk with wijk :∑i∈I

wijk = xjk ,∑k∈K

wijk ≤ Cjqj

Much better relaxations!

Takeaway: Adding redundant constraints can significantly improve relax-ations of nonconvex MINLPs.

Modeling and Applications Pooling

The Pooling Problem with Binary Variables

Arcs connecting the input streams toother nodes may be used only bypaying a fixed cost.

Network Design

Ship assignment

Add binary variables zij for these arcs:

xij ≤ Uijzij

A real nonconvex, integer problem.

Modeling and Applications Pooling

Practical Advice About the Luedtke Theorem

We have seen many cases where we have continuous variable0 ≤ x ≤ U and associated binary variable z ∈ {0, 1}.We see many models that write the expression w = zx .

On The Horror!

Never multiply a binary variable by abounded continuous variable

You can linearize this:

0 ≤ w ≤ Uz

−U(1− z) ≤ x − w ≤ U(1− z)

Modeling and Applications Pooling

Other MINLP Applications

1 Portfolio Management—Convex MIQP

Discrete: Trading StrategyNonlinear: Utility

2 Reactor Core Reloading—Nonconvex MINLP

Discrete: Bundle ReorderingNonlinear: Neutron Transport

3 Power Grid Operation—Nonconvex MINLP

Discrete: Unit CommittmentNonlinear: AC Power Flow

4 Building Co-Generation—Nonconvex MINLP

Discrete: Unit CommittmentNonlinear: Ramping Constraints

5 Oil-Spill Response—Nonconvex MINLP

Discrete: Resource AllocationNonlinear: Fluid Flow & Chemistry

2. Core-Reloading

4. Building Demand

5. Oil-Spill Response

Algorithms – Convex

Algorithms for Convex MINLP

Algorithms for convex MINLP are very similar to those for MILP

NLP-based branch-and-bound

Solve NLP relaxation at each node

Cutting plane based algorithms:

Use linear inequalities to “outer approximate” continuous relaxation

Solve LP relaxations instead of NLP relaxations

Update LP relaxations by adding cuts

Benders decomposition [Geoffrion, 1972], Outer approximation[Duran and Grossmann, 1986], LP/NLP branch-and-bound[Quesada and Grossmann, 1992], Extended cutting plane[Westerlund and Pettersson, 1995], Algorithmic refinements, e.g.[Abhishek et al., 2010, Bonami et al., 2008]

Algorithms – Convex

Valid Inequalities for Convex MINLP

Many classes of inequalities for MILP have been extended/studied forconvex MINLP

Disjunctive/split cuts[Ceria and Soares, 1999, Stubbs and Mehrotra, 1999,Belotti et al., 2012, Bonami, 2011, Modaresi et al., 2014b,Kılınc et al., 2010, Kılınc-Karzan and Yildiz, 2014]

Chvatal-Gomory rounding and mixed-integer rounding for conicMINLP [Cezik and Iyengar, 2005, Drewes, 2009]

Conic mixed-integer rounding [Atamturk and Narayanan, 2010]

Intersection cuts [Andersen and Jensen, 2013, Modaresi et al., 2014a]

Minimal valid inequalities and cut generating functions[Kılınc-Karzan, 2013, Moran R. et al., 2012]

Closures [Dadush et al., 2011b, Dey and Moran R., 2012,Dadush et al., 2011a]

Algorithms – Nonconvex Constructing Relaxations of Nonconvex Constraints

General Approach to Nonconvex MINLP

minimizex

f (x) subject to g(x) ≤ 0, x ∈ X , xi ∈ Z ∀ i ∈ I

Use our old trick: convex (often polyhedral) relaxation!

Relax integrality as before: xi ∈ R ∀ i ∈ I

New: R(Θ) ⊇ Θ = {x ∈ X : g(x) ≤ 0}Ensure relaxation is tractable: e.g. R(Θ) is convex

Algorithms – Nonconvex Constructing Relaxations of Nonconvex Constraints

Factorable Functions and MINLP

Consider MINLP with nonconvex, factorable f (x) and g(x)

minimizex

f (x) subject to g(x) ≤ 0, x ∈ X , xi ∈ Z ∀ i ∈ I

Factorable Functions

g(x) is factorable if it can be expressed as a combination of functionsfrom finite set of operators O = {+,×, /, , sin, cos, exp, log, | · |} whosearguments are variables, constants, or other factorable functions.

Excludes integrals∫ xξ=x0

h(ξ)dξ and black-box functions

Represented as expression trees

Algorithms – Nonconvex Constructing Relaxations of Nonconvex Constraints

Expression Tree Example

+

log

^

31

2

2x x

x

*

Expression tree of f (x1, x2) = x1 log(x2) + x32

Modeling languages (e.g. AMPL, GAMS) have expression tree “API”

Algorithms – Nonconvex Constructing Relaxations of Nonconvex Constraints

Reformulation of Factorable MINLP

Reformulate factorable MINLP as

minimizex

xn+q

subject to xk = ϑk(x) k = n + 1, n + 2, . . . , n + qli ≤ xi ≤ ui i = 1, 2, . . . , n + qx ∈ X ,xi ∈ Z, ∀i ∈ I ,

see e.g. [Smith and Pantelides, 1997]

q new auxiliary variables, xn+1, . . . , xn+q

ϑk is operator from O{+,×, /, , sin, cos, exp, log}

Algorithms – Nonconvex Constructing Relaxations of Nonconvex Constraints

Example of Reformulation of Factorable MINLP

min x1 + x1x22

s.t. x1x2 + sin x2 ≤ 4,x1 ∈ [−4, 4] ∩ Z,x2 ∈ [0, 10] ∩ Z.

Reformulation

min x7s.t. x3 = sin x2 −4 ≤ x1 ≤ 4 0 ≤ x5 ≤ 45

x4 = x1x2 0 ≤ x2 ≤ 10 −400 ≤ x6 ≤ 400x5 = 4− x3 − x4 −1 ≤ x3 ≤ 1 −404 ≤ x7 ≤ 404x6 = x2x4 −40 ≤ x4 ≤ 40x7 = x1 + x6 x1, x2, x4, x6, x7 ∈ Z.

Factorization is not unique!Integrality inherited from function and integrality on argumentsBounds inherited from function and argument bounds

Algorithms – Nonconvex Constructing Relaxations of Nonconvex Constraints

Reformulation of Factorable MINLP

Factorable form allows systematic construction of convex relaxation:

Nonconvex sets, k = n + 1, n + 2, . . . , n + q

Θk = {x ∈ Rn+q : xk = ϑk(x), x ∈ X , l ≤ x ≤ u, xi ∈ Z, i ∈ I}

Let R(Θk) ⊃ Θk be a convex relaxation

minimize

xxn+q

subject to x ∈ R(Θk) k = n + 1, n + 2, . . . , n + qli ≤ xi ≤ ui i = 1, 2, . . . , n + qx ∈ X .

... convex relaxation ... only look at simple sets!

Algorithms – Nonconvex Constructing Relaxations of Nonconvex Constraints

Examples of Relaxations

Construct relaxation for each operator ∈ O{+,×, /, , sin, cos, exp, log}Odd-degree monomials, xk = x2p+1

i , see[Liberti and Pantelides, 2003]

Bilinear functions xk = xixj , [McCormick, 1976]Let x = (xi , xj), L = (li , lj),U = (ui , uj)get convex hull of Θk = {(x , xk) : xk = xixj , L ≤ x ≤ U}:

xk ≥ ljxi + lixj − li lj xk ≤ ljxi + uixj − ui ljxk ≥ ujxi + uixj − uiuj xk ≤ ujxi + lixj − liuj

Important!

Tightness of convex hull depends on bounds li , lj , ui , uj

Algorithms – Nonconvex Constructing Relaxations of Nonconvex Constraints

Examples of Relaxations

Polyhedral relaxation, R(Θk), of xk = x3i and xk = xixj

l

u

xi

i

i

xk

x

k

j

i

x

x

Algorithms – Nonconvex Constructing Relaxations of Nonconvex Constraints

Examples of Relaxations

Polyhedral relaxation, R(Θk), of xk = x2i with xi continuous/integer

k

i

i

x

x

uli

i

liui

kx

x

... if xi ∈ Z then add inequalities violated at x ′i 6∈ Z

Algorithms – Nonconvex Spatial Branch-and-Bound

Spatial Branch-and-Bound (BnB)

To enforce constraints violated by relaxation solution use spatial BnB

Implicit enumeration technique like integer BnB

Recursively define partitions of feasible set into two sets by changingcontinuous and integer variable bounds

Solve LP (or convex NLP) relaxations (⇒ lower bounds)... and nonconvex NLPs (⇒ upper bound if feasible)

Lower bounding problem at node with bounds (l , u), e.g. LP(l , u)minimize

xxn+q

subject to x ∈ R(Θk) k = n + 1, n + 2, . . . , n + qli ≤ xi ≤ ui i = 1, 2, . . . , n + qx ∈ X .

Algorithms – Nonconvex Spatial Branch-and-Bound

Spatial Branch-and-Bound (BnB)

If LP(l , u) infeasible, then prune node.

Otherwise, x optimal solution of LP(l , u):

If x integer feasible and feasible in NLP(l , u) (hence MINLP), thenfathom node (new incumbent)

Otherwise ... branch ...

Two possible ways to branch (integer / nonlinear):

1 x not integral: xi ≤ bxic ∨ xi ≥ dxie like integer BnB2 ∃k : xk 6= ϑk(x) nonlinear infeasible:

Choose branching variable xi from arguments of ϑk(x)Branch xi ≤ xi ∨ xi ≥ xi ... two subproblemsRefine convex relaxation in each branch ... tighter bounds

Algorithms – Nonconvex Spatial Branch-and-Bound

Branching for Spatial Branch-and-Bound

Branching example xk = ϑk(xi ) = (xi )2 violated

x

^^ji

j

i

ii ul

(x ,x )

x

(x ,x )

x

x

^^ji

j

i

ii b ul

Algorithms – Nonconvex Spatial Branch-and-Bound

Branching for Spatial Branch-and-Bound

Branching based on xi ≤ b ∨ xi ≥ b

Good performance depends on good choice of i and b

Goal: Increase both bounds LP(l−, u−) and LP(l+, u+)

Strong branching, pseudocost branching, and reliability branchinggeneralized from convex MINLP

Other techniques (e.g., violation transfer) specific to nonconvexMINLP

Finding continuous branching candidates xi :

xi not fixed in parent problem

xi is argument of violated function xk 6= ϑk(x)

Algorithms – Nonconvex Tightening Bounds and Relaxations

Tightening Bounds and Relaxations

Bound tightening to reduce range of bounds xi ∈ [li , ui ]... because tighter bounds ⇒ tighter relaxations ⇒ smaller trees

Spatial BnB + Bound tightening = Branch-and-reduce[Ryoo and Sahinidis, 1996, Belotti et al., 2009]

Conceptual bound-tightening procedure:Feasible set F = {x ∈ [l , u] : g(x) ≤ 0, x ∈ X , xI ∈ Zp}Solve 2n (global) optimization problems, given upper bound U:

l ′i = min{xi : x ∈ F , f (x) ≤ U}; u′i = max{xi : x ∈ F , f (x) ≤ U}.

... nonconvex MINLPs just as hard ⇒ use relaxations:

1 FBBT: feasibility-based bound tightening

2 OBBT: optimality-based bound tightening

Algorithms – Nonconvex Tightening Bounds and Relaxations

FBBT: Feasibility-Based Bound Tightening

FBBT broadly used:

Artificial intelligence community & constraint programming

NLP solvers [Messine, 2004]

MILP solvers [Savelsbergh, 1994]

Basic Principle of FBBT

Infer bounds on xi from tighter bounds on xj for j 6= i .

Example: xj = x3i and xi ∈ [li , ui ]

Tighten interval of xj to [lj , uj ] ∩ [l3i , u3i ]

Tightened l ′j on xj ⇒ tighter l ′i = 3√

lj for xi

FBBT is efficient and fast, but convergence can be slow

Algorithms – Nonconvex Tightening Bounds and Relaxations

OBBT: Optimality-Based Bound Tightening

Solving min /max xi s.t. x ∈ F (nonconvex MINLP) not practicalInstead, use current relaxation

F(l , u) =

x ∈ Rn+q :x ∈ R(Θ)k k = n + 1, n + 2, . . . , n + qli ≤ xi ≤ ui i = 1, 2, . . . , n + qx ∈ X

.

Now get bounds on xi for i = 1, . . . , n by solving 2n LPs:

l ′i = min{xi : x ∈ F(l , u)}u′i = max{xi : x ∈ F(l , u)}

... only apply at root node, or small number of nodes

Algorithms – Nonconvex Exploiting Nonconvex Structures

The Trouble with Factorable Relaxations

Factorable relaxations are primarily based on local structure ⇒ Potentiallyweak bounds

Example: Θ = {(x1, x2) : x21 + x2

2 ≥ 1, 0 ≤ xi ≤ 2}

Convex hull: x1 + x2 ≥ 1Best possible factorable relaxation:

x3 = x21 ⇒ (1/2)x3 ≤ x1

x4 = x22 ⇒ (1/2)x4 ≤ x2

x3 + x4 ≥ 1⇒x1 + x2 ≥ (1/2)(x3 + x4) ≥ 1/2

Remedy: Study relaxations of more global structures

Algorithms – Nonconvex Exploiting Nonconvex Structures

A Special Structure: Nonconvex Quadratic Functions

Quadratically constrained quadratic program (QCQP)minimize

xxTQ0x + cT

0 x ,

subject to xTQkx + cTk x ≤ bk , k = 1, . . . , q

Ax ≤ b,0 ≤ x ≤ u, xi ∈ Z, ∀i ∈ I ,

Qk not necessarily convex ⇒ nonconvex problem

Problem is NP-hard even if all variables are continuousMany important special cases: Bilinear programming, poolingproblem, integer linear programming, max cut,. . .

Extensively studied: E.g.,

[Poljak and Wolkowicz, 1995, Sherali and Adams, 1998, Yajima and Fujie, 1998,

Burer and Vandenbussche, 2009, Burer, 2009, Burer and Letchford, 2009,

Burer and Letchford, 2012, Vandenbussche and Nemhauser, 2005,

Saxena et al., 2008, Misener and Floudas, 2012, Buchheim and Wiegele, 2012]

Algorithms – Nonconvex Exploiting Nonconvex Structures

General Nonconvex Quadratic Functions

Equivalent reformulation of QCQP: introduce Xij for all i , j pairs

minimizex ,X

Q0 • X + cT0 x ,

subject to Qk • X + cTk x ≤ bk , k = 1, . . . , q

Ax ≤ b,0 ≤ x ≤ u, xi ∈ Z, ∀i ∈ I ,X = xxT ,

where X = [Xij ] matrix, Qk • X =∑

ij [Qk ]ijXij =∑

ij [Qk ]ijxixj .

X = xxT represents nonconvex constraint Xij = xixj... otherwise problem is linear!

Relaxing equality X = xxT gives convex (linear) relaxation

We show two approaches: RLT and SDP

Algorithms – Nonconvex Exploiting Nonconvex Structures

Reformulation-Linearization Technique (RLT)[Adams and Sherali, 1986]

Begin with standard polyhedral relaxation of Xij = xixj :

Xij ≥ 0, Xij ≥ uixj + ujxi − uiuj , Xij ≤ ujxi , Xij ≤ uixj

Strengthen using more structure

1 Exploiting binary variables: xi ∈ {0, 1} ⇒ x2i = xi

Add linear constraints Xii = xi

2 Multiply linear constraints (e.g., PQ formulation for pooling)Multiplying xi ≥ 0 and bt −

∑j atjxj ≥ 0 gives

btxi −∑j

atjxixj ≥ 0

Replace xixj with Xij yields linear inequality

btxi −∑j

atjXij ≥ 0

Algorithms – Nonconvex Exploiting Nonconvex Structures

Semidefinite Programming

Semidefinite Programming (SDP) relaxations of QCQPs

1 Relax X − xxT = 0 to X − xxT � 0

2

X − xxT � 0 ⇔[

1 xT

x X

]� 0

3 Improve by adding Xii = xi for binary xi ∈ {0, 1}4 Additional constraints by squaring & linearizing constraints

... here H � 0 means H positive semidefinite (xTHx ≥ 0, ∀x)

Both RLT and SDP provide good relaxations ... can also combine them

The Future of MINLP?

Some Open Research Question in MINLP

A very biased view of open research in MINLP

Strong relaxations for nonlinear structures

Implementation & software design issues

Open questions from previous slide

Going beyond traditional MINLP

Black-box functions or simulations ...unrelaxable variables

Partial-differential equation constraints

Bilevel problems, e.g., leader-follower games

Stochastic/robust optimization

Make impact in important applications

The Future of MINLP?

Take Home Messages

Modeling is extremely important for MINLP... strong relaxations are key

Effective techniques from MILP useful for MINLP... more work can be done!

MINLP software maturing,... but lags behind commercial MILP software

Linear better than convex better than nonconvex

Exploit structure whenever possible

The Future of MINLP?

Abhishek, K., Leyffer, S., and Linderoth, J. T. (2010).FilMINT: An outer-approximation-based solver for nonlinear mixedinteger programs.INFORMS Journal on Computing, 22:555–567.DOI:10.1287/ijoc.1090.0373.

Adams, W. and Sherali, H. (1986).A tight linearization and an algorithm for zero-one quadraticprogramming problems.Management Science, 32(10):1274–1290.

Andersen, K. and Jensen, A. N. (2013).Intersection cuts for mixed integer conic quadratic sets.pages 37–48.

Atamturk, A. and Narayanan, V. (2010).Conic mixed-integer rounding cuts.Mathematical Programming A, 122(1):1–20.

Beale, E. and Tomlin, J. (1970).

The Future of MINLP?

Special facilities in a general mathematical programming system fornon- convex problems using ordered sets of variables.In Lawrence, J., editor, Proceedings of the 5th InternationalConference on Operations Research, pages 447–454, Venice, Italy.

Beale, E. M. L. and Forrest, J. J. H. (1976).Global optimization using special ordered sets.Mathematical Programming, 10:52–69.

Belotti, P., Goez, J., Polik, I., Ralphs, T., and Terlaky, T. (2012).A conic representation of the convex hull of disjunctive sets and coniccuts for integer second order cone optimization.Technical report, n. 12T-009, Lehigh University, Department ofIndustrial and Systems Engineering.http:

//www.optimization-online.org/DB_FILE/2012/06/3494.pdf.

Belotti, P., Lee, J., Liberti, L., Margot, F., and Wachter, A. (2009).Branching and bounds tightening techniques for non-convex MINLP.Optimization Methods and Software, 24(4-5):597–634.

The Future of MINLP?

Ben-Tal, A., Eiger, G., and Gershovitz, V. (1994).Global minimization by reducing the duality gap.Mathematical Programming, 63(1):193–212.

Bonami, P. (2011).Lift-and-project cuts for mixed integer convex programs.In Gunluk, O. and Woeginger, G., editors, Integer Programming andCombinatoral Optimization, volume 6655 of Lecture Notes inComputer Science, pages 52–64. Springer, Berlin.

Bonami, P., Biegler, L., Conn, A., Cornuejols, G., Grossmann, I.,Laird, C., Lee, J., Lodi, A., Margot, F., Sawaya, N., and Wachter, A.(2008).An algorithmic framework for convex mixed integer nonlinearprograms.Discrete Optimization, 5(2):186–204.

Buchheim, C. and Wiegele, A. (2012).

The Future of MINLP?

Semidefinite relaxations for non-convex quadratic mixed-integerprogramming.Mathematical Programming, pages 1–18.10.1007/s10107-012-0534-y.

Burer, S. (2009).On the copositive representation of binary and continuous nonconvexquadratic programs.Mathematical Programming, 120:479–495.

Burer, S. and Letchford, A. (2009).On nonconvex quadratic programming with box constraints.SIAM Journal on Optimization, 20(2):1073 – 89.

Burer, S. and Letchford, A. (2012).Unbounded convex sets for non-convex mixed-integer quadraticprogramming.Mathematical Programming, pages 1–26.10.1007/s10107-012-0609-9.

The Future of MINLP?

Burer, S. and Vandenbussche, D. (2009).Globally solving box-constrained nonconvex quadratic programs withsemidefinite-based finite branch-and-bound.Computational Optimization and Applications, 43(2):181 – 195.

Ceria, S. and Soares, J. (1999).Convex programming for disjunctive optimization.Mathematical Programming, 86:595–614.

Cezik, M. T. and Iyengar, G. (2005).Cuts for mixed 0-1 conic programming.Mathematical Programming, 104:179–202.

Dadush, D., Dey, S., and Vielma, J. P. (2011a).The split closure of a strictly convex body.Operations Research Letters, 39(2):121 – 126.

Dadush, D., Dey, S. S., and Vielma, J. P. (2011b).The chvatal-gomory closure of a strictly convex body.Mathematics of Operations Research, 36(2):227 – 239.

The Future of MINLP?

Dantzig, G. B. (1960).On the significance of solving linear programming problems with someinteger variables.Econometrica, 28:30–44.

Dey, S. S. and Moran R., D. A. (2012).Some properties of convex hulls of integer points contained in generalconvex sets.Mathematical Programming, pages 1 – 20.

Drewes, S. (2009).Mixed Integer Second Order Cone Programming.PhD thesis, Technische Universitat Darmstadt.

Duran, M. A. and Grossmann, I. (1986).An outer-approximation algorithm for a class of mixed-integernonlinear programs.Mathematical Programming, 36:307–339.

Geoffrion, A. M. (1972).

The Future of MINLP?

Generalized Benders decomposition.Journal of Optimization Theory and Applications, 10(4):237–260.

Jeroslow, R. and Lowe, J. (1984).Modelling with integer variables.Mathematical Programming Studies, 22:167–84.

Kılınc, M., Linderoth, J., and Luedtke, J. (2010).Effective separation of disjunctive cuts for convex mixed integernonlinear programs.Technical Report 1681, Computer Sciences Department, University ofWisconsin-Madison.

Kılınc-Karzan, F. (2013).On minimal valid inequalities for mixed integer conic programs.

Kılınc-Karzan, F. and Yildiz, S. (2014).Two-term disjunctions on the second-order cone.

Liberti, L. and Pantelides, C. C. (2003).Convex envelopes of monomials of odd degree.

The Future of MINLP?

Journal of Global Optimization, 25(2):157–168.

Markowitz, H. M. and Manne, A. S. (1957).On the solution of discrete programming problems.Econometrica, 25(1):84–110.

McCormick, G. P. (1976).Computability of global solutions to factorable nonconvex programs:Part I — Convex underestimating problems.Mathematical Programming, 10:147–175.

Messine, F. (2004).Deterministic global optimization using interval constraint propagationtechniques.RAIRO-RO, 38(4):277–294.

Meyer, R. (1976).Mixed integer minimization models for piecewise-linear functions of asingle variable.Discrete Mathematics, 16(2):163–71.

The Future of MINLP?

Misener, R. and Floudas, C. (2012).GloMIQO: Global mixed-integer quadratic optimizer.Journal of Global Optimization, pages 1–48.

Modaresi, S., Kılınc, M., and Vielma, J. (2014a).Intersection cuts for nonlinear integer programming: convexificationtechniques for structured sets.

Modaresi, S., Kılınc, M., and Vielma, J. (2014b).Split cuts and extended formulations for mixed integer conic quadraticprogramming.

Moran R., D. A., Dey, S. S., and Vielma, J. P. (2012).A strong dual for conic mixed-integer programs.SIAM Journal on Optimization, 22(3):1136 – 1150.

Padberg, M. (2000).Approximating separable nonlinear functions via mixed zero-oneprograms.Operations Research Letters, 27(1):1–5.

The Future of MINLP?

Poljak, S. and Wolkowicz, H. (1995).Convex relaxations of (0-1) quadratic programming.Mathematics of Operations Research, 20:550–561.

Quesada, I. and Grossmann, I. E. (1992).An LP/NLP based branch-and-bound algorithm for convex MINLPoptimization problems.Computers & Chemical Engineering, 16:937–947.

Ryoo, H. S. and Sahinidis, N. V. (1996).A branch-and-reduce approach to global optimization.Journal of Global Optimization, 8:107–139.

Savelsbergh, M. W. P. (1994).Preprocessing and probing techniques for mixed integer programmingproblems.ORSA Journal on Computing, 6:445–454.

Saxena, A., Bonami, P., and Lee, J. (2008).

The Future of MINLP?

Disjunctive cuts for non-convex mixed integer quadraticallyconstrained programs.Manuscript, available on Optimization Online,(www.optimization-online.org).

Sherali, H. and Adams, W. (1998).A Reformulation-Linearization Technique for Solving Discrete andContinuous Nonconvex Problems.Kluwer, Dordrecht.

Smith, E. M. B. and Pantelides, C. C. (1997).Global optimization of nonconvex MINLPs.Computers & Chemical Engineering, 21:S791–S796.

Sridhar, S., Linderoth, J., and Luedtke, J. (2013).Locally ideal formulations for piecewise linear functions with indicatorvariables.Operations Research Letters, 41:627–632.

Stubbs, R. and Mehrotra, S. (1999).

The Future of MINLP?

A branch-and-cut method for 0-1 mixed convex programming.Mathematical Programming, 86:515–532.

Tawarmalani, M. and Sahinidis, N. V. (2002).Convexification and Global Optimization in Continuous andMixed-Integer Nonlinear Programming: Theory, Algorithms, Software,and Applications.Kluwer Academic Publishers, Boston MA.

Vandenbussche, D. and Nemhauser, G. L. (2005).A polyhedral study of nonconvex quadratic programs with boxconstraints.Mathematical Programming, 102:531–557.

Vielma, J. P. and Nemhauser, G. (2011).Modeling disjunctive constraints with a logarithmic number of binaryvariables and constraints.Mathematical Programming, 128(1):49–72.

Westerlund, T. and Pettersson, F. (1995).

The Future of MINLP?

A cutting plane method for solving convex MINLP problems.Computers & Chemical Engineering, 19:s131–s136.

Yajima, Y. and Fujie, T. (1998).Polyhedral approach for nonconvex quadratic programming problemswith box constraints.Journal of Global Optimization, 13(2):151 – 170.