A lambda-Dynamics Module for GROMACS

28
Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation Summary A λ-Dynamics Module for GROMACS R. Thomas Ullmann Theoretical and Computational Biophysics Max Planck Institute for Biophysical Chemistry Göttingen, Germany Gromacs Workshop Göttingen May 19 2016

Transcript of A lambda-Dynamics Module for GROMACS

Page 1: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation Summary

A λ-Dynamics Module for GROMACS

R. Thomas UllmannTheoretical and Computational BiophysicsMax Planck Institute for Biophysical ChemistryGöttingen, Germany

Gromacs WorkshopGöttingen May 19 2016

Page 2: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryIntroduction to the GromEx projectExample: Protonation Triggers Opening of the Bacterial

Pentameric Ligand Gated Ion Channel (pLGIC) GLIC I

withCarstenKutzner,RudolfSchemm &Julian T.Brennecke

pLGICs mediate fast synaptic transmission in brain and muscle

Page 3: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryIntroduction to the GromEx projectExample: Protonation Triggers Opening of the Bacterial

Pentameric Ligand Gated Ion Channel (pLGIC) GLIC II

110 Å

simulation system:

300,000 atoms

420 protonatable sites

Page 4: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryIntroduction to the GromEx projectThe Two Major Challenges in the GromEx Project

Increase model realism by adding variableprotonationScaling to 10,000s – 1,000,000 coreså Ongoing work to develop a combinedλ-dynamics/fast multipole method forGROMACS

Page 5: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation Summaryλ-Dynamics as Way to Model Chemically Variable SitesDynamical Protonation through λ-dynamics

protonatedform

deprotonated form

Smooth Interconversion Between Site Forms with λ-Dynamics• B. Tidor, 1993, J. Phys. Chem., 97, 1069–1073 • X. Kong & C. L. Brooks, 1996, J. Phys. Chem., 105, 2414–2423• S. Donnini, F. Tegeler, G. Groenhof & H. Grubmüller, 2011, J. Chem. Theory Comput., 7, 1962–1978• S. Donnini, R. T. Ullmann, G. Groenhof & H. Grubmüller, 2016, J. Chem. Theory Comput., 12, 1040–1051

Page 6: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation Summaryλ-Dynamics as Way to Model Chemically Variable Sitesλ-dynamics on unit spheres, circles & hyper-spheres

φ1

φ2

λ2λ1

λ2

λ3

λ1

λ3

π+

π-

π2λ2

λ1λ2

λ1

φ1π

π23

x2

x1

x3

λ2

λ1

λ3

2 forms on a circle 3 forms on a sphere• n forms on an n -dimensional hypersphere• n λ variables and n Cartesian coordinates xin∑ix2i =

n∑iλi = R 2 def

= L with λi def= x2i

• actual dynamics simulated in space ofm − 1independent, angular coordinates (φ1, ..., φn−1)

Page 7: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation Summaryλ-Dynamics as Way to Model Chemically Variable SitesComputing Electrostatic Interactions With the

Fast Multipole Method (FMM) Enables Efficient Simulations on

Future Supercomputers with 10,000s to 1,000,000 CPU Cores

scalingPME FMM

atoms (N) O(n log n) O(N)

sites (s) O(s) O(1)

communicationnodes (p) O(p2) O(p log p)~~

~~

scaling bottlenecks

• C. Kutzner, R. Apostolov, B. Hess & H. Grubmüller, in Parallel Computing: Accelerating Computational Scienceand Engineering (CSE) , 722–730, IOS Press • C. Kutzner, S. Páll, M. Fechner, A. Esztermann, B. de Groot & H.Grubmüller, 2015, J. Comput. Chem., 36, 1990–2008 • I. Kabadshow & H. Dachsel 2012, IAS Series, 11, FZ Jülich

Page 8: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryOverviewThe GromEx Project: Current State & Near Future

λ-dynamics

GROMACS

FMMenergies + forces

for λ

GPU-FMM

input format beyond

A/B topologies for λ sites

energies + forces

for λ

() CPU

reference implementation

to be discussed

enhance kernels for LJ + bonded

interactions for λ sites

Currently assembling the first workingversion:protonatable sites with two formsper site based on the current A & Btopologies, input files and freeenergy kernelscharge neutralization using aparsimonious proton bufferFMM with λ-support withnode-level parallelization

Page 9: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryOverviewThe GromEx Project: Next Step

λ-dynamics

GROMACS

FMMenergies + forces

for λ

GPU-FMM

input format beyond

A/B topologies for λ sites

energies + forces

for λ

enhance kernels for LJ + bonded

interactions for λ sites

to be done for full gain from the newframework (constant-µ):add support for multiple localtopologies to the topology datastructures, .rtp, .top and .tng files(to be discussed)adapt bonded & Lennard-Jonesfree energy kernels to multiplelocal topologiesfull parallelization of the FMMopen questions to be discussed:

How to tell the FMM whichresources it is allowed to use?How to best utilize the currentdomain decomposition

Page 10: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryOverviewThe GromEx Project: Next Step

GROMACS

FMMenergies + forces

for λ

GPU-FMM

input format beyond

A/B topologies for λ sites

energies + forces

for λ

enhance kernels for LJ + bonded

interactions for λ sites

λ-dynamics

to be done for full gain from the newframework (constant-µ):add support for multiple localtopologies to the topology datastructures, .rtp, .top and .tng files(to be discussed)adapt bonded & Lennard-Jonesfree energy kernels to multiplelocal topologiesfull parallelization of the FMMopen questions to be discussed:

How to tell the FMM whichresources it is allowed to use?How to best utilize the currentdomain decomposition

Page 11: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryCommon PrinciplesInvariants for all λ-Dynamics variants

A site a can have n site forms with their respectivedimensionless weight factors λ = λ1, . . . , λnEach site obeys the constraint∑ni λi = L , where,canonically, L = 1.The actual dynamics takes place in a derived coordinatespace formed by the so-called simulation coordinates

s = s1, . . . , sm and the maximum λ value L .Coordinate transformations map between the coordinatespacesφ1

φ2

λ2λ1

λ2

λ3

λ1

λ3

π+

π-

π2λ2

λ1λ2

λ1

φ1π

π23

x2

x1

x3

λ2

λ1

λ3

Page 12: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryCommon PrinciplesThe Algorithm for a λ-Dynamics Time Step

Compute the potential energy E pot as function of thephysical coordinates and λ coordinates obtained in theprevious step.→ Fast Multipole Method, Electrostatics sessionFor each site a , Compute the vector of forces F i acting onthe λ particle along the simulation coordinates

F a =

[∂E pot

∂λa

]T [∂λa∂s a

]+ β−1

∂ ln√∣∣λgij ∣∣∂s a

Propagate the positions and velocities in simulationcoordinate spaceTransform the updated simulation coordinates s back tothe λ coordinate space

Page 13: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryCommon PrinciplesThe Choice of Simulation Coordinates Distinguishes λ-dynamics variants

Simulation coordinates can be chosen with the following aimsimplicitly satisfy the constraint∑n

i λi = Lenhance sampling efficiency by facilitatinginterconversion between site forms

Fully implemented variants:λ-dynamics on hyperspheres

circles (2-sphere) for two forms, spheres (3-sphere) forthree forms, n -sphere for n formsm = n − 1 simulation coordinates

Nexp λ-dynamics (J. Knight & C. Brooks, 2011,J. Comput. Chem., 32, 3423–3432)no simple geometric interpretationm = n simulation coordinates

It’s easy to implement more variants.

Page 14: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryThe Bias PotentialA Bias Potential Designed to Ensure Sampling of Physical States

and to Optimize Sampling Efficiency

Ωi =

( ni∑k

(λi ,kL

)2)−1

E bias = E bias, global +Nsites∑iE bias, sitei

E bias, sitei = K site

(Ωi − 1)2s

E bias, global = K global

Nsites∏i

Ωi − 12t

Page 15: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryThe Bias PotentialThe Effect of Barrier Height And Shape on the Transition Rate as

Predicted by Eyring Theory for a Site with Two Forms

0

1

0.0 0.5 1.0

λ

S = 0.05

0

1

barr

ier

pote

ntial/barr

ier

heig

ht

S = 0.35harmonic

0

1 S = 4.0

harm

onic

S = 0.00

S = 0.01S = 0.02

S = 0.05

S =

0.1S

= 0

.35

S =

4.0

0

20

40

60

80

100

120

140tr

ansitio

n r

ate

[ns

−1]

0 1 2 3 4 5 6

barrier height [kcal/mol]

0102030405060708090

100110120130140

Eyring tra

nsitio

n r

ate

[ns

−1]

140140140 100100100

100

10

01

00

10

0

808080

80

80

80

80

606060

60

60

60

60

4040

40

40

40

40

2020

20

20

20

20

1010

10

10

10

10

55

55

55

2.52.5

2.5

2.5

2.5

2.5

1

1

11

11

0.5

0.5

0.5

0.5

0.5

0.25

0.25

0.2

50.2

50.2

5

0.1

0.1

0.1

0.1

0.1

0.025

0.0

25

0.0

25

0.0

25

0.0

25

0.01

0.0

1

0.0

10.0

10

.01

0.0025

0.0

025

0.0

025

0.0

025

0.001

0.0

01

0.0

01

0.0

01

0.0

0025

0.0

0025

0.0

0025

0.0

00

25

0.0

001

0.0

001

0.0

001

0.0

00

1

0.0

00025

0.0

00025

0.0

00025

0.0

00025

1e−

05

1e−

05

1e−

05

0.0

000025

0.0

000025

0.0

000025

1e−

06

0.0

0000025

1e−

08

0.0

00000025

1e−

09

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

shape p

ara

mete

r S

0 2 4 6 8 10 12

barrier height [kcal/mol]

Page 16: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryThe Bias PotentialThe Effect of Barrier Height And Shape on Physical Purity and

Simulation Stability for a Site with Two Forms

1.0

1.1

1.2

1.3

1.4

1.5

effectively

popula

ted s

tate

s

1.551.551.55

1.5

5

1.5

51.5

51.5

5

1.51.51.5

1.5

1.5

1.5

1.5

1.451.451.45

1.4

51.4

51.4

51.4

5

1.41.41.4

1.4

1.4

1.4

1.4

1.351.35

1.35

1.3

5

1.3

5

1.3

5

1.35

1.31.3

1.3

1.3

1.3

1.3

1.3

1.251.25

1.2

5

1.25

1.25

1.25

1.21.2

1.2

1.2

1.2

1.2

1.151.15

1.15

1.15

1.15

1.11.1

1.1

1.1

1.081.08

1.081.08

1.06

1.06

1.06

1.04

1.041.04

1.02

1.02

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

shape p

ara

mete

r S

0 2 4 6 8 10 12

barrier height [kcal/mol]

10−2

10−1

100

101

102

103

104

105

106

107

maxim

um

forc

e [kJ

−1]

111

11

1

101010

10

10

10

100100100

100

100

100

100100100

100

100010001000

100010001000

100001000010000

100001000010000

100000100000100000

100000 100000 100000 100000

1000000100000010000001000000 1000000 1000000

10000000202020

20

20

20

20

303030

30

30

30

30

404040

40

40

40

40

505050

50

50

50

5050

606060

60

60

60

6060

707070

70

70

70

7070

808080

80

80

80

808080

909090

90

90

90

909090

110110110

110

110

110

110110110

110

120120120

120

120

120

120

120120120

130130130

130130130

130

130

130

130

140140140

140140140

140

140

140

140

150150150

150150150

150

150

150

150

160160160

160160160

160

160

160

170170170

170170170

170

170

170

180180180

180180180

180

180

180

190190190

190190190

190

190

190

200200200

200200200

200

200

200

210210210

210210210

210

210

220220220

220220220

220

220

230230230

230230230

230

230

240240240

240240240

240

240

250250250

250250250

250

250

260260260

260260260

260

270270270

270270270

270

280280280

280280280

280

290290290

290290290

290

300300300

300300300

300

310310310

310310310

320320320

320320320

330330330

330330330

340340340

340340340

350350350

350350350

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

shape p

ara

mete

r S

0 2 4 6 8 10 12

barrier height [kcal/mol]

Page 17: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryImplementationThe Most Important Objects of the λ-Site Module

LambdaSiteModuleLambdaSiteData

global arraysLambdaSite

LambdaSiteInternalsImplHypersphereImplNexpLambdaSiteSimFunc

LambdaSiteBiasPotentialImplHarmonicImplOmega

outside code: FMM, ...coordinate transformations etc.

Page 18: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation SummaryImplementationImplementation Details

class templates for exchangeability of data types (e.g.,float or double) and allocatorspolymorphic objectsabstract parent classes define pure virtual functions toensure that all daughter objects provide the samefunctionality and function signaturesanalytical functions for all quantities (potentials,coordinates, metric tensor determinants and their firstand second partial derivatives)all math functions are tested against numerical referencevaluesunit testing is automated for all implementations and allfloating point data types – adding a new implementationdoes not require writing new testsgeneral, allocator-aware array classes for data exchange,e.g., with the FMM and for computationSIMD support started, complete: vector-matrix products

Page 19: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation Summary

Outlook: Full Chemical Variability in Molecular Simulations

Page 20: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation Summary

Summary – Constant-pH & Beyond

First working version close to release (constant-pH).

The developed general λ-dynamics/FMM method will allowfor:modeling (bio)molecules in greater physical detail byenabling chemical variabilityimplementing physically correct electrostatic long-rangeinteractions also in presence of chemically variable sitesλ-dynamics simulations at small, constant overheadrelative to standard MDO (#sites) → ≈ O (1) (< 20% overhead)

Page 21: A lambda-Dynamics Module for GROMACS

Introduction The GromEx Project The λ-Dynamics Module: Physics & Implementation Summary

Acknowledgements

Bartosz Kohnke, Plamen Dobrev,Carsten Kutzner & Helmut GrubmüllerMPI for Biophysical Chemistry, Göttingen, GermanyIvo Kabadshow, Andreas Beckmann,David Haensel & Holger DachselJülich Supercomputing Centre, GermanySerena Donnini & Gerrit GroenhofUniversity of Jyväskylä, FinlandBerk Hess and other GROMACS developersKTH Royal Institute of Techology, Stockholm, Sweden

Page 22: A lambda-Dynamics Module for GROMACS

Discussion

Discussion

Discussion

Page 23: A lambda-Dynamics Module for GROMACS

DiscussionFMMA Fast Multipole Method for the Exascale

(Current Implementation, ongoing progress)

Scalability to largecore-counts – avoidcommunication

Optimal utilization ofHeterogeneous Hardware(CPUs + GPUs)

para

llel e

ffici

ency

21 24 27 210 213 2160%

50%

100%

number of MPI ranks

replication factor

1 2 4 816

64

128

32256

0.11

10100

1000

0 5 10 15 20 25

seco

nds

multipole order

CPU baseline

GPU

Page 24: A lambda-Dynamics Module for GROMACS

Discussionλ-Dynamics on Hyperspheresλ-dynamics on unit spheres, circles & hyper-spheres

φ1

φ2

λ2λ1

λ2

λ3

λ1

λ3

π+

π-

π2λ2

λ1λ2

λ1

φ1π

π23

x2

x1

x3

λ2

λ1

λ3

2 forms on a circle 3 forms on a sphere• n forms on an n -dimensional hypersphere• n λ variables and n Cartesian coordinates xin∑ix2i =

n∑iλi = R 2 def

= L with λi def= x2i

• actual dynamics simulated in space ofm − 1independent, angular coordinates (φ1, ..., φn−1)

Page 25: A lambda-Dynamics Module for GROMACS

Discussionλ-Dynamics on HyperspheresThe Potential Energy Landscape on a Unit Sphere

E pot(λ) = E site + E bias

• E bias = K 22s (Ω− 1)2 s withbarrier steepness s and barrier height between two forms K• Ω = 1/∑n

i λ2i is the effective number of forms contributing to thecurrent state→ discourage population of unphysical states andprefer interconversion between pairs of forms

Page 26: A lambda-Dynamics Module for GROMACS

DiscussionNexp λ-DynamicsNexp λ-Dynamics

λi = L exp [C sin θi ]∑j exp [C sin θj ]

m = n angular simulation coordinates θ are mapped to n λcoordinates, where C is the steepness constant.The opposite mapping is, as in the hypersphere variant, not unique:reset all λ values to the realizable range of valuesλmin ≤ λi ≤ λmax, record the index j of the maximum λ value inthe course.

λmin =exp [−C ]

(N − 1) exp [+C ] + exp [−C ]

λmax =exp [+C ]

(N − 1) exp [−C ] + exp [+C ]

assign all λi according toθi = asin

[C−1 ln

[λiλj

]+ sin θj

]

Page 27: A lambda-Dynamics Module for GROMACS

DiscussionThe Extended Phase SpaceThe Extended Hamiltonian

two global topologies– conceivable as asingle site with twoforms

Nsites sites withmultiple localtopologies

H = λ1E pot1 + λ2E pot2 + E bias

+

realparticles∑r

[ p2r2mr]

+pλ22mλ

H =

Nsites∑a

na∑i

[λa ,i E site

a ,i

]

+

Nsites∑b

nb∑j

Nsites∑c 6=a

nc∑k

[λb ,jλc ,kWb ,j ,c ,k

]+ E bias

+

realparticles∑r

[ p2r2mr]

+

Nsites∑d

md∑l

[p sd ,l2

2mλd ,l

]

Page 28: A lambda-Dynamics Module for GROMACS

DiscussionThe Extended Phase SpaceThe Extended Phase Space: Momentum Integrals Cancel from

Free Energy Differences in Classical Statistical Thermodynamics

z = q, q, λ, s z′ = q, q, λ, λ

Zi =

∫Γi

exp [−βH (z)]dz

Zi =

∫Γi

(E pot (q, λ)dqdλ

) +∞∫−∞

(E kin,q (q)dq

) +∞∫−∞

(E kin,s (s)ds

)

∆FA→B = −β−1 ln[ZBZA

]