The DD-αAMG Solver LibraryA Multigrid Solver Library for Wilson-Clover Fermions

Matthias Rottmann

contributions by: A. Frommer, K. Kahl, B. Leder, S. Krieg, A. Strebel, S. Heybrock,S. Bacchio, K. Szabo, W. Soldner and many others

Bergische Universitat Wuppertal

Lattice 2016, Southampton

Introduction Library Features Progress Report Summary & Outlook

Outline

Introduction

Library Features

Progress Report

Summary & Outlook

M. Rottmann, DD-αAMG solver library 1/14

Introduction Library Features Progress Report Summary & Outlook

Adaptive Algebraic Multigrid Approach

Two-grid error propagator for ν steps of post-smoothing

E(ν)2g = (1−MD)ν︸ ︷︷ ︸

smoother

(1− PD−1c P †D)︸ ︷︷ ︸coarse grid correction

, Dc := P †DP︸ ︷︷ ︸coarse operator

I low accuracy for D−1c and M is sufficient

I allows for introducing recursive construction for Dc

To Do: Define interpolation P and smoother M

DD-αAMG

M : Schwarz Alternating Procedure (SAP)[Hermann Schwarz 1870; Martin Luscher 2003]

P : Aggregation Based Interpolation[Brannick, Clark et al. 2010]

M. Rottmann, DD-αAMG solver library 2/14

Introduction Library Features Progress Report Summary & Outlook

The Algebraic Multigrid Principle

Smoother: 1−MD

I effective on “large” eigenvectors

I “small” eigenvectors remain

SAP

0

0.2

0.4

0.6

0.8

1

0 500 1000 1500 2000 2500 3000

||(I−MD)v i||/||v

i||

number i of eigenvalue λi

Dvi = λivi with |λ1| ≤ . . . ≤ |λ3072|

M. Rottmann, DD-αAMG solver library 3/14

Introduction Library Features Progress Report Summary & Outlook

Aggregation Based Interpolation

Construction

I define aggregates: domain decomposition A1, ...,As

A2

A1

A4

A3

P

P †

I calculate test vectors w1, ..., wN

I decompose test vectors over aggregates A1, ...,As

(w1, . . . , wN) = =A2

A1

As

→ P =

A1

A2

As

M. Rottmann, DD-αAMG solver library 4/14

Introduction Library Features Progress Report Summary & Outlook

The Algebraic Multigrid Principle

Coarse-grid correction: 1− PD−1c P †D

I small eigenvectors w1, . . . , wN built into interpolation P

⇒ effective on small eigenvectors

SAP

0

0.2

0.4

0.6

0.8

1

0 500 1000 1500 2000 2500 3000

||(I−MD)v i||/||v

i||

number i of eigenvalue λi

Aggregation + Low Modes

0

0.2

0.4

0.6

0.8

1

0 500 1000 1500 2000 2500 3000

||(I−PD−1

cP† D

)v i||/||v

i||

number i of eigenvalue λi

Dvi = λivi with |λ1| ≤ . . . ≤ |λ3072|

M. Rottmann, DD-αAMG solver library 5/14

Introduction Library Features Progress Report Summary & Outlook

The Algebraic Multigrid Principle

Coarse-grid correction: 1− PD−1c P †D

I small eigenvectors w1, . . . , wN built into interpolation P

⇒ effective on small eigenvectors

SAP

0

0.2

0.4

0.6

0.8

1

0 500 1000 1500 2000 2500 3000

||(I−MD)v i||/||v

i||

number i of eigenvalue λi

Aggregation + Low Modes

0

0.2

0.4

0.6

0.8

1

0 500 1000 1500 2000 2500 3000

||(I−PD−1

cP† D

)v i||/||v

i||

number i of eigenvalue λi

Dvi = λivi with |λ1| ≤ . . . ≤ |λ3072|

M. Rottmann, DD-αAMG solver library 5/14

Introduction Library Features Progress Report Summary & Outlook

The Algebraic Multigrid Principle

Two-grid method: E2g = (1−MD)(1− PD−1c P †D)

I complementarity of smoother and coarse-grid correction

I effective on all eigenvectors!

DD-αAMG

0

0.2

0.4

0.6

0.8

1

0 500 1000 1500 2000 2500 3000

||E2gv i||/||v

i||

number i of eigenvalue λi

Dvi = λivi with |λ1| ≤ . . . ≤ |λ3072|

M. Rottmann, DD-αAMG solver library 5/14

Introduction Library Features Progress Report Summary & Outlook

Adaptive Setup: How to Obtain Information about Small Eigenvectors

Inverse iteration with the method itself

start with randomtest vectors w1, . . . , wN

smooth all wia few times

build P and Dc

P good?apply method itself and

solve Dϕi = wi ∀i

set wi ← ϕi andorthonormalize w1, . . . , wN

stop

no

yes

add recurrence scheme → multilevel setup

M. Rottmann, DD-αAMG solver library 6/14

Introduction Library Features Progress Report Summary & Outlook

Performance and Setup Costs

M. Rottmann, DD-αAMG solver library 7/14

BMW-c cnfg, 3HEX-smeared 64× 643, 128 cores

30

50

200

400

100

1000

10000

−0.02−0.03−0.04−0.05mdmumcrit

mudti

me

toso

luti

on(i

nse

con

ds)

m0

mp oe BiCGStabtwo-level DD-αAMG

three-level DD-αAMGfour-level DD-αAMG

A. Frommer, K. Kahl, S. Krieg, B. Leder, R.→ arXiv:1303.1377,1307.6101

Introduction Library Features Progress Report Summary & Outlook

Performance and Setup Costs

M. Rottmann, DD-αAMG solver library 7/14

BMW-c cnfg, 3HEX-smeared 64× 643, 128 cores

30

50

200

400

100

1000

10000

−0.02−0.03−0.04−0.05mdmumcrit

mudse

tup

tim

e+

tim

eto

solu

tion

(in

seco

nd

s)

m0

mp oe BiCGStabtwo-level DD-αAMG

three-level DD-αAMGfour-level DD-αAMG

A. Frommer, K. Kahl, S. Krieg, B. Leder, R.→ arXiv:1303.1377,1307.6101

Introduction Library Features Progress Report Summary & Outlook

The DD-αAMG Solver Library

Technical Features

I C-code

I MPI + OpenMP

I SSE intrinsics

I library interface

I integrated profiling

I small parameter set forusers

I large parameter set fordevelopers

I debug information

I GNU GPL

Code Features

I Wilson+Clover

I AMG with arbitrarynumber of levels

I idling processes

I different smoothers:SAP, additive Schwarz,GMRES

I reference methods:BiCGStab, CGN,GMRES

I mixed precision,odd-even preconditioned

M. Rottmann, DD-αAMG solver library 8/14

available on github.com/DDalphaAMG

Introduction Library Features Progress Report Summary & Outlook

The DD-αAMG Library Interface

Interface Features

I conversion routines for configurations and vectors

I setup routine (initial setup)

I configuration update routine (→ HMC)

I setup update routine (→ HMC)

I mass update routines (→ Hasenbusch trick)

I solver routine (inversion for 1 RHS)

I GNU GPL → feel free to adjust it to your needs

M. Rottmann, DD-αAMG solver library 9/14

available on github.com/DDalphaAMG

Introduction Library Features Progress Report Summary & Outlook

Progress Report: Existing Branches

DD-αAMG for KNC/KNL

I SAP smoother implemented from scratch

I coarse grid correction from DD-αAMG

I DD-αAMG inherited SIMD vectorization from KNC-code

S. Heybrock, D. Richtmann, P. Georg, T. Wettig, R.→ arXiv:1512.04506,1601.03184

DD-αAMG for twisted mass fermionsI the interpolation P in DD-αAMG respects γ5P = Pγc5=⇒ D + iµγ5 Dc + iµγc5

S. Bacchio, J. Finkenrath, C. Alexandrou, A. Frommer, K. Kahl, R.→ Talk by S. Bacchio

M. Rottmann, DD-αAMG solver library 10/14

Introduction Library Features Progress Report Summary & Outlook

Progress Report: Future Branch – Eigensolver for γ5D

48× 243 CLS cnfg, mπ ≈ 290MeV

I Jacobi-Davidson with amultigrid solver

I outperforms PARPACK for400 eigenpairs

I improved scaling with latticevolume

100

1000

10000

50 100 150 200 250 300 350 400

solv

eti

me

(in

core

-h)

number of eigenpairs (λ, vλ)

PARPACKJacobi-Davidson

A. Strebel, J. Simeth, G. Bali, S. Collins, A. Frommer, K. Kahl, I. Kanamori,B. Muller, R.→ arXiv:1509.06865

M. Rottmann, DD-αAMG solver library 11/14

Introduction Library Features Progress Report Summary & Outlook

Outlook

I eigensolver: improve algorithm(clever restart strategies, etc.)=⇒ publication

100

1000

10000

50 100 150 200 250 300 350 400

solv

eti

me

(in

core

-h)

number of eigenpairs (λ, vλ)

PARPACKJacobi-Davidson

I overlap solver: prepare code for publicationJ. Brannick, A. Frommer, K. Kahl, B. Leder, A. Strebel, R.→ arXiv:1410.7170

−2

−1

0

1

2

−1 0 1 2 3 4 5 6 7

imag

inar

yax

is

real axis

spec(DW )spec(DN )

spec(DND−1W )

I vectorization: generalization, integrate AVX

Instruction Unit

ALU ALU . . . ALU

Mem Mem . . . Mem

M. Rottmann, DD-αAMG solver library 12/14

Introduction Library Features Progress Report Summary & Outlook

Summary & Outlook

M. Rottmann, DD-αAMG solver library 13/14

DD-αAMGSFB/TR55

Wilson Solver

OverlapSolver

Eigensolverfor γ5D

DynQCDBMW-c

KNC-codefor QPACE2/3

OpenQCDLuscher/CERN

Twisted MassHPC-LEAP

github.com/DDalphaAMG

Introduction Library Features Progress Report Summary & Outlook

DD-αAMG

M. Rottmann, DD-αAMG solver library 14/14

github.com/DDalphaAMG