-extrapolation on 3D semi-structured nite element meshes · ˝-extrapolation on 3D semi-structured...

The HHG Framework τ -extrapolation Scalability

τ -extrapolation on 3D semi-structured finiteelement meshes

European Multi-Grid Conference EMG 2010

Bjorn GmeinerJoint work with: Tobias Gradl, Ulrich Rude

September, 2010

Department for Computer Science 10 (System Simulation)

2


Contents

The HHG Framework

τ -extrapolation

Scalability


3


The HHG Framework


4


Combining finite element and multigrid methods

FE mesh may be unstructured.What nodes to remove for coarsening? Not straightforward!Why not start from the coarse grid?

The Hierarchical Hybrid Grids (HHG)concept

• Benjamin Bergen*: prototype

• Tobias Gradl: tuning, extensionsand adaptivity

* Dissertation in Erlangen, ISC award in 2005. Currently at Los Alamos Labs.


5


Properties of the HHG approach

Advantages

• Multigrid is straightforward

• Very memory efficient

• Massive performance benefits oncurrent computer architectures

• Subserves parallelization

⇒ 1012 unknowns are possible

Limitation

• Coarse input grid needed

• Adaptivity (ongoing work by TobiasGradl)


6


τ -extrapolation


7


Overview

• Achieving higher accuracy by the combination of two gridsand error expension 1

• For 2-D unstructured triangular meshes it was shown:One step of multigrid τ -extrapolation for piecewise linear C 0

finite element methods is equivalent to using quadraticelements 2

• A lack of regularity causes larger discretization errors at theinterfaces between two structured regions 3

• How severe are these discretization errors on semi-structuredmeshes in praxis?

1A. Brandt, Multigrid Techniques: 1984 Guide with Applications to FluidDynamics, GMD-Studien, Bonn (1984)

2M. Jung, U. Rude, Implicit Extrapolation Methods for Multilevel Finite ElementComputations, SIAM J. Sci. Comput., Vol 17 (1996)

3H. Blum, Asymptotic Error Expansion and Defect Correction in the Finite ElementMethod, Institut fur Angewandte Mathematik, Heidelberg (1990)


8


Motivation: The (local) truncation error

PDE with a differential operator N:

N(u) = f

Restriction to a grid by restriction R:

R(N(u)) = R(f )

Adding a discretized differential operator applied to the solutionNk(R(u)):

Nk(R(u)) = R(f ) + Nk(R(u))− R(N(u))︸︷︷︸local truncation error τh

Can we estimate the local truncation error in order to improveour approximation?


9


Full Approximation Storage Formulation

Solve the coarse grid equation

Nk−1(wk−1) = R(fk − Nk(uk))︸︷︷︸defect dk−1

+Nk−1(R(uk)).

and correct the solution on the fine grid by (wk−1 − R(uk)).

Rearranging the right hand side yields

Nk−1(wk−1) = R(fk) + Nk−1(R(uk))− R(Nk(uk))︸︷︷︸relative trunction error τ2hh

.

Using τ2hh as an approximation of the local truncation error

τh = Nk−1(R(u))− R(N(u)) for the coarse grid.


10


τ -Extrapolation on Structured Tetrahedral Grids...

1 2 3 4 5 6 710

−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

Level of Refinement

Dis

cret

izat

ion

Err

or

Correction Schemeτ−extrapolation

Regular Tetraeder


11


Two structured regions connected by its faces

I

I 3I

Problem: u = sinh(x)cos(y)cos(z)


12


Correction Scheme on a line

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.9

−0.8

−0.7

−0.6

−0.5

−0.4

−0.3

−0.2

−0.1

0x 10

−7

x−Axis

Dis

cret

izat

ion

Err

or

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−2.5

−2

−1.5

−1

−0.5

0

0.5x 10

−6

x−Axis

Dis

cret

izat

ion

Err

or

I

I 3I


13


τ -extrapolation on a line

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−3

−2.5

−2

−1.5

−1

−0.5

0

0.5x 10

−10

x−Axis

Dis

cret

izat

ion

Err

or

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−14

−12

−10

−8

−6

−4

−2

0

2

4x 10

−10

x−Axis

Dis

cret

izat

ion

Err

or

I

I 3I


14


Numerical Experiments on the European SupercomputerBlue Gene/P in Julich (Jugene)

• Compute node: 4-way SMP processor

• Processortype: 32-bit PowerPC 450 core 850 MHz

• Processors: 294 912

• Overall peak performance: 1 Petaflops

• Main memory: 2 Gbytes per node (aggregate 144 TB)


15


A complexer geometry (coarsest mesh)

Top view

Side view


16


Discretization Errors

• Struct. Regions: 16028

• Utilized number of cores: 16028

• Problem: −∆u = sin(x)sin(y)sin(z)

• Smoother: µ(6, 0) GS

• Coarse grid solver: CG

Levels Unknowns Correction Scheme τ -extrapolation1 1.59 · 105 9.74 · 10−5 -2 1.32 · 106 2.53 · 10−5 8.85 · 10−6

3 1.00 · 107 6.38 · 10−6 1.48 · 10−6

4 8.68 · 107 1.59 · 10−6 3.20 · 10−7

5 6.97 · 108 3.97 · 10−7 7.79 · 10−8

6 5.59 · 109 9.90 · 10−8 1.95 · 10−8

7 4.48 · 1010 2.47 · 10−8 4.90 · 10−9


17


Scalability


18


Scalability of HHG on Blue Gene/P (Jugene)

512 1024 2048 4096 81920

2

4

6

8

10

12

14

16

Cores

Sp

eed

up

no

rmal

ized

to

512

co

res

strong scaling

ideal

Figure: Strong Scaling behavior of HHG on PowerPC 450 cores of a BlueGene/P at Julich. This test case was performed solving 2.14 · 109

unknowns.


19


E-Scalability

Multigrid is optimal but not E-scalable!

• This means the parallel efficiency decreases with increasingnumber of processors, while keeping the number of grid pointsper processor constant. :-(

The problem: What to do with the coarsest grids?

• A simple strategy: Truncate the coarsest grids and then justapproximate on the coarsest grid!

But is it applicable for very many processors and thus quite finecoarsest grids?!?


20



Cores Struct. Regions Unknowns CG Time128 1 536 534 776 319 15 5.64

256 3 072 1 070 599 167 20 5.66512 6 144 2 142 244 863 25 5.69

1024 12 288 4 286 583 807 30 5.712048 24 576 8 577 357 823 45 5.754096 49 152 17 158 905 855 60 5.928192 98 304 34 326 194 175 70 5.86

16384 196 608 68 669 157 375 90 5.9132768 393 216 137 355 083 775 105 6.1765536 786 432 274 743 709 695 115 6.41

131072 1 572 864 549 554 511 871 145 6.42262144 3 145 728 1 099 176 116 223 180 6.81


20



Cores Struct. Regions Unknowns CG Time128 1 536 534 776 319 15 5.64256 3 072 1 070 599 167 20 5.66512 6 144 2 142 244 863 25 5.69

1024 12 288 4 286 583 807 30 5.712048 24 576 8 577 357 823 45 5.754096 49 152 17 158 905 855 60 5.928192 98 304 34 326 194 175 70 5.86

16384 196 608 68 669 157 375 90 5.91

32768 393 216 137 355 083 775 105 6.1765536 786 432 274 743 709 695 115 6.41

131072 1 572 864 549 554 511 871 145 6.42262144 3 145 728 1 099 176 116 223 180 6.81


20




1024 12 288 4 286 583 807 30 5.712048 24 576 8 577 357 823 45 5.754096 49 152 17 158 905 855 60 5.928192 98 304 34 326 194 175 70 5.86

16384 196 608 68 669 157 375 90 5.9132768 393 216 137 355 083 775 105 6.1765536 786 432 274 743 709 695 115 6.41

131072 1 572 864 549 554 511 871 145 6.42

262144 3 145 728 1 099 176 116 223 180 6.81


20




1024 12 288 4 286 583 807 30 5.712048 24 576 8 577 357 823 45 5.754096 49 152 17 158 905 855 60 5.928192 98 304 34 326 194 175 70 5.86

16384 196 608 68 669 157 375 90 5.9132768 393 216 137 355 083 775 105 6.1765536 786 432 274 743 709 695 115 6.41

131072 1 572 864 549 554 511 871 145 6.42262144 3 145 728 1 099 176 116 223 180 6.81


21


Observations

• τ -extrapolation seems to work on 3d structured FE meshes

• We see no limitation for the scalability of geometric MG onFE up to 262k cores

Outlook

• Understanding the effects of τ -extrapolation at the faces,edges and vertices on semi-structured meshes

• Finding a solution to keep O(h4) order consistency withincreasing size of the structured regions


22


Thank you for you attention!

Any questions?

The development of HHG was funded by

• the Elite Network of Bavaria within the InternationalDoctorate Program ”Identification, Optimization and Controlwith Applications in Modern Technologies”

• KONWIHR


-extrapolation on 3D semi-structured nite element meshes · ˝-extrapolation on 3D semi-structured...

Documents

Transcript of -extrapolation on 3D semi-structured nite element meshes · ˝-extrapolation on 3D semi-structured...