Accelerating Shallow Water Flow and Mass Transport Using Lattice Boltzmann Methods on GPUs

Kevin Tubbs

Dell | Global HPC Solutions Engineering

HPC Engineering

Introduction

HPC Engineering

Background and Overview

3

Role of CFD

• Improve Environmental Predictions

• Used to Improve Decision and Policy Making

• Test and Evaluate Preventative Methods

HPC Engineering

Dell Global HPC Solutions Engineering

4 Confidential

Performance Analysis & Characterization - Node level - Cluster level

Power Consumption and Energy Analysis

Benchmark and Application Analysis - Industry Standard Benchmarks & Apps: HPL, NPB, NAMD - Focus Area Specific: CFD, Climate & Weather, Molecular Dynamics, etc.

HPC Engineering

Why Accelereyes ? Programmability

Faster

Slower

Time - Consuming Easy-to-use

Writing Kernels

Using Libraries

Instruction Set Optimization

Compiler Directives

HPC Engineering

Lattice Boltzmann Methods

HPC Engineering

Lattice Boltzmann Method Overview:

• Modeling Shallow Water and Mass Transport

Shallow Water Equation

( )1 ( )eqf f f ft τ

∂+ ⋅∇ = − −

∂c

LB Numerical Formulation

Advection Dispersion Equation for Solute Concentration

( ) 0i

i

huht x

∂∂+ =

∂ ∂

( ) ( )jij C

j j

hu ChC CD h Ft x x x

∂ ∂ ∂ ∂+ = + ∂ ∂ ∂ ∂

( ) ( )22( )2

i ji ii

i i j j

hu uhu huhg Ft x x x x

ν∂∂ ∂ ∂

+ = − + + ∂ ∂ ∂ ∂ ∂

HPC Engineering

• Boltzmann equation

• Particle distribution function

• EDF

• LBM Grid

Background: LBM

D2Q9 LBM

( , , )f tx e

( )( ) ( )

( ) ( )

0,0 0

1 1cos ,sin , 1, 2,3, 4

4 4

1 12 cos ,sin , 5,6,7,8

4 4

e

e

α

α

α π α πα

α π α πα

= − − = =

− − =

e

( )1 ( )eqf f f ft τ

∂+ ⋅∇ = − −

∂e

4 7 8

3

6 2 5

1 0

( )22

02 2 4 20

3 1 , 02 2

eq eq eqscf h f h fe e e e

ααα α α

α

ω α>

⋅⋅ ⋅= + + − > = −

∑u eu e u u

( )22

2 2 4 20

3 1 , 02 2

eq eq eqs hc h h hg C g hC ge e he he

αα αα α α α

α

ω α>

⋅⋅ ⋅= + + − > = −

∑u eu e u u

( , , )g tx e

HPC Engineering

• Zeroth Moment

• First Moment

• LBM evolves in a two-step process – Step 1. Collision

– Step 2. Streaming

Background: LBM (cont.)

h fαα=∑i ihu e fα αα=∑

( ) ( ) ( ) ( )( )

( )2

1ˆ, , , ,

,6

eq

i i

f t f t f t f t

t e F te

α α α α

α

τ= − −

∆+

x x x x

x

( ) ( )ˆ, ,f t t t f tα α α+ ∆ + ∆ =x e x

hC gαα=∑i iChu e gα αα=∑

HPC Engineering

Results: 1) Model Validation 2. Simulation Results

HPC Engineering

Domain: 12 m x 12 m Dam Break Initial Conditions

Upstream: water depth 5 m Downstream: water depth 1m

BC: Free – Slip at walls for flow

2D – Dam Break: Backward Facing Step

t = 0.5 s

HPC Engineering

Domain: 200 m x 200 m Dam Break Initial Conditions

Upstream: water depth 10m Downstream: water depth 5m

BC: Free – Slip at walls for flow, Impermeable for concentration

2D Partial Dam Break

Water Depth Surface t = 7.2 s Velocity Vector Field t = 7.2 s

HPC Engineering

Problem Setup Domain: Infinite Shallow Water

Pool Constant Water Depth 1 m

2D – Continuous Release Mass Transport

( )( ) 11 2 12 1 2s Aτ τ

−= + −

Velocity Anisotropic Dispersion

( )2

1 2ij

sijA

hDc

t τ=∆ −

xx LD h κ= u yy TD h κ= u / 10L Tκ κ =

HPC Engineering

Domain: 800m x 200m Dam Break with

Transport Initial Conditions Upstream: water depth 10m Downstream: water depth 5m BC: Free – Slip at walls for flow

o Time Snap shots t = 10 s t = 30 s t = 60 s t = 90 s

2D – Dam Break with Mass Transport

HPC Engineering

Domain: 800 m x 200 m Dam Break with Transport Initial

Conditions Upstream: water depth 10m , Concentration 0.7 Downstream: water depth 5m, Concentration 0.2 BC: Free – Slip at walls for flow,

Impermeable for concentration

2D – Dam Break with Mass Transport

( ) i jij L ij L T

z

u u h gD k k k

Cδ

= + −

uu

5.93Lk =

0.23Tk =

o Time Snap shots t = 10 s t = 30 s t = 60 s t = 90 s

HPC Engineering

Performance

HPC Engineering

2D – Dam Break: Backward Facing Step

HPC Engineering

2D – Continuous Release Mass Transport

HPC Engineering

2D – Dam Break with Mass Transport

HPC Engineering

Summary

• LBM is a flexible algorithm for various CFD simulations

• Tools from Accelereyes allow rapid development and validation of LBM based CFD algorithms

• The GPU accelerated LBM algorithm achieved maximum of 24X speedup on GPU based architectures.

• The GPU parallel performance depends on size of simulation.

HPC Engineering

Resources

• Blogs

• Whitepapers

• HPC Advisor Online

• www.dell.com/gpu • www.dell.com/hpc • www.hpcatdell.com • www.DellHPCSolutions.com • http://www.hpcadvisorycouncil.com/best_practices.php

HPC Engineering

Related Work

Publications Tubbs, K. R., and F. T.-C. Tsai, GPU Accelerated Lattice Boltzmann Model for Shallow Water Flow and Mass Transport. Int. J. Numer. Meth. Eng (2010) DOI: 10.1002/nme.3066 http://blog.accelereyes.com/blog/2010/12/01/lattice_boltzmann_model/ Tubbs, K. R., and F. T.-C. Tsai, Multilayer Shallow Water Flow using Lattice Boltzmann Model with High Performance Computing. Advances in Water Resources, 32(12), 1767-1776, 2009. Presentations Tubbs, K. R., and F. T.-C. Tsai, A GPU Accelerated Lattice Boltzmann Model for Shallow Water Flow and Mass Transport, Sixth International Conference for Mesoscopic Methods in Engineering and Science (ICMMES), Guangzhou City, Guangdong Province, China, July 13-17, 2009. Proceedings Tubbs, K. R., and F. T.-C.. Tsai. Simulation of Multilayer Shallow Water Fluid Flow Using Lattice Boltzmann Modeling and High Performance Computing. ASCE/EWRI World Water & Environmental Resources Conference, Kansas City, Missouri, May 17-21, 2009. Tubbs, K. R., and F. T.-C.. Tsai, C. White, G. Allen. Lattice Boltzmann Modeling Using High Performance Computing for Shallow Water Fluid Flow and Mass Transport, American Geophysical Union 2008 Fall Meeting, San Francisco, CA, Dec 15-19, 2008. http://www.accelereyes.com/success_story/shallow_water_fluid_flow.php

Publications, Proceedings & Presentations:

HPC Engineering

Questions