Simulation of Non-Equilibrium Flows: Plasma Turbulence and Radiation Transport

Juan Pablo Trelles

Department of Mechanical Engineering and Energy Engineering Graduate Program University of Massachusetts Lowell

1  

UMass HPC Day University of Massachusetts Dartmouth

November 14, 2014

Non-Equilibrium

2

Ein Eout

Ein ≈ Eout ⇒ What energy do we use?

Sin few high energy

photons (low entropy)

Sout many low (thermal)

energy photons (high entropy)

Sin << Sout ⇒ “Useful Energy” due to 2nd law → Non-Equilibrium

Equilibrium α Degree of Interaction

Thermodynamics & Equilibrium

3  

cs

s fDtDf !<<< Thermodynamics

•  temperature, pressure, …

cs

s fDtDf !≈ Non-Equilibrium

•  many “degrees of freedom”; highly non-linear & coupled

cs

s fDtDf !<<

Local Thermodynamic Equilibrium •  fluid models: incompressible, compressible, MHD, …

cs

s fDtDf !=Canonical transport model:

Boltzmann’s eqn. rate of change in

phase-space* change due to collisions

* DDt

=∂∂t+∂[x,v]∂t

∂∂[x,v]

•  Plasma turbulence •  Radiation transport

Examples:

Variational Multiscale Large Eddy Simulation

of Plasma Turbulence

4  

5  natural

Plasmas

1 Plasma Science: From Fundamental Research to Technological Applications, The National Academies Press, 1995 technological

1

This talk

The 4th state of matter: solid → liquid → gas → plasma

Nonequilibrium Plasma Flow Model Transport System: Transient + Advective + Diffusive + Reactive = 0

6  

Chemical equilibrium

Thermodynamic nonequilibrium

Electromagnetic coupling

10 highly coupled & non-linear eqns. in 3D

Transport Problems & Multi-Scale Phenomena

7  

0YSYSYKYAYA 010 ==+−∂∂−∂+∂ )()()(reactivediffusiveadvectivetransient

R!"!#$!"!#$"#$"#$ jijiiit

•  Nonequilibrium Plasma Flow Model as a generic transport problem:

•  Challenge for all numerical methods (Finite Differences, Finite Volumes, Finite Elements, etc.):

Multi-Scale Problems

any term >> others physical scale >> mesh size

chemical reactions, nucleation

x or t

boundary layers, sheaths, interfaces

shocks, chemical fronts,

phase change

turbulence, wave scattering

y

Numerical Model: VMS-FEM

•  Transport System:

8  

•  Variational Multiscale Finite Element Method (VMS-FEM):

A0∂tYtransient +Ai∂iY

advective −∂i (Kij∂ jY)

diffusive

− (S1Y+S0 )reactive

= L↑Y−S0 =R

↑(Y) = 0

W ⋅R (Y)dΩΩ∫ = 0Variational form:

total = large + small ' and ' WWWYYY +=+=Scale decomposition:

W ⋅R (Y)Ω∫ dΩ

large

+ L∗W ⋅Y 'Ω∫ dΩ

small

= 0

Y ' = −τR (Y ) τ ≈ L−1

Solve: large = f(small) Model: small =f(large)

•  Solution Approach: TransPORT solver (TPORT) •  C++ •  Implicit Alpha Method •  Globalized Inexact Newton-Krylov •  Parallel Preconditioned GMRES

transport operator

residual

Consistent & Complete

Second-order Accuracy

Space & Time

1 Trelles J P, Modirkhazeni S M 2014 Comp Meth Appl Mechanics Engrng 282 87-131

1

Turbulence Modeling: A Matter of Scales

9  

LES: Large Eddy Sim.  

cut-off  

(3) Inertial

subrange

LES: mesh fine enough for inertial subrange

(1) Most energy in

large-scale motion

DNS: Direct Numerical Simulation  

(2) ~ Universal

behavior

Energy Cascade

1 Brown G L and Roshko A 1974 J Fluid Mech 64 353

1

VMS-LES for Plasma Turbulence

10  

DNS: no model High resolution & cost

Fluid Dynamics: Aerodynamics Heat Transfer Combustion

…

1 Bazilevs Y, Calo V M, Cottrell J A, Hughes T J R, Reali A, Scovazzi G 2007 Comp Meth Appl Mechanics Engrng 197 173

1

   

Plasma Physics:

MHD Low Temperature

Fusion …

Anisotropy Non-­‐equilibrium High  non-­‐linearity

???

3 Favier B, Godeferd F S, Cambon C 2012 Geophys. Astrophys Fluid Dyn. 106 89

3

LES: model small scales Medium resolution & cost

   

Isotropy Equilibrium

Mild  non-­‐linearity

Filters Eddy  viscosity

2 You D, Ham F and Moin P 2008 Center for Turbulence Research Annual Research Briefs

2

?

VMS    

- New flow regimes - Unexplored phenomena - Novel technologies - Industrial applications - …

VMS-LES:

1 Trelles J P, Pfender E, Heberlein J V R 2009 J Thermal Spray Tech 18 5-6 728

•  Essentially simple devices … –  Yet exceedingly rich & complex phenomena

1

Application: Plasma Torches

11  

Anode phenomena

Cathode phenomena

Balance drag – Lorentz forces Arc

attachment

MHD instabilities

Fluid instabilities

Radiative transfer

Cold flow entrainment

Turbulence

Chemical non-equilibrium

Thermodynamic non-equilibrium

•  Arc Torches: –  Core of diverse technologies:

spraying, waste treatment, gasification, …

http://www.progressivesurface.com

torch

jet workpiece

Arc & Jet Dynamics

12  

Heavy-species temperature Th [K]

1 Trelles J P 2013 J Phys D Appl Phys 46 25 255201 2 Trelles J P et al 2008 IEEE Trans. Plasma Sci. 36(4) 1026

1

fluctuations amplitude ~

high-speed images

2

Constant & Symmetric Boundary Conditions

No forcing

Nonequilibrium & Coherent Structures

13  

•  Long-lived flow features (e.g. vortices)

•  Temperature (heavy-species) •  Non-equilibrium:

“cold” gas + “hot” electrons

high nonequilibrium @ plasma – gas interface

H criterion

Q criterion 1 Trelles J P 2014 IEEE Trans. Plasma Sci. 42(10), 2852-2853

1

3 Outcalt & Heberlein, 2005

3

2 Trelles J P 2013 J Phys D Appl Phys 46 25 255201

2

Fluid Instabilities

14  

•  Shear Instability (Kelvin – Helmholtz)

HOT plasma

COLD gas

Ø  Ongoing: ab-initio nonlinear VMS formulation, higher resolution, integration PETSc solvers

1 Van Dyke An Album of Fluid Motion

1

Angular – Multiscale Finite Element Methods

for Radiation Transport

15  

Concentrated Solar Energy Utilization •  Solar concentrators:

–  Direct energy to desired target

16  1 Agrafiotis, et al. Renewable and Sustainable Energy Reviews 29 (2014) 656-682 2 DOE, 2014: The Year of Concentrating Solar Power, DOE/EE-1101 • May 2014

1

•  Applications: –  Power generation (CSP) –  Industrial thermal processes

2

Decarbonization of fossil fuels

Fossil fuel

Concentrated solar

H2-rich gas

C-rich condensate

Solar thermo-chemical process

CSP Plant

Radiation Transport Modeling

17  

Microscopic: Maxwell’s equations - Comprehensive; unfeasible - macroscopic / complex material systems

Macroscopic: Radiative Transfer Equation (RTE) - Approximate, “statistical”; describes radiative intensity I

R (Iλ ) = c−1∂t Iλaccumulationdue to finitespeed of light

!"# + si∂iIλrate of changealong direction s

! + (κλ +σ sλ )Iλeffectiveextinction

! "# $# −σ sλ

4πIλ

S2∫ (s ')Φλ (s, s ')dΩs '

scattering! "#### $####

− κλIbλblackbody emission

! = 0

RTE: Function of time, space, and direction (5-dimensional)

s

s

Iλ (t,x+ sds, s)

Iλ (t,x, s)

ds dA

x x y

z

Angular Finite Element Method for the RTE

18  

•  Strong form of RTE:

Recall Finite Element Method:

•  Weak form:

•  Galerkin FEM:

W (s) ⋅R (I )S2∫ dΩs = 0

R (I ) = 0

W (s) =N(s)

AFEM: transport system for a vector of discrete intensities I

R (I) = c−1M∂tItransient!"# $# + Si∂iI

advective! + ((κ +σ s )M−σ sF)I−κ IbL

T

reactive! "##### $##### = 0

“basis function”

M = N(s)TS2∫ N(s)ds Si = N(s)T

S2∫ N(s)sids

L = N(s)S2∫ ds F = 1

4πN(s)TΦ(s, s ')

S2∫ N(s ')ds

S2∫ 'ds

transport matrices

1 Castro R O, Trelles J P, 2014 J. Quant. Spec. Rad. Transf. (submitted)

1

Angular Basis Functions & Discretization

19  

•  Angular Basis Functions: -  ~ FEM for spherical triangles

-  Extendible to other types of elements, e.g. quadratic, quadrilaterals, NURBS, etc.

-  DOM as special case

N(s) = [ N1(s) N2(s) N3(s) ]

•  Discretization - Unitary Sphere: -  AFEM orders 0 (P0) and 1 (P1)

Discrete RTE System

20  

AFEM: discrete intensity I:

c−1M∂tI+Si∂iI+ ((κ +σ s )M−σ sF)I−κ IbLT = 0

64 coupled 3D PDEs on unstructured meshes

to be solved!

Example:

transport matrices for

64 directions

Radiation In a Solar Receiver-Reactor

21  

Increasing scattering

Increasing absorption

Incident radiation α •  Solar thermochemical processes (CO2 fuels)

•  Effect of intermittency, misalignment, etc. •  AFEM level 3 (64 directions)

Ongoing: coupling RTE – fluid flow à plasmas

Summary & Conclusions

•  Modeling & simulation of Nonequilibrium Flows •  Challenges (and opportunities) for HPC:

–  plasma turbulence modeling: inertial sub-range industrial VMS-LES

–  Radiative transport: high accuracy radiatively-coupled flows

•  Future directions:

–  Plasma – radiation coupling, complex kinetics

–  Distributed-memory parallelization – domain decomposition

–  Implementation in PETSc (KSP ✓ SNES)

22  

Thank You!

Ø  HPC: Enabling technology for scientific & engineering exploration of Nonequilibrium Energy Transport

Additional

23  

Simulation of Non-Equilibrium Flows: Plasma Turbulence and Radiation Transport

Juan Pablo Trelles Department of Mechanical Engineering

University of Massachusetts Lowell Transport of energy under thermodynamic non-equilibrium conditions is found in diverse technological applications, particularly in those related to sustainable energy conversion, such as the use of solar energy or electrical energy in the form of plasmas for the sustainable synthesis of fuels. The computational modeling and simulation of non-equilibrium flows can provide an unparalleled level of understanding in those applications. Two major challenges in non-equilibrium flow simulation are plasma turbulence and radiation transport. Simulation of turbulent transport is prone to the curse of resolution, i.e. increasing spatial and temporal resolution is required to describe transport through a myriad of scales, from macroscopic flow features, such as vortices, to molecular diffusion. In contrast, simulation of radiation transport is cursed by dimensionality, i.e. the problem is casted in a five-dimensional domain (three spatial and two angular dimensions). This talk will present current efforts in the computational modeling and simulation of plasma turbulence and radiation transport. Both types of problems are solved by parallel time-implicit solvers on unstructured meshes capable to describe flows through complex domains and at industrially-relevant operating conditions. Key challenges related to solver implementation and scalability will also be discussed.

24  

Ø  Solution of system:

Loop: Time stepping - Implicit predictor-multicorrector

Loop: Solution non-linear system - Globalized inexact-Newton

Loop: Solution linear system - Krylov method

end end

end

0YRes0YYXRes =→= )( ),,,( !t

11

1

++

+

Δ+=

≤Δ+

kkkk

kk

YYY

ResYJacRes

λ

γ

)()(

11 bPxAP

bAx−− =

=

0YYXRes →),,,( !t

Jac ≈ ∂Res∂Y

1 J. P. Trelles, S. M. Modir Khazeni, “Variational Multiscale Method for Nonequilibrium Plasma Flows”, Computer Methods in Applied Mechanics and Engineering (2014) No. 282, pp. 87-131.

1

Unstructured Data (Mesh) + Fully-Implicit Solver à Multiple parallel synchronization/communication

Parallelization

26  

•  Parallelization challenging: unstructured mesh + implicit solver

•  Current implementation: o  Shared memory using OpenMP o  Tested up to 64 cores @ MGHPCC

Parallelization: element-by-element

multi-coloring

•  Domain Decomposition: o  Multi-coloring o  “Greedy” algorithm

)~(_ ,~~~~ ~~ 11 APbxAbPxAP 000 diagblock==→= −−

•  Parallelization core: Linear Solver o  Krylov method à only Matrix – Vector products needed o  Challenge: Preconditioning o  Solution: Scaling (before linear solver)