19/07/2016

High Performance Computing for Modeling Fluid-Transport

and Electromagnetic Phenomena

Assoc. Prof. Antonis Papadakis

Purpose of Presentation

Educational Background

Employment History

Team Members

Hardware Resources

Numerical Models

Results - High Voltages

Future Work

Vision Forward (ΚΥΑ)

Present latest results and future plans on HPC for fluids and electromagnetic

phenomena

Exploit possibility of collaboration in HPC for Space and other applications

such as Computational Fluid Dynamics, Electromagnetics, Multiphase flows,

Plasmas, Renewables and Heat and Mass Transfer

Identify which areas are niche through collaborative effort

Explore possibility on participating in European and Research Promotion

Foundation (RPF) funded projects

Listen to suggestions regarding current and future work

University of Warwick, Department of Electrical Engineering (1996 - 1999)

BENG in Electrical Engineering

Cambridge University, Trinity College, Department of Electrical Engineering- Full Time

Scholarship (1999 - 2004)

PhD on Plasma MultiPhysics modeling

Skills developed

Maths (Linear algebra)

Modeling of partial differential equations using Finite Difference, Finite Element

and Finite Volume methods

Plasma physics

Software development (Programming language C++, Matlab )

University of Cyprus, Department of Physics, High Energy Physics group (2004 - 2007)

Cyprus representative for CERN – CMS experiment

Part of CMS group that discovered Higgs particle (Nobel Prize)

Electromagnetics (EM) of the ECAL detector of the CMS

Very Front Electronics (VFE’s) of the ECAL detector of the CMS

Fusion simulations for ITER/Hellenic Association of EURATOM

Frederick University, Department of Electrical Engineering (2007 - Today)

High Voltages

Electromagnetic Radiation (Overhead and underground transmission lines,

Substations of open and closed type, Mobile telephony antennas)

Plasmas (Fusion, Microplasmas, Atmospheric plasmas)

Renewables (Photovoltaic, Wind energy)

Computational Analysis – High Performance Computing, Adaptive mesh

techniques, Linear algebra and Computational fluid dynamics

EM

FE-FV

Fluid

PhD programme at Frederick University has recently begun – End of 2012

Started gathering a significant number of PhD students and Post-doctoral staff

PhD students – Erasmus INTACT and LEADERS funding

Mr. Muhammad Naveed Shaikh (Pakistan) – Computational fluid dynamics and

solar cells

Mr. Parash Acharya (Nepal) – Computational fluid dynamics and wind turbines

Mr. Almasy Wasif (Palestine) – Computational fluid dynamics

Mr. Gobinta Panta (Nepal) – Dielectric Barrier Discharges plasmas

Mr. Mujahid Hussain (Pakistan) – Computational fluid dynamics and solar cells

Mr. Kastrounis Nikolaos (Greece) – Computational fluid dynamics in Oil & Gas

Postdoctoral positions

Zafar Qayyum – Transport properties of solar cells

MSc candidates

Total of 10 MSc students

Birendra Suwal (PhD potential candidate) – Computational Fluid Dynamics

PERSONAL High Performance Computing Cluster (HPC)

Windows Server 2008 R2 Enterprise

Setup of a cluster

5 servers x 24 cores = 120 cores

RAM capabilities (228 Gbytes)

Visual Studio Environment: Automation, IntelliSense, Parallel

debugging

MPI already installed/enabled through HPC Pack

Private network (1 GBit/s)

Personal High Performance Computing Cluster (HPC) - Continued

Matlab incorporated in Visual Studio environment for

Viewing result purposes

Simulation purposes

Code written in object orientated software programming language C++

Existing EM/Fluid code is approximately 800 C++ files

Consists code for plasma, electromagnetic, photovoltaic and wind energy through

fluid models and EM solvers

Commercial software will also be used for validation purposes (COMSOL

Multiphysics, Ansys Fluent, PSCAD, Matlab)

Recently gained access to CYTERA supercomputer at CYI for analyzing large scalephenomena:

CYTERA (116 servers x 12 cores = 1392 cores) with large RAM capabilities(116 x 48 Gbytes = 5,568 GBytes)

Characterisation of plasma dynamics by solving:

Poisson equation for the electric field

Continuity equations of charged particles (electrons, positive and negative ions)

Navier-Stokes equations for the neutral gas (mass, momentum and energyconservations)

First to:

Couple charge and neutral gas dynamics in plasma applications back in 2000

Introduce neutral gas heating effects in plasmas from a single electron as aninitial condition

Numerically characterize secondary streamer propagation in Dielectric BarrierDischarges

Analyze streamer branching in long point-plane gaps from a single electron as aninitial condition

Numerically characterize normal and abnormal glow discharges from a singleelectron as initial condition

Analyze positive and negative streamer propagation in long RF gaps

Develop a highly efficient 2D-Adaptive mesh generator producing nearly idealtriangular elements

Poisson’s Equation

Continuity Equations

( ) ( )

r

o

p n ee

N N N 0

e2

eeeeppeeeeee

ND)WN.(NNWNWNSt

N

)WN.(NNNNWNSt

Npppnpneppeee

p

)WN.(NNWNSt

Nnnpnpnee

n

Poisson and Continuity model is coupled with the Navier-Stokesmodel via E/N on which , , , We, Wp, Wn depend on

nppneppeeeee NN2NN|W|N|W|N)v.(t

ssCP)vv.(

t

)v(

Navier-Stokes equations :

Coupled Problem with Continuity equations via

E.JfvC))T(k.()vP.()v.(t

ths

s

Solution Procedure

PO

Next Step

Nn

Start

PO

TPCON

NS

NS

TP CON

n

En

TRn

n+1/2

TRn+1/2

En+1/2

n+1

Nn+1

Nn+1/2

Finite Element-Flux Corrected Transport method (FE-FCT)

Discretization occurs using the Galerkin Finite Element method

Two-step Lax–Wendroff technique is used

Utilizes an accurate High-Order scheme

Utilizes a diffusive Low-Order scheme

Combines the High and Low-Order schemes in some clever way

Produce positive, monotonic, highly conservative results withnon-physical oscillations

Create New Mesh

Initial Coarse Mesh

Error Calculation

Edge swapping and Node Addition/Removals

Mesh Jiggling

Apply Initial Conditions to the Adapted New Mesh

Run the Simulation Decision Time to Re-mesh

Interpolation from the Adapted New Mesh back to the Initial Coarse Mesh

Error Calculation

Create New Mesh1

Edge Swapping and Node Addition/Removals

Mesh Jiggling

Interpolation from Adapted New Mesh to the Adapted New Mesh1

Create Adapted New Mesh

Create Adapted New Mesh1

Bar chart displaying the number ofelements that have similar elementquality values of the Mesh2 createdin a commercially mesh generationsoftware.

Bar chart displaying the number ofelements that have similar elementquality values of the Mesh2 created ina commercially mesh generationsoftware after being treated by theelement quality improvementalgorithm.

Test

Number

Before treatment After treatment Computer

Time (s)

RAM Memory

(kBytes)

Minimum

Value

Average

Value

Minimum

Value

Average

Value

1 0.785 0.960 0.859 0.971 2.568 3504

2 0.614 0.951 0.859 0.971 4.145 3788

3 0.605 0.949 0.875 0.982 5.961 5568

4 0.605 0.948 0.854 0.984 23.781 15736

5 0.602 0.936 0.851 0.981 49.109 22916

6 0.602 0.933 0.836 0.981 63.970 27016

Schematic diagram of the Delaunaytriangulation of a point-planeconfiguration mesh created in acommercially mesh generationsoftware.

Schematic diagram of theDelaunay triangulation of a point-plane configuration mesh createdin a commercially meshgeneration software.

1x10-30.50-1x10-3 -0.5

1x10-3

0.5

0

2

4

6

8

10

12x1021

0

Radial (R) – Coordinates (m)

m-3

2x10-40.2 0.4 0.80300

380

360

340

320

400

Axial (Z) – Coordinates (m)

0.6 1.81.61.0 1.2 1.4

420

440

460

480

Tools to be implemented

MODULAR 3D MIXED FE-FV TVD CODE

3D-ADAPTIVE MESH GENERATOR

PARALLEL PROCESSING - MPI

Utilized in a number of research areas under investigation such as simulation

of:

Electromagnetics (Maxwell’s equation solution)

Plasmas (Fusion, Atmospheric microplasmas, Atmospheric plasmas)

Renewables (Thin film and organic photovoltaics, Wind energy)

MODULAR 3D FE-FV-TVD CODE

C++ templated classes

MPI Implementation

Time-dependent, convection-diffusion schemes for laminar and

turbulent flows

Developing a combination of FV and FE method in 3D

3D-ADAPTIVE MESH GENERATOR

Tetrahedral elements

h and r-refinement

Laplacian smoothing

PARALLEL PROCESSING - MPI

Modular 3D FE-FV-TVD

3D-Adaptive Mesh Generator

Work already done:

Parallel 3D-FE-FCT code using OPENMP software

Intermediate stage into MPI processing

Mesh decomposition software METIS for MPI purposes

Installed and working to provide load balancing between the server cores

Poisson equation using MPI

Currently validating Helmholtz equation using MPI

Currently developing 3rd order FV–TVD and 3rd and 5th order WENO schemes

ELECTROMAGNETICS

Maxwell equation solution using Finite Elements, Finite Difference and Finite

Volume methods:

Human electromagnetic radiation absorption from mobile phones at high

frequencies

Simulation of the EM radiation generated from mobile telephony antennas

using either:

Commercial software (HFFS, COMSOL Multiphysics, OPERA-ELEKTRA)

Developed software

Active and passive shielding of EM sources both at the source and close to the

object under protection

Electromagnetic radiation from overhead transmission and distribution lines,

underground power lines and within substations of closed and open type

THIN FILM AND ORGANIC SOLAR CELLS

Capture the physics of the sunlight propagation and charged particle’s generation

(electrons and holes) and of the background material heating behaviour in

semiconductors

Optical modeling using Helmholtz equation at different frequencies of the

sunlight radiation (Already done using commercial software)

Maxwell equations for light propagation in solar cells

Simultaneous solution of the conservation of mass, momentum and energy

equations for electron, holes (excitons)

Lattice heat equations for the background material by solving a fully coupled

non-isothermal energy balance model

Materials of interest (Perovskites) – Efficiency surpassed silicon since discovered in

2009 - Very promising

ATMOSPHERIC MICROPLASMAS

Analyze atmospheric microplasmas by moving at smaller sizes by operating at

higher microwave frequencies: - Microstrip technology and Ring resonators

Proposed model:

Simultaneous solution of:

Maxwell equations for the calculation of the EM field

Charged and neutral gas particle conservation equations

ATMOSPHERIC PLASMAS

Arc characterization: Avalanche, Primary streamer, Normal glow (Redistribution of

the field on electrode), Abnormal glow, Secondary streamer, Spark and Arc

Incorporate into the existing code: (a) Momentum and energy balance equations

for the charged particles (b) Monte-Carlo simulations for the sheath-region near

the electrodes

Head of Center of Computational Analysis (KYA) – (http://www.kya.frederick.ac.cy)

Vision: Group all numerical modeling researchers of the University under one umbrella

Form a research funded center specialized in certain areas of Engineering such as:

Computational Fluid Dynamics, High Performance Computing, Numerical

modeling, Plasmas, Power Systems, Energy, Artificial Intelligence, Software, Oil &

Gas, Control, etc…

Provide scholarships to attract PhD students

Generate fast training programmes and seminars on High Performance Computing,

Linear algebra, Computational Fluid Dynamics, Adaptive mesh techniques

Setup of a new cluster for running MPI and GPU processing

Permanent staff running daily administrative jobs

Funding through collaboration with industry and through European and local funded

projects

CFD can be used into the design and characterization of the unsteady aerodynamic

environments in existing rocket engine turbines and rocket engine pumps

CFD Applications include modelling of (a) various types of plasma waves and instabilities

in planetary magnetospheres and radiation belts, (b) the confinement and stability of

plasmas in fusion devices, and (c) the propagation of discontinuities and shock waves in

the solar wind

Modelling of spacecraft behaviour due to experiencing of (a) energetic tenuous plasma

‘storms’ under geostationary orbit (b) energetic, dense, directed Auroral plasmas under

polar orbits and (c) cold, dense plasmas under low Earth orbits

Plasma wind tunnel testing for modeling spacecraft during uncontrolled atmospheric re-

entry at the end of their working lifetimes – Comparison with plasma wind tunnel tests

being performed in the L2K and L3K facilities of the German Aerospace Center (DLR) at

its Cologne facility

Simulation of temperatures, pressures, and flow patterns (velocities, accelerations, and

directions) in space vehicles and their propulsion systems

Complex internal flows

Turbopump flows with high speed rotation effects with tip vortex, leakage and

cavitation

Combustion instability for chemical propulsion systems

Simulation of launch vehicles at ascent:

Separated flows, typically along the long missile-type body

Shear layer interaction during stage separation

Plume induced flow separation

Plume with multiple nozzles (engines)

Jet impingement into cavity (flame trench ) during launch

Transient flows at launch

Source: Dochan Kwak and Cetin Kiris, “Current CFD Practices in Launch Vehicle

Applications”, International Workshop on “Future of CFD and Aerospace Sciences”, 23-

25th of April, 2012

High Performance Computing – Advection schemes

Mesh Decomposition software (METIS) for load balancing between the processors

Finite Volume - Total Variation Diminishing (FV-TVD) schemes using MessagePassing Interface (MPI)

Highly accurate advection schemes both in 2D and 3D

Produce positive, monotonic, highly conservative results with non-physicaloscillations

Can capture shock waves extremely well

Standard benchmark tests using MPI (square wave, shock tube, shock on a wedge)

Parallel linear algebra and optimization such as GMRES for asymmetric matricesand Conjugate Gradient methods for symmetric matrices

THANK YOU VERY MUCH

FOR LISTENING TO MY PRESENTATION