The Einstein Toolkit: A Community Computational Infrastructure for Relativistic Astrophysics

Gabrielle AllenSkolkovo Institute of Science & TechnologyMoscow, Russia

http://www.einsteintoolkit.org

Gravitational Wave Physics

Observations

Models & Simulation

Theory

Scientific Discovery!

G = 8TCompact binaries, supernovae collapse, gamma-ray bursts, oscillating NSs, gravitational waves, …

Basic Formalism: ADM1. Choose initial spacelike surface

and provide initial data (3-metric, extrinsic curvature)

2. Choose coordinates: Construct timelike unit normal to

surface, choose lapse function Choose time axis at each point on

next surface (shift vector)

3. Evolve 3-metric, extrinsic curvature

Usual numerical methods:

Structured meshes (including multi-patch), finite differences (finite volumes for matter), adaptive mesh refinement (since ~2003). High order methods.

Some groups use high accuracy spectral methods for vacuum space times

Black Holes & Vacuum Spacetimes 40+ year effort to model … big

advance in 2005

Now: Accurate waveforms for arbitrary masses, spins, momentum

Numerically generated waveforms now used with gravitational wave data analysis, analytic GR (NINJA, NR-AR)

B.Aylott et al 2009 Class. Quantum Grav. 26 165008

• Opens the door to general relativistic hydrodynamics

New Frontiers: Relativistic Matter General relativity

Nuclear equations of state

Relativistic magnetohydrodynamics (GRMHD)

Radiation Transport (neutrinos/photons) Expensive and complicated! Requires opacities/emissivities

Chemical reactions (thermonuclear, chemical)

Computation: Multiphysics!! GRMHD: petascale problem Radiation transport beyond this

Resolve 102m to 1010m, 500 grid variablesSchnetter et al, PetaScale Computing:

Algorithms and Applications, 2007

Software

Ecosystem

Software: Component Framework

Einstein Toolkit

Cactus Computational Toolkit

Cactus Flesh(APIs and Definitions)

MPI, Threads, New Programming Models

Driver Thorns (Parallelisation)

Group A Thorns

Group B Thorns

CS

CDSE

Computational

Relativists

Domain Scientist

s

Cactus

Framework

Software Platform

Cactus: www.CactusCode.org Open source component framework for HPC

Modular system with high level abstractions Components (“thorns”) defined by parameters, variables,

methods Cactus “flesh” binds together Cactus Computational Toolkit: general thorns

Different application areas Numerical relativity, CFD, coastal science, petroleum,

quantum gravity, cosmology, …

Key Features

Driver thorn provides scheduling, load balancing, parallelization

Application thorns deal only with local part of parallel mesh

Different thorns can be used to provide the same functionality, easily swapped.

Cactus: 1997-today

History: Black Hole Grand Challenge (‘94-’98): multiple codes,

groups trying to collaborate, tech/social challenges, NCSA (USA) group moves to AEI (Germany).

New software needed! Vision … Modular for easy code reuse, community sharing and

development of code Highly portable and flexible to take advantage of new

architectures and technologies (grid computing, networks)

Higher level programming than “MPI”: abstractions Emerging: general to support other applications, better

general code, shared infrastructure

AMR: Carpet Set of Cactus thorns

Developed by Erik Schnetter

Berger-Oliger style adaptive mesh refinement with sub-cycling in time High order differencing

(4,6,8) Domain decomposition Hybrid MPI-OpenMP

2002-03: Design of Cactus means many groups, even competing ones, suddenly had AMR with little code change

AEI (Rezzolla, Kaehler)

E. Schnetter

Carpet Scaling

Schnetter, 2013, arXiv:1308.1343

Numerical Relativity with Cactus 1997: 1st version of Cactus just for relativity

(Funding from MPG/NCSA)

1999: Cactus 4.0: “Cactus Einstein” thorns

1999-2002: EU Network “Sources of Gravitational Waves” Led to Whisky Code for GR Hydro in Cactus

Groups develop codes based on Cactus Einstein

2007: LSU/RIT/PennState/GeorgiaTech: NSF XiRel Improve scaling for multiple codes using Cactus

2009-: LSU/RIT/GeorgiaTech/Caltech/AEI: NSF CIGR Shared cyberinfrastructure including matter Einstein Toolkit from community contributions Sustainable, community supported model

So far …

Over 200 science papers

Over 30 student thesis

Around 50

contributors

14

Coastal Science: CaFUNWAVEPorted from a mature community code:

FUNWAVE (phase-resolving, time-stepping

Boussinesq model for ocean surface

wave propagation in the nearshore)

Now uses total variation diminishing method and

shock capturing scheme

J. Chen (Civil Eng), J. Tao (CCT/LSU), S.Brandt

(CCT/LSU), F. Shi (Delaware)

General applications for CactusCareer implications for staffIncreased funding opportunitiesImproved code base and communityMore support apparatus needed!

Einstein Toolkit

Community

Einstein Toolkit

“The Einstein Toolkit Consortium is developing and supporting open software for relativistic astrophysics. Our aim is to provide the core computational tools that can enable new science, broaden our community, facilitate interdisciplinary research and take advantage of emerging petascale computers and advanced cyberinfrastructure.”

Consortium: 94 members, 49 sites, 14 countries

Sustainable community model: 9 Maintainers from 6 sites: oversee technical developments,

quality control, verification and validation, distributions

and releases Whole consortium engaged in directions, support, development Open development meetings Governance model: still being discussed (looking at CIG, iPlant)

http://www.einsteintoolkit.org

Einstein Toolkit Members

Components

Currently 150 Cactus thorns: Initial data, evolution, analysis, AMR, … Tools, viz, etc Provide extensible standard interface for general relativity

variables (e.g. variables, parameters, data model for output)

Software contributions from other groups More than Cactus

Examples and tutorials Complete production codes for black holes, neutron stars New users: Test account on LONI supercomputer (Louisiana)

Community support

http://www.einsteintoolkit.org

Groups Concentrate on New Science

Einstein Toolkit

Cactus Computational Toolkit

Cactus Flesh(APIs and Definitions)

MPI, Threads, New Programming Models

Driver Thorns (Parallelisation)

Group A Thorns

Group B Thorns

CS

CDSE

Computational

Relativists

Domain Scientist

s

Example: Codes in Analytical Relativity Collaboration (NRAR)

6 out of 9 codes using Einstein Toolkit

Hinder et al (2013), arXiv:1307.5307

Issues

Incentives for software sharing and reuse

Credit for software (and data) as part of e.g. promotion and tenure

How to cite use of Einstein Toolkit in publications and ensure that contributions are cited?

http://www.einsteintoolkit.org 22

Education

Undergraduate Research (LSU REU in Computational Science) Undergraduates can do do

sophisticated AMR simulations on parallel machines with Einstein Toolkit

Graduate Scientific Computing Course at LSU Einstein Toolkit used in

teaching real world simulation related scientific computing concepts

CS, ECE, Civil Eng, Geoscience, Mech Eng

3D Full GR Simulations of Core-Collapse Supernovae

with the Einstein Toolkit

• Multi-Physics, Multi-Scale Simulations ongeneralized cubed-sphere grids with AMR.

• Scaling: time-evolving GR, no elliptic PDEs -> full physics sims scale strongly to O(10000)cores with hybrid OpenMP/MPI.

• Monte-Carlo Boltzmann solver &and efficient M1 radiation-hydro schemeunder development.

• MHD implementation (Moesta et al. 2013)-> study GRB central engines.

Ott et al. 2013, ApJ

Jet-Driven Supernovaewith the Einstein Toolkit

• Full GR -> get special relativity for free.

• GRMHD with full microphysics (EOS, neutrinos) from http://stellarcollapse.org

• Full adaptivity; shock/jet tracking framework.

• Fully reproducible thanks to open source.

Moesta et al. 2013 in prep.

Supermassive Stars and Black Holeswith the Einstein Toolkit

• Making massive seed black holes from supermassive stars:Simulation of fragmentation of differentially rotating polytrope.

• Implementation is full GR:simulate black hole formation self-consistently.

• Get gravitational-waves for “free” from spacetime metric -> probe formation of the first black holes.

Reisswig et al. 2013, PRL

New: GRHydro ET Thorn Base: GRHD public version of Whisky code (EU 5th Framework)

Much development plus new MHD Caltech, LSU, AEI, GATECH, Perimeter, RIT (NSF CIGR Award)

Full 3D and dynamic general relativity

Valencia formalism of GRMHD: Relativistic magnetized fluids in ideal MHD limit

Published text results, convergence arXiv: 1304.5544 (Moesta et al, 2013)

All code, input files etc part of Einstein Toolkit

User support

StrategyExascale

Automatic Code Generation

Einstein equations very complex Coding cumbersome, error prone Deters experimentation

Kranc: Mathematica tool to generate Cactus thorns from PDEs, specify differencing methods

Vision: Generate entire codes from underlying equations/problem specification, optimize codes for target architectures Revolutionize HPC Opportunity to integrate

verification/validation/data description

Governing

Equations

Numerics

KrancEngine

Custom Code

Resource

Chimora Use large scale CPU/GPU systems efficiently for

complex applications

Reduce code rewrite, new programming paradigms

Strategy uses: High level code transformations Loop traversal strategies Dynamically selected data/instruction cache JIT compiler tailored to application

Blazewicz et al (2013), arXiv:1307.6488

1265449

Chimora

Software Partnerships NSF S2I2 Software

Institutes Conceptualization awards

for institute level software development and support

Software Institute for Abstractions and Methodologies for HPC Simulation Codes on Future Architectures (Anshu Dubey)

High-Performance Computational Science with Structured Meshes and Particles (Phil Colella)

Katz, NSF

Some Challenges for Petascale and Beyond

New physics: neutrino transport, photon radiation transport

Massive scalability Local metadata, remove global operations Extend Cactus abstractions for new programming models Robust automatically generated code Multithreading, accelerators

Tools: real time debuggers, profilers, more intelligent application-specific tools

Data, visualization, profiling tools, debugging tools, tools to run codes, archive results, …

Growing complexity of application, programming models, architectures.

Social: how to develop sustainable software for astrophysics? CDSE and supporting career paths? Edcuation?

Conclusions Numerical relativity community generally now comfortable with

sharing software Didn’t happen overnight Some fundamental issues resolved first (BH-BH evolutions) Some trade-offs, flexibility/support

Einstein Toolkit approach Mechanism for injecting new science (e.g. GRHydro) and taking full

benefit of new CS opportunities Need to focus on implications for young researchers, motivation to

contribute, scientific aims Focus on modularity/abstractions reduces dependence on Cactus

Funding Need lightweight governance model to better target funding, help

funding agencies make decisions, enable leveraging international funding

Target limited science funding where it will make a difference, leverage CS funding

Cactus: broader application base has potential to coordinate with other disciplines

Einstein Toolkit Credits

Frank Loeffler (Louisiana State University) Erik Schnetter (Perimeter Institute) Christian Ott (Caltech) Ian Hinder (Albert Einstein Institute) Roland Haas (Caltech) Tanja Bode (Tuebingen) Bruno Mundim (Albert Einstein Institute) Peter Diener (Louisiana State University) Christian Reisswig (Caltech) Joshua Faber (RIT) Philipp Moesta (Caltech) And many others

AdvertisementSunday @ SC13, Denver

http://wssspe.researchcomputing.org.uk