Build your own simulation: The Einstein...

Build your own simulation: The Einstein Toolkit

Frank Loffler

Center for Computation and TechnologyLouisiana State University, Baton Rouge, LA

Jul 30th 2011

Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30

Solving Einstein’s Equations

Einstein equations:Gµν = 8πTµν

12 fully 2nd order PDE evolution equations4 coupled constraint equations4 gauge conditionsGR hydrodynamics, MHD, radiation transport

Fully numerical 3D models needed

Automated code generation


Computational Requirements

Unigrid scales to hundreds of thousands of cores

Productions runs use ≈ 10 levels of mesh refinement, nested grids ofsize ≈ 60x60x60

Current mesh refinement runs scale up to ≈ 10k cores

Runtime from weeks to few months


Key Challenge

Many scientific/engineering components

PhysicsMathematicsCFD

Many numerical algorithm components

Finite difference, finite volume, spectral methodsStructured or unstructured meshes, mesh refinementsMultipatch and multimodel

Many different computational components

Parallelism (MPI, OpenMP, ...)Parallel I/O (e.g. Checkpointing)Visualization

Challenge

Defining good abstractions to bring these together in a unified, scalableframework, enabling science


The Einstein Toolkit

Collection of software components and tools

Targeted at simulating and analyzing general relativistic astrophysicalsystems

Freely available as open source

See http://einsteintoolkit.org


http://einsteintoolkit.org

Guiding Principles

Open, community-driven software development

Separation of physics software from computational scienceinfrastructure

Well thought out and stable interfaces

Providing core computational tools to

facilitate interdisciplinary researchenable new sciencebroaden communitytake advantage of emerging Petascale computers and advancedcyberinfrastructure


Current State

Last release on April 21, 2011

New releases roughly every six months

Atm. mostly Cactus (framework), Carpet (AMR)

Production quality features (Ninja, NRAR)

Not all new - grown from other projects


Science Capabilities

McLachlan (BSSN, up to 8th order)

GRHydro (formerly based on WhiskyCode; Valencia formulation)

BH/NS initial data (TwoPunctures, Lorene)

Turduckening

MoL + e.g., Runge-Kutta

Carpet (Adaptive Mesh Refinement)

Black hole horizon finder (AHFinderDirect)

Gravitational wave extraction

Parallelization: MPI, OpenMP

Tools: e.g., EOS reader, HDF5 output, Visualization import


Quality Control

Open source 6= offering download

Requirements for (new) Einstein Toolkit components:

sufficient quality (actually being used)documentation (e.g. publication)community interest (maintained)

Regular, tested releases

Providing easy step-by-step instructions for first-time users


Einstein Toolkit testsuite status


People

Supported by NFS grants 0903973/0903782/0904015 (CIGR)

State-of-the-art set of tools for numerical relativity, open source

49 contributors over the past decade, both physics and CS

Currently 64 members from 24 sites in 10 countries

9 maintainers from 6 sites

> 200 publications, > 30 theses building on these components

Regular, tested releases

User support through various channels


Tutorial for New Users

http://docs.einsteintoolkit.org/et-docs/Tutorial for New Users

Get account on cluster “QueenBee” (fill in web form)

Download (4 shell commands)

Configure (3 commands) [user name, email address, allocation]

Build (1 command)

Run simulation (1 command)


http://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users

Linda Holyoke - REU undergraduate student

Linda Holyoke: AMR visualization

Science vs. Simulation

Doing science >> Running a simulation

Need to know about meaningful initial conditions, numerical stability,accuracy/ resolution, have patience, have curiosity, have a “gutfeeling” for what is “right”...

Einstein Toolkit cannot give that!

But, with open codes that are easy to use, can concentrate on science!


Convenience Tools

GetComponents Simfactory Formaline


Tools: GetComponents

Task: Collect software from various repositoriesat different sites

Example simulation assembly:

Core Einstein Toolkit (svn.einsteintoolkit.org)

Cactus Flesh and Toolkit (svn.cactuscode.org)

Carpet AMR (carpetcode.org, hg)

Tools, Parameter Files and Data (svn.einsteintoolkit.org)

Group Modules (x.groupthorns.org)

Individual Modules (x.mythorns.org)

x: cvs, svn, darcs, git, hg, http


Tools: Simulation Factory

http://www.simfactory.org/

Task: Provide support for common, repetitive steps:

Access remote systems, synchronize source code trees

Configure and build on different systems semi-automatically

Provide maintained list of supercomputer configurations

Manage simulations (follow “best practices”, avoid human errors)


http://www.simfactory.org/

Tools: Formaline

Task: Ensure that simulations are and remain repeatable, rememberexactly how they were performed

Take snapshots of source code, system configuration; store it inexecutable and/or git repository

Tag all output files


General Guideline

The Einstein Toolkit

Does not want to provide “the” best codeRather wants to offer different codes with are

of high qualitymaintainedopen sourceeasy to use

→ Computational General Relativity for the Masses


Support

Web site http://einsteintoolkit.org

Mailing list [email protected]

Bug tracking system http://trac.einsteintoolkit.org

Weekly public phone meetings

Also blog, wiki, code repositories, ...


http://einsteintoolkit.org

mailto:[email protected]

http://trac.einsteintoolkit.org

Future

Looking for users and contributions

Don’t want to take over software

Currently Cactus-centric, want to expand in other directions

Also looking for analysis tools / scripts(e.g. gravitational wave postprocessing tools)

Working on GRMHD

Improve existing numerical methods (scaling)


Prepare Tools

https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users

Setup on individual machines can be tricky

Usage of particular machine at LSU for this tutorial (QueenBee)

Need account? → http://einsteintoolkit.org/documentation/request

Using account: login (ssh)

Need git: soft add +git

Download GetComponents

wget --no-check-certificate \

https://github.com/gridaphobe/CRL/raw/ET_2011_05/GetComponents

chmod 755 GetComponents


http://einsteintoolkit.org/documentation/request

Download and Configure

Checkout (download) the Einstein Toolkit (may take a few minutes)

./GetComponents -a \

http://svn.einsteintoolkit.org/manifest/branches/ET_2011_05/\

einsteintoolkit.th

Includes Cactus, thorns, simfactory and example parameter files

Configure Simfactory

cd Cactus

cp simfactory/udb.example.pm simfactory/udb.pm

vi simfactory/udb.pm

Replace

YOUR_LOGIN\verb with your username on Queen [email protected] with your usual email addressYOUR_ALLOCATION with your project allocation on Queen Bee


Build and Run

Compile Executable

./simfactory/sim build --thornlist=manifest/einsteintoolkit.th

Run a Simulation

./simfactory/sim create-submit static_tov \

--parfile=par/static_tov.par --procs=32 --walltime=8:0:0

Check simulation status

./simfactory/sim list-simulations


Look at results

Make sure X-forwarding is enabled

ssh -Y qb.loni.org

Use gnuplot to visualize data

cd $SCRATCH/simulations/static_tov/output-0000/static_tov

gnuplot

p ’hydrobase::rho.maximum.asc’ u 2:3 w lp


Additional Example Parameter Files

par/ks-mclachlan.par:single isolated rotating black hole

qc0-mclachlan.par:binary black hole system using the moving puncture technique(12 orbit

)


Future

Looking for users and contributions

Don’t want to take over software

Currently Cactus-centric, want to expand in other directions

Also looking for analysis tools / scripts(e.g. gravitational wave postprocessing tools)

Working on GRMHD

Improve existing numerical methods (scaling)


Build your own simulation: The Einstein...

Documents

Transcript of Build your own simulation: The Einstein...