ATLAS Overview Week - 7 October 2004 Software Status David R. Quarrie LBNL [email protected].

ATLAS Overview Week - 7 October 2004

Software Status

David R. Quarrie

LBNL

[email protected]

David R. Quarrie: Software Status

2ATLAS Overview Week - 7 October 2004

Introduction

Review recent history and current status

Emphasis on those aspects not already covered by other sessions this week Aspects that span detector subsystems

Simulation, reconstruction, core services, data management, etc.

The short term schedule Dario will be covering the longer term schedule and plans

Briefly touch on some open issues



Recent History (1/2)

We’ve been stress-testing our developers and production team over the last few months

Combined Testbeam Highest priority

A testbed for calibration & alignment (databases), monitoring; HLT & TDAQ integration; simulation; infrastructure (rapid turn-around of bug-fixes)

Data Challenge 2 Primary goal is test of computing model

Descoped as a result of CTB priority Full chain test

Generationsimulationpile-updigitizationmixingreconstructionanalysis

Tier-0 exercise (phase II)

10% scale test - 1 days data in 10 days processing Distributed Physics Analysis (phase III)



Recent History (2/2)

HLT testbed Integration and validation testbed for CTB

Preparations for Rome Physics Workshop Initial layout studies

Complicated release schedule 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7 (8.0.8?)

Targetting DC2 simulation production “Just in time” functionality for generation, simulation, pile-up, digitization,

mixing “Enough” reconstruction in order to validate Extensive physics validation feedback

Generally very successful - a couple of problems took longer to discover than

expected 8.1.0, 8.2.0, 8.3.0, 8.4.0, 8.5.0, 8.6.0, 8.7.0, 8.8.0 (9.0.0)

Targetting CTB, development for DC2 Phase II+III and initial layout studies

Incremental release build procedures developed for CTB



Core Services

Athena Migrated to LCGCMT glue packages and most recent LCG package versions

Ensures consistency with LCG/external package versions Pile-up and event mixing production ready (I expect further optimizations)

History service in place apart from persistency support & readback

Extended EDM support (navigation, persistency)

Python configuration and scripting; SEAL dictionary; ROOT integration

Full integration when ROOT 4 available

Detector Description Migration to Oracle DB essentially completed; versioning in place

Deployment issues being worked on (e.g. Oracle client, MySQL plug-in)

Graphics Atlantis available for both offline (file-based) and online (network) use

Go now to: http://atlas-php.web.cern.ch/atlas-php/DDDB/node_tag_browser.php

Note, data does not appear until “commit” is issued from sqlplus or sqlplus isexited.

This web page is forcollecting geometry tags.

N.Konstantinidis – Graphics report20/09/2004

Real Data(Picture made by M.J.Costa)

N.Konstantinidis – Graphics report20/09/2004

Usefulness of an Event Display

Before bug fix… …after bug fix



Infrastructure

New SIT leader (Fred Luehring) and librarian (Emil Obreshkov) in place

Lots of reliability problems with release builds Local disk builds; local CVS cached repositories; local CMT versions; more

error detection and recovery

Full nightly builds taking too long Incremental builds still being tested; additional build machines; parallel builds

Platform support SLC3 port in place (awaits build machine for nightlies to start)

Mac OS X port still incomplete (but close)

64-bit port about to start

A lot of work has gone into distribution kits and validation Extending to developers (source) kits

Lack of (and fragmented) personnel still a major concern



Data Management

New organization in place Richard Hawkings and Torre Wenaus as co-leaders

Detailed work plan in place 12 subprojects, including detector production installation and geometry,

online databases, calibration and alignment, conditions database, event store, distributed data management, bookkeeping, distributed database services and servers

Project management and meetings Steering group; offline, online and technical coordination meetings

Infrastructure for DC2 Tier0 exercise in place Availability of a first data sample for testing is a high priority

Personnel shortfall in several areas



Calibration and Alignment

Combined Testbeam Primary technology testbed

Detector Inventory What data is stored where - where is it produced and consumed?

Data volumes and rates for different types of storage

Relational database, streamed objects in files, histogram files, etc. Survey incomplete - please respond if you haven’t already done so

Information is crucial for planning Some running totals with ~50% of detectors covered

Configuration data: 10 GB/year (dominated by SCT)

Conditions data in relational DB: 100 GB/year (dominated by TRT, LAr)

Conditions data in streamed files: 2400 GB/year (dominated by SCT)

Monitoring histograms: 24 TB/year (SCT, LAr, MDT Barrel)

Data distribution/feedback strategy



Simulation

Geant4 is working extremely well for production “This also means that in more than 35000 simulation jobs, we did not

observe a single consistent G4-failure.” (NorduGrid feedback)

Some low level issues being investigated

Multiple geometries available Full and initial ATLAS layouts; Combined testbeam

While memory and cpu useage are ok, still need some work

Digitization fully functional

Simulation is another area where lack of personnel is a concern Several crucial developers are moving to other ATLAS activities or leaving

Adele Rimoldi - University of Pavia & INFN, I - 24 September,2004

14G4ATLAS (from A.Dell’Acqua)

Not much progress/evolution : ensure continuity vs. revolution Increased support for CTB-related activities GeoModel extensively used Time now for more extensive changes

Framework Full support for MC truth Implementation of missing pieces/features

Full deployment of MC truth support machinery with rel. 8.0.3 MC Truth strategies and machinery for the IDET introduced with

8.3.0 The MC truth machinery MUST be explicitly activated, by

running a macro)! Thresholds for the various processes are user-defined and

depend on the application one is running Still missing: strategies for hadronic processes

Use HepMC format for storing information about secondary particles/vertices

Ongoing validation of MC truth (IDET radiography based on truth information (by F.Gianotti, Phys. Validation meeting 15/9/04))

Tracker radiography : i.e R vs Z at which secondaries are produced

middle Pixel layermissing

TRT C-wheelsmissing

R (cm)

Z (cm)

Adele Rimoldi - University of Pavia & INFN, I - 24 September,2004

15New Validation on 8.0.7 / 8.7.0 (from A.Nairz)

G4 Memory Usage (Rel. 8.0.7)

Process Size (1)Configuration

185 MB

146 MB

272 MB

181 MB

338 MB

351 MB

218 MB

171 MB

308 MB

213 MB

386 MB

406 MB

158 MBTile Calorimeter

206 MBMuon System + Toroids

299 MBLAr Calorimeter

202 MBInner Detector

379 MBFull Detector, no B-Field

397 MBFull Detector (DC1 Layout)

“Max Swap” (LSF) Memory Snooper BOR Memory Snooper EOR

(1) The numbers are averages over all runs in the respective configuration.



Reconstruction

Strong focus on Event Data Model ESD/AOD Task Force Draft Report available

Initial prototypes in place for DC2 and initial layout studies

Important to gain experience and iterate after Rome Physics Workshop

Strong focus on Physics Analysis Tools Earlier talk by Kyle

A prototype that again needs active use and feedback

More tools are almost ready (Python bindings to ROOT & Athena)

We need to understand how best to use the available tools

And multiple strategies might well be appropriate

Implementation of Reco Task Force recommendations well underway but not yet completed

Need to add emphasis on performance (cpu & memory)

David Rousseau, ESRAT summary, plenary III, 22 Sep 2004

RTF follow-upRTF recommendations

implementation well underway and we start to see the benefits e.g.:o Common calorimeter code⇒new

« calorimeter blind » clustering, and jet reconstruction

o Common tracking EDM⇒new tracking algorithms developed for test beam

o Persistence solution applies simultaneously to ID+Muon and Lar+Tile

o 5 different clustering algorithm in test beam with a few jobOption switches



Physics Validation

Part of Physics Coordination, not the Computing project, but...

Absolutely crucial feedback, and excellent feedback

Progress from gross detector geometry errors in early 8.0.x releases to detailed tuning on later releases Although one major deficiency slipped through

And no doubt we’ll continue to find and fix problems

Active for 7 months in the DC2 cycle Regular meetings

Encourage attendance from software developer community

19

Events with ET(3x5)/ET (generated) > 0.85

Events with ET(3x5)/ET (generated) < 0.85

Radius where hardest Brem has been produced (|η|<1)

BARREL

Pix1

Pix2Pix3

Pixel support

SCT1

SCT2

SCT3SCT4

TRT

Cables and services

TRTsupport

20Inner Detector Initial Layout

EtaEta

Pixel

SCT

Pixel+SCT Red: InitialBlack: Final

Red: InitialBlack: Final

PT=50GeV Muons

21Jet Reconstruction

DC2 sample B1 (DiJets PT>180)

KT Jets Cone Jets

CaloTowers: RedTopoCluster: BlackTruth: Blue

(Paige)



Short Term Release Plan

9.0.0: 27 Oct 2004 Clients are CTB, DC2 Tier-0, Physics Workshop Initial Layout

Lots of potential conflicts from multiple clients

9.1.0: 24 Nov

9.2.0: 15 Dec

9.3.0: 12 Jan 2005

9.4.0: 02 Feb

10.0.0: 23 Feb Primary client is Rome Physics Workshop

Releases 9.1.0-9.4.0 will revert to being developer releases (unlike

8.x.0) Some things will be broken while we restructure them

See Dario’s talk for long term planning and milestones



Issues and Concerns

We are driving the developer community very hard Juggling conflicting priorities and deliverables for CTB and DC2 has been

hard

Extremely useful (if sometimes painful) feedback from both

It’s crucial that we assimilate the lessons Need to step back and learn, not just rush onto the next

problem/deliverable

Still short of personnel The situation has improved since Manpower Review in Sept. 2003

But we are still ~25% short

Promised manpower taking longer to arrive than expected

Long ramp up times

We have to continue to exert pressure to fill critical areas



Summary

A tremendous amount of progress in the last few months

Geant4 has been a great success as a production simulation tool

Juggling conflicting priorities between CTB and DC2 has been complex But in general managed successfully

Albeit at the expense of lots of “just in time” releases

Very good feedback from the Physics Validation Teams

Physics analysis tools and data model are available in prototype form Experience using them over the next few months will be essential so we can

understand better how to use them, what’s missing, superfluous etc.

A major focus in next months must be usability and performance Includes remote use and development

Prototype Physicist Workbook (cf. BaBar) available in December

ATLAS Overview Week - 7 October 2004 Software Status David R. Quarrie LBNL [email protected].

Documents

Transcript of ATLAS Overview Week - 7 October 2004 Software Status David R. Quarrie LBNL [email protected].