LHCb Turbo

The LHCb Turbo StreamSean Benson

S. Benson

S. Benson LHCb Turbo

The LHCb experiment

2

• LHCb is a forward arm spectrometer (pseudo-rapidity range: 2 < η < 5), • Accurate decay time resolution through vertex locator (VELO), • Accurate particle ID provided by RICH detectors, • High muon reconstruction efficiency from muon stations.

VELO%primary%and%

secondary%ver3ces%%

RICH%par3cle%ID%%

Tracking%sta3ons%

momentum%resolu3on%%

Muon%chambers%trigger,%μDID%

E/HCAL%trigger,%p,e,γDπ0DID%

S. Benson LHCb Turbo

Data readout

3

5000 optical/analog links ~4Tb/s

320 ROBs (TELL1s/UKL1s): 24 inputs @ 1.5Gb/s 4 outputs @ 1Gb/s

3000 GbE ports 35 Gb/s

50 subfarms of 40 nodes

S. Benson LHCb Turbo

Run I data taking

!

4

• LHCb trigger separated in to three distinct levels:

• L0 - High energy single particle triggers (also with di-muon combinations), reduces rate to 1MHz.

• HLT1 - Looks for single displaced high quality tracks (again allowing di-muon combinations), reduces rate further to 80kHz.

• HLT2 - Full event reconstruction, with both inclusive and exclusive beauty and charm triggers, producing an output rate of 5kHz.

2012 Trigger strategy

S. Benson LHCb Turbo

Run I data taking

5

• LHCb relies on a large amount of computing power to reconstruct interesting decays inside the trigger.

• LHC delivered collisions ~30% of the time⇒ computing power would be idle 70% of the time.

• Deferring allows for the data to be buffered in local storage, an processed later between LHC fills.

• Introduced in 2012, this gave more time so that increasingly complex algorithms could be used in the trigger (such as online Ks reconstruction ).

2012 Trigger strategy

S. Benson LHCb Turbo

Run II data taking

6

• Introducing full deferral allows even more time in the second level of the software trigger.

• Move buffer stage from L0 to after first software level ⇒ buffer at level of kHz rather than MHz.

• With this time, we can: • calibrate and align the detector (see

Markus Frank’s talk presented earlier today),

• run an offline-like reconstruction (see Barbara Storaci’s talk presented on Monday),

• calculate offline-like particle identification likelihoods.⇒ gives the ability to be more selective in HLT2 (i.e. pre-scale Cabbibo favoured charm decays while keeping 100% of doubly Cabibbo suppressed charm).

S. Benson LHCb Turbo

Increasing the buffer size…

7

Benefits of collaborative working…

S. Benson LHCb Turbo

Run II calibrations

8

Run II philosophy: • Bring online and offline reconstructions as close as possible, • Use the same calibrations offline as online ⇒ no reprocessing before the end of Run II • Requirement for a turbo stream.

General strategy: • Use HLT1 special selection lines. • At the beginning of a fill, use farms nodes (1500) for track reconstruction. • Perform χ2 minimisation done on a single node.⇒ time needed to perform all the alignment tasks is a few minutes.

Similar to ALICE approach (J. Phys.: Conf. Ser. 219 022010), though LHCb approach to filter events (rather than compress - ALICE goal 100x).

RICH calibration: per run, all events, online analysis task

OT t0 calibration: per fill, 100k-1m events,

online analysis task

VELO alignment: per fill, 100k events,

online alignment framework

Tracker alignment: per fill, 50k D0 events,

online alignment framework

RICH alignment: per fill, 50k events,

online alignment framework

Muon alignment: per day/week, 20k J/ψ events,

online alignment framework

S. Benson LHCb Turbo

Turbo stream strategy

9

• New concept: Turbo stream - use the trigger reconstruction (of offline quality) to perform physics measurements.

• Full stream (see talk of Stefan Rosier in WLCG workshop):

• Calculate luminosity information, • reconstruction, • analysis pre-selections, • merging.⇒total time for single 3GB raw file: ~30h.

• Turbo stream chain: • Calculate luminosity information, • resurrect trigger candidates.⇒ time < 1h

• Removing what we don’t need reduces the size of the event and allows a much higher rate for the same bandwidth.

S. Benson LHCb Turbo

Turbo stream strategy

10

Full stream Turbo stream

A high rate of smaller events is sometimes preferable to a lower rate of larger ones.

S. Benson LHCb Turbo

Saving trigger objects• Run I:

• In Run I limited information on selected candidates was placed in to the raw banks. • These contained particle momenta and mass hypotheses (imposed by the trigger line)

information, along with track χ2 information where appropriate. • Very useful for monitoring, though not enough information to make and publish a

measurement. • Also required for data-driven trigger efficiencies calculated via the TISTOS method (arXiv:

1310.8544):εTOS = NTOS&TIS/NTIS

TOS = Triggered on selection, TIS = triggered independent of selection.

11

HLT1 muon triggersL0 muon triggers

HLT2 muon triggers

S. Benson LHCb Turbo

Implementation• Run II:

• For trigger lines that are sent to the Turbo stream, much more information on the decay will be saved.

• This includes (to name a few): • particles - full position and momentum covariance matrices • PID - Δln(𝓛) for all mass hypotheses, ingredients for neural network PID variables • Intermediate vertices - χ2, position, full covariance matrices

• By imprinting a certain pattern inside the raw data, full information about reconstructed decays can be saved.

• Typical decay costs <1kB in raw event.

12

mother - particle

non-basic daughter - particle basic daughter - particle mother vertex

basic daughter - particle basic daughter - particle Track/CALO level info -proto-particle

track Rich/Muon PIDs, etc.Track/CALO level info -

proto-particletrack

Rich/Muon PIDs, etc.

Track/CALO level info - proto-particle

track Rich/Muon PIDs, etc.

vertex

S. Benson LHCb Turbo

Resurrecting trigger objects -Tesla• LHCb has published over 230 papers since the beginning of LHC data taking. • This has been made possible by the common analysis framework allowing us to easily gain

access to information on our reconstructed candidates. • Simple addition of tools to user dataset production jobs allows easy access to:

• Kinematic information (4-momenta,…), • geometric information (impact parameter, flight distance,…), • global information (run number, event number, …), • …

• It is important that analysts are able to use existing infrastructure to allow Turbo measurements to be made just as easily.

• Tesla program has therefore been written to: • Resurrect decays from the raw event of unlimited complexity: A➝m*B(➝n*C(…)). • Persist a small fraction of the raw event. • Calculate luminosity information. • Create DSTs that appear of the same form as found after the “Analysis pre-selection” stage

(shown on slide 6).

• Resulting output can used for measurements in the same way as the Full stream ⇒ this means analysis pre-selection stage is essentially moved inside the trigger.

13

S. Benson LHCb Turbo

2015 plans• Initial use of the Turbo stream in 2015 early

measurements with a view to extending the principle in the upgraded LHCb experiment (Run III).

• Turbo stream is a new concept for Run II ⇒ for Run II, precautions will be put in place.

• Through routing bits, events selected by the Turbo stream can then be separated from the full stream.

• For initial commissioning of the Turbo stream, the complete raw event will be archived as a precaution.

• As confidence is gained, Turbo stream analyses are expected to be more common than Full stream counterparts (see talk of Marco Cattaneo in WLCG workshop)

14

S. Benson LHCb Turbo

Trigger quality• LHCb makes measurements of CP violation, through asymmetries and angular analyses. • This means that momentum, decay time, and mass resolutions should be comparable between

the trigger and offline calculations in order to produce analyses of the same quality as those in the Full stream.

• Shown below: Bs➝φφ - a golden mode for the upgraded LHCb experiment in determining CP violation through decay time dependent angular analyses (arXiv:1407.2222, arXiv:1303.7125, arXiv:1204.2813).

• Resolutions consistent between online and offline reconstructions.

15

]2 mass [MeV/csB5300 5400 5500

)2Ev

ents

/ (9

MeV

/c

0

100

200

300

400

500

600

700 Offline reconstructionOnline reconstruction 2015 simulation

LHCb

[MeV/c]trueT

-pT

psB-100 -50 0 50 100

Even

ts / (

6 M

eV/c

)

0

50

100

150

200

250

300

350

400

450

500

Offline reconstructionOnline reconstruction 2015 simulation

LHCb

[fs]trueτ-τ sB-100 0 100

Even

ts / (

11 fs

)

0

50

100

150

200

250

300

350

400

450

500

Offline reconstructionOnline reconstruction 2015 simulation

LHCb

LHCb TurboS. Benson

• Full deferral in the LHCb trigger gives enough time for offline quality calibrations and therefore allows offline quality selection criteria to be calculated.

• Saving only the candidates selected by the trigger allows the raw event to be an order of magnitude smaller than for the full stream.

• This also allows for a further reconstruction to be avoided and the trigger reconstruction used for analysis.

• The LHCb Turbo Stream is the means by which decays reconstructed in the trigger can be persisted and used for offline analysis.

• When the trigger objects are resurrected by the Tesla program, the usual analysis LHCb analysis framework can be used to perform measurements.

• 2015 presents itself as the earliest opportunity to put the concept to the test, with potential expansion in the upgraded LHCb experiment.

Summary

16