1Dirk Wiedner TWEPP2013

The trigger-less TBit/s readout for the

Mu3e experimentDirk Wiedner

On behalf of the Mu3e collaboration

24 Sep 2013

Dirk Wiedner, Mu3e collaboration 2

The Mu3e Signal

9/20/2012

• μ→eee rare in SM• Enhanced in:

o Super-symmetryo Grand unified modelso Left-right symmetric

modelso Extended Higgs sectoro Large extra dimensions

Rare decay (BR<10-12, SINDRUM)• For BR O(10-16)

>1016 muon decaysHigh decay rates

O(109 muon/s)

Dirk Wiedner, Mu3e collaboration 3

The Mu3e Experiment

9/20/2012

• Target double hollow cone• Silicon pixel tracker• Scintillating fiber tracker• Recurl station• Tile hodoscope

• Muon beam O(109/s)• Helium atmosphere• 1 T B-field

Dirk Wiedner, Mu3e collaboration 4

The Mu3e Experiment

9/20/2012

• Target double hollow cone• Silicon pixel tracker• Scintillating fiber tracker• Recurl station• Tile hodoscope

• Muon beam O(109/s)• Helium atmosphere• 1 T B-field

Dirk Wiedner, Mu3e collaboration 5

The Mu3e Experiment

9/20/2012

• Target double hollow cone• Silicon pixel tracker• Scintillating fiber tracker• Recurl station• Tile hodoscope

• Muon beam O(109/s)• Helium atmosphere• 1 T B-field

Dirk Wiedner, Mu3e collaboration 6

The Mu3e Experiment

9/20/2012

• Target double hollow cone• Silicon pixel tracker• Scintillating fiber tracker• Recurl station• Tile hodoscope

• Muon beam O(109/s)• Helium atmosphere• 1 T B-field

Dirk Wiedner, Mu3e collaboration 7

The Mu3e Experiment

9/20/2012

• Target double hollow cone• Silicon pixel tracker• Scintillating fiber tracker• Recurl station• Tile hodoscope

• Muon beam O(109/s)• Helium atmosphere• 1 T B-field

Dirk Wiedner TWEPP2013 8

Readout Requirements

24 Sep 2013

• 2.5 GHz muon decays• 50 ns readout frames (pixel)• O(5000) pixel chips• O(7000) scintillating fibers• O(7000) timing tiles• Online filtering

9

Timing Detectors

24 Sep 2013Dirk Wiedner TWEPP2013

• Scintillating fiber hodoscope

• Timing tiles• On detector zero-

suppression• Poster Session:

o STiC - A Mixed Mode Silicon-Photomultiplier Readout ASIC for Time-of-Flight Applications (Tobias Harion)

O(7000) fibersO(7000) tiles

Dirk Wiedner TWEPP2013 10

Silicon Pixel Detector

24 Sep 2013

• Inner double layer• Outer double layer• Re-curl layers

o Both sides (x2)• Sensor size

o 1x2 cm2 inner layerso 2x2 cm2 outer layers 180 inner sensors

4680 outer sensors

Dirk Wiedner, Mu3e collaboration 11

HV-MAPS

2/5/2013

• High Voltage Monolithic Active Pixel Sensors• HV-CMOS technology• Reversely biased ~60V

o Charge collection via drift Fast O(100 ns)o Thinning to < 50 μm possible

by Ivan PericI. Peric, A novel monolithic pixelated particle detector implemented in high-voltage CMOS technology Nucl.Instrum.Meth., 2007, A582, 876

Dirk Wiedner, Mu3e collaboration 12

HV-MAPS

2/5/2013

• High Voltage Monolithic Active Pixel Sensors• HV-CMOS technology• Reversely biased ~60V

o Charge collection via drift Fast O(100 ns)o Thinning to < 50 μm possible

by Ivan PericI. Peric, A novel monolithic pixelated particle detector implemented in high-voltage CMOS technology Nucl.Instrum.Meth., 2007, A582, 876

Dirk Wiedner, Mu3e collaboration 13

HV-MAPS

2/5/2013

• High Voltage Monolithic Active Pixel Sensors• HV-CMOS technology• Reversely biased ~60V

o Charge collection via drift Fast O(100 ns)o Thinning to < 50 μm possible

• Integrated readout electronicso Zero suppressiono 800Mbit/s serial LVDS outputs

by Ivan PericI. Peric, A novel monolithic pixelated particle detector implemented in high-voltage CMOS technology Nucl.Instrum.Meth., 2007, A582, 876

Dirk Wiedner, Mu3e collaboration 14

HV-MAPS

2/5/2013

• High Voltage Monolithic Active Pixel Sensors• HV-CMOS technology• Reversely biased ~60V

o Charge collection via drift Fast O(100 ns)o Thinning to < 50 μm possible

• Integrated readout electronicso Zero suppressiono 800Mbit/s serial LVDS outputs

by Ivan PericI. Peric, A novel monolithic pixelated particle detector implemented in high-voltage CMOS technology Nucl.Instrum.Meth., 2007, A582, 876

Dirk Wiedner TWEPP2013 15

Pixel Readout Scheme

24 Sep 2013

Dirk Wiedner TWEPP2013 16

Pixel Readout Scheme

24 Sep 2013

• Pixel logico Pixel address (8 bit)o Frame number (4 bit)o 50 ns frames

• Column logico Pixel datao Column addresso Coarse time

• Frame logico Super Frameo Contains 16 x 50 ns readout

frameso + Sensor header

• Readout buffer• Serializer and fast link(s)

Pixel address

Pixel Logic

Column Logic

Frame logicReadout buffer

Serializer

Fine time

Coarsetime

Columnaddress

Dirk Wiedner TWEPP2013 17

Pixel Readout Scheme

24 Sep 2013

• Pixel logico Pixel address (8 bit)o Frame number (4 bit)o 50 ns frames

• Column logico Pixel datao Column addresso Coarse time

• Frame logico Super Frameo Contains 16 x 50 ns readout

frameso + Sensor header

• Readout buffer• Serializer and fast link(s)

Pixel address

Pixel Logic

Column Logic

Frame logicReadout buffer

Serializer

8 bit

Fine time

4 bit12 bit

Coarsetime

Columnaddress

Dirk Wiedner TWEPP2013 18

Pixel Readout Scheme

24 Sep 2013

• Pixel logico Pixel address (8 bit)o Frame number (4 bit)o 50 ns frames

• Column logico Pixel datao Column addresso Coarse time

• Frame logico Super Frameo Contains 16 x 50 ns readout

frameso + Sensor header

• Readout buffer• Serializer and fast link(s)

Pixel address

Pixel Logic

Column Logic

Frame logicReadout buffer

Serializer

8 bit

Fine time

4 bit12 bit

Coarsetime

Columnaddress

12 bit8 bit

32 bit

Dirk Wiedner TWEPP2013 19

Pixel Readout Scheme

24 Sep 2013

• Pixel logico Pixel address (8 bit)o Frame number (4 bit)o 50 ns frames

• Column logico Pixel datao Column addresso Coarse time

• Frame logico Contains 16 x 50 ns readout

frameso + Sensor header Super Frame

• Readout buffer• Serializer and fast link(s)

Pixel address

Pixel Logic

Column Logic

Frame logicReadout buffer

Serializer

8 bit

Fine time

4 bit12 bit

Coarsetime

Columnaddress

32 bit

4 x serial @ 800 Mb/s

12 bit8 bit

Dirk Wiedner TWEPP2013 20

Data Link SchemeFrom detector slices

to time slices

24 Sep 2013

Dirk Wiedner TWEPP2013 21

Link Overview

24 Sep 2013

• Front end links o Pixel sensor to on-detector

FPGA• 400 – 800 Mbit/s• LVDS

o Timing detector readout• Optical links from detector

o Front end FPGAso … to readout boardso 5 Gbit/s

• Optical links in counting roomo Off-detector read out boardso …to PC Farm

Dirk Wiedner TWEPP2013 22

Link Overview

24 Sep 2013

• Front end links o Pixel sensor to on-detector

FPGA• 400 – 800 Mbit/s• LVDS

o Timing detector readout• Optical links from detector

o Front end FPGAso … to readout boardso 5 Gbit/s

• Optical links in counting roomo Off-detector read out boardso …to PC Farm

Pixel Sensor

Silicon FPGAs

x86

Readout board

x12

PCx48

Dirk Wiedner TWEPP2013 23

Link Overview

24 Sep 2013

• Front end links o Pixel sensor to on-detector

FPGA• 400 – 800 Mbit/s• LVDS

o Timing detector readout• Optical links from detector

o Front end FPGAso … to readout boardso 5 Gbit/s

• Optical links in counting roomo Off-detector read out boardso …to PC Farm

Pixel Sensor Fiber Tile Pixel

Sensor Fiber Tile Pixel Sensor Fiber Tile Pixel

Sensor Fiber Tile

Silicon FPGAs

x86

FiberFPGAs

x48

TileFPGAs

x48

Readout board

x16

Readout board

x8

Readout board

x8

x6336 x7000 x7000

PCx48

O(8Tbit/s)

Dirk Wiedner TWEPP2013 24

Tile

Link Overview

24 Sep 2013

• Front end links o Pixel sensor to on-detector

FPGA• 400 – 800 Mbit/s• LVDS

o Timing detector readout• Optical links from detector

o Front end FPGAso … to readout boardso 5 Gbit/s

• Optical links in counting roomo Off-detector read out boardso …to PC Farm

Pixel Sensor Fiber Pixel

Sensor Fiber Tile Pixel Sensor Fiber Tile Pixel

Sensor Fiber Tile

Silicon FPGAs

x86

FiberFPGAs

x48

TileFPGAs

x48

Readout board

x16

Readout board

x8

Readout board

x8

PCx48

x376

Dirk Wiedner TWEPP2013 25

Link Overview

24 Sep 2013

• Front end links o Pixel sensor to on-detector

FPGA• 400 – 800 Mbit/s• LVDS

o Timing detector readout• Optical links from detector

o Front end FPGAso … to readout boardso 5 Gbit/s

• Optical links in counting roomo Off-detector read out boardso …to PC Farm

Pixel Sensor Fiber Tile Pixel

Sensor Fiber Tile Pixel Sensor Fiber Tile Pixel

Sensor Fiber Tile

Silicon FPGAs

x86

FiberFPGAs

x48

TileFPGAs

x48

Readout board

x16

Readout board

x8

Readout board

x8

PCx48

x376 x192 x192O(4Tbit/s)

Dirk Wiedner TWEPP2013 26

Link Overview

24 Sep 2013

• Front end links o Pixel sensor to on-detector

FPGA• 400 – 800 Mbit/s• LVDS

o Timing detector readout• Optical links from detector

o Front end FPGAso … to readout boardso 5 Gbit/s

• Optical links in counting roomo Off-detector read out boardso …to PC Farm

Pixel Sensor Fiber Tile Pixel

Sensor Fiber TilePixel Sensor Fiber TilePixel

Sensor Fiber Tile

Silicon FPGAs

x86

FiberFPGAs

x48

TileFPGAs

x48

Readout board

x16

Readout board

x8

Readout board

x8

PCx48x192

x96x96

O(4Tbit/s)

Dirk Wiedner TWEPP2013 27

Front End FPGAs

24 Sep 2013

• FPGAs on detectoro 86 (+96) pieces

• Receive sensor datao 108 LVDS inputs

• 5 Gbit/s outputso 8 optical linkso … to counting house

• Switching data between readout boards farms A-D

Front end FPGA

800 Mbit/sLVDS inx 108

5 Gbit/s optical

Readout board

A

Pixel Senso

r

Readout board

B

Readout board

C

Readout board

D

Dirk Wiedner TWEPP2013 28

Front End FPGAs

24 Sep 2013

• FPGAs on detectoro 86 (+96) pieces

• Receive sensor datao 108 LVDS inputs

• 5 Gbit/s outputso 8 optical linkso … to counting house

• Switching data between readout boards farms A-D

Front end FPGA

800 Mbit/sLVDS inx 108

5 Gbit/s optical

Readout board

A

Pixel Senso

r

Readout board

B

Readout board

C

Readout board

D

Dirk Wiedner TWEPP2013 29

Front End FPGAs

24 Sep 2013

• FPGAs on detectoro 86 (+96) pieces

• Receive sensor datao 108 LVDS inputs

• 5 Gbit/s outputso 8 optical linkso … to counting house

• Switching data between readout boards farms A-D

Front end FPGA

800 Mbit/sLVDS inx 108

5 Gbit/s optical

Readout board

A

Pixel Senso

r

Readout board

B

Readout board

C

Readout board

D

Dirk Wiedner TWEPP2013 30

Front End FPGAs

24 Sep 2013

• FPGAs on detectoro 86 (+96) pieces

• Receive sensor datao 108 LVDS inputs

• 5 Gbit/s outputso 8 optical linkso … to counting house

• Switching data between readout boards farms A-D

Front end FPGA

800 Mbit/sLVDS inx 108

5 Gbit/s optical

Readout board

A

Pixel Senso

r

Readout board

B

Readout board

C

Readout board

D

Dirk Wiedner TWEPP2013 31

Front end

FPGA

Readout Board

24 Sep 2013

• FPGA readout boardso 4 per sub-detector

• 5 Gbit/s optical inputso 16-28 inputs

• 10 Gbit/s optical outputo 12 outputs to PCs

• Switching networko A-D sub-farmso One output per PC

Readout board

5 Gbit/s Optical

x28

PC

10 Gbit/s Optical

PC

Sub-farm A

Front end

FPGA

Front end

FPGA

Front end

FPGA

PCx12

Dirk Wiedner TWEPP2013 32

Readout Board

24 Sep 2013

• FPGA readout boardso 4 per sub-detector

• 5 Gbit/s optical inputso 16-28 inputs

• 10 Gbit/s optical outputo 12 outputs to PCs

• Switching networko A-D sub-farmso One output per PC

Front end

FPGA

Readout board

5 Gbit/s Optical

x28

PC

10 Gbit/s Optical

PC

Front end

FPGA

Front end

FPGA

Front end

FPGA

PC

Sub-farm A

x12

Dirk Wiedner TWEPP2013 33

GPU-PC

24 Sep 2013

• PC with GPU• 10 Gbit/s Fiber input

o 8 inputs from sub-detectors

• Data filteringo Timing Filter on FPGAo Track filter on GPUo Data to tape < 100

MB/s

FPGA PCIe board

GPU computer

Optical mezzanine connectors

Dirk Wiedner TWEPP2013 34

GPU-PC

24 Sep 2013

• PC with GPU• 10 Gbit/s Fiber input

o 8 inputs from sub-detectors

• Data filteringo Timing Filter on FPGAo Track filter on GPUo Data to tape < 100

MB/s

GPU computer

Dirk Wiedner TWEPP2013 35

Readout board

Readout board

GPU-PC

24 Sep 2013

• PC with GPU• 10 Gbit/s Fiber input

o 8 inputs from sub-detectors

• Data filteringo Timing Filter on FPGAo Track filter on GPUo Data to tape < 100

MB/sPC

10 Gbit/s Optical

x8

PC

10 Gbit/s Optical

x8

A

Readout board

B

x8 Readout board

Readout boardReadout board

Dirk Wiedner TWEPP2013 36

Timing Filter

24 Sep 2013

• Entire event on PCIe FPGA• Tile and Fiber data

o Easy to matcho Look for three tracks

• Reject data without three hitso … inside time interval

1

3

2

Dirk Wiedner TWEPP2013 37

Timing Filter

24 Sep 2013

• Entire event on PCIe FPGA• Tile and Fiber data

o Easy to matcho Look for three tracks

• Reject data without three hitso … inside time interval

1

3

2

Dirk Wiedner TWEPP2013 38

Vertex Filter

24 Sep 2013

• Entire event on GPU• Large target

o Large spread of muonso Easy vertex separation

• Reject data without three trackso … inside area interval on target

1

3

2

Dirk Wiedner TWEPP2013 39

Vertex Filter

24 Sep 2013

• Entire event on GPU• Large target

o Large spread of muonso Easy vertex separation

• Reject data without three trackso … inside area interval on target

1

3

2

Dirk Wiedner TWEPP2013 40

Summary• Mu3e has 280M pixels @ >109 muons/s• >1 Tbit/s data • 0-suppressed serial data from active pixel sensors• Switched optical network• GPU filter farm with optical inputs

24 Sep 2013

+ +

Dirk Wiedner TWEPP2013 41

Backup Slides

24 Sep 2013

Dirk Wiedner, Mu3e collaboration 42

Physics Motivation

9/20/2012

Standard model:• No lepton flavor violation

Lepton flavor violation?

Dirk Wiedner, Mu3e collaboration 43

Physics Motivation

9/20/2012

Standard model:• No lepton flavor violation, but:

o Neutrino mixingo Branching ratio <10-50 →unobservable

Lepton flavor violation: μ+→e+e-e+

Dirk Wiedner, Mu3e collaboration 44

The Mu3e Signal

9/20/2012

• μ→eee rare in SM• Enhanced in:

o Super-symmetryo Grand unified modelso Left-right symmetric

modelso Extended Higgs sectoro Large extra dimensions

Dirk Wiedner, Mu3e collaboration 45

The Mu3e Background

9/20/2012

• Combinatorial backgroundo μ+→e+νν & μ+→e+νν & e+e-

o many possible combinations

Good time and Good vertex resolution

required

Dirk Wiedner, Mu3e collaboration 46

The Mu3e Background

9/20/2012

• μ+→e+e-e+ννo Missing energy (ν) Good momentum resolution

(R. M. Djilkibaev, R. V. Konoplich,Phys.Rev. D79 (2009) 073004)

Dirk Wiedner TWEPP2013 47

Pixel Sensor Links

24 Sep 2013

• Vertex Sensor chipso 180 chipso 4 LVDS linkso 800 Mbit/s per link

• Central Silicon Trackero 936 chipso 2 LVDS linkso 800 Mbit/s

• Recurl stationso 3744 chipso 1 LVDS linko 400 Mbit/s

Dirk Wiedner TWEPP2013 48

Pixel Sensor Links

24 Sep 2013

• Vertex Sensor chipso 180 chipso 4 LVDS linkso 800 Mbit/s per link

• Central Silicon Trackero 936 chipso 2 LVDS linkso 800 Mbit/s

• Recurl stationso 3744 chipso 1 LVDS linko 400 Mbit/s

Pixel Sensor

800 Mbits/s

Front end FPGA

Dirk Wiedner TWEPP2013 49

Front End FPGAs

24 Sep 2013

• FPGAs on detectoro 86 (+96) pieces

• Receive sensor datao 108 LVDS inputs

• 5 Gbit/s outputso 8 optical linkso … to counting house

• Switching between readout boards A-D

Optical transceiver FE board

Dirk Wiedner TWEPP2013 50

Average Occupancies

24 Sep 2013

• All numbers per frame of 50 ns • Vertex detector

o 2 hits per sensor• Central silicon tracker

o 0.6 hits per sensor• Recurl stations

o 0.13 hit per sensor• Fiber hodoscope

o 0.16 hits per fiber• Timing tiles

o 0.09 hits per tileTile occupancies

Dirk Wiedner TWEPP2013 51

Maximum Occupancies

24 Sep 2013

• All numbers per frame of 50 ns • Vertex detector

o 5 hits per sensor• Central silicon tracker

o 2 hits per sensor• Recurl stations

o 1 hit per sensor• Fiber hodoscope

o 0.24 hits per fiber• Timing tiles

o 0.14 hits per tile

φ

zVertex occupancies