Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card...

30
CERN, November 2019 - John Gill

Transcript of Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card...

Page 1: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

CERN, November 2019 - John Gill

Page 2: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Overview

•Overview

•Core technology

•Available technology

•Beam Synchronous Timing

•Next steps

Page 3: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Overview - RFoWR

RFCapture

RF in

Page 4: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Overview - RFoWR

Deterministic Latency - Δt

RFCapture

Phase

Amplitude

Frequency

RF in RF out

Page 5: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Overview - RFoWR

Deterministic Latency - Δt

RFCapture

Phase

Amplitude

Frequency

RF in RF out

Page 6: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Available technology - RFoWR systems

RF over White Rabbit is not new. Two types of system have been developed at CERN:

• The first uses an ADC to digitise an RF signal. Currently unsupported.

https://wikis.cern.ch/display/HT/SPS+RF+distribution+over+White+Rabbit

• The second uses a PLL subsystem to lock and track the incoming RF signal.

https://www.ohwr.org/projects/wr-d3s/wiki

• The third is virtual ! Just transmit phase and FTW setpoints for a future TAI timestamp. Enables arbitrary RF signal or clock distribution.

PLL based-systems are used at the ESRF (production) facility in Grenoble and are being developed at SPring-8.REFURBISHMENT OF THE ESRF ACCELERATOR SYNCHRONISATIONSYSTEM USING WHITE RABBIThttp://inspirehep.net/record/1656131/files/tucpl01.pdf?version=1

Page 7: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Available technology - RFoWR systems

FMC-DAC600M Version 3.0 achieves:

• < 3 ps RMS jitter (100 Hz to 5 MHz) • < 7 ps RMS jitter ( 10 Hz to 5 MHz)• < 12 ps RMS jitter ( 1 Hz to 5 MHz)

for RF signals between 10 - 70 MHz.

https://www.ohwr.org/project/fmc-dac-600m-12b-1cha-dds/wikis/home

A cleaner PLL can also be employed on this receiver to multiply the DAC frequency up to ~500 MHz.

FORM FACTORS: PCIe and VME with SPEC and SVEC carriers

÷ X

Page 8: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Core technology - Direct Digital Synthesis overview

Page 9: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Core technology - Direct Digital Synthesis - phase to amplitude

Time

Time

Addr

ess

of L

UT

FTW

PHASE ACCUMULATOR

PHASE TO AMPLITUDE CONVERSION

LUT

Valu

es

FTW

Output frequency of a DDS can be described as function of the phase accumulator size, frequency tuning word and the clock frequency of the DDS (e.g. WR clock frequency @ 125 MHz).

OUTPUTFREQUENCY

Page 10: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Core technology - Direct Digital Synthesis and White Rabbit

+

ACCUMULATOR

FTW

LUT

DAC

FPGA

If each slave replays FTWs when using free running local oscillators/PLLs, their Numerically Controlled Oscillators (NCO) cannot be guaranteed to be aligned.

NCOs without WR timing

NCOs with WR timing

Page 11: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing (BST) - SPS + LHC RF distribution system

Beam Synchronous Timing (BST) system.

BST information transmitted via a Timing Trigger and Control (TTC) network.

With SPS - LLRF upgrade we need a new card to generate bunch and turn clocks to drive the BST master.

BST Master

TTC network

BST Receivers (BOBR)

BST-Master BOBR TTC

Page 12: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - WR2RF card proposal

Page 13: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - WR2RF card proposal

FTW

Page 14: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - WR2RF card proposal

BST Master

TTC network

BST Receivers (BOBR)

FTW

WR2RFBunch + Turn Clocks

Page 15: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - WR2RF card proposal

BST Master

TTC network

BST Receivers (BOBR)

FTW

WR2RFBunch + Turn Clocks

FTW

WRswitch network

In the future, a BOBR replacement card could

receiveFTWs directly

from the SPS LLRF.

RFoWR receivers

Page 16: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Next steps

Page 17: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Conclusion

The RFoWR user base is increasing: ESRF SHINE SPring-8 and CERN

Ongoing effort with Radio over Ethernet (RoE) IEEE 1914.3 committee to accept our proposals for a structure-aware DDS protocol.

SPS - LLRF system will be a high profile user of RFoWR and a new WR2RF card will be developed to drive the existing BST system.

Page 18: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - WR2RF card functions

WR NetworkFTW

FTW

FPGA - Kintex 7

Osc 10 MHz

Osc 1 GHz

RF Signal

SFP

WRCore

ConfigurableNCO + IQ

Clean-upPLL

Clock Multiplier

DACDDS

OCXO

Clk125/250 MHz

FTW

LO

IF

WR2RF card

LLRF

Page 19: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - SPS LLRF upgrade

• WR2RF receiver card to be a hybrid version of the eTRM-14/15 cards.

• Plan to use kintex 7 FPGA as on the eTRM-14.

• These cards are already part of the planned SPS LLRF system, so reusing analog circuitry and HDL logic should lower risk.

Page 20: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - SPS - LLRF upgrade

• Prototype boards developed for the eRTM-14/15 project have shown:

• White Rabbit clocks are generated with 70 fs RMS jitter (10 Hz and 5 MHz).

• Additional noise from the DDS increases the jitter to 100 fs RMS jitter (10 Hz to 5 MHz).

• We anticipate that the noise for the whole system will be < 1 ps RMS jitter (10 Hz to 5 MHz).

Page 21: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - bunch and turn clock generation

• Clocking FPGA logic with the recovered RF signal provides a method to create beam synchronous logic.

• Generating an in-phase bunch clock requires using the regenerated RF signal as a clock.

• Once we have a counter in the RF clock domain, we can coordinate triggers + pulses throughout a WR network.

• RF time.

ACCUMULATOR

FTW

LUT

DAC14b

FPGA

+

RFtime Counter

WR Clock Doman

RF Clock Domain

Div m

Bunch Clock

+

Page 22: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - What is the BOBR - main functions

• The BOBR card produces local copies of the bunch and turn clocks.

• Distribute triggers and produces pulses that are beam synchronous.

• Distributes LHC telegram information.

BST-Master BOBR TTC

LHC-

B2

L

HC-

B1

SPS

Page 23: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - time of flight compensation

+

ACCUMULATOR

FTW

LUT

DAC+

LUT

Valu

esAd

dres

s of

LUT

• Protons and particularly ions in the SPS accelerator are subject to changes in their Time of Flight (ToF).

• Correcting the RF (and bunch clock) for ToF should be straightforward as it can be expressed as a phase offset to be applied to the output of the NCO.

Page 24: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - SPS LLRF is tricky

• SPS RF is dynamic. This makes it difficult to utilise simple NCO driving a DAC approach.

• The BE-RF have developed an approach using a multi-level NCO that compensates for:

• Slip-stacking for ion fills.

• Frequency Shift Keying (FSK).

Page 25: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - TTC issues

• TTC network subject to diurnal and seasonal changes in signal delay.

• TTCrx PLL cannot operate with FSK that is utilised in SPS ion fills. PLLs unlock, SPS BST uses average of FSK RF signals.

• Ions experience large accelerations in the SPS and their time of flight is not compensated.

All of the above results in the bunch clock, and the bunch, no longer being synchronous with each other. This makes beam monitoring (position and loss) difficult.

Page 26: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

RFtime - As a Coordinate System for Synchrotrons Beam Synchronous Timing - RF Synchronous Events, Pulses and Triggers

Flat Top

Flat Bottom

Ramp

Time

Freq

uenc

y

The input RF is not necessarily constant. The frequency has to change as particles are accelerated.

The RF at injection (flat bottom) into the LHC is ~800 Hz (protons) or ~4600 Hz (ions) less than the frequency reached at flat top.

Bunch clock with varying frequency

Bunch clock with FSK

0 1 2 3 4 5

... 69 70 71 72 73 74 75 76

By expressing events and triggers in RFtime, we remove the need to express events in TAI time and avoid complicated calculations for each receiver.

RFtime

RFtime = #turn x RF harmonic + #bunch

Page 27: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - LHC telegram data requirements

Note update frequencies• TURN• CHANGE• 1 Hz - LHC Telegram

GPS Timestamp can be deletedWR provides this.

PLUS BI SPECIFIC DATA

Page 28: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Beam Synchronous Timing - Latency

• Each White Rabbit switch will contribute up to 10 usec of additional latency to the output signal. In some respects the switch appears as a long cable (up to 2 km).

• A reasonable estimate of this latency is assumed to be 100 usec, or approximately 4 turns of the SPS.

• The interaction of a dynamic RF and latency in reproducing that RF will result in a systematic and a random error to the delay to this recovered RF signal.

• Smaller latency is better.

Page 29: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Core technology - Flexible Frequency Synthesis

+

ACCUMULATOR

FTW

LUT

DAC

FPGA

LPF

Page 30: Overview Core technology Available technology Next steps...Beam Synchronous Timing - WR2RF card proposal BST Master TTC network BST Receivers (BOBR) F T W WR2RF Bunch + Turn Clocks

Core technology - Flexible Frequency Synthesis

+

ACCUMULATOR

FTW

LUT

DAC

FPGA

LPF

BPFPLL