1580 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 36, NO. 5, OCTOBER 1989

Trigger Input Output Level Rate Rate

1 45 kHz < 500 Hz

REAL-TIME DATA REDUCTION USING A PVAX PROCESSOR FARM

Based on

ITC, HCal, ECal

Beat Jost CERN/EP

CH-1211 Geneva 23 Switzerland

2 3

Abstract

The applicability of a coiiiiiiercial micro-processor farm to reduce the large data voluiiie of a High-Energy physics experiment has been investigated. The system has been iiiipleiiiented and is now being used in the DAQ system of the ALEPH experiment a t CERN and first experiences will be reported. This talk presents the results of a joint research project between CERN and Digital Equipment Corporation.

- 5 500 Hs - 10 Hz T P C Trigger pads

Full events - 10 HZ - 1 HZ

Introduction

The detectors of modern high-energy physics experi- ments consist of typically several 100000 readout channels producing several 100 MBytes of raw data per second. A p plying several levels of hardware trigger circuits and zero suppression of analog channels reduces the data rate to a few MBytes per second. Since this rate still exceeds

, the recording capabilities and the reconstruction capacity available, further reduction is necessary. This last reduc- tion is usually performed by software algorithms running in dedicated processors. Software is used because an imple- mentation in hardware is both too costly and too inflexible.

In this talk we will report on the inipleiiientation of the software trigger for the ALEPH experiment' currently un- der construction in the tunnel of the LEP e+e- collider a t CERN. The detector consists of several subdetectors (Time Projection Chamber (TPC), Hadron Calorimeter (HCal), Electroiiiagnetic Calorimeter (ECal) etc.) with altogether approxiiiiately 500,000 readout channels. After the sup- pression of analog zeroes, a typical event has a size of about 100 kBytes.

The ALEPH experiment has a trigger system consisting of three levels (see also Table 1).

Levels 1 and 2 are hardware triggers whereas the third level is a software selection. The beam crossing rate of the LEP accelerator is 45 kHz. The output rate of the Level 1 trigger should be below 500 Hs. This trigger is based iiiainly on calorimetry: HCal and ECal plus information froiii the Inner Tracking Chamber (ITC). The Level 2 trig- ger uses inforiiiation froiii the trigger pads of the TPC. Its

output rate is estimated to be of the order of 10 Hz but could well be a factor of 2 to 3 different.

In order to reiiiove background triggers and to cope with the recording capacities it is necessary to further reduce the event rate. The task of the Event-Processor (EP) is to perform the reduction to the real physics rate which is calculated to be about 1 Hz. The application software run- ning in the EP partly reconstructs the events to confirm the decision of the previous two trigger levels. It iiiakes use of the full inforination contained in the coiiipletely asseiiibled data.

We have adopted the concept of a processor fariii for the EP. The idea is to give each processor one full event to treat independently froiii the other processors. This is different froiii a real multi-processor system where many processors would work on a single event. Using a processor farm is a valid approach since the events arriving from the detector are independent and need not be resynchronized after the processing stage.

We will first describe the requirements iiiiposed on the system which led to the choice of a coiiiiiiercial product. Next, the hardware used will be presented. A large part of the talk will be devoted to the software aspects of the system and finally we will present soiiie performance mea- sureiiients of the system software.

Requireiiients Since we do not yet know how high the rate of back-

ground triggers will be and what the background will con- sist of, the system must obviously be very flexible from the software point of view. Equally important, however, is its expansibility with respect to the CPU power avail-

i~~~ a generel of the ALEPH experiment its DAQ able. Since the software trigger is located in the main data stream, the system must be very reliable to avoid unneces- system see e.g. [I].

1581

sary dead-time. We were therefore studying the possibility of using a coiiiiiiercial product for the EP. The main crite- ria were

good technical support for the hardware

good software support and maintenance even across releases of the host operating system

excellent prograiii development and debugging tools

identical code and results for the application software whether running on the host or on the E P

input data rate 2 1 MBytes/second

reasonable price

Hard war e To meet the requirenients we have chosen to use single-

board coiiiputers (KA800) froiii DEC iiiounted directly on the VAXBI bus of our main DAQ machine (VAX8700) (see Figure 1).

These processors are based on the rtVAX chip which offers the saiiie CPU power as a pVAX I1 with the restric- tion that they cannot run the VMS operating system. The KA800 has 1 MByte of local RAM on board with the pos- sibility of expanding it to 11 MBytes in increments of 2 MBytes. The local memory is connected to the CPU via a private bus. It is therefore not necessary to access the VAXBI bus for references to the local memory. However, the local RAM is accessible from all devices (interfaces, other KA800s) connected to the same BI bus. On the other hand a KA800 processor can access all iiieniory connected to the saiiie VAXBI bus it is located on. It can even access the iiieiiiory of the iiiain machine, which turns out to be a valuable feature since it allows all processors simultaneous access to rarely used constants and configuration parame- ters (see later). The KA800 has in addition a datamover capability iiiipleiiiented in hardware. This utility is used to transfer data from one KA800 to another KA800 or to the host a t rates up to 2.5 MBytes per second.

The expansibility of the system is guaranteed by the fact that up to 3 additional VAXBI busses can be connected to a VAX 8700 which (fully filled) adds up to a total CPU power available of about 30 VAX 780 equivalents.

Software

Software Developiiient Environment As nientioned earlier the KA800 processors cannot run

the VMS operating system. The system software has there- fore to be based on the VAXELN kernel routines. VAX- ELN has the advantage of being easy to use and optiiiiised for speed. In addition it uses little iiieiiiory (-90 kBytes). It consists of a set of services for communication, synchro- nization, device drivers and process iiianageinent. The nice feature of VAXELN is that all software can be written in high-level languages. This holds even for device drivers

and other code running in kernel mode. At the iiioiiient (Extended)Pascal, C, Ada and FORTRAN are supported. The application software can be written in any language supported by VAX/VMS (in our case it is written in FOR- TRAN) provided that no VMS specific code is used (e.g. $QIO).

All software is written, coiiipiled and linked on a VAX coiiiputer running the VMS operating system. This im- mediately gives access to all the existing tools for software development like language-sensitive editor, module man- agement systeiii, code management system, etc.

The debugging and testing of the system software takes place, of course, on a KA800 or another processor running under VAXELN. There exists a reiiiote full screen sym- bolic debugger which allows even the debugging of device drivers and other code running in kernel mode. This is an extremely useful feature to develop device driver protocols.

Typically the debugging and optimizing of the applica- tion software can take place under VMS, allowing use of tools such as the debugger, PCA, etc. Since the KA800 confornis to the VAX architecture, the developed code does not need to be recompiled. There are no special compil- ers or linkers. Only relinking is necessary since the system routines are different. This guarantees that the results ob- tained are bit-by-bit identical whether a prograiii runs un- der VAX/VMS or VAXELN. This makes the coiuparison and verification of results very easy. In addition access to files on the host is provided, which allows log files to be written for later inspection.

DRB32 Driver The processors running the filter tasks receive their

events from the data acquisition system through a DRB32 parallel interface from DEC. The DRB32 is a high speed 32-bit parallel interface between a VAXBI bus and a user device. It features

0 3'2 bit bi-directional half-duplex data bus

0 two 8 bit uni-directional control busses

transfer rate up to 5 MBytes per second

1919 mapping registers for scatter-gather transfers thus allowing data transfers of up to 950 kBytes with- out processor intervention

dual buffering capabilities

To allow the different processors to receive events, a spe- cial purpose driver for this interface has been written. The driver handles the protocol with the ALEPH event builder CPU2 and iiiipleiiients dispatching criteria so that each processor only receives events it is interested in. The driver runs in a dedicated KA800 processor which acts as a iiias- ter. This allows us to circumvent synchronization probleiiis

'A 68020 CPU residing in a FASTBUS-crate which assembles the data from the subdetectom to a full event. See Ref. (21.

1582

Central Fastbus Readout

. . .

Figure 1: Scheiiiatic view of the hardware configuration of the iiiain DAQ VAX with the Event Processor. The gray shaded areas represent the basic coinponents of the system. The unshaded areas show possible extensions. Note that the nuniber of KA800s is liiiiited to 10 processors per BI (for 3 Mbytes of memory per KA800) plus one KA800 with 1 MByte of nieiiiory due to the liiiiited nuiiiber of BI slots available.

which would arise from the fact that there is only one inter- face but iiiany processors which could access its registers siniult aneously.

In order to receive an event the filtering processors send a request to the master specifying the details of the buffer to receive the event (”address”, size) as well as the requireiiient the event has to meet (event-type, trigger mask). Once the main event builder sends a readout re- quest matching the specified requireiiients, the driver loads the iiiapping registers of the DRB32 in such a way that the data flow directly into the requesting processor’s memory3. No nieiiiory to iiieiiiory data copying is necessary. After the transfer has finished, the filtering processor gets inforiiied about the status of the transfer, the event length, etc. The driver is then ready for the next event. The system al- lows for multi-buffering in the filtering processors. At the nionient we foresee three buffers of 300 kBytes each. This allows us to smooth bursts of events from the main Event- Builder as well as shortcomings in sending good events to the iliain machine. These can arise when the main machine is unable to take new events because it is busy with other tasks or no buffer space for a new event is available. If no buffer is present, the filtering processors are idle and can, in principle, do other work. This is, however, not foreseen a t the monient due to the aiiiount of iiieinory available.

the interfacing between FASTBUS and VAXBI based ma- chines, see [3].

Process Architecture in Filterinn Processors

several processes, each having a special task. They are The software in the filtering processors is organised in

a main process used to coiiiniunicate with the iliain machine

a request sender process to queue event requests to the master

a receiver process retrieving event buffers containing raw events

a filter process that does the event selection

a process sending the filtered events to the main iiia- chine

a process that updates status inforniation

A scheiiiatic view of the process architecture is shown in Figure 2 together with the coniniunication paths.

The different processes coiiiniunicate with each other via event-flags and queues: An event is passed from one process to the next by putting it in a queue and signalling the receiving process that something is ready.

The actual event selection takes place in the filter pro- cess. The algorithm is iinpleiiiented in a subroutine called by the system software. This subroutine has to return a

1583

Figure 2: Scheiiiatic picture of the process architecture in a filtering processor (gray underlaid area). The thick ar- rows represent the flow of the buffer descriptors froin the event reader via the Master KA800 to the event receiver and the filtering process. If the event is accepted the buffer is passed to the process sending it to the iiiain VAX oth- erwise it goes directly to the reader (dashed arrow) to be overwritten by the next event. At the iiioment we fore- see three buffers per processor. The iiiain process accepts steering coiiiiiiands from the iiiain VAX and the status u p dater writes status inforiiiation into a shared iiieiiiory re- gion in the VAX 8700 (represented by thin arrows).

status code which determines whether or not the event is sent to the iuain iiiachine for further processing and record- ing.

The distribution of rarely accessed constants to the fil- tering processors takes place via shared iiieiiiory located in the VAX 8700. Upon reception of a message each proces- sor iiiaps over this iiieiiiory and copies the contents into the local iiieiiiory. Hence it is not necessary to send the con- stants to each processor in a large message. With the same iiiechanisiii we transfer status information to the main iiia- chine for displaying: Each processor has assigned a region of iiieiiiory to put the status information and the status display program maps over this region to read the data.

Integration in DAQ Svsteiii The integration into the DAQ systeiii takes place via a

set of interface tasks running on the iliain VAX. These are

0 a control task to interface to the run controller and the operator which also loads the processors with the appropriate systems

0 an event receiver to retrieve the good events and fill theiii into a buffer

0 a status display prograiii that updates status informa- tion

The control task and the event receiver use the VAXRTA coiiiniunication tools provided by DEC. These are general

purpose routines for sending and receiving iiiessages to and froiii the KA800 processors. The routines are an iiiipleiiien- tation of the VAXELN coiiiiiiunication iiiechanisiii under the VMS operating system. It is therefore very easy to apply them once one knows the ELN world.

The status display task accesses iiieiiiory in the VAX 8700 shared with the KA800s. The KA800 status updater writes into this iiieiiiory region and the display prograiii reads the inforiiiation and updates the status screens. This iiiethod has the advantage of keeping the number of nies- sages in the systeiii low. Since no synchronization is needed for this purpose this is a very efficient way to pass infor- iuation between the KA800s and the main machine.

Perforiiiance There are three performance figures of interest in our

application

0 the iiiaximum input rate froin the DAQ systeiii

0 the iiiaxiiiiuiii output rate froiii the Event Processor

0 the processing time of the filtering algorithiii

The input rate into the systeiii is given by the proto- col overhead between the Event-Builder and the Master KA800 (-9iiis per event) and by the speed of the KA800 memory (-2.5 MBytes/s). The rates measured are shown in Table 2

Event Events Sise Transferred

4 Bytes 0.440 4 kBytes

100 kBvtes 2000

Table 2: Perforiiiance of the event readout (preliminary). The iiiaximuiii transfer speed is limited by the KA8OO hard- ware to 2.5 MBytes/sec.

As can be seen by this table the input rate requested is satisfied. It should be noted that this rate can be doubled if two interfaces are used instead of one. Since the total band- width of the VAXBI-bus is specified to be 10 MBytes/s this should not iiiipose any problems. If the liiiiitation is iiii- posed by the maximuin hardware speed, several interfaces could be controlled by the saiiie master. If, however, the protocol overhead is the limiting factor, one master proces- sors would have to be used per interface. Nevertheless the events transferred have to be analysed which is probably the limiting factor in this case.

The output rate from the KA8OO is supposed to be only about 10 percent of the input rate. The requireiiients on this part are therefore not very high. The rates measured between a VAX 8700 and a KA800 are shown in Table 3.

The output rate froiii the EP is by far sufficient for our purposes.

1584

Message Host CPU size Usage

[ k B y t es] [ %] 0.5 15 5 20

50 35 100 40

‘Transfer rate

[ k B y t es / sec] 125 900

2500 2500

Transfer

[ Mess/sec]

Table 3: Results of perforiiiance iiieasureiiients for message transfers between the host (VAX 8700) and one KA800. The iiiaxiiiiuiii transfer speed is liiiiited by the KA800 hard- ware to 2.5 MBytes/sec.

The processing t h e of the filtering algorithiii is of course very hard to estiiiiate since we do not yet know what the real background triggers look like. The present algorithm takes less than 500 iiis to accept a Monte-Carlo event. This allows us to process about 20 events per second with the present hardware configuration (1 Master and 10 ~ l a v e s ) ~ .

Suiiiiiiary We have described the iiiipleiiientation of the software

trigger for the ALEPH experiiiient a t CERN. We use a microprocessor fariii based on the rtVAX chip. The hard- ware and part of the software is coiiiiiiercially available and therefore fully maintained. The perforiiiance obtained is coiisidered to be sufficient for our application. The startup of LEP will show whether this holds.

Acknowledgeiiients Many people have contributed in the course of this

work. The project was initiated by W. von Riiden and G. Liitjens coordinated the application software develop- iiient. A. Miotto and V. Mertens helped to design the event readout. I would also like to thank R. Vignoni and his team from DEC Engineering for supplying us with the necessary hard- ware and software during the project. Last, but not least, I alii indebted to E. Gerelle and J . Atnold for their support and collaboration in the fraiiie- work of the CERN-Digital Joint Project.

References

W. von Ruden, ”The ALEPH Data Acquisition Sys- teiii” , these proceedings.

A. Marchioro et al., ”The ALEPH Event Builder - A Multi-User FASTBUS Master”, IEEE Trans. Nucl. Science 35, vol 1, Feb 1988, pp 316-320.

B. Jost et al., ”A Multi-Function FASTBUS Front- End for VAXBI Coiiiputers”, presented a t this confer- ence.

‘Due to the fact that the KA800 does not implement cache mem- ory it is possible that the execution of the application program is slowed down during a DMA transfer. At the moment, however, we have no estimate of this effect.