High frame rate video image of a milk drop taken at 300 fps. Illumination source was a 300 Hz strobe with a 20 μs duration. The Image was taken with a Vision Research Phantom v 2 . 0 ™ camera using a Sarnoff VCCD512 high frame rate CCD Imager.

Snapshot: Hughes discusses high-speed

imagers, their economic

and technological impact as well as

requirements

for high- speed applications.

By Gary Hughes

Motion analysis, defined as the capture, dis­play, recording, and analysis of moving images from high-speed events, is under­

going a transition from a film-based medium to an all-electronic medium. High-speed film cameras, which have been used for many years in the recording of high-speed events such as automobile crash testing and aircraft flight

Optics & Photonics News/April 1995 21

1047

-693

8/95

/4/0

021/

06-$

06.0

0 ©

Opt

ical

Soci

ety

of Am

eric

a

Glossary Bus: A parallel set of conductors in a cable or printed circuit board that carries a digital data stream in a parallel format.

RS-170 format: A black and white video transmission standard specifying 525 video lines per TV frame in a 2:1 interlace format, at 30 frames per second. Used in the U.S., Japan, and other countries.

Scan conversion: The process of converting a video data stream of one spatial res­olution and frame rate to another resolution and frame rate.

Transfer smear: An image artifact that appears as a vertical bright column sur­rounding an intense highlight in the scene.

dynamics, are now competing with electronic charge-coupled devices (CCDs) and metal oxide semiconductors (MOS) imaging cameras. Although photographic film will contin­ue to be used for many years, the time is rapidly approaching when high-speed film cameras will be found in only a few spe­cialty applications. The unre­lenting drive towards lower cost electronic memory of all sorts, and the continual advancement in high-speed imagers will soon provide per­formance, cost, and conve­nience advantages that will make high frame rate CCD cameras the dominant choice for high-speed imaging.

The battle for supremacy between moving picture film and electronic video cameras was won long ago in the consumer marketplace by low-cost MOS and CCD cameras. The combination of modest resolution and frame rates, and the develop­ment of an enormous worldwide market helped make this transition happen at a extremely rapid rate. In the high-speed motion analysis area however, technology developments and market pull have not enabled the same rate of progress.

Film cameras have many advantages for high-speed photography. Because of the very high spatial informa­tion density of film, high-speed film transport technol­ogy allows the capture of large numbers of high density images with relatively low cost, compact cameras. On the other hand, the handling of film and processing of this data leaves a lot to be desired. Film canisters must be loaded and unloaded from cameras (sometimes in hostile environments or remote locations), then the film must be chemically processed, viewed, and physi­cally measured or digitized to enable precise analysis of the recorded motion. This process is time consuming and uses large volumes of toxic chemicals. Since there is

no instant replay, whole experiments must sometimes be repeated at great expense.

While these disadvantages create an opportunity for high-speed video, the challenges to implement this technology are daunting. To achieve the spatial resolu­tion of today's high-speed film cameras requires CCD formats well in excess of 1000 pixels on a side. To achieve comparable speeds requires frame rates of 150 -1000 frames per second (fps). A camera capable of 1024 X 1024 pixels at 1000 fps will produce over a gigabyte per second of video data without data compression.

This creates formidable chal­lenges not only in image cap­ture, but also in data trans­mission, display, manipula­tion, and storage.

CCD Fundamentals To provide a better under­standing of these problems, it is first necessary to under­stand some CCD fundamen­tals. Figure 1 shows a schematic representation of several types of CCD imagers designed for high frame rate (HFR) imaging. The basic strategy behind all of these approaches is to achieve high frame rate operation by pro­viding some form of parallel readout architecture to mini­mize amplifier bandwidth and noise.

Figure la shows a schematic representation of a split-frame transfer (FT)

CCD imager,1 Figure lb is a split-interline transfer (IT) CCD imager,2 and Figure 1c is one type of high frame rate MOS/CCD imaging array. The cross-hatched areas represent the photo detection areas of the chip, and the regions with arrows the charge transfer areas. With the FT imager, the photo detection and transfer region ("A" register) are one and the same. During the charge collec­tion or integration period, photons create electron-hole pairs in the silicon and the electrons are collected and temporarily stored in the CCD potential wells of the A-register. At the end of this integration period (1/30 sec for RS-170 video), (for a definition of this and other terms see glossary this page) these charges—an elec­tronic spatial image of the video scene—are transferred to the B-registers. This happens during the vertical retrace time of the TV display tube. The empty A-regis­ter is immediately used to integrate up the charge repre­senting the second video field while the stored field in each B-register is transferred one line at a time (during the horizontal retrace of the display) into the two

Continued on page 24

Figure 1. (a) FT CCD, (b) IT CCD, and (c) CCD/MOS imager architectures.

22 Optics & Photonics News/April 1995

Continued from page 22 C-registers. Here the charge packets are transferred to a number of parallel on-chip amplifiers (8-16 per edge) where they are converted to a voltage representing the video signal. In the IT imager, photo detection is done with separate photodiodes. These are reset at the begin­ning of each video field and the image pattern creates a charge pattern in these diodes (one per pixel). At the end of the integration time, the photo charge in the diodes are simultaneously transferred to the B-register columns and the reset and integration of the next video field begins. This charge is stored in the B-registers until the end of each video line when the whole pattern is shifted one line at a time into the C-registers (during the horizontal retrace time) and read out as in the FT imager.

In the MOS/CCD array, the photosites are addressed in blocks. For example, each block may consist of 32 lines of detectors that are either CCD wells or photodiodes that can be addressed by an MOS shift register. After a block is addressed, the 32 lines of detectors are read out in parallel by a column shift register, and then multi­plexed to the 32 parallel readout ports as shown, pro­viding another means of achieving parallel readout.3,4

The FT imager, if it is back illuminated, has the advantage of the highest overall quantum efficiency of all of the approaches, since it has 100% optical fill factor. As Figure 2 shows, the back-illuminated FT CCD has a marked advantage over a front-illuminated CCD or MOS array with respect to sensitivity in the blue and green region of the spec­trum. This is particularly important in HFR imaging because of the short integra­tion times and resulting low exposure levels. The FT approach is also a simpler struc­ture, so it can be manufactured in very large sizes with good yields. On the other hand, it has a larger transfer smear component, that at high frame rates must be removed with a high-speed shutter. The IT and MOS

imager fill factor is lower, and this can be improved by fabricating micro lenses on each pixel to focus the light, although approaching 100% fill factor without pixel to pixel optical crosstalk is difficult. Both are also more complex structures, and thus for a given chip size will

generally have a lower yield (% good chips per wafer) than a FT imager. However, since neither array needs separate B-registers (they are interleaved with the pix­els), they have a smaller chip area and more sites can be placed on a silicon wafer.

High-speed challenges The major issues of con­cern in the development of HFR cameras involve image size, frame rate, sensitivity, dynamic range, and the resulting impact on data transmission, storage, and retrieval. The multiplica­tion of the four significant performance parameters

(the two image dimensions, frame rate, and dynamic range) produces a tremendous data throughput burden as these parameters are pushed to ever increasing val­ues. The near-term solution for most applications is to use brute force and deal with all of the data produced. However, in the long run, the more cost-effective solu­

tion may be to use video and data compression to limit the quantity of data to more manageble numbers.

To illustrate how these issues interact it is useful to consider an example of a leading edge HFR camera. To keep within size, weight, and power budgets, most HFR CCD cameras are con­structed with at least two subassemblies as shown in Figure 3. The camera head typically contains the lens, CCD imager, drive electron­ics, and A/D conversion cir­cuitry. The camera head is connected to the video processor by a data link, either copper or optical

fiber. The video processor holds the digital circuitry required to reformat the image in the proper spatial rela­tionship, scan convert it for display, and provide a means of mass storage of the image data stream. In many appli­cations the camera is in a hostile environment and all of

Figure 2. Quantum efficiency of typical thinned back-illuminated CCD array1 (no anti-reflection coat­ing), and front-illuminated CCD array.2

Figure 3. Block diagram of a 1024 x 1024 pixel 210 fps HFR CCD camera.

24 Optics & Photonics News/April 1995

the subassemblies must be able to provide immunity from external electrical noise.

In the above example, a 1024 X 1024 pixel FT CCD imager is used in a camera that delivers a 12-bit digital data stream from the CCD at 210 fps. To read out the video signal from the CCD array at this frame rate requires 32 parallel output ports on the CCD array clocked at ~16 Mhz. Each port has its own on-chip correlated double sampling circuitry to allow low noise readout. After readout, these 32 channels of analog video must be digitized to 12 bits and transmitted to a video processor subassembly some dis­tance away. To transmit this data in a physically manageable data link (a differential 12 bit bus would require almost 800 wires), the data is serial­ized at a higher data rate and trans­mitted via multiple optical fibers to the video processor. This not only makes the physical link more man­ageable, it also provides needed noise immunity in hostile environ­ments.

In the video processor, the seri­alized data must be decoded, the 12-bit image reformatted, and then transmitted over another wideband data link for storage in high-speed, solid-state memory or on digital tape for later analysis. The image can also be decimated to 512 X 512 pixels and scan converted to an RS-170 format for display purposes during setup and experiment mon­itoring.

Although this example is a spe­cific one, the issues discussed are of concern for most high frame rate applications. In general, the major issues/needs that have to be addressed for high-speed applica­tions can be summarized as fol­lows:

Imager/camera head High frame rate Large image format (pixel size

and number) High sensitivity High dynamic range Low noise Color capability

Data link High data rate Noise immunity Link length

Video processor and storage High data rate Video compression Storage capacity

Display Format Scan conversion

Added to this list are the addi­tional concerns about size, weight, power consumption, ruggedness, and of course, cost.

Most of the issues/needs listed interact and can be traded for per­formance in one area at the expense of another. For example, data rate is a primary consideration in these cameras. Reducing the image for­

mat has a major impact on data rate. Limiting dynamic range to 8 bits will reduct the data even fur­ther and may be a good trade in some applications. For very large data throughput, image and data compression may be a cost-effective solution. Adding color capability provides attractive images, but adds complexity, and reduces signal dynamic range.

Table 1 (page 26) lists a number of commercially available digital HFR CCD cameras and several per­formance parameters of interest, particularly image format and data throughput rate, the major complex­ity and cost driver for a complete camera system. Figure 4 shows image size as a function of frame rate for these cameras. The inclusion of data throughput as a parameter illustrates the usefulness of this fig­ure of merit. As the figure shows,

Optics & Photonics News/April 1995 25

several state-of-the-art cameras are pushing the 3,000 Mbit/s frontier of data throughput with vastly different frame rates. With the data parameters shown, some of the trades made in these cameras are obvious. For exam­ple as Figure 4 illustrates, the Kodak Ektapro 4540™ and the Sarnoff SAR 1024 are near 3,000 Mbit/s in data throughput. However the Kodak camera has a 256 X 256 image format at 4500 fps, whereas the Sarnoff camera is at the other end of the spectrum with a 1024 X 1024 pixel format at 210 fps.

Applications Automobile crash testing is one application familiar to everyone, and it is an important part of indus­try-wide automobile man­ufacturing safety programs. A typical auto manufactur­er conducts about a dozen impact simulator tests weekly, with each test involving up to eight crash­es that demonstrate the performance of a particu­lar component under test. For these types of tests the cameras are sometimes mounted right on the test sled itself, and so must withstand the high "G" forces of the crash.

High frame rate video cameras can, and are being, used in a wide variety of important applications outside of the automotive industry. These include the analysis of mechanical dynamics of all kinds, fluid flow and aerosol dispersion problems, the analysis of electrical breakdown and circuit-breaker dynamics, combustion analysis, analysis of the ballistics of munitions, the dynamics of stores separation during airborne ord­nance and supply drops, the development of medical

equipment and procedures such as surgical lasers and angioplasty, and in biomechanical analysis during physical therapy.

Future The future market for cameras of this type will be lim­ited less by the imagination of potential users, than by the functionality, cost, size, and weight of available camera systems. This is a technology in need of low-

cost imagers, electronic memory, and fiber optic data transmission systems, while at the same time one that demands leading edge performance in each area. These are mutually exclusive requirements and will have a major impact on cost.

As the market grows, one would expect the development of products over a broad spectrum of performance and cost. Leading edge camera tech­nology will continue to strive to increase spatial resolution and frame rate at the same time in a 12-bit data format, a virtual

explosion of data handling capability. This is where image and data compression can provide a cost effec­tive solution to these ever increasing demands. At the other end of the spectrum, lower resolution 8-bit cam­eras will become available that will take advantage of lower cost component technology and will allow low cost HFR cameras to be available to a larger number of users. In the years ahead, one can expect that this type of camera will become as commonplace as the optical microscope; just another analysis tool available to sci­

entists, engineers, and technicians everywhere.

References 1. W. Lawler et al., "Performance of high frame-rate back-illuminated CCD imagers," SPIE Proc. 2172, 90 (February 1994). 2. "Image sensors and solid state camer products 1994/1995," EG&G Optoelectronics Catalogue, 194 (HS0512J). 3. J. Bixby, U.S. patent #4,332,752. 4. T.H. Lee and R. Khosla, U.S. patent #4,327,638.

Gary Hughes is head of Imaging Technology at David Sarnoff Research Center, Princeton, NJ.

Figure 4. Image size and data throughput rate for various commercialize digital high-frame rate CCD cameras.

Table I. High frame rate CCD/MOS video cameras with digital output.

26 Optics & Photonics News/April 1995