Enabling Terabit Per Second Switch Linecard Design Through Chip/Package/PCB Co-design

Qinghua Bill Chen #1, Jianmin Zhang #2, Kelvin Qiu#3, Darja Padilla#4, Zhiping Yang #5,

Antonio C. Scogna ψ6, Jun Fan*7 #CISCO SYSTEMS, INC

170 West Tasman Dr., San Jose, CA, USA billchen@cisco.comjianmin@cisco.com kqiu@cisco.com darja@cisco.com zhiping@cisco.com

ψCST of America 492 Old Connecticut Path, Framingham, MA, USA

antonio.ciccomancini@cst.com*Electrical and Computer Engineering, Missouri University of Science and Technology

1870 Miner Circle, Rolla, MO, USA jfan@mst.edu

Abstract—Widespread use of the Web 2.0 Internet

applications such as video streaming and social networking are continuously demanding higher bandwidth network equipment. Electrical designers increasingly face more and more challenges to deliver higher speed products within short development cycle due to design complexity and new multi-GHz signal integrity problems. This paper presents a modeling and simulation methodology through chip/package/PCB (printed circuit board) co-design and co-optimization to enable a terabit per second network switch linecard design. Channel design techniques such as BGA (Ball Grid Array) pin backdrill, via tuning, and low loss interconnects are outlined. Full wave 3D modeling techniques with optimal model segmentation, model cascading and model optimization are discussed. At the end, correlation between lab measurement and simulation in both frequency and time domains are investigated.

I. INTRODUCTION A network switch generally can be classified into a chassis-

based modular switch or a fixed configuration switch. The linecard is an important part of the modular switch and it integrates functions of port interfacing, packet processing and other networking intelligent services. With the emergence of content-rich web 2.0 applications such as wikis, blogs, social-networking and streaming video, the underlining internet infrastructure needs to support higher bandwidth and lower latency. This in turn demands network linecards to scale to higher throughput and higher port density--especially for data center and internet backbone applications. Today, a high performance linecard needs to support terabit per second throughput.

From an electrical design point of view, high speed differential serial interface technology is becoming the predominant technology of choices for data transmission between ICs and/or modules on a linecard. Comparing with

the traditional parallel interface technology, the serial link technology possesses advantages of lower power consumption per Gbps and higher bandwidth density per pin. The serial channel characteristics and its behavior can greatly impact network interface performance such as interface reach distance, interoperability with various modules, bit error rate, and so on. As data-rate increases to multi-Gbps, detailed design and accurate modeling are needed for such type of high performance channels.

The main factors limiting serial channel performance include reflection, transmission line loss and signal crosstalk [1-7]. Signal reflection occurs whenever a signal encounters impedance mismatches along its transmission path. Impedance discontinuities cause signal attenuation, distortion, standing wave, ringing and other effects; all of them result in loss of transmission energy. Channel loss includes copper loss and dielectric loss while dielectric loss becomes a dominant factor at higher frequency. Low loss dielectric material becomes essential as signal frequency increases. Signal crosstalk causes degradation to serial channel design due to chip-package-PCB geometry shrinks, and signal frequency increases. These factors need to be properly handled in order to meet the increased data throughput requirement.

This paper presents design, modeling and lab validation of a 25 Gbps serial channel of a next generation terabit per second switch card. The rest of the paper is organized as follows: Section II introduces the 25 Gbps serial channel design highlighting new design technologies such as die-package return loss optimization, via stub reduction, linecard backdrill, and low loss material application. Section III discusses 3D full wave modeling techniques for chip-package-PCB channel. Section IV gives a comparison between full-wave modeling and measurements in frequency domain are

978-1-4244-6307-7/10/$26.00 ©2010 IEEE 585

correlated up to 20 GHz. Section V validates the channel design using statistic eye simulator up to 25 Gbps and everything is summed up in the conclusion.

II. CHANNEL CO-DESIGN A typical end-to-end serial channel includes transmitter

(TX) integrated circuit (IC) and its package, PCB traces, vias, connector, passive components such as DC blocking capacitors, and receiver (RX) IC and its package. Although each segment of the serial channel path has its own unique physical constraints, design requirements and electrical characteristics, the overall goal of high speed design is to ensure small channel reflection, low channel loss and minimal crosstalk from adjacent structures so that the signal-to-noise-ratio (SNR) can be high enough to meet overall system bit error rate (BER) requirement. To meet this goal, a chip/package/PCB co-design methodology is used. In this methodology, all pieces of the chip/package/PCB end to end channel are designed and modeled concurrently with design co-optimization through coordination, iteration, and performance/cost tradeoff.

Fig. 1. Die-package fan-out area

A. Chip-Package Interface Design The interconnect from the serializer-deserializer (SerDes)

TX/RX buffer pads to package substrate are the start and the end of a serial channel as shown in Fig. 1. This area is typically the most congested area due to hundreds or even thousands of I/Os needing to fan out in a small area. Crosstalk is a big concern here due to very tight spacing between signal traces. What makes things even worse is that sometimes a TX differential pair and a RX differential pair are right next to each other due to poor silicon design practice. To reduce crosstalk, ground shielding traces could be the only feasible option unless the signals can be separated more. In order to avoid high frequency resonance and coupling, the shielding traces have to be tied to ground plane through stitching vias.

As illustrated in Fig. 1, the immediate interfacing area uses minimal manufacturing allowed trace width to fan out signals. Once a signal is routed out, its trace width is optimized to meet the overall return loss and insertion loss requirement through 3D full wave solver simulations taking into consideration the chip on-die circuits, package design and portion of PCB structure.

Another area of concern in package design is the return path disruption of high speed signals due to reference plane cut(s) and planar “Swiss cheese” effect due to an excessive number of via anti-pads.

B. PTH Via Tuning and Linecard backdrill In package and PCB design, plated through hole (PTH)

vias are extensively used to connect sub-circuits, traces and components to form completed signal paths. The discontinuity associated with PTH vias becomes a big performance limiting factor as signal speeds increase to multi-GHz ranges. The three main impacts of PTH vias are: impedance mismatch, stub resonance and via crosstalk.

The characteristic impedance of a via is determined by its geometry, anti-pad size, ground via location and the stackup of the PCB. A two step via optimization methodology is applied. The first step uses a physics-based via model [4, 6-9] and equivalent transmission-line via model to generate all via design parameters. The second step involves a 3D field solver to verify the design parameters. This methodology greatly speeds up the via tuning process and at the same time yields best design accuracy.

Fig. 2. Via tuning and via stub effects

For multi-Gbps signaling, a serial channel could fail miserably if via stub resonance is not taken into consideration. As shown in Fig. 2, a pair of differential vias is studied for their stub effects up to 20 GHz in field solver with a 22-layer PCB. The vias connect IC chip pins located on the top layer of the PCB to traces in middle layer of the stackup. From simulation results, it can be observed that the insertion loss is small for all stub lengths varying from 17.05 mil to 60.75 mil at frequencies lower than 8 GHz. However, as frequencies increase, the insertion loss increases dramatically for longer stub cases. Table I shows differential via insertion loss with

586

various stub lengths at frequencies of 1 GHz, 3 GHz, 6 GHz, 12 GHz and 18 GHz. For 60.75 mil stub, the insertion loss is 17.971 dB at 18 GHz which alone could prevent a design from working. Therefore, the PTH via stub length must be controlled for high frequency design.

TABLE I DIFFERENTIAL INSERTION LOSS DUE TO VIA STUB

Signal Layer

Stub Length (mil)

Differential Insertion Loss (dB)

1GHz 3GHz 6GHz 12GHz 18GHz

7 60.75 0.128 0.226 0.575 2.012 17.971 8 50.10 0.125 0.231 0.550 1.586 16.240 11 39.65 0.124 0.232 0.549 1.007 5.982 12 34.00 0.125 0.226 0.482 0.979 3.730 14 22.70 0.122 0.237 0.491 0.952 2.162 15 17.05 0.123 0.238 0.506 0.896 1.973

One obvious way to control a via stub is not to use PTH

via and, instead, use blind via or buried via found in high density interconnect (HDI) board technology. Most chip packages in high-end networking products use HDI, and therefore via stubs of packages can be minimized. Up until now, HDI technologies have not been widely used in network linecard designs due to issues such as higher cost and maximum layer count restrictions. Another way to reduce via stub is by routing signals on deeper routing layers.

However, for a terabit per second linecard design, due to the large numbers of high speed signals, it is impossible to route all high speed signals in deeper layers. A new methodology and manufacturing processes to remove linecard PTH via stub is illustrated in Fig. 3. This new methodology uses high precision drills to remove stubs at the specific location through accurate drill depth control. The via backdrill can be applied to both sides of PCB. Comparing to backplane counterboring [1], this linecard backdrill has more challenges and design consideration. Linecards have much finer design features and higher routing density. PTH vias placed under application specific integrated circuits (ASICs) on linecards are smaller compared with PTH holes on backplanes. This backdrill technology needs to tolerate all PCB manufacturing variations such as mis-registration and drill wander.

Fig. 3. Linecard PCB via backdrill technology

Fig. 4. Mixed mode insertion loss lab measurement

Fig. 5. Mixed mode return loss lab measurement

As a case study for this linecard backdrill technology, a 26-

layer test board was designed and built to verify electrical performance improvement and manufacturability. Good test results on both categories have been achieved. Fig. 4 and Fig. 5 give the differential insert loss S12 and differential return loss S11 measurement results for two 10” long differential signal pairs with identical layout but one with PTH via stub removed and the other with full stub of 72.6 mil. From the results, it can be observed that the IL difference of 0.8 dB at 3 GHz and maximum difference of 43 dB around 11.8 GHz. The return loss difference is around 5 dB across most of the frequency spectrum below 14 GHz as shown in Fig. 5.

C. Low Loss Dielectric Material Signal loss along transmission lines has become

increasingly a concern as frequency increases. Both conductor loss and dielectric loss contribute to the overall loss while dielectric loss become a dominate factor at higher frequency. As of now, FR4 is the main PCB material for most of network products mainly due to its low cost advantage. Fig. 6 shows loss characteristics comparison of 5 different PCB

587

material including FR4 and 4 types of lead free PCB materials (A, B, C and D) as the network industry is undergoing leaded to lead-free product conversion. From the figure, it can be seen that FR4 or its equivalent material has about 20 dB loss at 14 GHz for an 8” well-designed channel; this could be at the pivotal point of not being able to support a 25 Gbps design unless advance equalization technology is provided by TX/RX chips.

2 4 6 8 10 12 14 16 18 20-35

-30

-25

-20

-15

-10

-5

0

Frequency (GHz)

|Sdd

12| (

dB)

Material-A 8-INCHFR4 8-INCHMaterial-B 8-INCHMaterial-C 8-INCHMaterial-D 8-INCH

Fig. 6. Insertion loss comparison of different dielectric material

Fortunately, there are other higher performance lead-free PCB materials available, such as material D with its typical loss about 12 dB for an 8” trace at 20 GHz.

D. Other Design Considerations As bandwidth and frequency increase, other channel design

factors need to be studied and considered including but not limited to: PCB trace surface roughness, tapered cross section and dielectric material glass/fiber weave.

III. CHANNEL MODELING AND CORRELATION An end-to-end serial channel on a 22-layer PCB with two

packages (Pkg1 and Pkg2) and a pair of AC coupling caps, as illustrated in Fig. 7., is used for a chip/package/PCB modeling and measurement correlation study. The target data rate for the channel is 25 Gbps. The fundamental signal frequency is 12.5 GHz, and the 3rd harmonic of the fundamental is 37.5 GHz. In order to obtain accurate channel characterizations, simulations at very high frequency are needed, and excessive model simplifications must be avoided. These modeling requirements push the limits of most full wave 3D modeling tools available today. Due to the complexity of the channel, it is not feasible to model the whole structure together at one time. One commonly used approach is to implement a “divide and conquer” cascading modeling methodology where the complete channel is segmented into sub-models at “appropriate” locations. Each sub-model is characterized independently, and the entire channel model is then cascaded using equivalent-circuit models or S-parameter models from the sub-models in a circuit simulator.

Fig. 7. The end-to-end serial channel.

One basic consideration for dividing the entire model into

sub-models is the segmentation location where TEM waves are expected to be preserved at the dividing boundary on the signal ports. The other consideration is the simulation time, which is related to the mesh size. If two segmentation locations are defined at “M” and “N” respectively, referring to Fig. 7, TEM wave propagation is preserved on the signal ports at the segmentation boundary due to the nature of stripline, and three sub-models are built. The first model is the Pkg1 plus the left portion of the PCB traces segmented at “M”, and the second model is the Pkg2 plus the right portion of the traces divided at “N”. The third model is the rest portion of the traces between “M” and “N”. In this segmentation topology, TEM waves are preserved at the segmentation boundary, which is good for accurate simulation. The drawback, however, is longer simulation time for the first and the second models. This is because the dimensions of the traces and vias inside a package are much smaller than the size of the traces on a PCB. To meet the minimum mesh size requirement for the packages, the extra PCB portions added to the package models will significantly increase the total mesh amount of the model including both package and part of the PCB traces.

Another segmentation topology to divide an entire channel model into sub-models is to choose the boundary between the package BGA pins and the PCB soldering pads. This segmentation method separates the entire channel model into three sub-models as well. The advantage of this segmentation method is easy to work on, and is to save simulation time. The drawback is the TEM waves may not be preserved on the signal ports at the segmentation boundary. Simulation accuracy may be degraded. For the channel modeling studied herein, the dividing locations are defined at the BGA-PCB pin interfaces. The reason of using this segmentation approach is the simulation results can be obtained quickly while the modeling accuracy is acceptable for the application.

The package modeling started with importing package design in MCM file format with CST Link within the Cadence APD tool without geometrical simplifications. A rectangular shape portion and an irregular shape portion including the channel under study were then extracted for Pkg2 and Pkg1, respectively. The extracted models for the two packages and

Pkg1 Pkg2

AC CAP

M

N

588

the signal traces from the PCB are shown in Fig. 8. Since the full-wave modeling procedure for the three sub-models are similar, the simulation is only detailed for the Pkg1 model below for simplicity.

Fig. 8. Sub-models extracted from PCB and package files.

Fig. 9. Details of the Pkg1 model.

Fig. 10. Simulated single-ended S-parameters for Pkg1 model.

Two zero-thick virtual metals were built for better signal launch at the excitation ports in full-wave modeling, which could be easily realized by assigning the metal sheet as a PEC (Perfect Electrical Conductor) wall. The red solid line box and

the dashed blue line box were the PEC walls assigned at the BGA side and the die side, respectively, referring to Fig. 9. The virtual PEC wall at the BGA side was used to connect four surrounding ground BGA pins, while it connected to the C4 pads on top of the package at die side. Four face ports were defined on the two signal BGA pins and two C4 signal pads. A FIT time-domain solver and a causal constant loss tangent approximation with a second order Debye model were used in the full-wave modeling [10, 11]. Simulated single-ended S-parameters including insertion loss and return loss are shown in Fig. 10. It can be seen that the simulated insertion loss meets the reciprocity, and the return loss looking into die side and BGA side are different. The observations above indicate that the simulated results are reasonable.

S-parameters for the remaining two models are got in the similar way. The entire channel S-parameters are then obtained by cascading the three S-parameter blocks in a circuit simulator.

The measurement for the entire channel was done on a probe station with four articulating arms to provide the capability of 360º rotation and three dimensional movements for each probe independently. Four single-ended probes with pitch of 250 µm were used in the measurement. Differential probes could not be used because of the arrangement of the available surrounding reference pads/pins. The S-parameters were measured from 0.1 GHz to 40 GHz.

The comparison of single-ended insertion loss between the measurement and the simulation is shown in Fig. 11. It is observed that the simulated insertion loss catches the resonant frequencies from measurement around 3, 11, 18, 24 and 27 GHz, and matches the magnitude well. The overall correlation with the measured results and the loss prediction is good.

0 5 10 15 20 25 30 35 40-90

-80

-70

-60

-50

-40

-30

-20

-10

0

Inse

rtion

Los

s (dB

)

Frequency (GHz)

MeasuredSimulated

Fig. 11. Comparison between measurement and simulation.

IV. CHANNEL PERFORMANCE SIMULATION The time-domain performance of the full channel can be

predicted based on the cascaded S-parameters and statistic eye simulation tool such as StatEye. The main goal of the simulation is to validate the possibility of enabling 25 Gbps data rate for the entire chip/package/PCB channel designed. The simulation results are also used to study the potential implementation requirements of future SerDes technologies.

Pkg1 extraction Pkg2 extraction

PCB extraction

589

Fig. 12. 25 Gbps statistic eye simulation results (FR4 PCB)

The TX model in the simulations has the transmission amplitude of 800 mV differential peak to peak. Two types of RX models are used in the study. One has the decision feedback equalization (DFE) circuit with 5 tap, and the other has the DFE with 15 taps. Simulated eyes based on an FR4 PCB board for the entire channel are shown in Fig. 12. For a bit error rate (BER) target of 1E-17, about 100 mV vertical eye opening and 0.32 horizontal unit interval (UI) can be seen at the receiver for the 5-tap DFE scheme, while a 180 mV vertical eye opening and 0.43 UI horizontal opening is observed for the 15-tap DFE scheme. The huge eye opening differences of 80 mV and 0.11 UI are contributed by the DE scheme nothing to do with the channel characteristics.

Fig. 13. 25Gbps statistic eye simulation results (low loss PCB)

Low loss PCB material can reduce the insertion loss of the entire channel so that the RX can receive signal with bigger amplitude. This benefit can be observed from simulation results as illustrated in Fig. 13. It is clearly observed that the low loss channel eye margin is much better than that of the FR4 channel at the same data rate with the same simulation configuration. With a 5 tap DFE receiver, the eye height is 100 mV and 0.32 UI wide; and with a 15 tap DFE receiver, the eye becomes 180 mV high and 0.43 UI wide, which should be good enough for the SerDes to operate properly.

V. DISCUSSION AND CONCLUSION This paper presents a chip-package-PCB co-design,

modeling and lab validation methodology of a 25 Gbps serial channel for a next generation terabit per second switch card. As data rate increases to 25 Gbps and beyond, it is critical to optimize the end to end channel design by reducing all

discontinuities, minimizing channel loss and controlling high frequency crosstalk along the signal path. Special attention should be paid to ensure the IC die to package interface, PTH via stub length, and via impedance to meet the design target. Linecard backdrill technology is introduced and proven effective to improve channel performance dramatically at higher frequency. A combination of advanced TX/RX equalization technique and low loss material is essential for high bandwidth serial communication design. Finally, modeling techniques are discussed in the paper and good correlation results between simulation and measurement are obtained, which demonstrates that correct modeling and simulation can predict end-to-end serial channel performance over wide frequency spectrum accurately.

REFERENCES

[1] A. C. Scogna, J. Zhang, K. Qiu, Q. B. Chen, “Modeling issues and possible solutions in the design of high speed systems with signals at 20Gb/s,” DesignCon 2008, Santa Clara, CA, Feb. 4-7, 2008.

[2] J. Zhang, J. L. Drewniak, D. J. Pommerenke, Z. Yang, W. Cheng, J. Fisher, S. Camerlo, B. Archambeault, “Influence of an extended stub at connector ports on signal launches and TRL de-embedding,” Proc. of the 2006 IEEE International Symposium on Electromagnetic Compatibility, Portland, OR, Aug. 2006, vol. 1, pp. 172-177.

[3] J. Zhang, D. J. Pommerenke, J. L. Drewniak, R. E. DuBroff, Z. Yang, W. Cheng, J. Fisher, S. Camerlo, “A hybrid approach to decrease port influence in transmission line characterization,” Proc. of the 2005 IEEE International Symposium on Electromagnetic Compatibility, Chicago, IL, Aug. 2005, vol. 3, pp. 684-689.

[4] J. Zhang, B. Chen, J. L. Drewniak, A. Orlandi, “Physics-Based Via Model Development and Verification,” Proc. of the 2010 IEEE Asia -Pacific Symposium on Electromagnetic Compatibility, Beijing, China, 12 - 16 April, 2010

[5] H. Chen, Q. Li, L. Tsang, C.C. Huang, and V. Jandhyala. Analysis of a large number of vias and differential signaling in multilayered structures. IEEE Transactions on Microwave Theory and Techniques, 51:818-829, March 2003.

[6] Y. Zhang, J. Fan, G. Selli, M. Cocchini, D. P. Francesco, “Analytical Evaluation of Via-Plate Capacitance for Multilayer Printed Circuit Boards and Packages,” IEEE Trans. Microwave Theory Tech., vol. 56, no. 9, pp. 2118-2128, Sep. 2008.

[7] H. Wang, W. Cheng, J. Zhang, J. Fisher, L. Zhu, J.L. Drewniak, and J. Fan, “Investigation of mixed-mode input impedance of multi-layer differential vias for impedance matching with traces,” Proc. of the 2009 IEEE International Symposium on Electromagnetic Compatibility, Austin, TX, 17-21 Aug. 2009, pp. 203-207.

[8] A. Ciccomancini Scogna, "Signal Integrity Analysis of a 26 Layers Board with Emphasis on the Effect of Non-Functional Pads", on Proc. of IEEE Int. Symposium on EMC, Detroit, Aug 2008, USA.

[9] M. Cocchini, W. Cheng, J. Zhang, J. Fisher, J. Fan, J.L. Drewniak, Y. Zhang, “Differential vias transition modeling in a multilayer printed circuit board,” Proc. of the 2008 IEEE International Symposium on Electromagnetic Compatibility, Detroit, MI, 18-22 Aug. 2008, pp. 1-7.

[10] J. Zhang, M. Y. Koledintseva, J. L. Drewniak, D. J. Pommerenke, R. E. DubRoff, Z. Yang, W. Chen, K. N. Rozanov, G. Antonini, and A. Orlandi, “Reconstruction of dispersive dielectric properties for PCB substrates using a genetic algorithm,” IEEE Trans. Electromagn. Compat., vol. 50, no. 3, pp. 704-714, Aug. 2008.

[11] J. Zhang, J. L. Drewniak, D. J. Pommerenke, R. E. DuBroff, Z. Yang, W. Cheng, J. Fisher, S. Camerlo, “Signal link-path characterization up to 20 GHz based on a stripline structure,” Proc. of the 2006 IEEE International Symposium on Electromagnetic Compatibility, Portland, OR, Aug. 2006, vol. 2, pp. 356-361.

590