Table 2 Key aspects of the CP chip (from [3], reproduced with permission; © 1997 IEEE).

Transistor count	7.8 million
Logic	3.8 million
Array	4.0 million
Die size	17.35 mm × 17.30 mm
Power dissipation	37 W @ 2.5 V 300 MHz
System bus frequency	Half of processor
On-chip decoupling cap	102 nF
C4 chip interconnections	1600
Off-chip signal I/O	448
Maximum frequency	400 MHz

Dedicated thin-oxide capacitors totaling 102 nF are provided for on-chip decoupling. This, combined with the inherent nonswitching well-to-substrate and diffusion-to-well capacitances, provides about 200 nF of on-chip decoupling capacitance. Each capacitor has a gated n-FET control device with an external decap-enable pin for leakage current measurements during test. Additionally, each capacitor contains a "built-in" fuse which, in the presence of a large current resulting from oxide defects, results in an open in the first level of chip metallization (M1). The decoupling capacitor cell (Figure 2) is designed to fit under the dataflow wiring tracks. The cell is double-bit-pitch wide (43.2 µ) and 14 tracks tall (25.2 µ). Two of the fourteen horizontal wiring tracks are specifically blocked for the decoupling capacitor wiring so that the capacitor can fit under the wiring. A low-resistance layout of the capacitor cell provides a fast time constant, approximately 85 ps.

Figure 2

Chip floorplan and global design strategy

Each unit consists of many smaller design elements called macros, and within each unit macros are classified as dataflow or control. Dataflow macros are custom-designed and are arranged in bit stacks. Most of the dataflow and control logic was implemented with static CMOS circuits. Dynamic circuits were used only in isolated, performance-critical cases. Approximately 75% of the logic macros were implemented with custom schematics and layouts, and the remaining 25% were synthesized.

Control macros (also designated as random logic macros, or RLMs) are synthesized logic. This logic was implemented using a standard-cell approach in which gates are selected from a common fixed library and are placed and routed with automated tools. In most cases the RLMs also included gate array backfill of spare logic in the unused area within the macro. The gate array's spare logic consists of unwired transistors which can be personalized at the metal-only levels to facilitate quick design changes.

Functional descriptions in VHSIC Hardware Description Language (VHDL) and physical design hierarchies match at the macro level. The physical partitioning of the chip was largely influenced by the logical partition at the macro level. The chip physically comprises at least three levels of hierarchy (chip, unit, and macro), and design implementations were achieved concurrently at each level. Thus, interconnections between units and interconnections between macros were implemented in parallel with the macro layouts. Our concurrent design approach supported both a top-down and bottom-up design methodology, with constraints being passed at each level of the hierarchy. The shape and size of arrays and width of dataflow stacks were designed from the bottom up. The aspect ratio of control macros and placement of control pins were designed from the top down. Design constraints were passed in both directions of the hierarchy in the implementation of the chip power distribution. The power mesh on higher metal levels (M4 and M5) was predefined at the chip level and considered fixed at the unit and macro levels. The power mesh on the M3 level and below was designed from the bottom up, giving the most flexibility to the macros and units. In some cases, such as over the dataflow areas, the M2 power mesh was predefined.

The power distribution supports an average dc voltage drop of 23 mV. The current transients were managed by including additional on-chip decoupling capacitors around large noise sources such as the off-chip drivers, clock buffers, and on-chip drivers with large loads. Since a large amount of switching capacitance occurs in the dataflow stacks, decoupling capacitors were also placed under the wiring tracks with little or no area impact.

Phase-locked loop (PLL)

Figure 3

A primary barometer of PLL performance is jitter. In the CP, the PLL aligns its clock with other chips in the system. Consequently, accumulated phase error (the phase difference between the PLL output and the reference clock) is important. Other critical measurements are cycle-to-cycle jitter and long-term jitter, which reduce the cycle period available for logic operations. In the present design, there are three places where PLL performance can be severely degraded by power supply noise--the constant-current source (I_ref), the current injected into the ICO by the voltage-to-current converter, and the ICO itself. The I_ref supplies the constant current which biases the entire PLL. Any fluctuation in the current provided results in both cycle-to-cycle and long-term jitter. This error is reduced by minimizing the sensitivity to supply voltage (dc shifts and low-frequency noise), and by the use of capacitive coupling (high-frequency noise). The voltage-to-current converter supplies bias current to the oscillator. Noise added to this bias results in both types of jitter. The output of the converter is single-ended and can be sensitive to power-supply shifts. The present design not only minimizes the injected noise, but also utilizes common-mode noise-rejection technique so that the noises are added as common mode and cancel out at the output. The ICO is a differential cascoded current source design which relies on the variation of the input current to control the frequency.

To minimize the noise and add flexibility to the PLL usage, the analog V_DD is generated on-chip with the simple RC filter shown in Figure 4. This approach eliminates the need for an additional power supply and the associated package-coupled noise, and makes the design compatible with the existing power-supply systems. The on-chip coupling noise on the analog V_DD is minimized both by careful placement and wiring and by the low-impedance path to V_DD through the RC filter.

Figure 4

High-frequency design strategy

Standard cell library
A completely full custom implementation was not feasible because of the complexity of the S/390 architecture and our resource and schedule constraints. Thus, two carefully tuned standard cell booksets were developed for use with synthesis. One of these contained fixed schematics and layouts, and the other was parameterizable with capabilities for automatic layout generation [5]. Both bookset designs were limited to relatively simple functions: INVERTERs and BUFFERs; two-, three-, and four-way NANDs and ANDs; two- and three-way NORs and ORs; and simple AOIs, OAIs, AOs, and OAs (2 by 1 and 2 by 2). Many nonbuffered power levels were designed for each book type. These power levels were created both by reducing the width of the FET fingers and by adding FET fingers. Within the fixed bookset, each book type was individually simulated and beta-tuned (ratio of p-FET device width to n-FET device width) to achieve nearly equal rising and falling output delays.

The parameterized bookset offered the most flexibility and highest performance. Multiple beta ratio books were designed for the book types with the highest usage. The beta ratio was changed by both increasing and decreasing the ratio of p-FET width to n-FET width. Thus, rising input delay was improved to speed up the falling output delay, and vice versa. (Synthesis exploited these various beta ratios as it attempted to minimize both falling-input-to-falling-output delay and rising-input-to-rising-output delay for a path.) All of the books in this two-dimensional bookset (the two dimensions being power level and beta ratio) were timing-characterized for synthesis. Timing characterization included input-pin-to-output-pin base delays, and delay sensitivities to input-pin slew and output- pin capacitive load. Synthesis using these carefully tuned booksets yielded schematics which were nearly as optimal as full custom schematics.

Much effort was expended to minimize the internal parasitic resistances and capacitances of each book. Polygate resistances were reduced by placing the input pin between the n-FET and p-FET fingers. Source and drain diffusion capacitances were reduced by sharing as much of the diffusion as possible between adjacent-series-connected p-FETs and n-FETs. The local interconnection level was leveraged for the internal book connections of FET drains and sources.

Full custom circuit design
Each custom circuit designer had full control of circuit schematics and circuit topologies, and this permitted them to exploit fully the delay advantages of transmission gates, AOIs, and OAIs. Transmission gates were used primarily to implement selector functions when the data path was more timing-critical than the select path. Manual logic transforms were applied to the schematics to reduce the number of stages and critical path delays through the schematics. Transforms included buffer insertion and deletion, NAND-NAND-to-AOI conversions, NOR-NOR-to-OAI conversions, and Shannon expansions. The noncritical loads on a timing-critical net were buffered in a manner which reduced the load on the net's driver.

After the circuit schematic optimizations were completed, individual gates were tuned. Gates in the non-timing-critical paths were resized to achieve 10-90% rising and falling slew targets [5]. Next, stage gains in the critical paths were reduced to a value typically between 2 and 3, and equalized throughout the entire path. Stage gain was defined to be the ratio of a gate output load capacitance to its input pin capacitance. The output drivers (gates driving the macro output pins) were properly sized to drive the full lumped load capacitance instead of buffering the signal via multiple inverters to increase drive strength. The preceding gates (i.e., gates driving output drivers) were accordingly sized up to reduce their stage gain. Stage gains were also reduced by increasing the FET widths of the critical gate load on a timing-critical net and/or by lowering the FET widths of the noncritical gate loads on a timing-critical net. Equalizing stage gains was accomplished by increasing and/or decreasing the FET widths in the path until the slews of all of the nets in that path were approximately equal. (Excessive gain on a circuit stage causes an excessive slew on the output net of that stage.) The beta ratios of books were modified to equalize the rising-input-to-rising-output delay and falling-input-to-falling-output delay for the timing-critical paths, as synthesis did for the control functions.

The layout techniques employed during development of the standard cell booksets were also used during full custom circuit layouts. Additionally, to minimize wire capacitance in timing-critical custom paths, circuits were manually placed and wired. In situations where long wires were unavoidable (principally on nets with large fanout), wire RC delay was modeled with various wire widths to find the width which resulted in minimum path delay.

Dynamic circuits
Array circuits such as the L1 cache, L1 directory, TLBs, ROM, traces, and write buffer were designed with self-resetting CMOS (SRCMOS) circuits [4]. The only nonarray macros designed with dynamic circuits were the cache hit logic [6], the dynamic multiplexors used with latches, and the divide and square-root lookup ROM in the FPU. The rest of the macros were designed with static circuits. All known static techniques were applied before considering dynamic circuit solutions.

Timing considerations
The address-generation path was still timing-critical after application of all of the physical placement, wiring, and circuit techniques described above. Fortunately, the L1 cache and ROM access paths had been designed with timing margin; thus, some of this margin was transferred to the address-generation path. This was accomplished by simply adding a programmable clock delay block within the cache and ROM arrays. In other similar situations, where a path was timing-critical and the prior or subsequent cycle path had timing margin, we opted to reposition the latch to rebalance the delays of the back-to-back cycle paths [5].

Many critical paths arose because of excessive delay in the propagation of a signal between distant macros. The microprocessor floorplan was modified to move these macros closer together; even small movements improved timing, since (to first order) wire propagation delay is proportional to the square of wire length. If the wire was still timing-critical, several other solutions were explored. For wires with more than one load, we investigated the feasibility and timing impact of duplicating the net driver and net so that one of the drivers merely drives the timing-critical load. Solutions involving wide or tapered wires were analyzed via AS/X simulations. Widening a wire reduces its resistance at a faster rate than its capacitance increases, thus permitting a reduction in the RC delay of the wire. A tapering solution involved the use of a wider section of wire at the near (driver) end and a narrower section of wire at the far (receiver) end. These solutions further reduced its RC delay. Care was taken to ensure that no polysilicon wires were used on critical wiring nets.

In addition to implementing techniques to reduce cycle times, careful noise analysis was required to ensure that capacitive metal-to-metal noise couplings would not adversely affect the functionality of the circuits at high frequencies. All distant inputs to transfer gate latches were buffered locally with receivers. The parasitic coupling analysis tools identified nets that required grounded metal shields.

Design for reliability
It was challenging to design this high-speed microprocessor to operate functionally in an accelerated burn-in environment. In normal operation, the CP operates in a 2.5-V environment. However, to ensure high quality and reliability, the CP was required to pass high-voltage and temperature stress testing. A dynamic voltage screen (DVS) test consists of writing patterns at 4.25 V, while a 5.0-V bump is applied at the end of each DVS pattern. The CP was designed to be functional and pass I_DDQ leakage currents under these stress conditions. In addition, the CP was also designed to be functional under burn-in conditions at 140°C and 3.75 V.

Clock distribution and latches

Figure 5 Figure 6 Figure 7

Two types of latches are used on the microprocessor outside the arrays: L2-only latch and L1-L2 pair; both are cycle boundary latches (i.e., there are no midcycle or phase latches). The cycle boundary is defined to occur at 'CLKG' falling. Corresponding to the two latch types are two types of clock blocks. The first clock block/latch combination is shown in Figure 8(a). This clock block chops the global clock on the falling edge to create a short pulse 'CLKL' that triggers the latch. By using either a dynamic multiplexor in front of the latch [Figure 8(a)] or a preset static multiplexor [Figure 8(b)], a fast latch delay is achieved. This mux/latch combination interfaces smoothly with the static circuits, yet allows fast delays for multi-input high-fanout registers typical of the dataflow. While 'CLKL' is inactive (high), the dynamic multiplexor is in the precharge state while the latch is holding its state. Similarly, the static multiplexor is preset high (node mux_A is high) while the latch is holding its state. When the 'CLKL' low-going pulse arrives, the latch node 'LAT' begins to discharge. If the multiplexors' data/selects are such that the multiplexors remain in their precharged/preset state, node 'LAT' discharges fully. If, however, the multiplexors evaluate, node 'LAT' is driven high. The n/p transistors in the multiplexors are skewed to favor the transition launched by the arrival of the 'CLKL' pulse. Strength ratios in the preset static multiplexor are limited to 2.5:1 to maintain a reasonable noise margin and to allow adequate time for presetting all gates at the end of the clock pulse. The seven-input preset static multiplexor evaluates in about 225 ps compared to about 400 ps for a standard static multiplexor with the same input capacitance and area. When 'CLKL' is active, the latch is in the transparent mode. Full chip and macro wire RC extractions were carried out to guarantee no early-mode problems.

Figure 8

The second clock block/latch combination is shown in Figure 9. This clock block splits the global clock on the falling edge to create C1/C2 clocks. C1-C2 clock overlap at the cycle boundary is set close to 0 ps. Having a positive overlap would reduce the latch propagation delay, but it would also require more early-mode padding. These latches are used in non-timing-critical dataflow macros and in control macros where all latches are single-input and the speed advantage of an L2-only latch is reduced.

Figure 9

All of the latches used in the design are fully scannable and LSSD-compatible. The largest area penalty for fully scannable latches was in the general-purpose register (GPR) arrays at 25%. The overall chip area penalty is less than 5%. Having an LSSD-compatible design increased productivity during test vector generation and allowed greater than 99.5% dc stuck-at-fault test coverage. The goal of the design was also to achieve high ac transition test coverage, since ac test coverage often suffers from an inability to create appropriate transitions due to latch adjacency in the scan chain. This is most prevalent in dataflow macros where complicated logic functions are being done (e.g., multipliers). To eliminate the latch adjacency problems, nonfunctional scan-only latches were inserted into some macros for every register bit. With this addition, ac transition test coverage greater than 91% was achieved.

Specialized circuitry was added to all clock blocks to allow for edge shifting at the cycle boundary. Taking the clock splitter as an example, we are able to

• Completely un-overlap C1 falling and C2 rising by a large amount. This provides a work-around for potential early-mode problems.
• Delay C1 falling from its nominal value. This allows stressing of early mode and provides a determination of how much margin exists in the design.
• Delay C1 falling and C2 rising together from their nominal values. This allows cycle stealing from the previous cycle.

Figure 10

Microarchitecture optimization and circuit innovations

FXU binary-zero-detect circuit
The execution of instructions in the fixed-point unit involves computing results in the 64-bit binary adder, 32-bit decimal adder, and 64-bit logic unit, and setting the condition codes--all in a single cycle. The critical path was computing the 64-bit binary sum, doing a 64-bit zero-detect on the sum, and using the result to set binary instruction condition codes. A scheme was used where the 64-bit zero-detect was initiated and completed before the binary sum result was actually available [7]. This led to an execution cycle of less than 3 ns.

FPU 120-bit adder
The 120-bit floating-point adder was implemented as a carry-select adder. Care was taken to make sure that the adder would never produce a negative result. This eliminated the step of converting a two's-complement negative result to sign-magnitude format. The adder was split into 15 bytes, and each byte generated two sums. The critical path was the carry-merge tree, which was implemented in a binary tree fashion. This carry-merge tree produced the 15 selects for byte sums. The adder computed the result in 1.35 ns in eleven levels of logic.

Binary priority encoders
Several places in the logic design required the use of priority encoders. An example was a priority encoder in asynchronous interrupt logic where a 16-bit asynchronous interrupt vector had to be binary-encoded. Another use of this circuit was in leading-zero detectors. A fast priority encoder circuit was designed with pass-gate multiplexors, as shown in Figure 11. This circuit computes the result in about five levels of logic on 16-bit input vectors.

Figure 11

FXU 64-bit rotator
A 64-bit rotator in the fixed-point unit was implemented in two levels of 8-to-1 multiplexors. A straightforward implementation would look like that shown in Figure 12(a). In this implementation bits 0-6 of the first-level 8-to-1 multiplexor had to drive the entire width of the data stack to reach bits 57-63. This increased the wire capacitance of these bits eightfold and significantly increased the delay of the first-level multiplexor. The implementation that was used instead duplicated some bits of the first-level multiplexor shown in Figure 12(b). This increased the capacitance of the first-level multiplexor select and data lines by only 12.5%, while keeping the wire capacitance of the second-level multiplexor small. We found the latter implementation to be much faster.

Figure 12

Interleaving bits in the FXU data stack
The fixed-point-unit data stack was 64 bits wide to accommodate instructions that work on 64-bit operands. However, a large number of instructions in the architecture move data from the upper 32 bits to the lower 32 bits and vice versa. To minimize both area and cycle time, we avoided the many wiring channels that would be required for this by interleaving bits 0, 32, 1, 33,···, 31, 63. This had zero cost on most macros that are bit-sliced, such as registers, GPR, FPR, mask, and BLU. Even in the 64-bit rotator there was no penalty. The only macro that was made more complicated was the 64-bit binary adder in its carry-merge tree.

Clock checking in the RU unit
Every cycle datum that represents the architected state of the processor is sent to the RU unit from both copies of the processor to be compared, ECC-encoded, and written into the checkpoint array. Special precautions had to be taken to ensure that data are written into the checkpoint array at the appropriate time. There can be a malfunction in the write address decode circuit or the clock circuit that can prevent the clock from becoming active (thus not writing the checkpoint array), or cause it to become active when it should not, causing a spurious write. Therefore, the entire data vector cannot be written into the checkpoint array by a single clock--two clocks are required so that they can check each other. The data vector is partitioned into two halves, and a unique clock is sent to each half, as shown in Figure 13. The use of two toggle latches with an error detector guarantees that a single fault in the write address decode circuit or clock circuit will be detected.

Figure 13

Array circuits

To achieve these performance goals, the arrays utilized various innovative dynamic circuit techniques. Some of the key design attributes are

• A six-device CMOS SRAM array cell (33-µ² cell area).
• A high-density ROM array cell (4.4-µ² cell area).
• Multiport register file cells.
• Peripheral array circuits utilizing synchronized pipelined designs with dynamic and self-resetting circuit techniques for fast access and cycle times [4].
• Array circuits (all of them) designed for system robustness to be fully functional at extended power supply and temperature ranges with worst-case process parameters.
• Advanced ABIST (array built-in self-test) engines. These are used in the SRAM and ROM macros to provide comprehensive array test coverage. The ABIST engines are scan-programmable for pattern-generation flexibility and are capable of performing accurate cycle- and access-time measurements.