Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy...

18
Changyong Ahn & Nayoung Lee | March 2019 Expanding the Boundaries of the AI Revolution:

Transcript of Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy...

Page 1: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

Changyong Ahn & Nayoung Lee | March 2019

Expanding the Boundaries of the AI Revolution:

Page 2: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

1

Outline

Page 3: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

2

Page 4: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

3

Machine Learning/Deep Learning Use Cases

Page 5: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

4

Source: Standford

Deep Neural Network

Simple View

Σ(Activation function, Compute)

= Multiply & Accumulate sum

Weights x Input

Output

Layer

Weights x

Input

Weights x Input

……

MEM Write

MEM Read

Year CNN # of layers # of ParametersMemory size (MB)

Top5 ErrorRate

1998 LeNet 8 60K

2012 AlexNet 7 60 million 240 15.3%

2014 GoogleNet 19 4 million 6.67%

2014 VGG Net 16 138 million 574 7.3%

2015 ResNet 50/152 519 3.6%

Deep Neural Network Fundamental Concepts

Memory Challenges of Deep Learning

Page 6: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

5

Memory Solution for ML/DL Systems

* Source : SK hynix

Memory Sub system hierarchy change

Conventional DRAM IPM

Target Market/Price Broad & CheapSpecific & high

Premium

Standardization JEDEC Semi Custom

Qualification Period Relatively short Relatively long

Key factorsPrice

CompetitivenessReliability / Performance

TSV Region

Interposer

Substrate(“PCB”)

“In Package Memory”

Fast Storage(SSD)

HDD

LLC

DRAM

1) In-Package Memory 2) SCM(“Storage Class Memory”) : 3DXP, PCRAM

Storage Class Memory

Fast Storage(SSD)

HDD

LLC

IPM

DRAM

<10ns

50ns

100ns – 1us

50-100us

~10ms

Page 7: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

6

Page 8: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

7

HBM, What’s the difference?

GDDR/DDR/LPDDR HBM

FBGA KGSD

HBM in 2.5D SiP

PHYTSVDA ball

DRAM Slice

DRAM Slice

DRAM Slice

DRAM Slice

Interposer

SoC

PHY

Side

Mold

ing

Side

Mold

ing

Substrate

Directly soldered on PCB or used as a DI

MM

Mold

DRAM

DRAM

PCB Substrate

Page 9: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

8

HBM Advantages

More Bandwidth

High Power Efficiency

Small Form Factor

DDR4 LPDDR4(X) GDDR6 HBM2HBM2E (JEDEC)

HBM3(TBD)

Data rate 3200Mbps3200Mbps(up to 4266

Mbps)

14Gbps(up to 16Gb

ps)

2.4Gbps2.8Gbps

>3.2Gbps(TBD)

Pin count x4/x8/x16x16/ch

(2ch per die)x16/x32 x1024 x1024 x1024

Bandwidth 5.4GB/s 12.8(17)GB/s 56GB/s 307GB/s 358GB/s >500GB/s

Density(per package)

4Gb/8Gb8Gb/16Gb/24Gb/32Gb

8Gb/16Gb 4GB/8GB 8GB/16GB8GB/16GB/

24GB (TBD)

To Achieve 1TB Bandwidth ……

160ea of

DDR4-3200

40ea of

DDR4-3200 Module

4ea HBM2 in

a single 50mm x 50mm Sip

Note: Advil is

a registered trademark

Page 10: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

9

Page 11: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

10

HBM Architecture

HBM2 core die supports 4 pseudo channels or 2 channels

Each channel consists of 2 Pseudo Channels. Only BL4 is supported

Items Target

# of Stack 4/8(Core) + 1(Base)

Ch./Slice 2

Total Ch. for KGSD 8/16 (8ch based operation)

IO/Ch. 128

Total IO/KGSD 1024(=128 x 8)

Address/CMD Dual CMD

Data Rate DDR

CH-A CH-BB0 B1

B2 B3

B4 B5

B6 B7

64 I/OADDCMD

64 I/O

CH-A CH-BB0 B1

B2 B3

B4 B5

B6 B7

64 I/OADDCMD

64 I/O

Page 12: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

Next-Gen. System Architecture Leveraging HBM

HPC & Server(B/W & Capacity)

Network & Graphics(B/W)

Client-DT & NB(B/W & Cost)

+

Bandwidth

Solution

Cost Solution

+

Bandwidth

Solution

Bandwidth

Solution

+

Bandwidth

Solution Capacity Solution

Post-DDR4

+

Post-DDR4

+

B/W

B/W

B/W

B/W &

Capacity

B/W & Cost

HBM

HBM and 2.5D SiP integration unlock new system architecture

Page 13: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

HBM Test FlowGeneral DRAM Test Flow HBM Test Flow

Page 14: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

13

Quality and Reliability Features

SoC

Substrate

PMBIST

Cell Repair

BISS (Built In Self Stress)Microbump Repair

Error Correcting CodeStorage

HBM

PHY

DRAM

DRAM

DRAM

PHYLogic die PHY

TSV

micro bump

PKG Substrate

Proxy Package

HBM Features enable high quality and reliability at post 2.5D assembly

1

2

3 4 5

6

Interposer

1 2 3 4 5

6

Page 15: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

14

Collaterals Available from HBM vendors

Item Remarks

Functionality Datasheet (Jedec/Vendor)

Verilog (mission mode and DFT)

IBIS

Hspice

Mechanical/Interposer design GDS

Bump pad netlist

Bump Ballout

Thermal Simulation Flotherm

Icepak

Page 16: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

Future of HBM Solution

HBM would penetrate various market segments in the short future.

Auto

motiv

e

HBM1

GFX

HBM2

GFX

NT

W

HP

C

Client

HBM3

SVR

GFX

NT

W

HPC

SVR

Client

PC

Consum

er128GBps

1GB

256GBps

1G/2G/4G/8GB

105C

ECC

95C

512GBps

2G/4G/8G/16GB

105C

ECC/ In DRAM

ECCOptimized

Base die

Low Cost Ver.

Expansion to various

Applications

Moving to Volume Market

Con

sumer

15

Page 17: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

16

Page 18: Expanding the Boundaries of the AI Revolution...* Source : SK hynix Memory Sub system hierarchy change Conventional DRAM IPM Target Market/Price Broad & Cheap Specific & high Premium

17