Binarized ImageNet Inference in 29 s...BNN: Binarized neural network on FPGA. Neurocomputing 275...
1
Binarized ImageNet Inference in 29 μs Tong Geng *^, Ang Li*, Tianqi Wang^, Shuaiwen Leon Song*, Martin Herbordt^ *Pacific Northwest National Laboratory (PNNL), ^Boston University (BU) Introduction SC’18, Dallas, TX, USA Intra-layer Fusion Parameterized Architecture Inter-layer Fusion Figure 1. the difference between CNN and BNN [1] CPU GPU FPGA Platform Xeon E5- 2640 [2] Tesla K40 [2] Stratix V [2] Our Work: VCU118 Latency (ms) 10789 1257 1.16 (1x) 0.029 (40x) Performance (Image/s) 0.093 0.8 862 (1x) 34480 (40x) Energy Efficiency (Image/J) 0.00098 0.0034 32.9 (1x) 1078 (32.7x) Frequency (MHz) 2500 810 150 200 Utilization - - - 99.7% LUT FF BRAM DSP 635k/1182k (53.7%) 1527k/2364k (64.6%) 1459/2160 (67.5%) 5808/6840 (84.9%) Template Maxeler MPC-X2000 [2] Our Work: VCU118 Macro Layer Time (ms) Perf (GOP/s) Time (ms) Perf (GOP/s) speepup (times) CONV2 5.84x10 -2 15348 1.63x10 -2 54989 3.58 CONV3 2.03x10 -2 14712 1.62x10 -2 18435 1.25 CONV4 2.71x10 -2 16560 1.62x10 -2 27702 1.67 CONV5 1.81x10 -2 16546 1.63x10 -2 18373 1.11 FC1 4.31x10 -3 17503 1.57x10 -2 48050 2.75 BNN vs CNN: Easier Computation: floating-point-multiply-add (FMA) operations single-bit XNOR/POPCOUNT. Less Storage: floating-point parameters and activations single-bit except the first and last layer. Potential Low Latency: ultra-low latency is always the target of BNN. Experimental Results Utilization using different platforms to accelerate the inference of AlexNet with batch size of 10: × GPU: ~ 7%. CPU: ~ 10%. FPGA: >60%. The configurability enables millions of one- bit ALUs to be implemented on a single device. Challenges to make truly low-latency inference using FPGA 1. The critical Normalization Layer (NL) uses full-precision floating point (i.e., 2 FP MUL/DIV + 3 FP ADD/SUB). 2. Optimal designs for all layers need to be simultaneously configured on FPGA with no reconfiguration. 3. Existing works process layers sequentially. Hence, the their latencies are accumulated with no overlapping. Addressing the 1 st challenge: Intra-layer fusion • 3-to-1 fusion: Activation (ACT), NL and Binarization Layer (BL) are fused and replaced by a comparison layer. • Simplified Computation: Two comparison from ACT and BL and 5 floating-point operations from NL are eliminated and replaced by an integer-based comparison. • Less Storage: 4 floating-point variables (means, variance & 2 scaling factors) in NL become 1 integer threshold. • Lower Latency: for single CONV and FC layer, latency is reduced: results are shown in the Table below. Addressing the 2 nd challenge: the decent architecture design • A parameterized architecture which is flexible enough to support load balancing is proposed. • All layers are simultaneously configured on single FPGA using model parallelism. • ~100% utilization: Tuning the parameters of the proposed architecture, each CONV/FC layer can be easily designed so that producing and consuming rates match. • Minimal Resources: Keeping load balancing, parameters which are tuned are selected with the criteria of using minimal resources. • Efficient storage method: Data and parameters for the channels processed in parallel are packed in the shared memory; while the ones for the channels processed sequentially are stored in an interleaved manner. • All PEs work in lockstep manner and share the same Control Processor. Addressing the 3 rd challenge: inter-layer fusion: • Fine-grained pipelining to fuse CONV & 1 st FC layers. • An image is processed based on the data dependency. • Layers are processed in parallel overlapped latencies. [1] “Accelerating Neural Networks with Binary Arithmetic,” https://ai.intel.com/accelerating-neural-networks-binary-arithmetic. [2] Shuang Liang, Shouyi Yin, Leibo Liu, Wayne Luk, and Shaojun Wei. 2018. FP- BNN: Binarized neural network on FPGA. Neurocomputing 275 (2018) Acknowledgement: This research was funded by the Deep Learning for Scientific Discovery Investment under Pacific Northwest National Laboratory's Laboratory Directed Research and Development Program. The experimental platform was supported by U.S. DOE Office of Science, Office of Advanced Scientific Computing Research under the “CENATE” project (award No. 66150). The Pacific Northwest National Laboratory is operated by Battelle for the U.S. Department of Energy under contract DE-AC05-76RL01830. , =ቐ , , < ∗, − ∗ ∗, +
Transcript of Binarized ImageNet Inference in 29 s...BNN: Binarized neural network on FPGA. Neurocomputing 275...
Binarized ImageNet Inference in 29 μsTong Geng *^, Ang Li*, Tianqi Wang^, Shuaiwen Leon Song*, Martin Herbordt^
*Pacific Northwest National Laboratory (PNNL), ^Boston University (BU)
Introduction
SC’18, Dallas, TX, USA
Intra-layer Fusion Parameterized Architecture Inter-layer Fusion
Figure 1. the difference between CNN and BNN [1]
CPU GPU FPGA
PlatformXeon E5-2640 [2]
Tesla K40 [2] Stratix V [2]Our Work:
VCU118
Latency (ms) 10789 1257 1.16 (1x) 0.029 (40x)
Performance (Image/s) 0.093 0.8 862 (1x) 34480 (40x)
Energy Efficiency (Image/J)
0.00098 0.0034 32.9 (1x) 1078 (32.7x)
Frequency (MHz) 2500 810 150 200
Utilization - - - 99.7%
LUT FF BRAM DSP635k/1182k
(53.7%)1527k/2364k
(64.6%)1459/2160
(67.5%)5808/6840
(84.9%)
TemplateMaxeler MPC-X2000
[2]Our Work: VCU118
Macro Layer Time (ms) Perf (GOP/s) Time (ms) Perf (GOP/s) speepup (times)
CONV2 5.84x10-2 15348 1.63x10-2 54989 3.58CONV3 2.03x10-2 14712 1.62x10-2 18435 1.25CONV4 2.71x10-2 16560 1.62x10-2 27702 1.67CONV5 1.81x10-2 16546 1.63x10-2 18373 1.11
FC1 4.31x10-3 17503 1.57x10-2 48050 2.75
BNN vs CNN: Easier Computation: floating-point-multiply-add (FMA)
operations single-bit XNOR/POPCOUNT. Less Storage: floating-point parameters and activations
single-bit except the first and last layer. Potential Low Latency: ultra-low latency is always the
target of BNN.
Experimental Results
Utilization using different platforms to accelerate theinference of AlexNet with batch size of 10:× GPU: ~ 7%. CPU: ~ 10%. FPGA: >60%. The configurability enables millions of one-
bit ALUs to be implemented on a single device.
Challenges to make truly low-latency inference using FPGA1. The critical Normalization Layer (NL) uses full-precision
floating point (i.e., 2 FP MUL/DIV + 3 FP ADD/SUB).2. Optimal designs for all layers need to be simultaneously
configured on FPGA with no reconfiguration.3. Existing works process layers sequentially. Hence, the
their latencies are accumulated with no overlapping.
Addressing the 1st challenge: Intra-layer fusion• 3-to-1 fusion: Activation (ACT), NL and Binarization Layer
(BL) are fused and replaced by a comparison layer.• Simplified Computation: Two comparison from ACT and
BL and 5 floating-point operations from NL are eliminatedand replaced by an integer-based comparison.
• Less Storage: 4 floating-point variables (means, variance& 2 scaling factors) in NL become 1 integer threshold.
• Lower Latency: for single CONV and FC layer, latency isreduced: results are shown in the Table below.
Addressing the 2nd challenge: the decent architecture design• A parameterized architecture which is flexible enough to
support load balancing is proposed.• All layers are simultaneously configured on single FPGA
using model parallelism.• ~100% utilization: Tuning the parameters of the proposed
architecture, each CONV/FC layer can be easily designedso that producing and consuming rates match.
• Minimal Resources: Keeping load balancing, parameterswhich are tuned are selected with the criteria of usingminimal resources.
• Efficient storage method: Data and parameters for thechannels processed in parallel are packed in the sharedmemory; while the ones for the channels processedsequentially are stored in an interleaved manner.
• All PEs work in lockstep manner and share the sameControl Processor.
Addressing the 3rd challenge: inter-layer fusion:• Fine-grained pipelining to fuse CONV & 1st FC layers.• An image is processed based on the data dependency.• Layers are processed in parallel overlapped latencies.
[1] “Accelerating Neural Networks with Binary Arithmetic,” https://ai.intel.com/accelerating-neural-networks-binary-arithmetic.
[2] Shuang Liang, Shouyi Yin, Leibo Liu, Wayne Luk, and Shaojun Wei. 2018. FP-BNN: Binarized neural network on FPGA. Neurocomputing 275 (2018)
Acknowledgement: This research was funded by the Deep Learning for Scientific Discovery Investment under Pacific Northwest National Laboratory's Laboratory Directed Research and Development Program. Theexperimental platform was supported by U.S. DOE Office of Science, Office of Advanced Scientific Computing Research under the “CENATE” project (award No. 66150). The Pacific Northwest National Laboratory isoperated by Battelle for the U.S. Department of Energy under contract DE-AC05-76RL01830.
𝒚𝒊,𝒋 = ቐ𝟎
𝟏
𝒊𝒇𝐦𝐚𝐱 𝒙𝒊,𝒋, 𝟎 < 𝑬∗,𝒋 −𝜷𝒋 ∗ 𝑽𝒂𝒓 𝒙∗,𝒋 + 𝜺
𝜸𝒋
𝒐𝒕𝒉𝒆𝒓𝒔
![Plastic Film Capacitors - Panasonic · Insulation resistance (IR) Withstand voltage ±10 % (K), ±20 % (M) Category temp. range −40 ℃ to +110 ℃ Rated voltage [AC] 275 V Capacitance](https://static.fdocument.org/doc/165x107/5eaa86acb8948405283e9700/plastic-film-capacitors-panasonic-insulation-resistance-ir-withstand-voltage.jpg)
![arXiv:0909.1003v1 [math.CV] 7 Sep 2009 · Acknowledgements. We thank M. Bauer, D. Bernard, Ste en Rohde and Stanislav Smirnov for useful discussions. This work is partially funded](https://static.fdocument.org/doc/165x107/5f184aedba4adb33030b4cc1/arxiv09091003v1-mathcv-7-sep-2009-acknowledgements-we-thank-m-bauer-d-bernard.jpg)
